robots.txt tester
Paste a robots.txt file, name a crawler and a URL, and see whether that crawler may fetch it, which line decides, and what else in the file will not do what it looks like it does.
Paste-in tool. Your browser cannot fetch another site's headers for a web page (CORS
forbids it), and WebPeeks has no server to do it, so nothing you paste leaves your browser.
Open https://your-site/robots.txt in a browser and copy it, or fetch it with curl.
You can also paste a draft before publishing it.
curl -s https://example.com/robots.txt bingbot → /search?q=walking+boots
Blocked
The longest matching rule is "Disallow: /search" on line 4.
No group names this crawler, so the User-agent: * group applies.
Rules in the applicable group
| Line | Type | Pattern | Against this URL |
|---|---|---|---|
| 3 | Disallow | /admin/ | — |
| 4 | Disallow | /search | matches (wins) |
| 5 | Allow | /search/help | — |
| 6 | Disallow | /*.pdf$ | — |
Groups in the file
| Line | User agents | Rules |
|---|---|---|
| 2 | * | 4 rules |
| 8 | Googlebot | 2 rules |
| 12 | GPTBot | 1 rule |
Sitemaps
- https://www.example.com/sitemap.xml
Notes on the file
- InfoLine 15: crawl-delay: 10
Not part of RFC 9309. Bing and Yandex honour it; Google ignores it.
The matching rules, from RFC 9309
robots.txt was an informal convention from 1994 until the IETF published it as RFC 9309, the Robots Exclusion Protocol, in September 2022. This tester follows that document:
- Pick the group. A group is one or more
User-agentlines followed by rules. The crawler looks for groups naming its product token (GooglebotforGooglebot/2.1), ignoring case. If several groups name it, their rules are combined. If none does, theUser-agent: *group applies. If there is no such group either, everything is allowed. - Find the matching rules. Each
AlloworDisallowvalue is a path prefix, matched against the URL's path and query string.*matches any sequence of characters and a$at the end means "the URL ends here". An emptyDisallow:matches nothing. - Longest match wins. Of all matching rules, the one with the longest path (counted in bytes) decides. If an Allow and a Disallow are equally long, Allow wins.
- No match means allowed, and
/robots.txtitself is always allowed.
A worked example
In the example file, Bingbot has no group of its own, so it gets the * group. The URL
/search?q=walking+boots matches Disallow: /search (7 characters) and nothing
longer, so it is blocked. Change the URL to /search/help and it matches both
Disallow: /search and Allow: /search/help; the Allow is longer, so the page is
allowed.
Now change the crawler to Googlebot. It has its own group, so the * group no
longer applies to it at all: /search?q=… and even /admin/ are allowed, and only
/private/ is blocked. This is the most common robots.txt surprise. The file also shows two
smaller ones: Crawl-delay is not in RFC 9309 (Bing and Yandex honour it, Google ignores it),
and a Sitemap line applies to the whole file whichever group it sits in.
| Pattern | Matches | Does not match |
|---|---|---|
| /shop | /shop, /shop/, /shopping, /shop?id=2 | /Shop, /en/shop |
| /shop/ | /shop/, /shop/boots | /shop, /shopping |
| /*.pdf$ | /a.pdf, /docs/guide.pdf | /a.pdf?download=1, /a.pdfx |
| /*? | any URL with a query string | /page |
| / | every URL | — |
Paths are case-sensitive, and percent-encoding is normalised before comparing, so
/%7Euser and /~user are the same path.
Where real crawlers differ
RFC 9309 matches the product token exactly. Google documents one refinement: its specialised crawlers
fall back to the main crawler's group, so Googlebot-Image uses a Googlebot
group when there is no Googlebot-Image group. This tester does not apply that fallback, so
for Google's specialised crawlers, test the Googlebot token as well. Crawlers are also
allowed to stop reading after 500 KiB, to cache the file for up to a day, and to treat a server error
when fetching it as "disallow everything" for a while. And robots.txt only binds crawlers that choose
to obey it; it is not access control.
Common questions
Does Disallow stop a page appearing in Google?
Not reliably. Disallow stops crawling, not indexing. If other sites link to a blocked URL, Google can still list it, usually with no description because it never read the page. To keep a page out of search results, let it be crawled and add a noindex meta tag or X-Robots-Tag header; a crawler that cannot fetch the page never sees the noindex.
Is robots.txt a security measure?
No. It is a request that well-behaved crawlers honour, and it is public: anyone can read it, including people looking for the admin paths it lists. Protect private areas with authentication, not with robots.txt.
Why does my Googlebot group ignore the rules under User-agent: *?
Because a crawler obeys exactly one set of rules: the group or groups that name it. If there is a User-agent: Googlebot group, Googlebot reads only that, and the * group does not apply to it at all. Repeat any rule you want to apply to everyone inside each named group.
How do I block AI training crawlers?
Add a group for each crawler's token with Disallow: /. The operators document their own tokens, for example GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl) and Google-Extended (which controls use of content for Google's AI models without affecting Search). Compliance is voluntary, and new crawlers appear often, so the list needs maintaining.
Does the order of Allow and Disallow lines matter?
Not under RFC 9309. The rule with the longest matching path wins regardless of order, and Allow wins an exact tie. Some older or simpler crawlers use the first matching rule instead, so putting the more specific rules first is a harmless way to get the same answer from both.