How robots.txt really works: which rule wins, and what it cannot do

Three facts that explain most surprises

1. A crawler obeys exactly one group. The one that names it most specifically. Everything else in the file is ignored for that crawler.
2. The longest matching rule wins, not the first one in the file.
3. Disallow stops crawling, not indexing. A blocked page can still appear in search results.

The rest of this guide is those three facts with the edge cases attached, and then the things robots.txt is often expected to do and does not.

One crawler, one group

A group is a run of User-agent lines followed by the rules beneath them. Take this file:

User-agent: Googlebot
Disallow: /drafts/

User-agent: *
Disallow: /private/

Googlebot reads the first group and nothing else. /private/ is not blocked for it, because the wildcard group is for crawlers that have no group of their own. This is the usual cause of “we blocked it for everyone, but Google is crawling it anyway”: rules written for * were never going to apply to a crawler that is named elsewhere in the file.

Two groups for the same crawler are merged into one, so splitting a bot’s rules across the file does no harm. If a crawler’s own group exists but contains no rule that matches a URL, that URL is allowed; the crawler does not fall back to the wildcard group.

The longest rule wins

When more than one rule matches a URL, the one with the longest path wins. Order in the file does not matter. If two matching rules are the same length, Allow beats Disallow.

Take this file and test eight URLs against it:

User-agent: *
Disallow: /admin/
Allow: /admin/public/
Disallow: /*.pdf$
Disallow: /search

URL	Result	Decided by
/admin/settings	Blocked	Disallow: /admin/
/admin/public/logo.png	Allowed	Allow: /admin/public/ (longer than /admin/)
/report.pdf	Blocked	Disallow: /*.pdf$
/report.pdf?download=1	Allowed	no rule matches
/search	Blocked	Disallow: /search
/search-tips/	Blocked	Disallow: /search
/Admin/settings	Allowed	no rule matches
/blog/post/	Allowed	no rule matches

Four of those catch people out. Rules match from the start of the path and keep matching, so /search also blocks /search-tips/ and anything else that begins with those characters. A $ anchors the end of the URL, so /*.pdf$ stops matching the moment a query string follows. Paths are case-sensitive, so /Admin/ is a different place from /admin/. And Allow: /admin/public/ beats Disallow: /admin/ because it is the longer of the two, even though it comes second.

Wildcard characters count towards the length

Length is measured on the rule’s text, and that includes * and $. Google’s own open-source parser does exactly this, which has a consequence that is easy to miss:

User-agent: *
Allow: /docs/
Disallow: /*.pdf$

For /docs/guide.pdf both rules match. /docs/ is 6 characters, /*.pdf$ is 7, so the Disallow is longer and wins: the PDF is blocked even though it sits inside an allowed folder. Counting only the literal characters would give the opposite answer, 6 against 5, which is why testers that discount wildcards can disagree with Google on a file like this one.

What robots.txt cannot do

It does not remove pages from search. Blocking a URL prevents it from being fetched, not from being listed. A blocked page that other sites link to can still appear in results, usually without a description, because the crawler was never allowed to read it. To keep a page out of the index, allow crawling and serve a noindex tag. The crawler has to be able to fetch the page to see the instruction.

It has no noindex. Google stopped supporting noindex in robots.txt in September 2019, along with nofollow. It also ignores crawl-delay, although some other crawlers honour it. Lines like these are skipped without a warning, which makes them worse than absent: they look like protection.

It is not a lock. The file is public and well-behaved crawlers choose to follow it. A Disallow: /secret-admin/ line tells everyone, including anyone looking, that the path exists. Anything that must stay private needs authentication, not a request.

It does not enforce itself. A firewall in front of the site decides first. When this site’s AI crawlers were checked in August, every one of them, including OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot, ClaudeBot and CCBot, received a 403 on every page, while the robots.txt, which was the only thing that returned 200, said they were welcome. The block came from a bot-protection setting at the edge that runs before any file is read. Reading robots.txt told us nothing about it; only requesting the pages did.

The file you wrote may not be the file that is served

Until September, the robots.txt this site served was not the file in its repository. A managed setting at the CDN added a block of its own for AI-training crawlers on top. In September the provider migrated those settings to new controls, and today the served file is identical to the repository’s. Nobody edited the file in either case. The only reliable way to know what crawlers see is to fetch it.

A few more properties that decide whether the file is read at all:

• It must sit at the root of the host: https://example.com/robots.txt. Each subdomain and each protocol is its own host and needs its own file.
• A 404 means no restrictions. A server error (5xx) is treated as a temporary full block, so a deploy that breaks the file can pause crawling of the whole site.
• Google caches the file, generally for up to a day, and reads only the first 500 KiB.
• Blocking CSS or JavaScript stops Google rendering the page the way a visitor sees it.
• The Sitemap line takes an absolute URL, can appear anywhere, and is not part of any group.
• The commonest expensive mistake is a staging Disallow: / that survives a deploy to production.

Checking what a crawler actually gets

Fetch the file, and check its status code:

curl -s https://example.com/robots.txt
curl -s -o /dev/null -w '%{http_code}\n' https://example.com/robots.txt

Then request a real page the way the crawlers do, with their names in the user agent:

for bot in GPTBot ClaudeBot PerplexityBot OAI-SearchBot; do
  printf '%s ' "$bot"
  curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; $bot/1.0)" https://example.com/
done

A 200 across the board is what you want. A 403 for a bot name with a 200 for an ordinary browser points to a firewall or bot rule, whatever robots.txt says. One caution: this sends the bot’s name from your own address, not from the crawler’s. A firewall that verifies bots by their network address can treat the test differently from the real thing, so a clean result here is encouraging rather than proof. A 403 is a finding in either case. Run on this site today, all four return 200.

Before you ship a file

• Each crawler has one group, and you know which rules apply to it.
• Every Allow that is meant to override a Disallow is the longer of the two, counting * and $.
• Nothing you want removed from search is blocked; those pages are crawlable and carry noindex.
• No noindex, nofollow or crawl-delay lines are doing the job of something else.
• CSS, JavaScript and every language version are reachable. A blocked page cannot confirm its hreflang return link.
• The served file matches the one you wrote, and the status code is 200.

The robots.txt tester applies these rules to any file and URL and shows which line decided the result. Blocked pages that still appear in search are listed without a description. For pages that are crawlable, the SERP snippet preview shows how the title and description will look.