robots.txt tester
Test any URL against any crawler and see exactly which rule decides it — and why that rule wins.
Runs entirely in your browser. Nothing you paste is uploaded or stored.
Allowed by the most specific matching rule on line 5.
Allow: /admin/public-report.pdfMatched against the User-agent: * group. A crawler only ever obeys one group: the one that names it, or the wildcard if none does.
https://example.com/sitemap.xmlWhat this tool does
It parses a robots.txt the way a crawler does, then tells you whether a specific URL is allowed for a specific crawler and — the part that matters — which rule decided it. Alongside that it flags the syntax problems that make a file behave differently from how it reads.
The three rules people get wrong
The most specific rule wins, not the first one. Specificity is the length of the path pattern with wildcards discounted. So Allow: /admin/public-report.pdf beats Disallow: /admin/ even though it comes later in the file, because it is longer and therefore more specific. Reading robots.txt top to bottom like a firewall config produces the wrong answer almost every time.
A crawler obeys exactly one group. If your file has a User-agent: Googlebot group and a User-agent: * group, Googlebot reads only the first and ignores the wildcard completely. Rules you assumed were global do not apply. This is the usual cause of “we blocked it for everyone except Google, but Google is crawling it anyway”.
Disallow is not noindex. Blocking a URL prevents crawling, not indexing. A blocked page that other sites link to can still appear in search, listed without a description because the crawler was never allowed to read it. If the goal is removal, do the opposite: allow crawling and return a noindex tag, so the crawler can see the instruction.
Checks the tool runs on the file itself
Rules that appear before any User-agent line and therefore apply to nobody. Sitemap lines that are not absolute URLs. Noindex and Crawl-delay, both ignored by Google. Paths that do not start with a slash. And blocked CSS or JavaScript, which stops Google rendering the page as a visitor sees it — a page whose layout Google cannot render is a page it judges on incomplete evidence.
Questions
Which rule wins when two rules match?
The most specific one, measured by the number of characters in the path — not the one that comes first in the file. If two rules are equally specific, Allow beats Disallow. This is why the order of lines in robots.txt almost never matters, and why people misread their own file.
How does a crawler pick a group?
It obeys exactly one group: the one that names it most specifically. If a file has a group for Googlebot and a wildcard group, Googlebot follows only the Googlebot group and ignores the wildcard entirely — including any rules it would otherwise have inherited.
Does Disallow remove a page from Google?
No, and this is the most expensive misunderstanding in SEO. Disallow stops crawling, not indexing. A blocked URL with links pointing at it can still appear in results, usually with no description. To remove a page, allow crawling and serve a noindex tag so the crawler can actually see it.
Does Noindex work in robots.txt?
It has not since September 2019. Google announced it would stop supporting undocumented directives including noindex, nofollow and crawl-delay in robots.txt. Lines like that are silently ignored, which makes them worse than useless — they look like protection.
Are paths case-sensitive?
Yes. Disallow: /Admin does nothing about /admin. The file name itself must also be exactly robots.txt in lowercase, at the root of the domain.