Free Robots.txt Tester
Check if a URL is allowed by robots.txt.
What is the Robots.txt Tester?
A single line in robots.txt can accidentally hide an entire section of your site from search engines — or fail to block the pages you meant to keep out. Because the matching rules are fiddly, it's easy to get wrong and never notice.
This robots.txt tester fetches a site's robots.txt and tells you whether a specific URL is allowed or blocked for a given crawler, and which rule decides it.
Last updated: Aug 12, 2026
What does this tool do?
You enter a URL and pick a crawler, and the tool fetches that site's robots.txt, parses the rules for the chosen user-agent, and applies robots.txt matching to your path.
It shows a clear allowed-or-blocked verdict, the exact rule that applied, any declared sitemaps, and the full robots.txt content.
Key features
Allowed / blocked verdict
A clear answer for the URL you test.
Per-crawler
Test Googlebot, Bingbot, GPTBot, and others.
Shows the matching rule
See which Allow or Disallow decided it.
Sitemap detection
Lists sitemaps declared in robots.txt.
Full file view
Read the complete robots.txt.
Example
Common use cases
- SEO specialists — Confirm important pages aren't blocked.
- Web developers — Verify robots.txt rules work as intended.
- Site owners — Check a page is crawlable before wondering why it isn't indexed.
- Migrators — Ensure a new site isn't accidentally blocking everything.
- AI-SEO folks — Check whether GPTBot or ClaudeBot can access a page.
- Agencies — Audit a client's crawlability.
- Students — Learn how robots.txt matching works.
- Anyone debugging indexing — Rule robots.txt in or out as the cause.
Benefits
- It removes guesswork from robots.txt matching.
- It tests per-crawler, including AI bots.
- It names the exact rule responsible.
- It surfaces declared sitemaps too.
Tips
- robots.txt controls crawling, not indexing — use a noindex tag to keep a page out of results.
- A blocked page can still appear in results without a snippet if it's linked elsewhere.
- The most specific (longest) matching rule wins; Allow overrides an equally specific Disallow.
- Test with the actual crawler you care about — rules can differ by user-agent.
- Don't block CSS and JS that pages need to render.
- Blocking a page also stops crawlers seeing its noindex tag — allow it if you want it de-indexed.
- Add your sitemap URL to robots.txt for better discovery.
- Re-test after editing robots.txt to confirm the change.
Common mistakes to avoid
Using robots.txt to de-index a page
Fix: Blocking crawling doesn't remove a page from results — use a noindex tag and allow crawling so it's seen.
Accidentally disallowing everything
Fix: Disallow: / blocks the whole site — test before deploying.
Blocking CSS and JS
Fix: Crawlers need them to render pages; don't disallow those paths.
Testing the wrong user-agent
Fix: Rules can differ per crawler — test the one that matters.
How it works
- 1
Enter a URL
The page you want to test.
- 2
Pick a crawler
Default is Googlebot.
- 3
See the verdict
Allowed or blocked, with the matching rule.
Frequently asked questions
Whether the URL you entered is allowed or disallowed for the chosen user-agent, and which robots.txt rule decides it.
Googlebot by default; you can enter any user-agent to test its rules.
Whether the URL you entered is allowed or disallowed for the chosen user-agent, and which robots.txt rule decides it.
Googlebot by default; you can pick Bingbot, GPTBot, ClaudeBot, Google-Extended, or the wildcard, or any user-agent.
No. It controls crawling. To keep a page out of results, use a noindex meta tag and allow crawling so it's seen.
The longest matching path rule wins; an Allow overrides an equally specific Disallow. The tool follows this standard.
Then everything is crawlable by default, and the tool tells you so.
No. robots.txt is fetched only to evaluate your test.
Conclusion
A robots.txt tester takes the guesswork out of crawlability so you never accidentally hide (or fail to hide) a page. Pair it with the site audit and meta-tag generator to control indexing end to end.