What is robots.txt Tester?
robots.txt Tester is a free online tool that fetches a site's robots.txt file and shows its allow and disallow rules. Make sure you are not blocking Googlebot, Bingbot or AI crawlers from important pages.
Test robots.txt online. Fetch your site’s robots file and review disallow rules for crawlers — free robots.txt checker on Fah Swe Tools.
Key features
- Fetches /robots.txt from any public site
- Lists disallow and allow rules
- Shows sitemap lines
- Free and polite — one request per check
Popular uses
- Find accidental 'Disallow: /' that blocks the whole site
- Check rules after launching a new site
- Review competitors' crawl rules
Also known as: robots.txt Checker, robots.txt Validator, Crawl Rules Tester.
How it works
- Enter a site origin (https://example.com).
- Fetch robots.txt.
- Review rules.
FAQ
Is crawling polite?
Single GET with short timeout and rate limits.
Does robots.txt stop a page from being indexed?
No. It stops crawling, but a blocked URL can still be indexed from links. Use a noindex tag to keep a page out of Google.
Where must robots.txt be located?
At the root of the domain, for example https://example.com/robots.txt.