Back to tools
Robots.txt Generator
Create a crawler-friendly robots.txt file with clear rules and a sitemap reference.
Crawler rules
Generated robots.txt
User-agent: * Allow: / Disallow: /api/ Sitemap: https://example.com/sitemap.xml
Keep valuable public pages allowed. Do not use robots.txt to hide sensitive content—protect it with authentication instead.
Rule tester
Test a path against the Googlebot rule, falling back to the wildcard rule.
Blocked for *