Free Robots.txt Generator
Build a robots.txt file in seconds.
Loading tool…
About the Robots.txt Generator
Free robots.txt generator. Choose which crawlers to allow or block, add disallow rules and your sitemap URL, then copy a valid robots.txt file.
How it works
- 1
Set rules
Allow or block paths and user agents.
- 2
Add sitemap
Include your sitemap URL.
- 3
Copy file
Save it as robots.txt at your site root.
Frequently asked questions
It tells search engine crawlers which parts of your site they may or may not crawl. It's placed at the root of your domain.
Yes — including your sitemap URL helps search engines discover all your pages.