Robots.txt Generator
Set a crawler user-agent and add allow or disallow paths to build a robots.txt file you can download and upload to your site's root.
Create a robots.txt text file for your site. Rules guide cooperative crawlers; they do not secure private content.
User-agent: * Allow: /
🔒 All processing happens in your browser — files never leave your device. Your input stays in this browser and is not uploaded to Smart Online Tools.
How it works
- Enter the user-agent and crawler paths.
- Optionally add your sitemap URL.
- Review the preview, then download robots.txt.
Good to know: Robots.txt rules are public instructions for compliant crawlers, not access controls. They cannot keep private pages or files secure.
Privacy and processing
Your input stays in this browser and is not uploaded to Smart Online Tools.
Frequently asked questions
What does Robots.txt Generator do?
Create a robots.txt file with crawler allow, disallow, and sitemap rules. Set a crawler user-agent and add allow or disallow paths to build a robots.txt file you can download and upload to your site's root.
Does this tool upload my files or input?
Your input stays in this browser and is not uploaded to Smart Online Tools.
What should I know before using this tool?
Robots.txt rules are public instructions for compliant crawlers, not access controls. They cannot keep private pages or files secure.