Robots.txt Generator
Build a valid robots.txt file with a visual rule builder — set allow/disallow rules per crawler, add a sitemap and crawl-delay, then copy or download.
🔒 Processed locally in your browser
Loading…
How to use
- Pick a quick-start preset — "Allow all" lets every crawler index your whole site, "Block all" blocks every crawler entirely, or start with "Custom" to build your own rules.
- For each rule group, set the User-agent (use * for all crawlers, or a specific bot name like Googlebot) and add Allow or Disallow rules with the paths you want to target.
- Optionally set a Crawl-delay per group and add your Sitemap URL — both are appended to the generated file automatically.
- Copy the generated robots.txt from the preview box, or download it directly — then upload it to the root of your domain (e.g. https://example.com/robots.txt).
Features
- One-click "Allow all" and "Block all" presets for the most common cases
- Visual rule builder — add or remove user-agent groups and Allow/Disallow rules without writing syntax by hand
- Optional Crawl-delay per group and a Sitemap URL field
- Live-generated output that updates instantly as you edit the rules
- 100% client-side — your rules never leave your browser
Frequently asked questions
Where do I put the robots.txt file?
Upload it to the root of your domain, so it's reachable at https://yourdomain.com/robots.txt. Search engines only look for it at that exact location.
Does robots.txt actually block search engines from crawling my site?
It's a request, not enforcement. Well-behaved crawlers like Googlebot respect it, but nothing stops a malicious or poorly-behaved bot from ignoring it entirely. Don't rely on it to hide sensitive content.
What does the User-agent field mean?
It targets which crawler a rule group applies to. Use * to match every crawler, or a specific bot's name (like Googlebot or Bingbot) to give that one different rules from the rest.
Do I need a Sitemap line?
It's optional but recommended — it points crawlers to your sitemap.xml so they can discover your pages more efficiently. Leave the field blank if you don't have one yet.
Is Crawl-delay respected by all search engines?
No. It's supported by some crawlers (like Bing) but ignored by others (Google no longer honors it in robots.txt). Use Google Search Console to control Google's crawl rate instead.