Developer & Web Tools
robots.txt Generator
Build a valid robots.txt with crawl rules, sitemap references and common bot presets.
Privacy
Everything on this page runs inside your browser. Nothing you paste, type or upload is transmitted, logged or stored.
Tips
- robots.txt controls crawling, not indexing. To keep a page out of results, use a noindex meta tag — a blocked page can still be listed if others link to it.
- It must sit at the domain root: example.com/robots.txt. In a subfolder it is ignored.
- Well-behaved crawlers honour it; malicious scrapers do not. It is not a security control.
Frequently asked questions
Does blocking AI crawlers affect my Google ranking?
Blocking Google-Extended opts you out of Gemini training without affecting Google Search crawling or ranking, which uses the separate Googlebot agent.