robots.txt Generator

Generate a robots.txt file for your website directly in your browser.

Default rule for all crawlers (User-agent: *)

Set crawl delay

Block specific AI crawlers

GPTBot (OpenAI) ChatGPT-User (OpenAI browsing) ClaudeBot (Anthropic) Google-Extended (Gemini training) CCBot (Common Crawl) Bytespider (ByteDance) PerplexityBot (Perplexity) Applebot-Extended (Apple Intelligence)

Sitemap URL(s) — one per line

robots.txt Output

Runs entirely in your browser — nothing is sent to a server. robots.txt is a voluntary convention: well-behaved crawlers respect it, but it doesn't technically prevent access — use proper authentication or server-level blocking for content that genuinely must stay private. Place the downloaded file at your site's root (e.g. yoursite.com/robots.txt).

Frequently Asked Questions

Not by force — it's a voluntary convention that well-behaved crawlers respect, but it doesn't technically prevent access. Use proper authentication or server-level blocking for content that genuinely must stay private.

Yes — there's a dedicated section with checkboxes for GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, CCBot, Bytespider, PerplexityBot, and Applebot-Extended.

At your site's root, e.g. yoursite.com/robots.txt.