AI robots.txt generator

Decide which AI crawlers may use your content. Pick the bots to block, copy the generated rules into your robots.txt. Each bot links to a full explanation in the AI bots directory.

AI training crawlers

Bulk crawlers that collect page content to train or fine-tune AI models. Blocking them keeps your content out of future training datasets but does not remove anything already collected.

AI assistant fetchers

Fetchers that load a page on behalf of a user inside an AI assistant: to answer a question, summarize a link, or cite a source. Blocking them stops assistants from reading your pages live, which can also remove your site from their cited answers.

AI search indexers

Crawlers that build search indexes used to ground AI answers with links and citations. Blocking them removes your site from those AI search results.

robots.txt
# AI crawler rules - generated with glyphex.io/tools/robots-txt-generator

# GPTBot (OpenAI)
User-agent: GPTBot
Disallow: /

# ClaudeBot (Anthropic)
User-agent: ClaudeBot
Disallow: /

# Google-Extended (Google)
User-agent: Google-Extended
Disallow: /

# Applebot-Extended (Apple)
User-agent: Applebot-Extended
Disallow: /

# Meta-ExternalAgent (Meta)
User-agent: meta-externalagent
Disallow: /

# Bytespider (ByteDance)
User-agent: Bytespider
Disallow: /

# CCBot (Common Crawl)
User-agent: CCBot
Disallow: /

# Amazonbot (Amazon)
User-agent: Amazonbot
Disallow: /

# AI2Bot (Allen Institute for AI)
User-agent: AI2Bot
Disallow: /

# Diffbot (Diffbot)
User-agent: Diffbot
Disallow: /

# cohere-training-data-crawler (Cohere)
User-agent: cohere-training-data-crawler
Disallow: /

Append these rules to the robots.txt at the root of your site. Existing rules for other crawlers are unaffected. Not sure what a bot does? Look it up in the AI bots directory.

Frequently asked questions

Does robots.txt actually stop AI bots?

Robots.txt is a voluntary standard: well-run crawlers from OpenAI, Anthropic, Google, Apple and Meta document that they obey it, but it is not an enforcement mechanism. Bots with poor compliance records (and user-initiated fetchers that operators exempt) can only be stopped at the network level, e.g. with a WAF or CDN bot rules.

Will blocking AI bots hurt my SEO?

Blocking dedicated AI training bots (GPTBot, ClaudeBot, CCBot, Google-Extended) has no effect on your Google or Bing rankings; those use separate crawlers. But blocking AI search indexers like OAI-SearchBot or PerplexityBot removes you from AI answers that cite and link sources, which is a growing traffic channel.

Which AI bots should I block?

A common middle ground is to block bulk training crawlers (which collect your content without sending anything back) while allowing assistant fetchers and AI search indexers (which cite your pages and send referral visitors). The "Block training bots only" preset in this generator does exactly that.

Where does robots.txt go?

It must be served at the root of your domain: https://example.com/robots.txt. Append the generated AI rules to your existing file. Rules are per user-agent, so they will not affect your existing directives for other crawlers.

See which AI assistants already send you traffic

Glyphex classifies visits from ChatGPT, Perplexity, Claude, Gemini and more. Know your AI referral traffic before deciding what to block.

See your AI traffic