GPTBot
AI training crawlerOpenAI's web crawler for collecting training data. Content fetched by GPTBot may be used to train and improve OpenAI's foundation models, such as the GPT series.
| Operator | OpenAI |
| Type | AI training crawler |
| Respects robots.txt | Yes (documented) |
| Documentation | Official operator documentation |
GPTBot user-agent string
This is how GPTBot identifies itself in your server logs:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbotHow to block GPTBot in robots.txt
Add this to the robots.txt file at the root of your site:
Block GPTBot
User-agent: GPTBot
Disallow: /Explicitly allow GPTBot
User-agent: GPTBot
Allow: /What blocking changes: Your content is excluded from OpenAI's future training crawls. This does not affect whether ChatGPT can browse your site live (that is ChatGPT-User) or whether you appear in ChatGPT search (that is OAI-SearchBot).
Want to decide for all AI bots at once? Use the AI robots.txt generator.
Other ai training crawlers
- ClaudeBot (Anthropic)
- Google-Extended (Google)
- Applebot-Extended (Apple)
- Meta-ExternalAgent (Meta)
- Bytespider (ByteDance)
- CCBot (Common Crawl)
Is OpenAI sending you visitors?
Glyphex classifies AI referral traffic from ChatGPT, Perplexity, Claude, Gemini and more. See which assistants cite your pages and what those visitors do.
See your AI traffic