ClaudeBot
AI training crawlerAnthropic's web crawler for collecting publicly available content that may be used to train Claude models.
| Operator | Anthropic |
| Type | AI training crawler |
| Respects robots.txt | Yes (documented) |
| Documentation | Official operator documentation |
ClaudeBot user-agent string
This is how ClaudeBot identifies itself in your server logs:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; [email protected]How to block ClaudeBot in robots.txt
Add this to the robots.txt file at the root of your site:
Block ClaudeBot
User-agent: ClaudeBot
Disallow: /Explicitly allow ClaudeBot
User-agent: ClaudeBot
Allow: /What blocking changes: Your content is excluded from Anthropic's future training crawls. Live fetches inside Claude (Claude-User) and Claude's search index (Claude-SearchBot) are controlled separately.
Want to decide for all AI bots at once? Use the AI robots.txt generator.
Other ai training crawlers
- GPTBot (OpenAI)
- Google-Extended (Google)
- Applebot-Extended (Apple)
- Meta-ExternalAgent (Meta)
- Bytespider (ByteDance)
- CCBot (Common Crawl)
Is Anthropic sending you visitors?
Glyphex classifies AI referral traffic from ChatGPT, Perplexity, Claude, Gemini and more. See which assistants cite your pages and what those visitors do.
See your AI traffic