CCBot
AI training crawlerThe crawler of the Common Crawl non-profit, which publishes a free, open archive of the web. Many AI labs train models on Common Crawl data, which makes CCBot the single widest door into AI training datasets, but the same archive also supports academic research.
| Operator | Common Crawl |
| Type | AI training crawler |
| Respects robots.txt | Yes (documented) |
| Documentation | Official operator documentation |
CCBot user-agent string
This is how CCBot identifies itself in your server logs:
CCBot/2.0 (https://commoncrawl.org/faq/)How to block CCBot in robots.txt
Add this to the robots.txt file at the root of your site:
Block CCBot
User-agent: CCBot
Disallow: /Explicitly allow CCBot
User-agent: CCBot
Allow: /What blocking changes: Your content stays out of future Common Crawl snapshots and therefore out of the many AI training pipelines built on them. Past snapshots already published are unaffected.
Want to decide for all AI bots at once? Use the AI robots.txt generator.
Other ai training crawlers
- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- Google-Extended (Google)
- Applebot-Extended (Apple)
- Meta-ExternalAgent (Meta)
- Bytespider (ByteDance)
Is Common Crawl sending you visitors?
Glyphex classifies AI referral traffic from ChatGPT, Perplexity, Claude, Gemini and more. See which assistants cite your pages and what those visitors do.
See your AI traffic