cohere-training-data-crawler
AI training crawlerA crawler attributed to Cohere that collects web content for training its enterprise language models. Cohere publishes little documentation about it, so compliance behavior is not well documented.
| Operator | Cohere |
| Type | AI training crawler |
| Respects robots.txt | Not documented |
| Documentation | None published by the operator |
cohere-training-data-crawler user-agent string
This is how cohere-training-data-crawler identifies itself in your server logs:
cohere-training-data-crawlerHow to block cohere-training-data-crawler in robots.txt
Add this to the robots.txt file at the root of your site:
Block cohere-training-data-crawler
User-agent: cohere-training-data-crawler
Disallow: /Explicitly allow cohere-training-data-crawler
User-agent: cohere-training-data-crawler
Allow: /What blocking changes: Signals that your content should be excluded from Cohere's training data collection.
Want to decide for all AI bots at once? Use the AI robots.txt generator.
Other ai training crawlers
- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- Google-Extended (Google)
- Applebot-Extended (Apple)
- Meta-ExternalAgent (Meta)
- Bytespider (ByteDance)
Is Cohere sending you visitors?
Glyphex classifies AI referral traffic from ChatGPT, Perplexity, Claude, Gemini and more. See which assistants cite your pages and what those visitors do.
See your AI traffic