Diffbot
AI training crawlerA structured-data extraction service that crawls pages into a machine-readable knowledge graph sold to customers, including for AI applications. Because customers can request specific URLs, some fetches behave like user-initiated requests.
| Operator | Diffbot |
| Type | AI training crawler |
| Respects robots.txt | Partially |
| Documentation | Official operator documentation |
Diffbot user-agent string
This is how Diffbot identifies itself in your server logs:
Mozilla/5.0 (compatible; Diffbot/1.0; +https://www.diffbot.com)How to block Diffbot in robots.txt
Add this to the robots.txt file at the root of your site:
Block Diffbot
User-agent: Diffbot
Disallow: /Explicitly allow Diffbot
User-agent: Diffbot
Allow: /What blocking changes: Signals your content should stay out of Diffbot's knowledge graph and downstream AI uses; on-demand extraction requests may still occur, so pair with a network-level block if it matters.
Want to decide for all AI bots at once? Use the AI robots.txt generator.
Other ai training crawlers
- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- Google-Extended (Google)
- Applebot-Extended (Apple)
- Meta-ExternalAgent (Meta)
- Bytespider (ByteDance)
Is Diffbot sending you visitors?
Glyphex classifies AI referral traffic from ChatGPT, Perplexity, Claude, Gemini and more. See which assistants cite your pages and what those visitors do.
See your AI traffic