AI2Bot
AI training crawlerThe Allen Institute's crawler collecting web content for training open language models such as the OLMo family, used primarily for research.
| Operator | Allen Institute for AI |
| Type | AI training crawler |
| Respects robots.txt | Yes (documented) |
| Documentation | Official operator documentation |
AI2Bot user-agent string
This is how AI2Bot identifies itself in your server logs:
Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)How to block AI2Bot in robots.txt
Add this to the robots.txt file at the root of your site:
Block AI2Bot
User-agent: AI2Bot
Disallow: /Explicitly allow AI2Bot
User-agent: AI2Bot
Allow: /What blocking changes: Your content is excluded from AI2's research training datasets.
Want to decide for all AI bots at once? Use the AI robots.txt generator.
Other ai training crawlers
- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- Google-Extended (Google)
- Applebot-Extended (Apple)
- Meta-ExternalAgent (Meta)
- Bytespider (ByteDance)
Is Allen Institute for AI sending you visitors?
Glyphex classifies AI referral traffic from ChatGPT, Perplexity, Claude, Gemini and more. See which assistants cite your pages and what those visitors do.
See your AI traffic