Claude-SearchBot
AI search indexerAnthropic's crawler for improving Claude's web search: it indexes pages so Claude can surface them as linked results when answering with search.
| Operator | Anthropic |
| Type | AI search indexer |
| Respects robots.txt | Yes (documented) |
| Documentation | Official operator documentation |
Claude-SearchBot user-agent string
This is how Claude-SearchBot identifies itself in your server logs:
Claude-SearchBot/1.0; [email protected]How to block Claude-SearchBot in robots.txt
Add this to the robots.txt file at the root of your site:
Block Claude-SearchBot
User-agent: Claude-SearchBot
Disallow: /Explicitly allow Claude-SearchBot
User-agent: Claude-SearchBot
Allow: /What blocking changes: Your site is excluded from Claude's search-grounded answers and the referral clicks they generate.
Want to decide for all AI bots at once? Use the AI robots.txt generator.
Other ai search indexers
- OAI-SearchBot (OpenAI)
- Googlebot (Google)
- PerplexityBot (Perplexity)
- Bingbot (Microsoft)
- Applebot (Apple)
- YouBot (You.com)
Is Anthropic sending you visitors?
Glyphex classifies AI referral traffic from ChatGPT, Perplexity, Claude, Gemini and more. See which assistants cite your pages and what those visitors do.
See your AI traffic