Googlebot
AI search indexerGoogle's main search crawler. Included here because content indexed by Googlebot also powers AI Overviews and AI Mode in Google Search. There is no separate crawler or opt-out token for those features beyond standard Search controls.
| Operator | |
| Type | AI search indexer |
| Respects robots.txt | Yes (documented) |
| Documentation | Official operator documentation |
Googlebot user-agent string
This is how Googlebot identifies itself in your server logs:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)How to block Googlebot in robots.txt
Add this to the robots.txt file at the root of your site:
Block Googlebot
User-agent: Googlebot
Disallow: /Explicitly allow Googlebot
User-agent: Googlebot
Allow: /What blocking changes: Blocking Googlebot removes you from Google Search entirely, including AI Overviews. For almost every site this costs far more traffic than it protects. Use Google-Extended (training) or nosnippet/noindex controls instead.
Want to decide for all AI bots at once? Use the AI robots.txt generator.
Other ai search indexers
- OAI-SearchBot (OpenAI)
- Claude-SearchBot (Anthropic)
- PerplexityBot (Perplexity)
- Bingbot (Microsoft)
- Applebot (Apple)
- YouBot (You.com)
Is AI sending you visitors?
Glyphex classifies AI referral traffic from ChatGPT, Perplexity, Claude, Gemini and more. See which assistants cite your pages and what those visitors do.
See your AI traffic