Google-Extended
AI training crawlerA robots.txt control, not a separate crawler: Google-Extended has no user-agent of its own. Adding it to robots.txt tells Google not to use content that its regular crawlers fetch for training Gemini models and for grounding Gemini answers.
| Operator | |
| Type | AI training crawler |
| Respects robots.txt | Yes (documented) |
| Documentation | Official operator documentation |
No user-agent of its own
Google-Extended is a robots.txt control token, not a separate crawler: you will never see it in your server logs. It changes how content fetched by Google's regular crawlers may be used.
How to block Google-Extended in robots.txt
Add this to the robots.txt file at the root of your site:
Block Google-Extended
User-agent: Google-Extended
Disallow: /Explicitly allow Google-Extended
User-agent: Google-Extended
Allow: /What blocking changes: Your content is excluded from Gemini training and grounding. Regular Google Search indexing and ranking are unaffected; Googlebot keeps crawling as before. Note that Google's AI Overviews in Search are part of Search and are NOT controlled by Google-Extended.
Want to decide for all AI bots at once? Use the AI robots.txt generator.
Other ai training crawlers
- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- Applebot-Extended (Apple)
- Meta-ExternalAgent (Meta)
- Bytespider (ByteDance)
- CCBot (Common Crawl)
Is AI sending you visitors?
Glyphex classifies AI referral traffic from ChatGPT, Perplexity, Claude, Gemini and more. See which assistants cite your pages and what those visitors do.
See your AI traffic