Google-Agent
AI assistant fetcherGoogle's agent that browses websites on behalf of a user. As a user-triggered fetcher it generally ignores robots.txt. Google is experimenting with Web Bot Auth (signed requests) under https://agent.bot.goog.
| Operator | |
| Type | AI assistant fetcher |
| Respects robots.txt | Partially |
| Documentation | Official operator documentation |
Google-Agent user-agent string
This is how Google-Agent identifies itself in your server logs:
compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agentHow to block Google-Agent in robots.txt
Add this to the robots.txt file at the root of your site:
Block Google-Agent
User-agent: Google-Agent
Disallow: /Explicitly allow Google-Agent
User-agent: Google-Agent
Allow: /What blocking changes: A robots.txt block is generally not honored for user-triggered fetches; blocking at the network/WAF level is the reliable option. Doing so stops Google's agent from acting on pages for its users.
Want to decide for all AI bots at once? Use the AI robots.txt generator.
Other ai assistant fetchers
- ChatGPT-User (OpenAI)
- Claude-User (Anthropic)
- Perplexity-User (Perplexity)
- Amzn-User (Amazon)
- Meta-ExternalFetcher (Meta)
- MistralAI-User (Mistral AI)
Is Google sending you visitors?
Glyphex classifies AI referral traffic from ChatGPT, Perplexity, Claude, Gemini and more. See which assistants cite your pages and what those visitors do.
See your AI traffic