Meta-ExternalAgent
AI training crawlerMeta's crawler for collecting content used to train its AI models (such as Llama) and to index content for Meta AI products.
| Operator | Meta |
| Type | AI training crawler |
| Respects robots.txt | Yes (documented) |
| Documentation | Official operator documentation |
Meta-ExternalAgent user-agent string
This is how Meta-ExternalAgent identifies itself in your server logs:
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)How to block Meta-ExternalAgent in robots.txt
Add this to the robots.txt file at the root of your site:
Block Meta-ExternalAgent
User-agent: meta-externalagent
Disallow: /Explicitly allow Meta-ExternalAgent
User-agent: meta-externalagent
Allow: /What blocking changes: Your content is excluded from Meta's AI training crawls. Link previews on Facebook/Instagram use a different crawler (facebookexternalhit) and are unaffected.
Want to decide for all AI bots at once? Use the AI robots.txt generator.
Other ai training crawlers
- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- Google-Extended (Google)
- Applebot-Extended (Apple)
- Bytespider (ByteDance)
- CCBot (Common Crawl)
Is Meta sending you visitors?
Glyphex classifies AI referral traffic from ChatGPT, Perplexity, Claude, Gemini and more. See which assistants cite your pages and what those visitors do.
See your AI traffic