Webzio-Extended
AI training crawlerA robots.txt control token from Webz.io that marks whether data collected by Webzio may be used for AI and ML training. It is not a separate crawler and does not appear as a user agent in logs.
| Operator | Webz.io |
| Type | AI training crawler |
| Respects robots.txt | Yes (documented) |
| Documentation | Official operator documentation |
No user-agent of its own
Webzio-Extended is a robots.txt control token, not a separate crawler: you will never see it in your server logs. It changes how content fetched by Webz.io's regular crawlers may be used.
How to block Webzio-Extended in robots.txt
Add this to the robots.txt file at the root of your site:
Block Webzio-Extended
User-agent: Webzio-Extended
Disallow: /Explicitly allow Webzio-Extended
User-agent: Webzio-Extended
Allow: /What blocking changes: Signals that content collected by Webz.io must not be used for AI or ML training. Regular Webzio collection is unaffected unless you block Webzio too.
Want to decide for all AI bots at once? Use the AI robots.txt generator.
Other ai training crawlers
- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- Google-Extended (Google)
- Applebot-Extended (Apple)
- Meta-ExternalAgent (Meta)
- Bytespider (ByteDance)
Is AI sending you visitors?
Glyphex classifies AI referral traffic from ChatGPT, Perplexity, Claude, Gemini and more. See which assistants cite your pages and what those visitors do.
See your AI traffic