Webzio
AI training crawlerThe current crawler of Webz.io, which collects web content for commercial datasets used for data and AI products. Webz.io states it adheres to robots.txt exclusions but does not publish a user-agent string.
| Operator | Webz.io |
| Type | AI training crawler |
| Respects robots.txt | Yes (documented) |
| Documentation | Official operator documentation |
No user-agent of its own
Webzio is a robots.txt control token, not a separate crawler: you will never see it in your server logs. It changes how content fetched by Webz.io's regular crawlers may be used.
How to block Webzio in robots.txt
Add this to the robots.txt file at the root of your site:
Block Webzio
User-agent: Webzio
Disallow: /Explicitly allow Webzio
User-agent: Webzio
Allow: /What blocking changes: Your content is excluded from Webz.io datasets collected by this crawler.
Want to decide for all AI bots at once? Use the AI robots.txt generator.
Other ai training crawlers
- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- Google-Extended (Google)
- Applebot-Extended (Apple)
- Meta-ExternalAgent (Meta)
- Bytespider (ByteDance)
Is AI sending you visitors?
Glyphex classifies AI referral traffic from ChatGPT, Perplexity, Claude, Gemini and more. See which assistants cite your pages and what those visitors do.
See your AI traffic