Omgili
AI training crawlerThe older crawler of Webz.io (formerly Omgili, also called Omgilibot), which collects forum and news content. Webz.io delivers the data through its web data API to media monitors and researchers, and says the bot always complies with robots.txt.
| Operator | Webz.io |
| Type | AI training crawler |
| Respects robots.txt | Yes (documented) |
| Documentation | Official operator documentation |
Omgili user-agent string
This is how Omgili identifies itself in your server logs:
omgili/0.5 +https://omgili.comHow to block Omgili in robots.txt
Add this to the robots.txt file at the root of your site:
Block Omgili
User-agent: Omgili
Disallow: /Explicitly allow Omgili
User-agent: Omgili
Allow: /What blocking changes: Your content is excluded from Webz.io datasets collected by this crawler.
Want to decide for all AI bots at once? Use the AI robots.txt generator.
Other ai training crawlers
- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- Google-Extended (Google)
- Applebot-Extended (Apple)
- Meta-ExternalAgent (Meta)
- Bytespider (ByteDance)
Is Webz.io sending you visitors?
Glyphex classifies AI referral traffic from ChatGPT, Perplexity, Claude, Gemini and more. See which assistants cite your pages and what those visitors do.
See your AI traffic