Bytespider
AI training crawlerByteDance's (TikTok's parent company) web crawler, widely reported to collect training data for its AI models, including the Doubao assistant. It has repeatedly been reported crawling aggressively, and site owners have reported inconsistent robots.txt compliance.
| Operator | ByteDance |
| Type | AI training crawler |
| Respects robots.txt | Not documented |
| Documentation | None published by the operator |
Bytespider user-agent string
This is how Bytespider identifies itself in your server logs:
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])How to block Bytespider in robots.txt
Add this to the robots.txt file at the root of your site:
Block Bytespider
User-agent: Bytespider
Disallow: /Explicitly allow Bytespider
User-agent: Bytespider
Allow: /What blocking changes: A robots.txt block signals your intent, but given the compliance reports many sites also block Bytespider at the network/WAF level. Blocking has no effect on any user-facing search or referral traffic.
Want to decide for all AI bots at once? Use the AI robots.txt generator.
Other ai training crawlers
- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- Google-Extended (Google)
- Applebot-Extended (Apple)
- Meta-ExternalAgent (Meta)
- CCBot (Common Crawl)
Is ByteDance sending you visitors?
Glyphex classifies AI referral traffic from ChatGPT, Perplexity, Claude, Gemini and more. See which assistants cite your pages and what those visitors do.
See your AI traffic