AI bots & crawlers directory

OpenAI's web crawler for collecting training data. Content fetched by GPTBot may be used to train and improve OpenAI's foundation models, such as the GPT series.

Anthropic's web crawler for collecting publicly available content that may be used to train Claude models.

A robots.txt control, not a separate crawler: Google-Extended has no user-agent of its own. Adding it to robots.txt tells Google not to use content that its regular crawlers fetch for training Gemini models and for grounding Gemini answers.

Like Google-Extended, this is a robots.txt control rather than a separate crawler: it tells Apple not to use content already fetched by Applebot for training Apple's foundation models.

Meta's crawler for collecting content used to train its AI models (such as Llama) and to index content for Meta AI products.

ByteDance's (TikTok's parent company) web crawler, widely reported to collect training data for its AI models, including the Doubao assistant. It has repeatedly been reported crawling aggressively, and site owners have reported inconsistent robots.txt compliance.

The crawler of the Common Crawl non-profit, which publishes a free, open archive of the web. Many AI labs train models on Common Crawl data, which makes CCBot the single widest door into AI training datasets, but the same archive also supports academic research.

Amazon's web crawler. Amazon documents it as improving Alexa's ability to answer questions and other Amazon AI services, which can include training uses.

The Allen Institute's crawler collecting web content for training open language models such as the OLMo family, used primarily for research.

A structured-data extraction service that crawls pages into a machine-readable knowledge graph sold to customers, including for AI applications. Because customers can request specific URLs, some fetches behave like user-initiated requests.

A crawler attributed to Cohere that collects web content for training its enterprise language models. Cohere publishes little documentation about it, so compliance behavior is not well documented.

The fetcher ChatGPT uses when a user action requires loading a page, for example when a user asks ChatGPT to read, summarize, or check a specific URL. Requests happen on behalf of a user, not as bulk crawling, and OpenAI states this traffic is not used for training.

The fetcher Claude uses when a user asks it to open a specific page: reading an article, checking a link, or citing a source during a conversation.

The fetcher used when a Perplexity user asks about a specific URL. Perplexity documents that because these are direct user requests, this agent generally ignores robots.txt rules.

Meta's fetcher for user-initiated link loads inside Meta AI products. Because fetches happen on behalf of a specific user request, Meta documents that it may bypass robots.txt rules.

The fetcher Mistral's Le Chat assistant uses to load pages when answering user questions with web citations.

Fetches pages to generate DuckAssist AI answers in DuckDuckGo search, with source links back to the pages used.

OpenAI's crawler for the search index behind ChatGPT search. It discovers and indexes pages so they can be surfaced as linked, cited results. OpenAI states it is not used for training.

Anthropic's crawler for improving Claude's web search: it indexes pages so Claude can surface them as linked results when answering with search.

Google's main search crawler. Included here because content indexed by Googlebot also powers AI Overviews and AI Mode in Google Search. There is no separate crawler or opt-out token for those features beyond standard Search controls.

Perplexity's crawler for building the search index behind its AI answer engine. Perplexity states it is used for search indexing and not for training foundation models.

Microsoft's search crawler. Its index powers Bing search and grounds Microsoft Copilot answers. Microsoft offers a nocache/noarchive meta-tag control for limiting Copilot's use of content rather than a separate crawler token.

Apple's web crawler, feeding Siri and Spotlight suggestions. Content it fetches can also be used for Apple Intelligence features unless you opt out via Applebot-Extended.

The crawler behind You.com's AI search engine, indexing pages to ground its answers with cited sources.