What search crawlers and AI agents do on your site, and how to see it

A normal analytics script sees none of this traffic. Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and the fetchers behind ChatGPT, Claude and Perplexity request HTML and leave. They do not execute JavaScript, so the tracker never fires and the dashboard shows nothing. The visits that do show up, a person arriving from chatgpt.com, are the end of a chain that started days earlier with a crawl you never saw.

Glyphex now records that chain. The Agents page shows which crawlers and AI agents fetch which pages, whether they are who they claim to be, how fast they pick up new content, and which of those crawls turn into human visits. This article explains what is measured, how the data gets in, and what to do with it.

Three kinds of visitors

The split on the Agents page is human, agentic and crawler.

Crawlers fetch pages for an index or a training set. Search engines (Googlebot, Bingbot, Applebot), AI training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider), AI search indexers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and AI assistant fetchers that open a page because a user asked a question (ChatGPT-User, Claude-User, Perplexity-User). The catalogue has 35 named agents with operator, kind and the documented user-agent token for each.

Agentic browsers are real browsers driven by software on behalf of a person: ChatGPT agent, Comet, Claude in Chrome, a Playwright script. They run JavaScript, so they look human to a normal tracker. Glyphex detects them three ways: the browser's own navigator.webdriver flag, a Web Bot Auth signature on the request, and behaviour (sessions of three or more pages with no pointer, key, touch, click or scroll event at all). They are recorded as agents, never as humans, and their goal completions are counted separately so "agentic conversions" exist as a number.

Humans are everything else and stay in the normal reports. Suspected agent-driven sessions can be excluded from human stats with one filter.

How the data gets in

The tracker still catches JavaScript-capable agents at collect time. For everything else, the site forwards request metadata from the server:

• A Next.js proxy.ts or middleware.ts snippet, about 40 lines, that posts bot-like requests to the ingest endpoint.
• Express middleware or a Cloudflare Worker with the same shape.
• Cloudflare Logpush or a Vercel log drain pointed at the endpoint with ?format=cloudflare or ?format=vercel.
• An nginx access log piped through Vector, for self-hosters.

The snippets pre-filter on user-agent so human requests are never sent. The server classifies each request against the catalogue and discards anything it does not recognise as an agent, so you can forward raw logs without Glyphex keeping human data. Ingest keys work on every plan, including free.

Verified, spoofed, signed

Anyone can send Googlebot/2.1 in a user-agent header. Glyphex fetches the published IP ranges of Google, Microsoft, OpenAI, Anthropic, Apple, Perplexity and Common Crawl daily (2,029 ranges at the time of writing) and checks each hit against them. A hit that claims Googlebot from an address outside Google's ranges is shown as unverified, in its own row, with a one-click firewall rule to block it.

Agents that sign their requests with Web Bot Auth (RFC 9421 Ed25519 signatures, used by ChatGPT agent and Google's agent today) get cryptographic verification: Glyphex fetches the signer's key directory over HTTPS, rebuilds the signature base and verifies it. Valid signatures show a check mark. Invalid ones are flagged as bad signatures, which means someone is impersonating that agent.

What to do with it

The page opens with "Fix these", a short list of recommendations computed from your own data. Twelve rules today, each with the action and, where possible, the fix itself:

• A verified search crawler is getting 403s or 5xx responses: your firewall or rate limit is blocking indexing.
• robots.txt disallows an agent that keeps fetching content pages: robots.txt is advisory, block at the edge with the generated rule.
• Pages that assistants fetch 20+ times a month but that never earn an AI-referred visit: rewrite the opening into a quotable answer.
• Your top human pages were never crawled by any search or AI search engine: internal links and sitemap.
• Training crawlers from an operator whose assistant never cites you: consider blocking training while keeping search.
• Googlebot takes more than a week to fetch new pages: sitemap lastmod and IndexNow, with a generated key.

Resolved recommendations are tracked, so the page also shows how many fixes landed in the last 30 days.

The AX score

Glyphex fetches your top 50 pages once a day the way a crawler would, HTML only, and scores each one from 0 to 100 on ten factors: fetchable without JavaScript, title and meta description and a single h1, an answer paragraph of 40 or more words before the first list or sub-heading, JSON-LD, a canonical link, text-to-HTML ratio, response time, robots.txt not blocking verified crawlers, an llms.txt file, and no crawler errors in the last week. The site score and the worst pages with their failing factors are on the Agents page. The same checker runs on any URL at /tools/ax-checker.

Our own homepage scored 75 before this launch. The failing factor was the answer block: the hero had a headline and a 20-word sub-line, then a list. Adding one 60-word paragraph under the headline moved it to 90.

Operators and policy

The Operators section shows one row per company: OpenAI, Anthropic, Google, Microsoft, Perplexity, Meta, Apple. For each: hits, bytes served, waste (non-200 responses), assistant fetches, AI-referred human visits and conversions from that operator's products, your robots.txt stance per token, observed compliance, and the share of signed requests. Next to it, a suggested stance (keep, allow search only, block training) with the robots.txt lines to paste.

This is the view that answers "is letting GPTBot crawl us worth it". Cost on the left, value on the right, per company.

For your own agents

Everything on the Agents page is available over the MCP server (get_agents_overview, get_agent_recommendations, get_ai_visibility, get_operator_roi, get_ax_score) and the v1 API under /api/v1/stats/agents. The weekly email report has an agents section. Public dashboards can show an agents card.

Getting started

1. Open your site in Glyphex and go to Agents. If you have JavaScript-capable agent traffic already, you will see it.
2. Create an ingest key under API keys and paste the snippet for your stack. Googlebot, GPTBot and ClaudeBot appear within minutes of the first crawl.
3. Run /tools/ax-checker on your homepage and fix the first failing factor.
4. If you do not have an llms.txt, /tools/llms-txt-generator drafts one from your sitemap.

Self-hosters get all of it in the open-source edition, without plan limits.