None of 59,166 AI crawler requests were verified, and 914 could not have been real

Over 24 days, six web properties logged 59,166 requests carrying the user agent of an AI crawler. Not one of them was verified. Nine hundred and fourteen of them could not have been genuine, because the user agent they claimed does not exist.

That last figure is the one to start with, because it needs no statistical judgment. Google's crawler documentation states plainly that Google-Extended "doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity." Google-Extended is a preference token you put in robots.txt. It never appears in an HTTP request, because nothing sends it.

Our logs contain 914 requests that sent it, across 16 separate days. Every one is forged, by the publisher's own published contract. A further six requests claimed Applebot-Extended, which Apple documents the same way — an opt-out token for robots.txt, not a crawler.

This is the retrieval leg of machine relations, and it is the leg nobody outside a site can inspect. The Machine Relations Index measures which domains answer engines cite. Search Console measures what happened on a results page. Neither can see what actually arrived at your server and asked for a file. Only the site owner holds that, which is why this audit turns the instrument on our own logs — and finds the instrument wanting.

What was measured #

The dataset is 24 daily Cloudflare snapshots of AI-bot traffic across six properties: authoritytech.io, machinerelations.ai, jaxonparrott.com, paralabs.ai, paralax.ai and christianlehman.com. The window runs 2026-08-23 to 2026-09-17. That is 24 of 26 calendar days; 2026-09-03 and 2026-09-04 were never collected and are absent rather than empty.

The snapshots hold 25,730 grouped rows, each a combination of request path, HTTP status, claimed user agent and host, with a request count. They sum to 59,166 requests. Every figure below is computed from those rows.

Eight distinct bot labels appear:

A crawl-analytics dashboard would render that table as a picture of AI interest in these properties. It is not that. It is a table of what arriving software said about itself.

Every request is unverified, and the field that says so is a constant #

Each stored row carries three identity fields. Across all 59,166 requests, identityVerified is false without exception. Where any evidence is recorded at all, the value is the string user_agent_pattern_only.

Reading the collector explains why, and the explanation is worse than a low verification rate. identityVerified: false is written as a literal when the row is constructed. It is not the outcome of a check that failed. It is a constant, and no request in this dataset or any future one could ever be marked otherwise.

The cause is upstream of that line. The collector queries Cloudflare's GraphQL analytics API for four dimensions: request path, response status, user agent and host. It never requests a client IP. Identity verification for every AI crawler in existence is a reverse-DNS lookup or an IP-range match. A pipeline that does not retain the source address is not failing to verify; it is structurally incapable of verifying, and the identityVerified field is decoration on a measurement that was never attempted.

The filter that decides what counts as an AI bot is a single substring match — userAgent_like "%GPTBot%" and eight siblings. Anything that puts that substring in its user-agent header enters the dataset as an AI crawler. There is no second condition.

What is actually in the traffic #

If a meaningful share of this traffic is not crawlers, it should be doing something crawlers do not do. It is.

The cleanest test needs no keyword list and no judgment about what a path means. Across all 48 days of collection, 4,283 distinct host-and-path combinations returned a successful status at least once. Any request in the window for a combination outside that set asked for something these sites have never served to anyone.

7,553 requests — 12.77% of the total — asked for paths that have never once been served, across 2,180 distinct host-and-path pairs.

That could in principle be honest crawling of dead links. It is not. 69.2% of those requests match a credential or configuration artifact. Applying such a classifier directly, 5,803 requests (9.81%) over 515 distinct paths asked for files in these families:

None of these sites runs WordPress, PHP, Laravel, or a Vite dev server. This is credential enumeration, and it is inside an AI-crawler dataset for one reason: it put the right substring in its user-agent header.

The two measures disagree on the exact figure — 9.81% against 12.77% — because one depends on a keyword list and the other does not. That gap is the honest width of the answer, and every ranking below is identical under both.

Contamination runs inversely to real footprint #

Per claimed bot, the share of traffic asking for never-served paths:

The mechanism is visible in the arithmetic. Never-served volume per label spans 3.9x, from 376 to 1,473 requests. Real retrieval volume spans 869.7x, from 23 to 20,002. Attackers distribute themselves roughly evenly across the bot names they might wear; real crawlers do not.

So the contamination rate of any bot label is, to a first approximation, a function of how little genuine traffic that label carries. The smaller a crawler's true presence on your site, the more of its reported number is somebody else's attack traffic. This inverts the way such a table is normally read. A dashboard row showing modest activity from an emerging engine is the row most likely to be mostly noise.

The same law holds per property:

The first row is the finding in miniature. Every one of the 819 requests that reached paralax.ai claiming to be an AI crawler asked for a path the site has never served. Not one fetched a page. A crawl dashboard would have reported 819 AI-bot requests for that property over 24 days. The correct number is zero, and it took the site's own logs to say so.

The string is a filter only where the vendor maintains it #

Two of the 24 days retain the full claimed user-agent string — 5,115 requests across 25 distinct strings. That slice contains a natural experiment.

The fourth and seventh rows differ by one character. GPTBot/1.4 is the version OpenAI currently documents; across 570 requests it asked for a never-served path zero times. GPTBot/1.3 is the superseded version; across 202 requests, 83.7% were probes. Same structure, same suffix URL, one digit apart, opposite populations.

Apple gives the same result. The 718-request Applebot string is character-for-character the example in Apple's own documentation, including the +http://www.apple.com/go/applebot suffix, and carries no probe traffic. The 125-request variant drops that suffix and advertises Version/17.0 instead of 17.4, and is 75.2% probes. Apple notes that Applebot "will update the browser version that it advertises" — so the current version is a moving target, and the forgeries are working from a stale copy.

OpenAI's OAI-SearchBot splits the same way. The three variants carrying the documented Chrome/131.0.0.0 Safari/537.36 prefix account for 494 requests and 0.0% probes, including the robots.txt-marked form OpenAI documents for robots fetches. The three truncated variants account for 196 requests at 52% to 73%.

Then the rule breaks, and where it breaks is instructive.

PerplexityBot's 419-request string is exactly the one in Perplexity's documentation, and it carries 20.0% probe traffic. Perplexity has never moved past PerplexityBot/1.0, so there is no stale version for a forger to get wrong — the documented string is the only string, and copying it perfectly is free. ClaudeBot's dominant 807-request form carries 11.5%, and Anthropic publishes no user-agent string at all, directing site owners to its IP list instead.

The pattern, stated as a finding: version rotation is an accidental anti-forgery mechanism. Vendors who publish an exact string and keep bumping it (OpenAI, Apple) leave a clean current form and a contaminated stale one. Vendors who never rotate, or publish no string, hand out a permanently valid costume. In neither case is the string proof — it is a heuristic whose strength depends entirely on the vendor's release discipline, and it collapses the moment a forger refreshes its copy.

The fix exists, is free, and is published by every vendor #

Every operator of a bot in this dataset publishes the IP ranges its crawler uses. All six files were fetched on 2026-09-18:

Two things stand out. The ChatGPT-User file was refreshed the morning these figures were computed and carries ten times the prefixes of any other, which is consistent with it being the label carrying the most genuine traffic here. And Perplexity's file has not been touched in nineteen months while listing eight prefixes — so a publisher choosing to allowlist by IP should expect that list to be the least reliable of the six.

Every one of the 337 prefixes across all six files is IPv4. Not one IPv6 prefix is published by any vendor. A site reached over IPv6 by something claiming to be GPTBot cannot be confirmed or refuted from the published ranges at all, and that gap belongs to the vendors, not to site owners.

The complementary method is reverse DNS. Google documents verifying its crawlers by reverse lookup to googlebot.com, and Apple documents the same against *.applebot.apple.com with a forward confirmation. Both are two lookups per address and neither requires vendor cooperation beyond what is already published. Cloudflare additionally maintains a verified-bots list that performs this check at the edge — available to the same account whose analytics API produced this dataset, and not consulted by it.

Longer term the industry is moving to cryptography rather than address lists. The IETF's web bot auth work proposes that agents sign their requests using HTTP Message Signatures, which would make a forged bot identity computationally hard instead of a header edit. Until that ships and is adopted, IP verification is the only available answer, and it is available today.

Three defects this audit found in our own collector #

Stated plainly, because the same three are likely present in any log pipeline built the same way.

It cannot verify. No client-IP dimension is requested, so no identity check is possible and identityVerified is a hardcoded false. Adding the dimension and matching against the six published files above converts every number in this study from a claim into a measurement.

One of its nine patterns has never matched anything. The configured list includes Anthropic, intended to catch Anthropic's crawlers. In 48 days it has returned zero rows. Cloudflare's userAgent_like is case-sensitive, and of the 867 ClaudeBot requests whose full string was retained, 861 carry lowercase anthropic.com and none carry a capitalised Anthropic. The pattern as written can never match. It was not under-performing; it was dead, and the totals looked plausible enough that nobody checked.

Every total is a floor. The query is capped at 100 grouped rows per bot, per property, per day. Of the 788 queries that returned anything, 104 — 13.2% — came back with exactly 100 rows, concentrated on the busiest property and the highest-volume labels. Those days are truncated by an unknown amount. Wherever this study says a count, the true count is that or higher.

Checking your own logs #

1. Pull a day of AI-bot requests with the client IP included. If your pipeline does not retain the source address, stop here and fix that first; nothing else in this list is possible without it.
2. Match each address against the publisher's own range file from the table above. Reverse DNS works too, and for Google and Apple it is the documented method.
3. Split the unverified remainder by path. Requests for .env, .git, id_rsa, wp-json and their kin are not crawlers, whatever their header says.
4. Check for a Google-Extended or Applebot-Extended user agent. Any request claiming either is forged with certainty, and its presence tells you your log has never been filtered.
5. Look at your smallest bot labels first. That is where contamination concentrates, because attack volume is roughly constant per label while real volume is not.
6. Compare requested paths against paths your site actually serves. A label that has never fetched a real page has a true retrieval count of zero, regardless of its request count.
7. Check whether your query is truncating. If a grouped result comes back at exactly your row limit, that day's total is a floor.

Step 4 is the fastest and the most diagnostic. It takes one search and returns a yes or no about whether any filtering has ever been applied to the numbers you have been reading.

What this does and does not establish #

It does not establish that AI crawler traffic is mostly fake. The opposite, for the labels that matter: ChatGPT-User, ClaudeBot and Applebot account for 41,468 of the 59,166 requests here, and 93.4% of their traffic asked for paths these sites genuinely serve. Real retrieval by real crawlers is the bulk of this dataset. The claim is narrower and harder: no individual request in it can be attributed, and the aggregate is overstated by an amount that varies from 5% to 100% depending on which row you read.

It does not establish that anything leaked. 295 requests to credential-family paths were logged with HTTP 200. All the paths involved were re-requested live on 2026-09-18 and every one returned 404 or 403. The 404 responses are served as a fully rendered 55,933-byte page, which is the likeliest explanation for the logged 200s: a framework catch-all route answering a miss with a page. That is worth naming as a hazard in its own right — a 200 in a crawl log does not establish that content was served.

It does not generalise cleanly to larger sites. Six properties in one subject area, 24 days, one CDN's analytics. The direction should generalise, because the mechanism is a missing verification step rather than anything specific to these sites, and third-party inventories of AI agent user agents such as Dark Visitors show the same labels in wide circulation. The magnitudes are ours.

One number is not laundered. The collector's own totals field sums to 59,415 against the 59,166 reconstructable from stored rows — an excess of 249 requests, 0.42%, spread across all four bot categories, which could not be accounted for from the rows. Every figure in this study uses the rows, and this paragraph exists because the discrepancy is small enough to have been quietly dropped.

Why this belongs in machine relations #

The discipline has a measurement problem shaped like a chain. An engine retrieves your page, decides whether to cite it, and a person may or may not arrive. Three legs, three instruments, and the first one turns out to rest on an unverified self-report.

That matters beyond log hygiene, because retrieval counts are load-bearing. They get used to argue that a page is working, that an engine has discovered a section, that a robots.txt change took effect. Every one of those arguments is only as sound as the identity behind the request, and in this dataset the identity behind every single request is an unchecked claim — with 914 of them claiming something that cannot exist.

This is also why the Machine Relations Index is built the way it is. The Index does not read crawler logs. It observes what answer engines actually cite, across 124,397 source events over 125 days and 22,179 domains in the 2026-09-18 release, and a domain's standing there depends on nothing any third party can spoof, because the evidence is the engine's own output rather than a header a visitor wrote about itself. A publisher can look up its own domain's position in that release. The citation leg is measurable from outside. The retrieval leg is measurable only from inside — and this study is what we found when we checked whether ours was measuring anything at all.

The useful conclusion is not that crawl analytics are worthless. It is that a crawl-analytics number without an identity check is a count of claims, that the error runs largest exactly where a publisher is most likely to be looking for early signal from a new engine, and that the correction costs one extra field in a query and a file every vendor already publishes.

Methodology #

Source: 24 daily Cloudflare httpRequestsAdaptiveGroups snapshots across six zones, window 2026-08-23 to 2026-09-17, collected daily at approximately 00:10Z for the preceding complete UTC day. 2026-09-03 and 2026-09-04 were not collected. 25,730 grouped rows totalling 59,166 requests. Dimensions retained per row: request path, edge response status, host, and (on 2026-09-16 and 2026-09-17 only) the claimed user-agent string. No client IP is retained, which is the subject of this study.

Bot attribution is the collector's own substring match on the user-agent header, userAgent_like "%pattern%", across nine configured patterns, of which eight ever matched. Requests are counted, not rows; where a figure counts rows or distinct paths it says so.

"Never-served" is defined without reference to path semantics: a host-and-path pair is never-served if no row in any of the 48 collected snapshots records a 2xx status for it. 4,283 pairs met that bar; requests for any pair outside it are counted as never-served. The credential-family classifier is a separate, stricter measure using explicit pattern families, listed in the table above with their counts; it excludes all content routes by prefix, so a content slug containing a word like "secret" is never classified as a probe. Both measures are reported because they bound the answer from either side and agree on every ordering.

Probe shares in the user-agent variant table use the credential-family classifier on the 5,115 requests from 2026-09-16 and 2026-09-17 that retain a claimed string. Vendor range files and vendor documentation were fetched on 2026-09-18 and prefix counts and timestamps are as of that date. Live status re-checks of probe paths were performed on 2026-09-18 against authoritytech.io and machinerelations.ai.

Limits that bound every figure: the 100-row query cap truncates 104 of 788 non-empty queries, so all totals are floors; two days are missing from the window; the collector's own totals exceed the row-level reconstruction by 249 requests (0.42%), and the row-level figures are used throughout; and no request in the dataset can be verified, which means the split between genuine and forged traffic is inferred from request behaviour rather than measured from identity.