# Most of What AI Agents Ask For and Cannot Find Is Not Content

Across six domains over 28 days, AI user agents requested 59 URLs that did not exist. Twenty-six of them were credential, configuration or remote-execution paths, and those 26 took 226 of the 314 requests. Ranked by demand, the first request for a page a reader could want sat 24th. An AI-crawler 404 list is not a content backlog until it is filtered.

Canonical URL: https://machinerelations.ai/research/ai-crawler-404-demand-probe-traffic-2026
Published: 2026-09-22
Research type: Study
Tags: measurement, ai-crawlers, methodology, machine-relations, content-strategy

## Source Body

Over 28 days across six domains we operate, requests carrying AI user-agent strings asked for 59 distinct URLs that returned no page. Twenty-six of those URLs were paths to credentials, configuration, server diagnostics or remote execution. Those 26 took 226 of the 314 requests. Twenty URLs — 22 requests — were for a page a reader could plausibly want.

Ranked by request volume, the most-wanted missing page on our estate was an SSH host key.

This matters because the missing-URL list has quietly become a content planning input. The advice is reasonable on its face: machines are telling you what they expected to find, so build it. A curated brief published at [authoritytech.io](https://authoritytech.io/curated/ai-crawler-logs-content-demand-signal) puts "AI crawler requested it and the URL is missing" at priority one of its refresh queue. We publish that advice. We also run the logs. Applied to our own logs without a filter, that priority-one rule points a content team at `host.key`.

## What we measured

The [Machine Relations](https://machinerelations.ai/glossary/machine-relations) crawl pipeline reads edge logs for six domains — authoritytech.io, machinerelations.ai, jaxonparrott.com, christianlehman.com, paralabs.ai and paralax.ai — and classifies each request by the user-agent string it carries. The window readable on 2026-09-21 ran 28 days, from 2026-08-23, and held 143,788 requests under AI identities, of which 24,833 were classed as assistant fetches rather than bulk crawling.

From that traffic the pipeline extracts every path that returned no page, which is the list this study is about: 59 distinct URLs, 314 requests.

We sorted all 59 by what the path is for, not by who asked:

| What the path targets | Distinct URLs | Requests | Share of requests |
|---|---|---|---|
| Credentials, configuration, diagnostics, remote execution, path traversal | 26 | 226 | 72.0% |
| Framework internals, routing artefacts, opaque tokens | 13 | 66 | 21.0% |
| A page a reader could want | 20 | 22 | 7.0% |

The first group is the standard contents of a vulnerability scanner's wordlist: private key files, environment and configuration endpoints, cloud service-account credentials, a WordPress REST route on sites that have never run WordPress, a [Spring Boot actuator](https://docs.spring.io/spring-boot/reference/actuator/endpoints.html) logger endpoint, a [Go pprof](https://pkg.go.dev/net/http/pprof) profiling route, an [nginx](https://nginx.org/en/docs/http/ngx_http_stub_status_module.html) status page, a [Vite](https://vite.dev/config/server-options) dev-server file route, and percent-encoded traversal sequences aimed at `/proc/self/environ` — [CWE-22](https://cwe.mitre.org/data/definitions/22.html), the classic [path traversal](https://owasp.org/www-community/attacks/Path_Traversal) shape, alongside [CWE-540](https://cwe.mitre.org/data/definitions/540.html) configuration disclosure.

None of them resolved. We re-checked the whole group live on 2026-09-22 and every path returned 404, including the two private-key filenames and the cloud service-account key. The probes found nothing. That is not the finding. The finding is what happened to them next.

## The demand score ranked the scanner first

Our pipeline scores each missing URL by request volume and recency, then labels it an opportunity with a suggested slug. Every one of the 59 came back labelled a content opportunity. There is no class in the classifier for a path that is not content.

So the ranked list reads:

| Rank by demand score | Path | Requests | What it is |
|---|---|---|---|
| 1 | `/curated/host.key` | 25 | SSH host key |
| 2 | `/blog/manage/env` | 23 | Environment variables |
| 3 | `/blog/manifest.webmanifest` | 21 | Framework asset |
| 4 | `/curated/api/openapi.json` | 21 | API schema |
| 5 | `/blog/api/fs/exec` | 16 | Remote execution |
| … | … | … | … |
| 24 | `/glossary/source-role` | 2 | A definition a reader wanted |

Twenty-three rows of scanner traffic and framework noise before the first request a human would recognise as demand. A weekly review that reads the top ten and stops — which is how a prioritised list gets used — sees no content demand at all, because there is none in the top ten.

The distribution is also concentrated in a way that has nothing to do with editorial interest. Of the 59 missing URLs, 51 were on authoritytech.io, 5 on machinerelations.ai, 2 on jaxonparrott.com, 1 on christianlehman.com and none on paralabs.ai or paralax.ai. That ordering tracks which hosts are exposed to broad internet scanning, not which hosts have gaps a reader noticed.

## The seven percent was worth reading

Filtered down, the 20 genuine rows were specific and useful in a way the raw list was not:

- **Four were definitions.** Requests for `/glossary/source-role`, `/glossary/pr-for-ai-search`, `/glossary/entity-resolution` and `/glossary/machine-relations-index-mri` on machinerelations.ai — our own category vocabulary, asked for by PerplexityBot and ClaudeBot. Re-requested on 2026-09-22, all four resolve.
- **One was an encoding defect.** A request for a research URL whose hyphens were non-breaking U+2011 characters rather than ASCII. The ASCII form of that URL returns 200. A machine held our slug and could not reach the page.
- **Five were near-miss or renamed slugs** on published pages — a truncated slug, a title variant, an article now living under a different filename. All five resolve today by redirect. Those are routing repairs, not articles.
- **Ten still return 404**, at one request each over 28 days.

Ten one-off requests is not a content plan. But it is the honest size of the thing, and knowing that is worth more than a ranked list of 59 that reads like demand and is not. Half of the useful rows were not writing work at all: they were a definition already owed, an encoding defect, and redirects that already exist.

## What this study does not claim

We did not verify that the agents were who their user-agent strings said they were, and nothing here should be read as saying Anthropic, OpenAI, Google, Apple, Perplexity or Cohere scanned our estate. All six publish their crawler identities and the IP ranges or reverse-DNS patterns that authenticate them — [OpenAI](https://developers.openai.com/api/docs/bots), [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers), [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), [Apple](https://support.apple.com/en-us/119829), [Perplexity](https://docs.perplexity.ai/guides/bots) and Cohere all document theirs, and the IETF is standardising a cryptographic replacement in [web bot auth](https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/). This dataset carries no client-IP dimension, so it cannot run those checks in either direction.

That limit is the subject of a separate measurement we published on [the AI-crawler user-agent verification gap](https://machinerelations.ai/research/ai-crawler-user-agent-verification-gap-2026), and it cuts the same way here: a user-agent string is a claim, and the [Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) it is usually paired with is honoured voluntarily by whoever chooses to. Whether the traffic was genuine agent activity or a scanner wearing the name, the operational conclusion is identical, which is why the study is worth running without the identity dimension: the path is not content either way.

For completeness, the requests against the 26 credential and execution paths arrived under eight different claimed identities: ChatGPT-User carried 62, Claude-User 49, ClaudeBot 39, GPTBot 24, Google-Extended 23, Applebot 14, Claude-SearchBot 12 and cohere-ai 3.

## The rule this produces

An AI-crawler 404 list is not a content backlog. It becomes one after a filter, and the filter is cheap:

| Drop before ranking | Because |
|---|---|
| Paths naming a credential or key file | `host.key`, `key.pem`, `id_ecdsa`, service-account JSON, `.boto` |
| Paths naming configuration or environment | `/env`, `/api/env`, config JSON, firebase config |
| Diagnostic and admin endpoints | actuator, pprof, status pages, health checks |
| Any path containing `..`, `%2e%2e` or `%252f` | Traversal, not a URL a reader typed |
| Platform routes for a stack you do not run | A WordPress route on a site with no WordPress |
| Framework internals and build artefacts | Manifests, source-map lookups, function routers |
| Opaque tokens with no word boundaries | Not a slug anyone would link |

What survives is small. On our estate it was 20 URLs and 22 requests out of 314, and half of those twenty were fixes rather than articles: four definitions, one character-encoding defect and five redirect targets. Every one of them was sitting underneath a ranked list whose first recommendation was to write a page called `host.key`.

The broader point is about the category, not about us. Crawler analytics is being sold as a demand instrument; [Cloudflare's AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/) and similar products put AI-agent request data in front of marketing teams who have never read a server log and have no reason to recognise `/actuator/loggers` as anything other than a page they forgot to write. A demand metric that ranks attack surface first is not a demand metric. Any tool that reports AI-agent 404s should state its filter, and any team reading one should ask for it.

## Methodology and sources

**Instrument.** Edge request logs for six domains, collected daily and consolidated by the Machine Relations crawl pipeline into `editorial/data/ai-crawl-intelligence.json`. Requests are classified as AI traffic by user-agent string against a maintained pattern set, then bucketed as assistant fetch, search fetch, crawler or agent.

**Window.** 2026-08-23 to 2026-09-21 inclusive, 28 readable days. The pipeline requested 30 days; the source returned 28, and the shorter window is the one reported. Artifact generated 2026-09-22T00:10:30Z.

**Population.** 143,788 requests carrying AI user-agent strings, of which 24,833 were assistant fetches. From these, 59 distinct request paths returning no page, with 314 requests against them.

**Classification.** Each of the 59 paths was assigned to exactly one of three groups by what the path targets, independent of which identity requested it. Group one is any path naming a credential or key file, an environment or configuration endpoint, a diagnostic or administrative endpoint, a file-system or template-execution route, or containing an encoded traversal sequence. Group two is any path belonging to a web framework, a service runtime or a routing artefact, plus opaque tokens with no word structure. Group three is everything else: a path shaped like a page a reader could ask for. The three groups are exhaustive and mutually exclusive, and sum to 59 URLs and 314 requests.

**Verification.** Every path in group one was re-requested over plain HTTPS on 2026-09-22 and returned 404. All twenty paths in group three were re-requested the same day: four glossary paths resolve (two directly, two by redirect), five resolve by redirect, one resolves once its non-breaking hyphens are written as ASCII, and ten still return 404.

**Limits.** Identity is unverified: this dataset has no client-IP dimension, so no user-agent claim in it was authenticated against a published range, and the per-identity counts are counts of claims. The estate is six domains under common operation, so the distribution of scanning against it is not a sample of the web. Request counts are a floor, not a total, where log collection capped a busy day. The demand score described here is our pipeline's own ranking field, not an industry metric.

**Context.** Index release `mri_score_v2.0+2026-09-22+4aa6d302f7ed`, covering 22,828 cited domains and 127,980 source events, is the current [Machine Relations Index](https://machinerelations.ai/glossary/machine-relations-index-mri) release at publication. It is not an input to this study, which is a log measurement, and is named so the reader can date the estate against it.

## Attribution

This research is published by Machine Relations Research, the research program of machinerelations.ai — the public research and standards initiative that publishes the glossary, research, evidence, and measurements for the Machine Relations discipline. Provenance and editorial standards: https://machinerelations.ai/about

## Machine-readable related links

### Related concepts

- [Machine Relations Index (MRI)](https://machinerelations.ai/glossary/machine-relations-index)
- [Machine Relations (MR)](https://machinerelations.ai/glossary/machine-relations)
- [MRI Score](https://machinerelations.ai/glossary/mri-score)
- [Source Role](https://machinerelations.ai/glossary/source-role)

### Supporting research

- [How to Run an AI Citation Gap Analysis: The Step-by-Step Methodology for Finding What AI Engines Won't Cite (2026)](https://machinerelations.ai/research/ai-citation-gap-analysis-methodology-2026)
- [Only 508 of 22,377 AI-Cited Domains Are Cited Often Enough to Score](https://machinerelations.ai/research/ai-citation-evidence-floor-scoreable-domains-2026)
- [Verifying AI crawler user agents: 0% for 48 days, then 73% once we kept the IP](https://machinerelations.ai/research/ai-crawler-user-agent-verification-gap-2026)
- [When an AI Citation Index Is Wrong: One Misclassified Field, 57 Corrected Pages](https://machinerelations.ai/research/ai-citation-index-correction-record-2026)

### Framework context

- [Machine Relations Stack](https://machinerelations.ai/stack)
- [Evidence Base](https://machinerelations.ai/evidence)