Blocked and Cited Anyway: What a robots.txt Disallow Actually Does to AI Citation

Summary: reddit.com is the most-cited domain in the Machine Relations Index. Its robots.txt contains exactly one rule group — User-agent: * followed by Disallow: / — and it was cited in 2,027 of 16,307 observed answer runs, on 123 of 129 observed days, by four of the six measured engines, at Confidence A. Across the 250 most-cited domains in the release generated 2026-09-22, 224 served a parseable robots.txt to our fetch. 30 of those 224 fully disallow at least one of the nine AI crawler tokens we tested. Most of them are cited by the corresponding engine anyway. The exception is instructive: a disallow on a search token tracks absence far better than a disallow on a training token, which is exactly what the operators' own documentation says should happen.

Almost every published AI-visibility checklist opens with the same instruction: check your robots.txt, unblock the AI crawlers, because a blocked crawler cannot read you and an engine cannot cite what it cannot read. It is offered as a binary gate — the cheap, obvious prerequisite before any content work is worth doing.

It is testable, and until now it has mostly been tested from one side. Several good surveys measure how often sites block AI crawlers: a robots.txt census of the Tranco top million found 9.33% of scanned domains fully blocking at least one major AI crawler, with blocking concentrated at the head and among publishers (Crawlora AI-Crawler Blocking Index, June 2026); a 500-site audit found 24.0% of top-ranked domains fully blocking at least one (Forage AI, September 2026); a 10,894-domain baseline found that half the publishers Google AI Mode cited in a sample of queries block at least one AI crawler (HasData AI Crawler Block Index, July 2026).

What those surveys cannot do is join a site's stated policy to a large, engine-by-engine record of whether that engine actually cited it. That join is what the Machine Relations Index makes possible, and it is what this study reports.

What was measured #

Two datasets, one key: the domain.

On the citation side, the Machine Relations Index public view generated 2026-09-22 under methodology version mri_score_v2.0. The window runs 2026-05-10 to 2026-09-22, 129 observed days, 22,828 cited source domains, 127,980 source events and 16,307 observed answer runs across six engines — ChatGPT, Claude, Gemini, Google AI Mode, Google AI Overviews and Perplexity. For every domain the release records which of the six engines cited it at least once in the window. The 250 domains carrying a published head ranking were taken as the population.

On the policy side, each of those 250 domains had its /robots.txt fetched live on 2026-09-23 from a datacenter host, www first and then the apex, with one retry per host. A response counted as readable only if it returned HTTP 200 and the body actually parsed as a robots file — at least one line beginning User-agent:, Disallow:, Allow: or Sitemap:, and no HTML document markers. That validation matters: four head domains return an HTML challenge or marketing page at HTTP 200 for the robots.txt path, and a census that trusts the status code alone silently records them as files that name no crawler at all.

224 of the 250 returned a readable file, or 89.6%. The remaining 26 are named in full at the end of this piece.

A domain counts as fully disallowing a token when its file contains a group naming that token whose rules include the exact rule Disallow: / and no Allow rule reopening the root. A path-level restriction is not counted. Nine tokens were tested, chosen because each operator documents them as the controls for its own products.

The result #

Of the 224 head domains with a readable file, 148 name none of the nine tokens — 66.1%. 30 fully disallow at least one, which is 13.4%. That minority is the population the rest of this study is about.

Read the first row and the second row together. They describe the same engine, measured over the same 224 domains, in the same window. A head domain that shuts OpenAI's training crawler out completely was still cited by ChatGPT roughly seven times in ten. A head domain that shuts OpenAI's search crawler out was cited three times in ten. Same operator, same file, different token, and the gap runs the full width of the measurement.

The same shape appears at Google. A full disallow on Google-Extended goes with citation by Gemini for 6 of 18 head domains, against 95.1% for everyone else — the sharpest association in the table.

And it disappears entirely at Perplexity. All 10 head domains that fully disallow Perplexity-User were cited by Perplexity, and 15 of the 17 that fully disallow PerplexityBot were too.

Why the gradient is there #

The operators say so themselves, and it is the part of their documentation that almost nobody reads before writing the checklist.

OpenAI runs its tokens as independent switches: "a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training." It states the consequence in one direction only — sites "opted out of OAI-SearchBot will not be shown in ChatGPT search answers" — and says plainly of the user-initiated agent that "because these actions are initiated by a user, robots.txt rules may not apply," and that ChatGPT-User "is not used to determine whether content may appear in Search" (OpenAI crawler documentation, read 2026-09-23). Disallowing GPTBot is an instruction about a training corpus. It was never an instruction about citation, and the measurement agrees.

Anthropic splits its fleet the same way, into ClaudeBot for training, Claude-User for user-initiated requests, and Claude-SearchBot, which it says is the one whose removal "prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results" (Anthropic crawler documentation, read 2026-09-23).

Google's crawler documentation describes a fleet split between common crawlers, special-case crawlers and user-triggered fetchers, with Google-Extended as a control over Gemini grounding rather than over inclusion in Google Search (Google crawler documentation, read 2026-09-23). A domain removed from Gemini's grounding path can still reach an answer through the Search index that a separate token governs, which is the likeliest reading of the six head domains that carry the disallow and were cited anyway.

Perplexity is the one that does not fit. Its documentation describes PerplexityBot as "designed to surface and link websites in search results on Perplexity," and says Perplexity-User "controls which sites these user requests can access" (Perplexity crawler documentation, read 2026-09-23). Those are descriptions of a control over visibility, and at the head of this index the citation record does not move with it. We are reporting that gap, not explaining it.

Underneath all of this is the matching rule that makes per-token blocking work at all: a crawler obeys the group that names it, or the * group if none does (RFC 9309, section 2.2.1). A directive addressed to one product token reaches no other.

The ten that block everyone #

Ten of the 224 head domains, 4.5%, carry a User-agent: * group with Disallow: / — the rule that applies to any crawler the file does not name specifically. Every one of them is cited by at least three engines.

Eight of the ten also name AI tokens in their own groups, so for those agents the blanket rule is not the operative one and the named group decides. Two do not, and they are the cases with nothing else to read.

reddit.com is the cleanest. Its entire robots.txt is one group: User-agent: *, Disallow: /. It names no AI crawler, no search crawler, nothing. Every automated fetcher on the web falls through to a rule that asks it not to fetch any path — and reddit.com is the most-cited domain in a 22,828-domain index, present on 123 of 129 observed days.

linkedin.com is the other end of the same point. It carries the blanket rule and names ten AI tokens in their own groups, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, PerplexityBot, Perplexity-User and Google-Extended, under a full disallow. It is ranked third in the release and cited by all six engines in 1,104 of 16,307 runs on 120 of 129 days, at Confidence A. There is no token left to add.

What this does not say #

It does not say blocking is free, and nobody should read a row of this table as advice to block. The association runs the way the operators' documentation predicts wherever a search token is involved, and the one clean lever in the table — OAI-SearchBot at 30.0% against 83.2% — points the other way.

It does not establish causation in either direction. These are 250 of the most-cited domains on the web, which is precisely the population with the most alternative routes into an answer: brand-name recall, partner feeds, licensing agreements, syndicated copy on other hosts, and the search indexes that separate tokens govern. What survives at the head may not survive in the tail, and this study measures only the head.

It does not measure fetching. The Index records that an engine cited a domain, not that a crawler successfully retrieved a page that day. A citation can be produced from content an engine already holds. robots.txt is also a published request rather than an enforced wall, and enforcement studies measure a different thing than policy studies do.

It compares one reading of a file against a 129-day record. A domain's policy may have changed inside that window, and a site that began blocking in August would still carry citations earned in June. That limitation is structural and cannot be removed without a policy time series.

Three of the nine tokens are the ones the operators name as governing search visibility, and the token Anthropic identifies as the one that matters for Claude — Claude-SearchBot — was not in the tested set. The Claude rows therefore measure training and user-initiated tokens only, and understate what a deliberate Claude opt-out would look like.

Methodology and sources #

Citation side. Machine Relations Index public view, contract machine_relations_index_public_view_v2.0, methodology version mri_score_v2.0, generated 2026-09-22, read whole from https://machinerelations.ai/data/machine-relations-index.json in an isolated compute environment on 2026-09-23. Window 2026-05-10 to 2026-09-22, 129 observed days. 22,828 cited source domains; 127,980 source events; 16,307 observed answer runs; 947 eligible queries; six engines. The evidence floor for a published segment is at least 10 observations across at least 7 distinct run dates, and confidence grades A, B and C are domain-level. The population is the 250 domains carrying a published head ranking, read from the release rather than re-derived by sorting. Per-domain engine presence, answer runs cited, days cited and confidence grade are read from the release's own fields.

Policy side. One GET /robots.txt per domain on 2026-09-23 from a datacenter host, www host first and apex second, one retry each, 20-second timeout, identified user agent. A file counted as read only on HTTP 200 with a body that parses as robots syntax and carries no HTML document markers. Groups were parsed per RFC 9309: consecutive User-agent lines open a group, and the following Allow and Disallow lines belong to it. A token counts as fully disallowed only where a group naming it carries Disallow: / with no Allow rule reopening the root.

Vantage. A live census of third-party files is vantage-dependent and a single vantage loses rows silently. Every figure above was produced from the isolated-compute vantage, and the ten blanket-disallow domains were then re-fetched independently from a second host on 2026-09-23; nine reproduced identically. The tenth, sciencedirect.com, served its file to the first vantage and returned HTTP 403 to the second in the same hour, so its row rests on the single reading that succeeded and carries no link a reader could follow to the same result. Rates use the 224 readable files as the denominator, never the 250 attempted.

Domains that served no readable file, with head rank: nih.gov (6), gartner.com (9), grandviewresearch.com (28), fortunebusinessinsights.com (32), peoplemanagingpeople.com (34), openvc.app (68), oracle.com (71), businesswire.com (74), allaboutcookies.org (80), mckinsey.com (82), mayoclinic.org (84), pwc.com (88), 6sense.com (98), galengrowth.com (103), ieee.org (125), mdpi.com (127), prowly.com (139), safetydetectives.com (140), muckrack.com (152), sirion.ai (193), openbankingtracker.com (198), gusto.com (214), hrexecutive.com (215), schwab.com (233), github.io (239), wolterskluwer.com (244). Twenty-two returned an HTTP error to both hosts, and four returned HTTP 200 with a body that was not a robots file.

Operator documentation read 2026-09-23: OpenAI, Anthropic, Perplexity, Google. Protocol: RFC 9309. Prior policy-side censuses referenced for context: Crawlora, HasData, Forage AI. Named domains' policies are checkable by any reader at the robots.txt links in the table above, and at amazon.com, crunchbase.com, statista.com and nytimes.com for the named cases outside it.

For the live instrument and its current release, see the Machine Relations Index.