# Blocked and Cited Anyway: What a robots.txt Disallow Actually Does to AI Citation

Joining the 250 most-cited domains in the Machine Relations Index release of 2026-09-22 to their live robots.txt files, a full disallow on an engine's training crawler is a weak predictor of whether that engine cites the domain: 18 of the 26 head domains that fully disallow GPTBot were cited by ChatGPT across the 129-day window, and 15 of the 17 that fully disallow PerplexityBot were cited by Perplexity. The disallow that does track citation is the one on the search token: 3 of 10 for OAI-SearchBot and 6 of 18 for Google-Extended. Ten head domains, including the first and third ranked, serve a blanket Disallow for every crawler and are cited by at least three engines each.

Canonical URL: https://machinerelations.ai/research/robots-txt-disallow-ai-citation-head-index-2026
Published: 2026-09-23
Research type: Study
Tags: machine-relations, ai-search, citations, robots-txt, crawlers, measurement

## Source Body

# Blocked and Cited Anyway: What a robots.txt Disallow Actually Does to AI Citation

> **Summary:** reddit.com is the most-cited domain in the Machine Relations Index. Its robots.txt contains exactly one rule group — `User-agent: *` followed by `Disallow: /` — and it was cited in 2,027 of 16,307 observed answer runs, on 123 of 129 observed days, by four of the six measured engines, at Confidence A. Across the 250 most-cited domains in the release generated 2026-09-22, 224 served a parseable robots.txt to our fetch. 30 of those 224 fully disallow at least one of the nine AI crawler tokens we tested. Most of them are cited by the corresponding engine anyway. The exception is instructive: a disallow on a **search** token tracks absence far better than a disallow on a **training** token, which is exactly what the operators' own documentation says should happen.

Almost every published AI-visibility checklist opens with the same instruction: check your robots.txt, unblock the AI crawlers, because a blocked crawler cannot read you and an engine cannot cite what it cannot read. It is offered as a binary gate — the cheap, obvious prerequisite before any content work is worth doing.

It is testable, and until now it has mostly been tested from one side. Several good surveys measure how often sites block AI crawlers: a robots.txt census of the Tranco top million found 9.33% of scanned domains fully blocking at least one major AI crawler, with blocking concentrated at the head and among publishers ([Crawlora AI-Crawler Blocking Index, June 2026](https://doi.org/10.5281/zenodo.20774249)); a 500-site audit found 24.0% of top-ranked domains fully blocking at least one ([Forage AI, September 2026](https://forage.ai/blog/ai-crawler-blocking-statistics/)); a 10,894-domain baseline found that half the publishers Google AI Mode cited in a sample of queries block at least one AI crawler ([HasData AI Crawler Block Index, July 2026](https://hasdata.com/blog/ai-crawler-block-index)).

What those surveys cannot do is join a site's stated policy to a large, engine-by-engine record of whether that engine actually cited it. That join is what the Machine Relations Index makes possible, and it is what this study reports.

## What was measured

Two datasets, one key: the domain.

On the citation side, the Machine Relations Index public view generated 2026-09-22 under methodology version `mri_score_v2.0`. The window runs 2026-05-10 to 2026-09-22, 129 observed days, 22,828 cited source domains, 127,980 source events and 16,307 observed answer runs across six engines — ChatGPT, Claude, Gemini, Google AI Mode, Google AI Overviews and Perplexity. For every domain the release records which of the six engines cited it at least once in the window. The 250 domains carrying a published head ranking were taken as the population.

On the policy side, each of those 250 domains had its `/robots.txt` fetched live on 2026-09-23 from a datacenter host, `www` first and then the apex, with one retry per host. A response counted as readable only if it returned HTTP 200 **and** the body actually parsed as a robots file — at least one line beginning `User-agent:`, `Disallow:`, `Allow:` or `Sitemap:`, and no HTML document markers. That validation matters: four head domains return an HTML challenge or marketing page at HTTP 200 for the robots.txt path, and a census that trusts the status code alone silently records them as files that name no crawler at all.

224 of the 250 returned a readable file, or 89.6%. The remaining 26 are named in full at the end of this piece.

A domain counts as **fully disallowing** a token when its file contains a group naming that token whose rules include the exact rule `Disallow: /` and no `Allow` rule reopening the root. A path-level restriction is not counted. Nine tokens were tested, chosen because each operator documents them as the controls for its own products.

## The result

Of the 224 head domains with a readable file, 148 name none of the nine tokens — 66.1%. 30 fully disallow at least one, which is 13.4%. That minority is the population the rest of this study is about.

| Token | Operator and documented role | Head domains fully disallowing it | Of those, cited by that engine | Share cited | Share cited among all other head domains |
| --- | --- | --- | --- | --- | --- |
| GPTBot | OpenAI, model training | 26 | 18 | 69.2% | 82.3% |
| OAI-SearchBot | OpenAI, ChatGPT search index | 10 | 3 | 30.0% | 83.2% |
| ChatGPT-User | OpenAI, user-initiated fetch | 14 | 7 | 50.0% | 82.9% |
| ClaudeBot | Anthropic, model training | 25 | 18 | 72.0% | 97.0% |
| anthropic-ai | Anthropic, legacy training token | 20 | 14 | 70.0% | 96.6% |
| Claude-User | Anthropic, user-initiated fetch | 9 | 4 | 44.4% | 96.3% |
| PerplexityBot | Perplexity, search and linking | 17 | 15 | 88.2% | 98.6% |
| Perplexity-User | Perplexity, user-initiated fetch | 10 | 10 | 100% | 97.7% |
| Google-Extended | Google, Gemini grounding control | 18 | 6 | 33.3% | 95.1% |

Read the first row and the second row together. They describe the same engine, measured over the same 224 domains, in the same window. A head domain that shuts OpenAI's training crawler out completely was still cited by ChatGPT roughly seven times in ten. A head domain that shuts OpenAI's *search* crawler out was cited three times in ten. Same operator, same file, different token, and the gap runs the full width of the measurement.

The same shape appears at Google. A full disallow on Google-Extended goes with citation by Gemini for 6 of 18 head domains, against 95.1% for everyone else — the sharpest association in the table.

And it disappears entirely at Perplexity. All 10 head domains that fully disallow Perplexity-User were cited by Perplexity, and 15 of the 17 that fully disallow PerplexityBot were too.

## Why the gradient is there

The operators say so themselves, and it is the part of their documentation that almost nobody reads before writing the checklist.

OpenAI runs its tokens as independent switches: "a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training." It states the consequence in one direction only — sites "opted out of OAI-SearchBot will not be shown in ChatGPT search answers" — and says plainly of the user-initiated agent that "because these actions are initiated by a user, robots.txt rules may not apply," and that ChatGPT-User "is not used to determine whether content may appear in Search" ([OpenAI crawler documentation, read 2026-09-23](https://developers.openai.com/api/docs/bots)). Disallowing GPTBot is an instruction about a training corpus. It was never an instruction about citation, and the measurement agrees.

Anthropic splits its fleet the same way, into ClaudeBot for training, Claude-User for user-initiated requests, and Claude-SearchBot, which it says is the one whose removal "prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results" ([Anthropic crawler documentation, read 2026-09-23](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)).

Google's crawler documentation describes a fleet split between common crawlers, special-case crawlers and user-triggered fetchers, with Google-Extended as a control over Gemini grounding rather than over inclusion in Google Search ([Google crawler documentation, read 2026-09-23](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)). A domain removed from Gemini's grounding path can still reach an answer through the Search index that a separate token governs, which is the likeliest reading of the six head domains that carry the disallow and were cited anyway.

Perplexity is the one that does not fit. Its documentation describes PerplexityBot as "designed to surface and link websites in search results on Perplexity," and says Perplexity-User "controls which sites these user requests can access" ([Perplexity crawler documentation, read 2026-09-23](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)). Those are descriptions of a control over visibility, and at the head of this index the citation record does not move with it. We are reporting that gap, not explaining it.

Underneath all of this is the matching rule that makes per-token blocking work at all: a crawler obeys the group that names it, or the `*` group if none does ([RFC 9309, section 2.2.1](https://www.rfc-editor.org/rfc/rfc9309.html)). A directive addressed to one product token reaches no other.

## The ten that block everyone

Ten of the 224 head domains, 4.5%, carry a `User-agent: *` group with `Disallow: /` — the rule that applies to any crawler the file does not name specifically. Every one of them is cited by at least three engines.

| Domain | Head rank | Engines citing it | Answer runs cited | Days cited of 129 | Confidence |
| --- | --- | --- | --- | --- | --- |
| [reddit.com](https://www.reddit.com/robots.txt) | 1 | 4 of 6 | 2,027 | 123 | A |
| [linkedin.com](https://www.linkedin.com/robots.txt) | 3 | 6 of 6 | 1,104 | 120 | A |
| [facebook.com](https://www.facebook.com/robots.txt) | 23 | 4 of 6 | 203 | 52 | B |
| sciencedirect.com | 47 | 5 of 6 | 134 | 58 | B |
| [instagram.com](https://www.instagram.com/robots.txt) | 59 | 3 of 6 | 114 | 43 | B |
| [tracxn.com](https://www.tracxn.com/robots.txt) | 64 | 4 of 6 | 109 | 37 | B |
| [fortinet.com](https://www.fortinet.com/robots.txt) | 106 | 6 of 6 | 76 | 39 | C |
| [wsj.com](https://www.wsj.com/robots.txt) | 112 | 4 of 6 | 74 | 23 | C |
| [sap.com](https://www.sap.com/robots.txt) | 154 | 6 of 6 | 63 | 38 | C |
| [quora.com](https://www.quora.com/robots.txt) | 166 | 4 of 6 | 60 | 23 | C |

Eight of the ten also name AI tokens in their own groups, so for those agents the blanket rule is not the operative one and the named group decides. Two do not, and they are the cases with nothing else to read.

reddit.com is the cleanest. Its entire robots.txt is one group: `User-agent: *`, `Disallow: /`. It names no AI crawler, no search crawler, nothing. Every automated fetcher on the web falls through to a rule that asks it not to fetch any path — and reddit.com is the most-cited domain in a 22,828-domain index, present on 123 of 129 observed days.

linkedin.com is the other end of the same point. It carries the blanket rule *and* names ten AI tokens in their own groups, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, PerplexityBot, Perplexity-User and Google-Extended, under a full disallow. It is ranked third in the release and cited by all six engines in 1,104 of 16,307 runs on 120 of 129 days, at Confidence A. There is no token left to add.

## What this does not say

It does not say blocking is free, and nobody should read a row of this table as advice to block. The association runs the way the operators' documentation predicts wherever a search token is involved, and the one clean lever in the table — OAI-SearchBot at 30.0% against 83.2% — points the other way.

It does not establish causation in either direction. These are 250 of the most-cited domains on the web, which is precisely the population with the most alternative routes into an answer: brand-name recall, partner feeds, licensing agreements, syndicated copy on other hosts, and the search indexes that separate tokens govern. What survives at the head may not survive in the tail, and this study measures only the head.

It does not measure fetching. The Index records that an engine cited a domain, not that a crawler successfully retrieved a page that day. A citation can be produced from content an engine already holds. robots.txt is also a published request rather than an enforced wall, and enforcement studies measure a different thing than policy studies do.

It compares one reading of a file against a 129-day record. A domain's policy may have changed inside that window, and a site that began blocking in August would still carry citations earned in June. That limitation is structural and cannot be removed without a policy time series.

Three of the nine tokens are the ones the operators name as governing search visibility, and the token Anthropic identifies as the one that matters for Claude — Claude-SearchBot — was not in the tested set. The Claude rows therefore measure training and user-initiated tokens only, and understate what a deliberate Claude opt-out would look like.

## Methodology and sources

**Citation side.** Machine Relations Index public view, contract `machine_relations_index_public_view_v2.0`, methodology version `mri_score_v2.0`, generated 2026-09-22, read whole from `https://machinerelations.ai/data/machine-relations-index.json in an isolated compute environment on 2026-09-23. Window 2026-05-10 to 2026-09-22, 129 observed days. 22,828 cited source domains; 127,980 source events; 16,307 observed answer runs; 947 eligible queries; six engines. The evidence floor for a published segment is at least 10 observations across at least 7 distinct run dates, and confidence grades A, B and C are domain-level. The population is the 250 domains carrying a published head ranking, read from the release rather than re-derived by sorting. Per-domain engine presence, answer runs cited, days cited and confidence grade are read from the release's own fields.

**Policy side.** One `GET /robots.txt` per domain on 2026-09-23 from a datacenter host, `www` host first and apex second, one retry each, 20-second timeout, identified user agent. A file counted as read only on HTTP 200 with a body that parses as robots syntax and carries no HTML document markers. Groups were parsed per RFC 9309: consecutive `User-agent` lines open a group, and the following `Allow` and `Disallow` lines belong to it. A token counts as fully disallowed only where a group naming it carries `Disallow: /` with no `Allow` rule reopening the root.

**Vantage.** A live census of third-party files is vantage-dependent and a single vantage loses rows silently. Every figure above was produced from the isolated-compute vantage, and the ten blanket-disallow domains were then re-fetched independently from a second host on 2026-09-23; nine reproduced identically. The tenth, sciencedirect.com, served its file to the first vantage and returned HTTP 403 to the second in the same hour, so its row rests on the single reading that succeeded and carries no link a reader could follow to the same result. Rates use the 224 readable files as the denominator, never the 250 attempted.

**Domains that served no readable file,** with head rank: nih.gov (6), gartner.com (9), grandviewresearch.com (28), fortunebusinessinsights.com (32), peoplemanagingpeople.com (34), openvc.app (68), oracle.com (71), businesswire.com (74), allaboutcookies.org (80), mckinsey.com (82), mayoclinic.org (84), pwc.com (88), 6sense.com (98), galengrowth.com (103), ieee.org (125), mdpi.com (127), prowly.com (139), safetydetectives.com (140), muckrack.com (152), sirion.ai (193), openbankingtracker.com (198), gusto.com (214), hrexecutive.com (215), schwab.com (233), github.io (239), wolterskluwer.com (244). Twenty-two returned an HTTP error to both hosts, and four returned HTTP 200 with a body that was not a robots file.

**Operator documentation** read 2026-09-23: [OpenAI](https://developers.openai.com/api/docs/bots), [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers), [Google](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers). Protocol: [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html). Prior policy-side censuses referenced for context: [Crawlora](https://doi.org/10.5281/zenodo.20774249), [HasData](https://hasdata.com/blog/ai-crawler-block-index), [Forage AI](https://forage.ai/blog/ai-crawler-blocking-statistics/). Named domains' policies are checkable by any reader at the robots.txt links in the table above, and at [amazon.com](https://www.amazon.com/robots.txt), [crunchbase.com](https://www.crunchbase.com/robots.txt), [statista.com](https://www.statista.com/robots.txt) and [nytimes.com](https://www.nytimes.com/robots.txt) for the named cases outside it.

For the live instrument and its current release, see the [Machine Relations Index](https://machinerelations.ai/index).

## Attribution

This research is published by Machine Relations Research, the research program of machinerelations.ai — the public research and standards initiative that publishes the glossary, research, evidence, and measurements for the Machine Relations discipline. Provenance and editorial standards: https://machinerelations.ai/about

## Machine-readable related links

### Related concepts

- [Machine Relations Index (MRI)](https://machinerelations.ai/glossary/machine-relations-index)
- [MRI Score](https://machinerelations.ai/glossary/mri-score)
- [Machine Relations (MR)](https://machinerelations.ai/glossary/machine-relations)
- [AI Visibility](https://machinerelations.ai/glossary/ai-visibility)

### Supporting research

- [Citation Absorption vs Citation Selection: Why Getting Cited Is Not the Same as Getting Used](https://machinerelations.ai/research/citation-absorption-vs-selection-ai-search-2026)
- [Ranked Without a Grade: Half of the AI Citation Index's Top-Ten Positions Belong to Ungraded Domains](https://machinerelations.ai/research/ranked-without-a-grade-ai-citation-index-domain-lookup-2026)
- [Top Publications AI Engines Cite for Healthtech Companies (2026)](https://machinerelations.ai/research/top-healthtech-publications-ai-search-2026)
- [AI Search Brand Visibility Study 2025 to 2026: What Changes, What Gets Cited, and Why (2026)](https://machinerelations.ai/research/what-is-machine-relations-marketing-discipline)

### Framework context

- [Machine Relations Index](https://machinerelations.ai/index)
- [Machine Relations Stack](https://machinerelations.ai/stack)
- [Evidence Base](https://machinerelations.ai/evidence)