Summary: Among the 250 highest-ranked domains in the Machine Relations Index, 229 served a readable robots.txt on 2026-10-14. Of the 25 that fully disallow OpenAI's training crawler, GPTBot, 18 were cited by ChatGPT in the 151-day window, 72.0%. Of the 11 that fully disallow OpenAI's search crawler, OAI-SearchBot, 4 were, 36.4%, against 89.9% for every other readable head domain. Anthropic shows the same split: 72.0% cited behind a ClaudeBot block, 25.0% behind a Claude-SearchBot block. Google-Extended, Google's Gemini grounding control, goes with Gemini citation for 7 of 18 domains. Perplexity is the exception: it cited 16 of the 17 domains that fully disallow PerplexityBot. The gradient is the one the operators' own documentation describes, and it held between two releases of the Index.
Almost every published AI-visibility checklist opens with the same instruction: check robots.txt and unblock the AI crawlers, because an engine cannot cite what it cannot read. It is offered as a binary gate, the cheap prerequisite before any content work is worth doing.
It is testable, and it has mostly been tested from one side. Several surveys measure how often sites block AI crawlers: a robots.txt census of the Tranco top million found 9.33% of scanned domains fully blocking at least one major AI crawler, with blocking concentrated at the head and among publishers (Crawlora AI-Crawler Blocking Index, June 2026); a 500-site audit found 24.0% of top-ranked domains fully blocking at least one (Forage AI, September 2026); a 10,894-domain baseline found that half the publishers Google AI Mode cited in a sample of queries block at least one AI crawler (HasData AI Crawler Block Index, July 2026).
What those surveys cannot do is join a site's stated policy to a large, engine-by-engine record of whether that engine actually cited it. The Machine Relations Index makes that join possible, and this study reports it.
What was measured #
Two datasets, one key: the domain.
On the citation side, the Machine Relations Index public view generated 2026-10-14 under methodology version mri_score_v2.1. The window runs 2026-05-10 to 2026-10-14, 151 observed days, 30,757 cited source domains, 190,582 source events and 25,323 observed answer runs on 1,206 eligible queries across six engines: ChatGPT, Claude, Gemini, Google AI Mode, Google AI Overviews and Perplexity. For every domain the release records which of the six engines cited it at least once in the window. The population is the 250 domains ranked 1 to 250 on the release's full-universe rank, read from the release rather than re-derived; 806 domains carry that rank. Of the 250, 20 are graded Confidence A, 107 B and 123 C.
On the policy side, each of those 250 domains had its /robots.txt fetched live on 2026-10-14 from a datacenter host, www first and then the apex, with one retry per host. A response counted as readable only if it returned HTTP 200 and the body parsed as a robots file: at least one line beginning User-agent:, Disallow:, Allow: or Sitemap:, and no HTML document markers. That validation matters: three head domains return a page that is not a robots file at HTTP 200, and a census that trusts the status code alone records them as files that name no crawler.
229 of the 250 returned a readable file, 91.6%. The remaining 21 are named at the end of this piece.
A domain counts as fully disallowing a token when a group naming that token carries Disallow: / (or Disallow: /*) and no Allow rule reopening the root. A path-level restriction is not counted. Ten tokens were tested, each documented by its operator as a control for its own products. One was added since the release of 2026-09-22: Claude-SearchBot, which Anthropic names as the token governing search visibility in Claude.
The result #
Of the 229 head domains with a readable file, 147 name none of the ten tokens, 64.2%. 31 fully disallow at least one in a group that names it, 13.5%. That minority is the population the rest of this study is about.
| Token | Operator and documented role | Head domains fully disallowing it | Of those, cited by that engine | Share cited | Share cited among all other readable head domains | Counting the * fallback |
|---|---|---|---|---|---|---|
| GPTBot | OpenAI, model training | 25 | 18 | 72.0% | 89.2% | 18 of 26 |
| OAI-SearchBot | OpenAI, ChatGPT search index | 11 | 4 | 36.4% | 89.9% | 4 of 14 |
| ChatGPT-User | OpenAI, user-initiated fetch | 15 | 8 | 53.3% | 89.7% | 8 of 18 |
| ClaudeBot | Anthropic, model training | 25 | 18 | 72.0% | 96.1% | 19 of 28 |
| anthropic-ai | Anthropic, legacy training token | 19 | 13 | 68.4% | 95.7% | 16 of 26 |
| Claude-SearchBot | Anthropic, search indexing | 8 | 2 | 25.0% | 95.9% | 3 of 13 |
| Claude-User | Anthropic, user-initiated fetch | 11 | 5 | 45.5% | 95.9% | 6 of 16 |
| PerplexityBot | Perplexity, search and linking | 17 | 16 | 94.1% | 99.1% | 18 of 19 |
| Perplexity-User | Perplexity, user-initiated fetch | 11 | 11 | 100% | 98.6% | 17 of 18 |
| Google-Extended | Google, Gemini training and grounding control | 18 | 7 | 38.9% | 94.3% | 9 of 20 |
The last column also counts domains that name no group for the token but whose User-agent: * group disallows everything, which is the rule a crawler follows when no group names it (RFC 9309, section 2.2.1). It moves no conclusion.
Read the first two rows together. They describe the same engine, over the same 229 domains, in the same window. A head domain that shuts OpenAI's training crawler out completely was still cited by ChatGPT about seven times in ten. A head domain that shuts OpenAI's search crawler out was cited about one time in three. Same operator, same file, different token.
Anthropic's rows show the same split, now that its search token is measured: 72.0% cited behind a ClaudeBot block, 25.0% behind a Claude-SearchBot block, against 95.9% for every other readable head domain. The user-initiated tokens sit between the two at both operators.
Google's control behaves like a search token for Gemini and like nothing at all for Google Search. Of the 18 head domains that fully disallow Google-Extended, 7 were cited by Gemini, against 94.3% of the rest. 16 of the same 18 were cited by Google AI Mode and 16 by Google AI Overviews, in line with the 219 of 250 head domains AI Mode cited overall.
Perplexity does not fit. 16 of the 17 head domains that fully disallow PerplexityBot were cited by Perplexity, and all 11 that fully disallow Perplexity-User were.
Two releases, the same shape #
The release of 2026-09-22 covered 129 days and 16,307 answer runs under mri_score_v2.0. The release of 2026-10-14 covers 151 days and 25,323 runs under mri_score_v2.1. Ranks shifted, one token was added and every robots.txt was fetched again. The shape held.
| Token | 2026-09-22 release: cited of fully disallowing | 2026-10-14 release: cited of fully disallowing |
|---|---|---|
| GPTBot | 18 of 26 (69.2%) | 18 of 25 (72.0%) |
| OAI-SearchBot | 3 of 10 (30.0%) | 4 of 11 (36.4%) |
| ClaudeBot | 18 of 25 (72.0%) | 18 of 25 (72.0%) |
| Claude-SearchBot | not tested | 2 of 8 (25.0%) |
| PerplexityBot | 15 of 17 (88.2%) | 16 of 17 (94.1%) |
| Google-Extended | 6 of 18 (33.3%) | 7 of 18 (38.9%) |
These are small groups, and a single domain moves a row by several points. The ordering is what reproduced: training tokens high, search tokens low, Perplexity high on both.
Why the gradient is there #
The operators say so themselves, in the part of their documentation few checklists quote.
OpenAI runs its tokens as independent switches: a webmaster "can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training." It states the consequence in one direction only, that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers", and says of the user-initiated agent that "robots.txt rules may not apply" and that ChatGPT-User is "not used to determine whether content may appear in Search" (OpenAI crawler documentation, read 2026-10-14). Disallowing GPTBot is an instruction about a training corpus. It was never an instruction about citation, and the measurement agrees.
Anthropic splits its fleet the same way, into ClaudeBot for training, Claude-User for user-initiated requests, and Claude-SearchBot, whose removal it says "may reduce your site's visibility and accuracy in user search results" (Anthropic crawler documentation, read 2026-10-14).
Google describes Google-Extended as a standalone token that governs whether crawled content may be used to train Gemini models and to ground Gemini Apps, and states that it "does not impact a site's inclusion in Google Search" (Google crawler documentation, read 2026-10-14). That is the split the AI Mode and AI Overviews counts show.
Perplexity describes PerplexityBot as "designed to surface and link websites in search results on Perplexity," and says Perplexity-User "controls which sites these user requests can access" (Perplexity crawler documentation, read 2026-10-14). Those describe a control over visibility, and at the head of this index the citation record does not move with it. This study reports that gap; it does not explain it.
Who blocks the search crawler #
The OAI-SearchBot result rests on a small and unusual group. Seven head domains fully disallow OAI-SearchBot in a named group and were not cited by ChatGPT in the window: nytimes.com, pcmag.com, cnet.com, everydayhealth.com, tracxn.com, usnews.com and quora.com. Three of them, PCMag, CNET and Everyday Health, belong to one publisher, Ziff Davis, which filed a copyright suit against OpenAI in April 2025 (PCMag, April 2025). Those three and nytimes.com fully disallow all seven OpenAI and Anthropic tokens tested, and both Perplexity tokens as well. ChatGPT and Claude cited none of the four in the window; Perplexity, Google AI Mode and Google AI Overviews cited all four.
Four domains fully disallow OAI-SearchBot and were cited by ChatGPT anyway: gartner.com, amazon.com, cnbc.com and builtin.com. A policy read on one day cannot say when it was set, and a domain that began blocking in September would still carry citations earned in June.
So the search-token rows describe deliberate opt-outs by a handful of publishers, and they track absence. They do not show that any site outside that group would lose citation by the same margin.
ChatGPT draws on the most distinctive pool #
One more fact from the same release bears on why the OpenAI search token carries weight. Of the 8,154 domains ChatGPT cited in the window, 4,727, or 58.0%, were cited by no other engine. The comparable shares are 49.3% for Perplexity (7,412 of 15,033), 41.2% for Google AI Mode, 37.0% for Gemini, 29.2% for Claude and 19.3% for Google AI Overviews. A source pool that overlaps least with the other engines is consistent with ChatGPT leaning on its own search index, the index OAI-SearchBot governs. The Index records citations, not retrieval paths, so it cannot show that directly.
The ten that block everyone #
Ten of the 229 readable head domains, 4.4%, carry a User-agent: * group with Disallow: /, the rule that applies to any crawler the file does not name. Every one of them is cited by at least three engines.
| Domain | Head rank | Engines citing it | Answer runs cited | Days cited of 151 | Confidence |
|---|---|---|---|---|---|
| reddit.com | 1 | 4 of 6 | 2,429 | 145 | A |
| linkedin.com | 3 | 6 of 6 | 1,453 | 142 | A |
| facebook.com | 26 | 4 of 6 | 266 | 71 | B |
| sciencedirect.com | 56 | 5 of 6 | 165 | 70 | B |
| instagram.com | 78 | 3 of 6 | 131 | 54 | B |
| wsj.com | 100 | 5 of 6 | 113 | 28 | B |
| tracxn.com | 108 | 4 of 6 | 109 | 37 | B |
| fortinet.com | 142 | 6 of 6 | 93 | 43 | C |
| sap.com | 147 | 6 of 6 | 92 | 47 | C |
| quora.com | 185 | 5 of 6 | 78 | 34 | C |
Nine of the ten also name AI tokens in their own groups, so for those agents the named group decides. One does not.
reddit.com is the cleanest case. The file it served is one group, User-agent: * and Disallow: /, naming no AI crawler and no search crawler. It is the most-cited domain in a 30,757-domain index, cited in 2,429 of 25,323 runs on 145 of 151 observed days. The four engines citing it are Gemini, Google AI Mode, Google AI Overviews and Perplexity. ChatGPT and Claude, whose search crawlers fall through to the blanket rule, cited it in none of the window's runs. The same rule covers Googlebot and PerplexityBot, and those engines cite it, so the blanket rule alone does not account for the split.
linkedin.com is the other end of the same point. It carries the blanket rule and names all ten tested tokens in their own groups. Eight of those groups are a full disallow. The two that are not are OAI-SearchBot and Claude-SearchBot, which get path-level rules instead. linkedin.com is ranked third and cited by all six engines in 1,453 runs on 142 of 151 days, at Confidence A.
Method note. Every absence above, including reddit.com's zero for ChatGPT and Claude, is read from the per-domain engine list in the release, which any reader can query at machinerelations.ai/data/machine-relations-index.json, together with the live files at the robots.txt links in the table.
What this does not say #
It does not say blocking is free, and no row of this table is advice to block. Wherever a search token is involved, the association runs the way the operators' documentation predicts, and the cleanest levers in the table, OAI-SearchBot at 36.4% and Claude-SearchBot at 25.0%, both point the other way.
It does not establish causation. These are 250 of the most-cited domains on the web, the population with the most alternative routes into an answer: brand recall, licensing agreements, syndicated copy on other hosts, and the search indexes that separate tokens govern. What holds at the head may not hold in the tail, and this study measures only the head.
It does not measure fetching. The Index records that an engine cited a domain, not that a crawler retrieved a page that day. A citation can come from content an engine already holds, and robots.txt is a published request rather than an enforced wall.
It reads one file per domain, once, against a 151-day record. A policy may have changed inside the window. A site may also serve a different file to a different user agent, and this census reads only what was served to an identified research agent.
Frequently asked questions #
Does blocking GPTBot remove a site from ChatGPT answers? Not at the head of the index. 18 of the 25 top-250 domains that fully disallow GPTBot were cited by ChatGPT in the 151 days to 2026-10-14. OpenAI documents GPTBot as a training control and OAI-SearchBot as the search control.
Which robots.txt token tracks AI citation? The search token. Head domains that fully disallow OAI-SearchBot were cited by ChatGPT 36.4% of the time and those that disallow Claude-SearchBot were cited by Claude 25.0% of the time, against about 90% and 96% for other head domains. Perplexity cited its blockers either way.
Does Google-Extended affect AI Overviews? Not in this data. 16 of the 18 head domains that fully disallow Google-Extended were cited by Google AI Overviews and 16 by AI Mode, while only 7 were cited by Gemini. Google states Google-Extended does not affect inclusion in Google Search.
Methodology and sources #
Citation side. Machine Relations Index public view, contract machine_relations_index_public_view_v2.0, methodology version mri_score_v2.1, generated 2026-10-14 (release mri_score_v2.1+2026-10-14+7a624b28cd41), read whole from https://machinerelations.ai/data/machine-relations-index.json on 2026-10-14. Window 2026-05-10 to 2026-10-14, 151 observed days. 30,757 cited source domains; 190,582 source events; 25,323 observed answer runs; 1,206 eligible queries; six engines. The evidence floor for a published segment is at least 10 observations across at least 7 distinct run dates, and confidence grades A, B and C are domain-level. The population is the 250 domains ranked 1 to 250 on the release's full-universe rank. Per-domain engine presence, answer runs cited, days cited and confidence grade are read from the release's own fields. The per-engine pool shares count every one of the 30,757 domains.
Policy side. One GET /robots.txt per host on 2026-10-14 from a datacenter host, www first and apex second, one retry each, 20-second timeout, identified user agent. A file counted as read only on HTTP 200 with a body that parses as robots syntax and carries no HTML document markers. Groups were parsed per RFC 9309: consecutive User-agent lines open a group, the following Allow and Disallow lines belong to it, and several groups naming the same token are combined. A token counts as fully disallowed only where a group naming it carries Disallow: / or Disallow: /* with no Allow rule reopening the root. Rates use the 229 readable files as the denominator, never the 250 attempted.
Vantage. A live census of third-party files is vantage-dependent. The full census was run twice from the same host on 2026-10-14, minutes apart, and every count above reproduced exactly. A different vantage, or a different declared user agent, can be served a different file.
Domains that served no readable file, with head rank: peoplemanagingpeople.com (32), fortunebusinessinsights.com (57), allaboutcookies.org (69), mayoclinic.org (73), oracle.com (76), businesswire.com (85), impulsec.com (105), globenewswire.com (107), mdpi.com (131), ieee.org (139), mckinsey.com (140), pwc.com (161), gusto.com (172), 6sense.com (178), github.io (192), demandbase.com (205), netsuite.com (218), analog.com (220), prowly.com (233), schwab.com (234), cybersecuritynews.com (240). Thirteen returned HTTP 403 on both hosts, two returned HTTP 502, three returned HTTP 202 with a challenge page, and three returned HTTP 200 with a body that was not a robots file.
Operator documentation read 2026-10-14: OpenAI, Anthropic, Perplexity, Google. Protocol: RFC 9309. Prior policy-side censuses referenced for context: Crawlora, HasData, Forage AI. Named domains' policies are checkable at the robots.txt links in the table above, and at nytimes.com, pcmag.com, cnet.com, gartner.com, amazon.com and cnbc.com for the named cases outside it.
For the live instrument and its current release, see the Machine Relations Index.