# The Citation Support Gap: Can AI Citations Be Pinned to the Claims They Anchor?

A receipt-bound measurement of whether AI answer-engine citations support the claims they anchor, with resolvable coverage stated next to every rate.

Canonical URL: https://machinerelations.ai/research/citation-support-gap
Published: 2026-07-10
Research type: Study

## Source Body

# The Citation Support Gap

**Do AI citations support the claims they anchor?** An AI answer engine cites a
source next to a claim. The citation looks like evidence. Whether the cited page
actually supports the specific claim it sits beside is a separate question — and
until you test it at the level of the individual (claim, source) pair, you cannot
answer it.

Machine Relations built an instrument to test exactly that, at scale, across four
AI answer engines. This piece reports what the instrument found and, in equal
measure, how it is built and where it is still weak. It is a methods-and-findings
report from a measurement instrument, not a scorecard.

> **Measurement status.** These are directional numbers from an instrument in open
> calibration. Inter-model agreement is κ 0.647; a primary-excluded holdout sits at
> 0.476 — both below the bar we would set for a headline metric. Two of the four
> engines are still below coverage floors and report no rate. The findings that hold
> regardless of that calibration are structural — the shape of the gap, not its third
> decimal. We state coverage next to every rate, and we withhold rates that fall below
> the evidence floor. We do not claim these are the first such numbers, or the only
> ones; we claim they are receipt-bound and reproducible.

> **Cohorts are partitioned by measurement semantics — never blended into one trend.**
> A cohort is a set of graded citations that share a *measurement semantics*: the same
> fetch basis, prompt and model pins, claim-map version, label set, adjudication policy,
> and denominator/evidence policy. Two runs pool into one cohort only when that semantic
> key matches; any semantic change opens a new cohort. Operational code hashes are
> provenance, not a boundary — a refactor that preserves the semantics keeps one cohort,
> and a genuine policy change splits it. This report keeps two ideas apart on purpose:
>
> 1. **Legacy baseline** — the frozen corpus of 10,066 events (the tables
>    below). Its cited pages were fetched **at grade time**, days after each answer was
>    produced, so it measures *current-page* support. It is a dated, one-time baseline
>    (graded once), not a moving series.
> 2. **Collection-time longitudinal series** — from cycle 0001 onward, every new event's
>    cited page is snapshotted **at collection time** (minutes after the answer is
>    produced) and grading consumes only that snapshot. This is the track that grows
>    nightly and the one that fixes the provenance bound below.
>
> The collection-time track is itself partitioned by semantic key. It currently holds
> **2** partitions, because the denominator/evidence policy
> changed mid-flight (a snapshot-failure was once terminalized immediately; it is now
> deferred and retried on a bounded backoff, which changes *which* events enter the
> denominator). We report each partition on its own track rather than pooling two
> different denominators into one flattering rate:
>
> | Partition (denominator / evidence policy) | Events | Resolvable coverage | Supported (of resolved) |
> |---|---|---|---|
> | bounded-retry-defer | 2,305 | 33.3% | 46.2% |
> | immediate-terminal | 220 | 81.4% | 48.0% |
>
> The headline numbers in this piece are the legacy baseline; the collection-time
> partitions are reported on their own tracks as they accrue, and a bridge sample
> reports cross-cohort comparability without ever merging them.

---

## Finding 1 — Engines differ ~12x in whether a citation can even be pinned to a claim

Before you can ask whether a citation supports a claim, you have to identify which
claim the citation is anchoring. In answers that present discrete, attributable
statements with inline markers, that mapping is clean. In answers that fuse many
claims into flowing prose with a citation dropped at the end, it often cannot be
done at all without guessing — and guessing is not measurement.

We call a (claim, source) pair **unmappable** when no honest disambiguating signal
exists, and we exclude unmappable pairs from every support-rate denominator. The
*rate* of unmappable pairs is itself a reported statistic. When we do this, the
engines separate sharply:

| Engine | Resolvable coverage | Resolved / total citations | Status |
|---|---|---|---|
| ChatGPT | 77.0% | 1,255 / 1,630 | reported |
| Perplexity | 47.6% | 1,298 / 2,726 | reported |
| Gemini | 9.0% | 392 / 4,349 | collecting — rate withheld |
| Claude | 6.5% | 88 / 1,361 | collecting — rate withheld |

Resolvable coverage ranges from **77.0% (ChatGPT) to 6.5%
(Claude)** — roughly a 12x spread. It is driven almost entirely
by answer style: some engines attach citations to discrete claims, which makes them
auditable; engines that answer in fused prose leave far fewer citations that can be
honestly tied to a single claim.

This is the lead finding, and it is prior to any support rate. **An engine's
citations can be trustworthy only to the extent they can be checked at all.** It is
also why a single "support rate for AI citations" is meaningless without per-engine
coverage: an overall number is dominated by whichever engines happen to be the most
mappable.

---

## Finding 2 — Among citations that can be pinned, roughly half fully support the claim

For the two engines above the coverage floor, the support rate among *resolvable*
citations is stated below with its coverage attached, because the rate means nothing
without it:

- **ChatGPT** — of the 1,255 resolvable citations, **54.8%
  were Supported**; resolvable coverage was 77.0% (1,255 /
  1,630).
- **Perplexity** — of the 1,298 resolvable citations,
  **50.1% were Supported**; resolvable coverage was
  47.6% (1,298 / 2,726).
- **Legacy-baseline corpus, across all four engines** — of 3,033 resolvable
  citations, **49.7% were Supported**; resolvable coverage was
  30.1% (3,033 / 10,066). This is the legacy
  baseline only; the collection-time partitions are reported on their own tracks and are
  never blended into this number.

Gemini (9.0% coverage) and Claude (6.5% coverage) fall
below the evidence floor and report **no** support rate — they are marked *collecting*.
Withholding a rate is not hiding it. It is the same discipline that governs the rest of
the instrument: below the floor, the honest output is "not yet," not a number dressed up
as one.

"Supported" here is the top of a five-label fidelity scale — *Supported, Amplified,
Contradicted, Misattributed, Fabricated* — with a separate operational label,
*Unreadable*, that isolates a page we could not fetch from a claim that was actually
fabricated. Roughly half of pinnable citations landing at the top of that scale is a
directional signal about how much of the remaining half amplifies, misattributes, or
overreaches beyond what the source states.

> **A provenance caveat we keep in front, not in a footnote — and which cohort it
> applies to.** For the **legacy baseline** (the corpus, the numbers in this section),
> cited pages were fetched *at grade time*, days after the answer was produced, so it
> measures whether the page *currently* supports the claim, not whether it did at
> citation time. Pages change; this is a known bound on that baseline. The
> **collection-time series** (cycle 0001 onward) closes exactly this gap — it snapshots
> the cited page minutes after the answer is produced and grades only that snapshot — so
> the bound is a property of the dated baseline, not of the ongoing measurement.

---

## Finding 3 — The instrument is measurably conservative on over-reach

An instrument's biases should be published, not assumed away. We validated the
committee against **AttrScore (AttrEval-GenSearch)**, a public, human-annotated
citation-attribution benchmark, on a balanced three-way sample.

Overall agreement with the human labels was **0.9166666666666666** (n = 60).
The agreement was not uniform across classes, and the pattern is the point:

- **Attributable** claims (source clearly supports): agreement **0.864**.
- **Not-attributable** claims (source clearly does not): agreement **0.95**.
- **Extrapolatory** claims (the claim reaches beyond the source): agreement **0.333**.

The extrapolatory arm is where the committee and the benchmark part ways, and they
part in one direction: of 18 extrapolatory claims, the committee called
**11 not-attributable** and let only **1** through as
attributable. In plain terms, when a claim reaches past what its source actually says,
our instrument almost always calls it unsupported and almost never waves it through.
Against genuinely supported claims it does the opposite rarely.

This is a directional bias, and it is the one we would choose: an instrument built to
answer "does the source support the claim?" errs toward *no* on the ambiguous middle
rather than toward false credit. It also caps what we can claim — the extrapolatory
class is the weakest arm of our external validation, and we report it as such rather
than averaging it away into the 0.9166666666666666 headline.

---

## The dogfooding spine

We did not build this to grade other systems first. The existence-versus-support
distinction — a source being *present* is not the same as a source *supporting* the
claim — first mattered inside our own knowledge system, where a citation existing was
never proof the citation held. The instrument that checks public answer engines is the
same test, turned outward. We measured ourselves before we measured anyone else, and
the measurement path is identical: atomic (claim, source) pairs, a committee, honest
abstention, receipts.

---

## How it is built — and where it is bounded

The instrument is a two-layer design. Rigor is bought **once**, in a reference set;
the production corpus then runs on calibrated economy.

**Layer 1 — the reference set.** A few hundred atomic (claim, source) pairs. Two
candidate graders from different model families label each pair independently; a
third-family judge adjudicates disagreements blind to grader identities; an
adversarial refuter attempts to overturn every accepted adjudication; unresolved
cases **abstain** rather than being forced to a label. Six deterministic control
fixtures anchor the scale (verbatim-supported, altered number, explicit
contradiction, unrelated source, unsupported amplification, unreadable page).
Control-fixture accuracy is **100%** (30/30).

**Layer 2 — the production corpus.** Every item is graded by two families
(**always-dual**), with a third family only on disagreements; unresolved outcomes
stay *uncertain* rather than being forced. Control fixtures are seeded throughout
each run so that a mid-run control failure halts the run — drift protection learned
from a real model-alias repricing incident. Because grading is **always-dual across
the entire corpus** — not a random audit slice — the inter-model agreement rate is a
property of every graded item, not a separately sampled QA pass.

**Calibration, stated honestly.** Inter-model raw agreement is 0.722; Cohen's
κ is 0.647. A genuine primary-excluded holdout — where the primary grader is absent
from the label it is scored against — sits at **0.476** (n = 21). That is
far below a solo bar, which is precisely why production **never** runs a single grader:
the primary is not fit to grade alone, so it never does. We report agreement, stability,
refutation survival, abstention, and control accuracy. We deliberately do **not** report
precision or recall against labels drawn from the same ensemble — that would be circular.

**Receipts.** Every number traces to a content-hashed receipt chain. Aggregation is a
pure function of frozen matter: a corpus receipt is re-verified against its own
`receipt_sha256`, the event manifest is bound by hash, and the output hash covers the
inputs so that re-running over identical matter is idempotent. No live database read
sits between the corpus and the published number. The receipt lineage is the dataset
spine: the dated **legacy baseline** is corpus receipt `29c9d69d…`
(graded once, never re-graded), and each nightly **collection-time cycle** chains its
own receipt to the one before it. The cumulative artifact partitions the lineage into
semantic cohorts by `cohort_key`, and each cohort is reported on its own track rather
than averaged into a single moving number.

**The bounds we will not paper over.** Legacy-baseline coverage overall is
~30.1% and two engines are still collecting. κ 0.647 and a 0.476
holdout are below any bar we would attach to a headline metric. Grade-time fetching
measures current-page support, not citation-time support. The extrapolatory validation
arm is weak. These are the reasons the numbers above are directional, and the reason the
structural findings — the ~12x mappability gap, the conservative-on-over-reach
bias — carry more weight than any single support percentage.

---

## Why this is a Machine Relations measurement

Machine Relations is the discipline of earning AI-engine citations. Answer-Source
Fidelity is the dimension that asks the harder follow-up: once a brand is cited, does
the citation hold up the claim it is attached to? A citation that cannot be pinned to
a claim, or that amplifies past its source, is a weaker relation than a raw citation
count suggests. Measuring fidelity — openly, with coverage attached and rates withheld
below the floor — is how the citation count becomes an accountable metric instead of a
vanity one.

The dataset behind this page regenerates nightly and is published in machine-readable
form alongside it. The methodology contracts are open. The instrument's biases are
stated. That is the standard we hold the measurement to before we hold any engine to it.

---

*Directional internal findings from an instrument in open calibration. Rates are
coverage-qualified; rates below the evidence floor are withheld. Dataset:
`fidelity-mr-research-data-v1.json` (`aggregate_sha256` `4012ebcb…`),
regenerated nightly. Every numeral above is produced from the receipt-bound aggregate
and benchmark artifacts by the nightly projection stage; the prose is authored.
Reproducibility: pinned models, prompt hashes, code hashes, and a content-hashed
receipt chain.*

## Attribution

This research is published by Machine Relations Research, the research program of machinerelations.ai — the public research and standards initiative that publishes the glossary, research, evidence, and measurements for the Machine Relations discipline. Provenance and editorial standards: https://machinerelations.ai/about

## Machine-readable related links

### Related concepts

- [Machine Relations (MR)](https://machinerelations.ai/glossary/machine-relations)
- [Machine Relations Index (MRI)](https://machinerelations.ai/glossary/machine-relations-index)
- [AI Citations](https://machinerelations.ai/glossary/ai-citations)
- [MRI Score](https://machinerelations.ai/glossary/mri-score)

### Supporting research

- [Brand24 Alternatives in 2026: The AI Citation Gap Every Social Listening Tool Shares](https://machinerelations.ai/research/brand24-alternatives-ai-citation-gap-2026)
- [BrightEdge Alternatives in 2026: The AI Citation Gap Every Enterprise SEO Platform Shares](https://machinerelations.ai/research/brightedge-alternatives-ai-citation-gap-2026)
- [Conductor Alternatives in 2026: The AI Citation Gap Every Enterprise SEO Platform Shares](https://machinerelations.ai/research/conductor-alternatives-ai-citation-gap-2026)
- [How to Measure AI Visibility ROI: The CMO Dashboard That Replaces Guesswork](https://machinerelations.ai/research/measure-ai-visibility-roi-cmo-dashboard-2026)

### Framework context

- [Machine Relations Index](https://machinerelations.ai/index)
- [Machine Relations Stack](https://machinerelations.ai/stack)
- [Evidence Base](https://machinerelations.ai/evidence)