The Citation Support Gap: Can AI Citations Be Pinned to the Claims They Anchor?

The Citation Support Gap #

Do AI citations support the claims they anchor? An AI answer engine cites a source next to a claim. The citation looks like evidence. Whether the cited page actually supports the specific claim it sits beside is a separate question — and until you test it at the level of the individual (claim, source) pair, you cannot answer it.

Machine Relations built an instrument to test exactly that, at scale, across four AI answer engines. This piece reports what the instrument found and, in equal measure, how it is built and where it is still weak. It is a methods-and-findings report from a measurement instrument, not a scorecard.

Measurement status. These are directional numbers from an instrument in open calibration. Inter-model agreement is κ 0.647; a primary-excluded holdout sits at 0.476 — both below the bar we would set for a headline metric. Two of the four engines are still below coverage floors and report no rate. The findings that hold regardless of that calibration are structural — the shape of the gap, not its third decimal. We state coverage next to every rate, and we withhold rates that fall below the evidence floor. We do not claim these are the first such numbers, or the only ones; we claim they are receipt-bound and reproducible.

Cohorts are partitioned by measurement semantics — never blended into one trend. A cohort is a set of graded citations that share a measurement semantics: the same fetch basis, prompt and model pins, claim-map version, label set, adjudication policy, and denominator/evidence policy. Two runs pool into one cohort only when that semantic key matches; any semantic change opens a new cohort. Operational code hashes are provenance, not a boundary — a refactor that preserves the semantics keeps one cohort, and a genuine policy change splits it. This report keeps two ideas apart on purpose:

1. Legacy baseline — the frozen corpus of 10,066 events (the tables below). Its cited pages were fetched at grade time, days after each answer was produced, so it measures current-page support. It is a dated, one-time baseline (graded once), not a moving series.
2. Collection-time longitudinal series — from cycle 0001 onward, every new event's cited page is snapshotted at collection time (minutes after the answer is produced) and grading consumes only that snapshot. This is the track that grows nightly and the one that fixes the provenance bound below.

The collection-time track is itself partitioned by semantic key. It currently holds 2 partitions, because the denominator/evidence policy changed mid-flight (a snapshot-failure was once terminalized immediately; it is now deferred and retried on a bounded backoff, which changes which events enter the denominator). We report each partition on its own track rather than pooling two different denominators into one flattering rate:

Partition (denominator / evidence policy)	Events	Resolvable coverage	Supported (of resolved)
bounded-retry-defer	1,551	29.7%	42.1%
immediate-terminal	220	81.4%	48.0%

The headline numbers in this piece are the legacy baseline; the collection-time partitions are reported on their own tracks as they accrue, and a bridge sample reports cross-cohort comparability without ever merging them.

Finding 1 — Engines differ ~12x in whether a citation can even be pinned to a claim #

Before you can ask whether a citation supports a claim, you have to identify which claim the citation is anchoring. In answers that present discrete, attributable statements with inline markers, that mapping is clean. In answers that fuse many claims into flowing prose with a citation dropped at the end, it often cannot be done at all without guessing — and guessing is not measurement.

We call a (claim, source) pair unmappable when no honest disambiguating signal exists, and we exclude unmappable pairs from every support-rate denominator. The rate of unmappable pairs is itself a reported statistic. When we do this, the engines separate sharply:

Resolvable coverage ranges from 77.0% (ChatGPT) to 6.5% (Claude) — roughly a 12x spread. It is driven almost entirely by answer style: some engines attach citations to discrete claims, which makes them auditable; engines that answer in fused prose leave far fewer citations that can be honestly tied to a single claim.

This is the lead finding, and it is prior to any support rate. An engine's citations can be trustworthy only to the extent they can be checked at all. It is also why a single "support rate for AI citations" is meaningless without per-engine coverage: an overall number is dominated by whichever engines happen to be the most mappable.

Finding 2 — Among citations that can be pinned, roughly half fully support the claim #

For the two engines above the coverage floor, the support rate among resolvable citations is stated below with its coverage attached, because the rate means nothing without it:

• ChatGPT — of the 1,255 resolvable citations, 54.8% were Supported; resolvable coverage was 77.0% (1,255 / 1,630).
• Perplexity — of the 1,298 resolvable citations, 50.1% were Supported; resolvable coverage was 47.6% (1,298 / 2,726).
• Legacy-baseline corpus, across all four engines — of 3,033 resolvable citations, 49.7% were Supported; resolvable coverage was 30.1% (3,033 / 10,066). This is the legacy baseline only; the collection-time partitions are reported on their own tracks and are never blended into this number.

Gemini (9.0% coverage) and Claude (6.5% coverage) fall below the evidence floor and report no support rate — they are marked collecting. Withholding a rate is not hiding it. It is the same discipline that governs the rest of the instrument: below the floor, the honest output is "not yet," not a number dressed up as one.

"Supported" here is the top of a five-label fidelity scale — Supported, Amplified, Contradicted, Misattributed, Fabricated — with a separate operational label, Unreadable, that isolates a page we could not fetch from a claim that was actually fabricated. Roughly half of pinnable citations landing at the top of that scale is a directional signal about how much of the remaining half amplifies, misattributes, or overreaches beyond what the source states.

A provenance caveat we keep in front, not in a footnote — and which cohort it applies to. For the legacy baseline (the corpus, the numbers in this section), cited pages were fetched at grade time, days after the answer was produced, so it measures whether the page currently supports the claim, not whether it did at citation time. Pages change; this is a known bound on that baseline. The collection-time series (cycle 0001 onward) closes exactly this gap — it snapshots the cited page minutes after the answer is produced and grades only that snapshot — so the bound is a property of the dated baseline, not of the ongoing measurement.

Finding 3 — The instrument is measurably conservative on over-reach #

An instrument's biases should be published, not assumed away. We validated the committee against AttrScore (AttrEval-GenSearch), a public, human-annotated citation-attribution benchmark, on a balanced three-way sample.

Overall agreement with the human labels was 0.9166666666666666 (n = 60). The agreement was not uniform across classes, and the pattern is the point:

• Attributable claims (source clearly supports): agreement 0.864.
• Not-attributable claims (source clearly does not): agreement 0.95.
• Extrapolatory claims (the claim reaches beyond the source): agreement 0.333.

The extrapolatory arm is where the committee and the benchmark part ways, and they part in one direction: of 18 extrapolatory claims, the committee called 11 not-attributable and let only 1 through as attributable. In plain terms, when a claim reaches past what its source actually says, our instrument almost always calls it unsupported and almost never waves it through. Against genuinely supported claims it does the opposite rarely.

This is a directional bias, and it is the one we would choose: an instrument built to answer "does the source support the claim?" errs toward no on the ambiguous middle rather than toward false credit. It also caps what we can claim — the extrapolatory class is the weakest arm of our external validation, and we report it as such rather than averaging it away into the 0.9166666666666666 headline.

The dogfooding spine #

We did not build this to grade other systems first. The existence-versus-support distinction — a source being present is not the same as a source supporting the claim — first mattered inside our own knowledge system, where a citation existing was never proof the citation held. The instrument that checks public answer engines is the same test, turned outward. We measured ourselves before we measured anyone else, and the measurement path is identical: atomic (claim, source) pairs, a committee, honest abstention, receipts.

How it is built — and where it is bounded #

The instrument is a two-layer design. Rigor is bought once, in a reference set; the production corpus then runs on calibrated economy.

Layer 1 — the reference set. A few hundred atomic (claim, source) pairs. Two candidate graders from different model families label each pair independently; a third-family judge adjudicates disagreements blind to grader identities; an adversarial refuter attempts to overturn every accepted adjudication; unresolved cases abstain rather than being forced to a label. Six deterministic control fixtures anchor the scale (verbatim-supported, altered number, explicit contradiction, unrelated source, unsupported amplification, unreadable page). Control-fixture accuracy is 100% (30/30).

Layer 2 — the production corpus. Every item is graded by two families (always-dual), with a third family only on disagreements; unresolved outcomes stay uncertain rather than being forced. Control fixtures are seeded throughout each run so that a mid-run control failure halts the run — drift protection learned from a real model-alias repricing incident. Because grading is always-dual across the entire corpus — not a random audit slice — the inter-model agreement rate is a property of every graded item, not a separately sampled QA pass.

Calibration, stated honestly. Inter-model raw agreement is 0.722; Cohen's κ is 0.647. A genuine primary-excluded holdout — where the primary grader is absent from the label it is scored against — sits at 0.476 (n = 21). That is far below a solo bar, which is precisely why production never runs a single grader: the primary is not fit to grade alone, so it never does. We report agreement, stability, refutation survival, abstention, and control accuracy. We deliberately do not report precision or recall against labels drawn from the same ensemble — that would be circular.

Receipts. Every number traces to a content-hashed receipt chain. Aggregation is a pure function of frozen matter: a corpus receipt is re-verified against its own receipt_sha256, the event manifest is bound by hash, and the output hash covers the inputs so that re-running over identical matter is idempotent. No live database read sits between the corpus and the published number. The receipt lineage is the dataset spine: the dated legacy baseline is corpus receipt 29c9d69d… (graded once, never re-graded), and each nightly collection-time cycle chains its own receipt to the one before it. The cumulative artifact partitions the lineage into semantic cohorts by cohort_key, and each cohort is reported on its own track rather than averaged into a single moving number.

The bounds we will not paper over. Legacy-baseline coverage overall is ~30.1% and two engines are still collecting. κ 0.647 and a 0.476 holdout are below any bar we would attach to a headline metric. Grade-time fetching measures current-page support, not citation-time support. The extrapolatory validation arm is weak. These are the reasons the numbers above are directional, and the reason the structural findings — the ~12x mappability gap, the conservative-on-over-reach bias — carry more weight than any single support percentage.

Why this is a Machine Relations measurement #

Machine Relations is the discipline of earning AI-engine citations. Answer-Source Fidelity is the dimension that asks the harder follow-up: once a brand is cited, does the citation hold up the claim it is attached to? A citation that cannot be pinned to a claim, or that amplifies past its source, is a weaker relation than a raw citation count suggests. Measuring fidelity — openly, with coverage attached and rates withheld below the floor — is how the citation count becomes an accountable metric instead of a vanity one.

The dataset behind this page regenerates nightly and is published in machine-readable form alongside it. The methodology contracts are open. The instrument's biases are stated. That is the standard we hold the measurement to before we hold any engine to it.

Directional internal findings from an instrument in open calibration. Rates are coverage-qualified; rates below the evidence floor are withheld. Dataset: fidelity-mr-research-data-v1.json (aggregate_sha256 cce903eb…), regenerated nightly. Every numeral above is produced from the receipt-bound aggregate and benchmark artifacts by the nightly projection stage; the prose is authored. Reproducibility: pinned models, prompt hashes, code hashes, and a content-hashed receipt chain.