Machine Relations · Measurement
Answer-Source Fidelity
Do AI citations support the claims they anchor?
An AI answer engine cites a source next to a claim. Whether the cited page actually supports the specific claim it sits beside is a separate question — one you can only answer by testing the individual claim/source pair. This page reports that test across four engines, refreshed nightly, with resolvable coverage stated next to every rate and rates below the evidence floor withheld rather than guessed.
Finding · the lead result
What the current collection-time series shows
A measurement series is a set of graded citations collected under one method: the same timing for capturing cited pages, the same prompts and models, the same claim-mapping rules, the same label definitions, and the same rules for which events count toward the denominator. Identical methods pool together automatically; any change of method opens a new series. Each series is reported on its own track — never averaged into another — and a small comparability sample reports how the series relate without ever merging them.
The framing rule
How every number on this page is stated
Never “X% of AI citations are supported.” Only: of the N resolvable citations for engine E, X% were supported; resolvable coverage was C%.
A support rate without its coverage is not a measurement. An engine whose citations mostly cannot be mapped to a claim can post a flattering rate on the sliver that can — so coverage travels with every rate, and any engine below the floor reports collecting, not a number.
Finding · claim pinnability
Engines differ sharply in whether a citation can even be pinned to a claim
Before asking whether a source supports a claim, you must identify which claim the citation anchors. In answers built from discrete, marked statements that mapping is clean; in answers that fuse many claims into prose it often cannot be done without guessing. We call an unmappable pair exactly that, exclude it from every rate, and report the unmappable rate itself. The engines then separate sharply — coverage below is the share of each engine's citations we could resolve and grade.
| Engine | Resolvable coverage | Supported (of resolved) | Status |
|---|
Finding · support among the mappable
Among citations that can be pinned, roughly half fully support the claim
“Supported” is the top of a five-label scale — Supported, Amplified, Contradicted, Misattributed, Fabricated — with a separate operational label, Unreadable, that isolates a page we could not fetch from a claim that was genuinely fabricated. The remaining share is where citations amplify, misattribute, or reach past what their source states.
Provenance caveat — kept in front, and scoped to its series
For the legacy baseline (the frozen corpus, the numbers above), cited pages were retrieved at grading time, days after the answer was produced — it measures whether the page currently supports the claim, not whether it did at citation time. The current collection-time series closes exactly this gap by capturing the cited page at the moment of collection. A known bound on the dated baseline, not a defect — and not a property of the ongoing measurement.
Finding · the instrument's own bias
The instrument is measurably conservative on over-reach
We validated the committee against AttrScore (AttrEval-GenSearch), a public, human-annotated citation-attribution benchmark, on a balanced three-way sample. Overall agreement with the human labels was —. The pattern across classes is the point: on claims clearly supported (—) or clearly unsupported (—) the committee tracks the humans closely; on extrapolatory claims — those that reach beyond the source — agreement drops to —, and it drops in one direction. Of — extrapolatory claims the committee called — unsupported and waved through —.
In plain terms: when a claim reaches past its source, our instrument almost always calls it unsupported and almost never lets it pass. Against genuinely supported claims it rarely does the opposite. That is a directional bias, and it is the one a fidelity instrument should have — it errs toward no on the ambiguous middle rather than toward false credit. It also caps what we claim: the extrapolatory arm is our weakest external result, and we report it rather than average it into the headline.
Calibration · stated honestly
Where the instrument stands
—
Inter-model agreement (Cohen’s κ)
two cross-family graders, real pairs
—
Control-fixture accuracy
deterministic known-answer fixtures
—
Primary-excluded holdout
primary grader absent from its own target
—
External benchmark agreement
AttrScore / AttrEval-GenSearch
The holdout sits far below a solo-grader bar, which is exactly why production never runs a single grader: every item is graded by two model families, a third family adjudicates disagreements, and unresolved cases stay uncertain. We report agreement, stability, refutation survival, abstention, and control accuracy — never precision or recall against labels drawn from the same ensemble, which would be circular.
Methodology
How it is built
Rigor is bought once, in a reference set, and the production corpus then runs on calibrated economy. Layer one — a few hundred atomic claim/source pairs: two cross-family graders label each independently, a third-family judge adjudicates disagreements blind to grader identity, an adversarial refuter attempts to overturn every accepted adjudication, and unresolved cases abstain. Six deterministic control fixtures anchor the scale. Layer two — the production corpus grades every item with two families (always-dual), a third only on disagreement, with control fixtures seeded throughout so a mid-run control failure halts the run.
The full methodology contracts are open: the machine-adjudicated reference set, the claim-map contract, the aggregate specification, and the calibration report.
Trust · the receipt chain
Every number traces to a hash
Aggregation is a pure function of frozen matter: a corpus receipt is re-verified against its own hash, the event manifest is bound by hash, and the output hash covers the inputs so a re-run over identical matter is idempotent — no live database sits between the corpus and the number. The dataset regenerates nightly.
Machine layer
Read this page as data
The measurement is published in machine-readable form alongside this page and regenerates nightly.