Source-Role Classification Sensitivity in AI Citation Rankings

A citation leaderboard can change when the source-role taxonomy changes, even when every observed citation count stays exactly the same. That is not a data error. It is a measurement-design issue: the numerator and denominator answer one question, while the source-role label answers a different interpretive question.

The September 12, 2026 Machine Relations Index (MRI) release is a clean anchor for the distinction. The public release covers May 10 through September 12, spans 119 observed dates, and reports six healthy answer engines in the release manifest. In the AI-infrastructure category, Medium is cited in 157 of 611 observed runs (25.70%, confidence A), arXiv in 103 of 611 runs (16.86%, confidence A), and Nvidia in 49 of 611 runs (8.02%, confidence B). Those are observed domain-level citation rates in the public index, not judgments about source quality, causality, or editorial influence. (MRI release manifest) (MRI public index)

This note uses those three published rows only as fixed inputs. The worked example below is hypothetical and deliberately changes only the grouping label. It does not recalculate the live MRI index, create a new composite score, or use the failed September 13 partial collection.

The invariant measurement #

Start with the domain observations exactly as published for AI-infrastructure in the September 12 release.

Nothing in the example below changes these six facts:

1. Medium remains 157 cited runs.
2. arXiv remains 103 cited runs.
3. Nvidia remains 49 cited runs.
4. The denominator remains 611 observed runs for each row.
5. The observation window remains May 10–September 12.
6. The release remains the September 12 MRI v2 release, not the rejected September 13 collection.

The only adjustable input is the analyst's source-role grouping rule.

Classification A: platform-function roles #

The first grouping keeps the published role labels as platform-function metadata.

Under this grouping, the role leaderboard for the three-domain example is:

1. Editorial publication: 157 of 611 observed runs.
2. Academic and government source: 103 of 611 observed runs.
3. Vendor-owned source: 49 of 611 observed runs.

This is a faithful presentation of the published role metadata. It answers: if each domain keeps its current role label, which role appears most often among these selected rows?

Classification B: research-distribution roles #

Now change only the role taxonomy. Suppose an analyst is not asking whether a domain is a publication, academic repository, or vendor site. Instead, the analyst is asking whether the cited source functions as a research-distribution surface or a vendor-owned surface in this narrow example.

That hypothetical rule groups Medium and arXiv together as research-distribution surface and leaves Nvidia as vendor-owned surface.

The arithmetic is intentionally simple: 157 plus 103 equals 260. The denominator is still 611 because the example is a same-category, same-release comparison. Medium is still 157/611. arXiv is still 103/611. Nvidia is still 49/611.

But the leaderboard changed. Under Classification A, Editorial publication leads and Academic and government source is separate. Under Classification B, a broader Research-distribution surface group leads with a combined 260 cited runs. No citation event was added. No citation event was removed. Only the metadata grouping changed.

What changed and what did not #

The domain-level measurement did not change.

This is the sensitivity point: role-based leaderboards are functions of both observations and classification rules. The citation numerator is evidence. The role label is measurement metadata. A method note must keep those layers separate.

Why declared metadata belongs in the result #

Measurement systems routinely keep denominator, metadata, and classification choices attached to the number. The NIST/SEMATECH engineering statistics handbook treats measurement context and uncertainty as part of technical interpretation, not decoration. AAPOR standards separate sampled units, response status, and reporting definitions. Census methodology publishes survey design and data-quality limits with the estimates. FDA missing-data guidance distinguishes observed outcomes from absent or incomplete evidence. (NIST/SEMATECH e-Handbook) (AAPOR standards) (U.S. Census Bureau methodology) (FDA missing-data guidance)

The same principle appears in data-governance references. OECD's statistical quality framework, World Bank metadata guidance, ISO data-quality vocabulary, Dublin Core terms, and W3C DCAT all make metadata part of how a dataset is described, exchanged, and interpreted. Schema.org's Article vocabulary and Google Search Central's structured-data guidance apply the same machine-readable logic to web publishing: labels help machines interpret a resource, but the label is not the same thing as the evidence inside the resource. (OECD quality framework) (World Bank metadata guidance) (ISO 8000-2 vocabulary) (DCMI Metadata Terms) (W3C DCAT 3) (Schema.org Article) (Google structured data intro)

That is the methodological reason source-role sensitivity should be disclosed. A role label is metadata that supports interpretation. It is not a hidden vote about source quality, a causal mechanism, or a replacement for the observed citation denominator.

Why this matters for AI citation reporting #

Source-role labels are useful because they make citation behavior interpretable. A domain-only list can tell an operator that Medium, arXiv, and Nvidia appear in AI-infrastructure answers. A role-aware view can tell the operator whether answer engines are drawing from editorial surfaces, academic repositories, vendor-owned documentation, community platforms, market databases, or other kinds of evidence. The existing Machine Relations research on industry citation patterns makes the same practical point: industries change the source pool that AI systems retrieve from, and broad sector labels are not enough to explain citation behavior. (AI citation patterns by industry)

But source-role labels are not proof of source quality. Calling Medium an editorial publication does not prove that editorial media caused its citations. Calling Nvidia vendor-owned does not prove that vendor-owned sources are weak. Calling arXiv academic does not prove that academic sources are always preferred. The labels describe how the measurement system groups domains for interpretation.

This is especially important when people compare platforms, vendors, and publishers. A leaderboard sorted by domain answers one question: which domains were cited most often in the eligible runs? A leaderboard grouped by source role answers another: which labeled source classes account for the observed citations? If the labels change, the second answer can change while the first remains fixed.

A reproducible sensitivity check #

Use this sequence before publishing any role-level citation ranking:

1. Freeze the observation table. Record the domain, cited-run numerator, observed-run denominator, confidence tier, category, engines, and release window.
2. Declare the role taxonomy. State whether the grouping is platform-function, ownership, publication type, research function, buyer-use case, or another explicit rule.
3. Aggregate only after labeling. Do not combine rows until every included domain has one declared role under the chosen taxonomy.
4. Run a label-only sensitivity pass. Reassign only the role labels under an alternative taxonomy and compare whether the role leaderboard changes.
5. Keep the domain table attached. If the grouped leaderboard changes, readers must still be able to see that each domain's numerator and denominator did not move.
6. State the boundary. A changed role leaderboard is not evidence that one source type caused citations, that a publication strategy worked, or that a domain's quality changed.

The companion measurement note on collecting versus zero citation rate addresses a prior evidence question: whether a row is eligible for interpretation at all. This note starts after evidence exists and asks a different question: whether the taxonomy used to interpret eligible rows changes the role-level story. (Why collecting is not a zero citation rate)

What not to conclude #

This example does not show that editorial media is better than vendor-owned sources. It does not show that Medium caused AI-infrastructure citations. It does not show that arXiv and Medium should be merged in the live MRI. It does not produce a new official AI-infrastructure index. It also does not use September 13 data: that collection failed the Google AI Mode quality threshold and was not released as a fresh daily MRI dataset.

The correct conclusion is narrower and more useful: if a ranking is grouped by source role, the grouping rule is part of the result. Report the underlying numerator and denominator first, then show how the source-role taxonomy turns those observations into a leaderboard. If the leaderboard changes under a defensible alternative grouping, publish that as classification sensitivity, not as a change in citation behavior.

FAQ #

Did the citation rates change in the hypothetical? #

No. Medium remains 157/611, arXiv remains 103/611, and Nvidia remains 49/611 for the AI-infrastructure category in the September 12 release.

Is source role a source-quality score? #

No. Source role is metadata about how a domain is classified for interpretation. Quality, reliability, extractability, and causal effect require separate evidence.

Can role grouping still be useful? #

Yes. Role grouping helps teams understand whether citations are coming from editorial publications, academic sources, vendor-owned pages, community platforms, market databases, or other surfaces. It is useful only when the taxonomy is declared and the domain-level observations remain visible.

Why not publish one blended score? #

A blended score would hide the exact sensitivity this note is meant to expose. The safer method is to publish the fixed domain observations, the declared role taxonomy, and any label-only sensitivity result separately.

Sources #

1. Machine Relations Research. "MRI release manifest." September 12, 2026. https://machinerelations.ai/data/mri-release-manifest.json
2. Machine Relations Research. "Machine Relations Index public data." September 12, 2026. https://machinerelations.ai/data/machine-relations-index.json
3. Machine Relations Research. "AI Citation Patterns by Industry: What Changes Across Vertical Search in 2026." https://machinerelations.ai/research/ai-citation-patterns-by-industry-2026
4. Machine Relations Research. "Why ‘Collecting’ Is Not a Zero Citation Rate." https://machinerelations.ai/research/collecting-is-not-zero-citation-rate
5. NIST/SEMATECH. "e-Handbook of Statistical Methods." https://www.itl.nist.gov/div898/handbook/
6. AAPOR. "Best Practices." https://aapor.org/standards-ethics/best-practices/
7. U.S. Census Bureau. "ACS Design and Methodology." https://www.census.gov/programs-surveys/acs/methodology/design-and-methodology.html
8. U.S. Food and Drug Administration. "Missing Data in Clinical Trials." https://www.fda.gov/regulatory-information/search-fda-guidance-documents/missing-data-clinical-trials
9. OECD. "Quality Framework for OECD Statistical Activities." https://www.oecd.org/sdd/qualityframeworkforoecdstatisticalactivities.htm
10. World Bank Data Help Desk. "Developing Data Metadata." https://datahelpdesk.worldbank.org/knowledgebase/articles/889386-developing-data-metadata
11. ISO. "ISO 8000-2 Data quality vocabulary." https://www.iso.org/standard/63555.html
12. Dublin Core Metadata Initiative. "DCMI Metadata Terms." https://www.dublincore.org/specifications/dublin-core/dcmi-terms/
13. W3C. "Data Catalog Vocabulary (DCAT) - Version 3." https://www.w3.org/TR/vocab-dcat-3/
14. Schema.org. "Article." https://schema.org/Article
15. Google Search Central. "Intro to structured data markup in Google Search." https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data