Publication Over Provenance
Careers and grants reward the claim while the reproducible chain is optional
Novel results publish faster, citation counts rise, funding and tenure cases strengthen — without paying the cost of rerunnable artifacts.
Publication over provenance is a dark pattern: the organisation or field successfully advances people and projects on published claims — papers, preprints, press releases, benchmark tables — while treating datasets, code, environment pins, and pre-registered protocols as optional extras that can follow “if reviewers ask.”
The pattern works under misaligned incentives: hiring committees read titles; grant panels read impact factors; product marketing reads leaderboard rank. Short-termism makes “accept first, reproduce never” the winning move when replication is nobody’s scored duty.
The pattern
Reward geometry favours throughput of novel positive results over quality of the causal chain. Authors learn that a polished PDF beats a Docker image for career returns. Reviewers under time pressure score novelty and clarity, not rerunnable lineage. Journals require “data available upon request” — a phrase that often means unavailable in practice.
The claimed effect
- Faster publication cycles and higher citation velocity
- Stronger tenure and funding portfolios
- Media and policy attention from headline findings
- Apparent progress on “AI for X” KPIs without platform cost
In software systems
MLOps maturity is declared in strategy decks while experiment tracking is per-team, optional, and stripped before export to slides. “Model cards” exist as markdown no CI gate reads. Benchmark numbers enter product pages without run IDs. The metric publishes; the commits do not.
In human organisations
Academic and industrial research groups optimise for paper count, patent filings, and demo day outcomes. Reproducibility checklists (Pineau et al., 2021) are satisfied cosmetically — seeds mentioned, repository link broken, environment unspecified.
Local optimization at lab level: the PI’s metric is grants; the student’s metric is graduation; neither is scored on independent replication.
In socio-technical systems
Social proof amplifies unreplicated claims: Twitter threads, arXiv banners, and vendor benchmarks circulate as L3 renderings at effective tier 2–3 while consumers treat them as established. The socio-technical stack rewards shareability over refutation propagation.
What it violates
Values: Truth — headline substitutes for chain. Coherence — public claim and private artifacts tell different stories. Accountability — no one owns rerunnability.
Principles: Document before apply — mechanism unspecified before the claim travels. Immutability of facts — when replication fails, non-propagating refutation follows.
How to protect the organisation
Score the chain, not only the abstract. Promotion and funding criteria include run IDs, artifact availability, and independent replication — not only citation count.
Pre-commitment before peeking. Experiment architecture with registered hypotheses and primary metrics before outcomes are inspected.
Honest alternatives: Blind peer review where feasible; commitment boundary for consequential model deployment; pinned epistemic harness so provenance is cheaper than deferral.
Related failure architecture
Primary structural wound: ruptured provenance chain. Field-level name: ML reproducibility crisis. Empirical scale: Kapoor & Narayanan (2022).