Ruptured Provenance Chain

Reproduce training run 7a3f9b should mean: retrieve the configuration, dataset version, environment image, and code hash that run referenced; launch with those exact inputs; get the same output within documented non-determinism. When that is impossible — because data was cleaned in place, config lived only in a notebook, requirements.txt was unpinned, or the paper’s “94.2%” has no link to any run — the provenance chain is ruptured.

This is the architectural core of the ML reproducibility crisis — culture (publication over provenance, artifact withholding, provenance deferred) and misaligned incentives make missing commits rational; the diagnosis here is still missing commits and refs at each step from capture to published result. The same failure wears infrastructure’s clothes in Immutable Infrastructure (The Overwritten Balance → mutable history): unpinned images and mutable state destroy the chain that makes stability composable.

In software systems

Mutable L0/L1 without events. Dataset rows updated in place; labels overwritten instead of AnnotationRevised. The cleaned dataset is “the dataset” — no record of what was removed or why.

Unpinned environments. transformers>=4.0 resolves differently six months later. The training environment was treated as L2 when it needed to be L1: pinned image SHA, lockfile as L0 spec.

Unrecorded configuration. Learning rate, seed, and batch size set interactively; never committed before the run. The L1 specification that determines weights is lost.

Metrics divorced from runs. Accuracy in a paper or dashboard with no run ID, checkpoint hash, evaluation script version, or dataset version ref. A metric without provenance is not science — it is an anecdote (Event Sourced Science).

Operational read paths bypass the log. Event store exists; production reads mutable projections with no lineage. Inference-as-fact completes the rupture when scores become ground truth without model version.

In human organisations

Experiment notebooks as source of truth. The “official” result is whatever slide was presented; lab notebooks, raw data, and protocol deviations never entered a durable register. Replication requires interviewing authors — not querying history.

Audit trails that stop at approval. A decision is logged; the evidence bundle, model version, and data snapshot that justified it are not retained or not linkable. Post-hoc review cannot reconstruct what was known when.

Versionless documents. Policy and analysis PDFs circulate without version commits; “which assumption set produced this forecast?” has no answer.

In socio-technical systems

ML-based science at scale. Kapoor & Narayanan (2022) document leakage and irreproducibility across fields — inflated claims when train/test discipline and reporting fail. Pineau et al. (2021) show computational irreproducibility even when authors intend rigour: code, data, seeds, and environment must be first-class, not optional appendices.

When replication does fail, non-propagating refutation often compounds the wound: the chain was already broken, and the refutation still does not reach dependents.

LLM outputs in permanent records. Summaries pasted into case files with no prompt hash, model version, or source snapshot — a provenance chain of length one, already severed.

Structural causes

CRUD as domain language

State-over-events storage makes “current row” authoritative and history optional. Provenance requires append-only commits at each transition: measure → commit (labels, config, deployment) → derive (weights, metrics).

Epistemic tier collapse

L3 renderings (papers, dashboards) treated as evidence without refs to L2 sources; L2 treated as fixed without L1 pins.

Diffuse write authority

Many actors can mutate datasets, configs, and projections — each “fix” severs lineage unless emitted as compensating or superseding events.

Remedy (via patterns and architecture)

Event-sourced ML lifecycle; pinned epistemic harness; experiment architecture with pre-commitment and separate streams. Tools (MLflow, DVC, W&B) enforce the chain when used as L1 registries, not optional metadata. Immutable containers make deployment inputs L1-U-low (Immutable Infrastructure). Reproducibility becomes a structural property — a query over committed history — not a cultural aspiration alone.