Event Sourced Science
Why the ML reproducibility crisis is an architecture problem
A significant fraction of machine learning research results cannot be reproduced. Papers publish results. Other researchers attempt to replicate them. The results differ — sometimes slightly, sometimes dramatically. The original authors attempt to reproduce their own work and cannot match their published numbers.
This is called the ML reproducibility crisis, and it is usually framed as a cultural problem: publication pressure (publication over provenance), inadequate documentation (provenance deferred), insufficient sharing of code and data (artifact withholding) — sustained by short-termism and local optimization under misaligned incentives. Empirical work documents the scale — data leakage alone affects hundreds of papers across scientific fields (Kapoor & Narayanan, 2022; arXiv:2207.07048), and computational irreproducibility persists even when authors intend rigour (Pineau et al., 2021).
Architecturally it is ruptured provenance chain — the causal chain from result to evidence was never preserved — and when replication fails, non-propagating refutation (The Retraction vignette, Art. 01): claims propagate widely and the demotion never reaches what depended on them.
These cultural factors are real — and rational given how careers and grants are scored. This article takes culture and incentive structure as given. We are not waiting for career scoring or institutional culture to reform before asking what architecture can do. Misaligned incentives explain why deferral persists without gates; they are not an excuse to defer provenance, but they are the environment any remedy must work inside. Underneath them is an architectural problem with a precise diagnosis: the causal chain from raw data to published result is broken at multiple points, because the artifacts at each step were not treated according to the facets of truth.
Fix the architecture — ruptured provenance chain healed with committed provenance chain at each step — and replayable reproducibility becomes a structural property workable under current incentives, not a bet on cultural reform first.
See concept maps: ML reproducibility crisis (this case) · Provenance discipline stack (abstract, transferable layering).
The taxonomy applied to ML
The capture modes from Atoms of Truth map cleanly onto the ML lifecycle.
Measure captures: Raw observations — sensor readings, images, text documents, user events — are the designated input boundary of the ML system. They are transductions of physical or social reality, not the reality itself (The Real). A pixel value is not the scene. A text string is not the intent. The fidelity of the capture apparatus matters and must be characterised.
Assert captures:
Hypotheses, notebook drafts, exploratory hyperparameter guesses, and model chain of thought (CoT) traces are assert — put forward without binding (Four Capture Modes). HypothesisLogged { claim, author, at } and NotebookCellExecuted { cellId, parameters, at } belong in the log as honest assertions, not as measurements or commits. A learning rate typed in a cell is not yet a L1 specification; a model’s CoT is not a measurement. Treating assert as commit is inference-as-fact; treating it as measure collapses instrument transduction with human conjecture. Reproducibility requires assertions to stay visible and separable until a named authority commits configuration at the commitment boundary — or they vanish when the notebook closes.
Commit captures:
Ground-truth labels for supervised training are L0 decision events, not observations. When a researcher annotates a benchmark example — marking a microscopy frame with a cell phenotype, or a spectrum with a material class for a classification study — they are making a principled committed decision: an epistemic commitment to treat that sample as belonging to that class in the training record. The label is a L0 event: TrainingLabelCommitted { sampleId, label, committedBy, expertise, protocolRef, at }. Writing a model’s argmax into the ground-truth column without human commit is inference-as-fact. This is not a correction to the measurement. It is a separate fact about what a named authority decided for the dataset.
The implication: when a labelling error is discovered, the correct response is not to overwrite the label in place. It is to record a new decision: TrainingLabelRevised { originalLabelId, newLabel, revisedBy, reason }. Both the original label and the revision remain in the record. Disagreement between annotators is itself information about label difficulty — information that should be preserved, not resolved by majority vote with the minority erased (When Facts Are Wrong).
Decision events — deployment:
A model deployed to production is the result of a principled committed decision at the commitment boundary. ModelDeployedToProduction is a L0 event, not an implementation detail. Who decided, when, on what evidence, under what policy — all permanent facts. Without this record, the connection between a model’s production behaviour and the human decision that put it there is invisible.
Derivations: Model weights are L2 — a deterministic derivation of the training data through the training process. Given the same data, the same architecture, the same hyperparameters, and the same random seed in a deterministic training environment, the same weights emerge. The reproducibility of the weights depends entirely on the fixedness of these L1 inputs.
Where the causal chain breaks
Break 1: Mutable datasets. Datasets are cleaned in place. Duplicates are removed. Corrupted samples are filtered. Private information is redacted. These operations overwrite the aggregate without recording what was removed or why. Individual measure captures may be L0 designated observations; the dataset as published is an L1 artifact — and in-place cleaning mutates that aggregate without append-only events, the mutable history failure mode. The correction should instead produce new L0 events: SampleRemovedFromDataset { sampleId, reason, removedBy, at }, and the cleaned dataset becomes a new L1 version with explicit references to the cleaning operations applied.
Break 2: Unpinned computational environments. A training run uses transformers>=4.0 in its requirements. Six months later, transformers is at version 4.35 and produces different results. The environment was L2 — it resolves differently over time — when it needed to be L1: a pinned Docker image with a specific SHA, containing every library at a specific version (Immutable Infrastructure). The Docker image is the authoritative artifact. The lockfile that defines it is the L0 specification.
Break 3: Assert never committed. The training run used a specific learning rate schedule, batch size, and random seed. These were set interactively in a notebook — assert captures, exploratory and non-binding. Nobody recorded them formally or crossed the commitment boundary before training. The configuration — the L1 specification that determines the L2 output — was never promoted from assert to commit and was lost when the session ended. Reproduction requires that every configuration parameter that influenced the training outcome be committed as a L0 fact before training begins.
Break 4: Results divorced from provenance. “94.2% accuracy on the validation set” appears in a paper. The link between this number and the exact run that produced it, the exact dataset it was evaluated on, the exact model checkpoint it came from — is not preserved. Every metric is a L2 artifact. It must carry explicit references to the L1 inputs (training run ID, evaluation dataset version, model checkpoint hash, evaluation script version) that produced it. A metric without provenance is not science — it is an anecdote. It is a claim with no path back to tier one.
Reproducibility as a structural property
This is event sourced science in the precise sense of Four Capture Modes: an immutable append-only log of what was measured, asserted, committed, and derived during reasoning, experiment design, and execution. Reproducibility is not a separate ritual layered on top — it is replaying that history to recompute outputs, the same move event-sourced systems use when they rebuild state from the log.
When the causal chain is intact — when every artifact carries references to the L1 inputs that produced it, and those inputs carry references to the L0 decisions that justified them — reproduction is not a discipline. It is a query over committed history.
“Reproduce training run 7a3f9b” means: retrieve the configuration referenced by that run, retrieve the dataset version it referenced, retrieve the Docker image it ran in, launch the training job with those exact inputs. The output should be identical, within the bounds of any non-determinism that was explicitly documented.
Tools like MLflow, DVC, and Weights & Biases exist precisely to enforce this causal chain. They are version control for L1 artifacts. They record which dataset version, which environment, which configuration, which code version produced which run. They make the causal chain explicit and queryable — the same architectural move as committed provenance chain at bind time.
The cultural problem — researchers not using these tools rigorously — is downstream of both incentive pressure and an architectural understanding problem. Given misaligned incentives, tooling only helps when it is cheaper than deferring provenance. When researchers understand that a training run is a L2 artifact whose reproducibility depends entirely on the fixedness of its L1 inputs, the discipline follows more naturally. The tools are the mechanism. The understanding is the motivation. And it is the same understanding the physicist’s ladder taught us at the very start: the experiments at the bottom must be preserved, or everything built above them is unmoored (The Epistemic Ladder).
When behaving well gets cheaper
The incentive geometry is real and, for this article, fixed. Under publication pressure, short-termism, and local optimization, scientists rationally favour publishing too soon and too often — deferring provenance until “after acceptance” and treating rigour as friction on throughput. That culture is not an illusion to be argued away; it is the constraint architecture must absorb.
But the remedy here is not moral suasion or “slow down and document more by hand.” Given misaligned incentives, the move is to lower the cost of behaving well so rigour and speed stop trading off so brutally.
Hypothesis: A thinking exoskeleton with an epistemic harness — ethics and adopted values encoded in gates and workflows, not poster slogans — can help scientists improve quality while publishing sooner, not instead of publishing. When the exoskeleton takes over the heavy lifting of maintaining the provenance chain — pinning environments, committing configuration before the run, linking every metric to committed inputs, blocking bind when refs are missing — the marginal cost of honest work falls. The barrier to delivering higher-quality work drops with it. If that holds, structural reproducibility — runs replayable from the log — becomes achievable while career scoring still rewards headlines over lineage. Architecture changes what “fast” means inside the lab; it does not by itself rewrite grants and tenure. That is sufficient for the architectural half of the ML reproducibility crisis; the publication-geometry half may lag until rewards change (see below).
In general: when the cost of epistemic hygiene is borne by structure rather than remembered under deadline, maintaining lineage ceases to be heroics and becomes the default path to submission.
An exoskeleton for ecosystems
Individual labs can heal local chains. Much of the crisis is cross-institutional: journals, reviewers, funders, preprint servers, and lab notebooks each hold fragments of lineage that rarely compose. Refutations stall in one silo while citations propagate in another.
An exoskeleton for ecosystems — a shared epistemic platform with provenance, refutation propagation, and contestability built in — could improve health across a wide range of sciences given existing cultural constraints, if institutions collaborate on interchange rather than competing silos. Journals and funders need not abandon their existing criteria wholesale; they need to require and consume queryable lineage the way they already require PDFs and DOIs. The platform is infrastructure that makes honest provenance cheap enough to mandate — not a replacement for peer review, but the substrate on which review, replication, and correction can actually travel.
This is speculative design, not a report on a deployed system. It is the direction the structural remedy points toward once reproduction is a query, deferral is blocked at bind time, and the exoskeleton does the lifting that culture currently punishes.
What if incentives aligned too?
Everything above treats incentive geometry as a given constraint — architecture lowers the cost of rigour and mandates queryable lineage inside misaligned systems, not after they have been reformed. A harder question sits beside it.
Open question: What if we could also change the incentive structure — not only make honesty cheaper through architecture, but honour honest, clean contributions in the scoring systems careers actually run on? What if economic interest could attach to work whose provenance chain survives scrutiny — on the same log that holds the science?
Today, grants, citations, and tenure mostly reward visible outputs at bind time, not downstream verification of lineage. A paper with a ruptured chain can still win; meticulous provenance that prevents future retractions may never appear on a CV. That is the geometry publication over provenance exploits.
Structural reproducibility can improve while incentives stay misaligned; publication geometry may not. Architecture can lower the cost of honesty. It cannot, by itself, reverse a reward function that still pays for headlines alone. Something has to connect queryable lineage to recognition — and perhaps to reward — without collapsing into another proxy to game (Goodhart’s Law). We do not know how. We suspect the incentive-facing face of the crisis will persist if shame and retraction carry all the weight while the upside of clean work stays invisible — even when individual runs become replayable.
If you have seen systems where verification or lineage actually moved career outcomes — or where it conspicuously failed — that experience belongs in the conversation. This article stops at the question.
Part VIII continues with how the same L0 foundation makes software cheap to extend — From Immutable Events to Immutable Software.