Leakage and the Reproducibility Crisis in ML-based Science
2022 — systematic survey of data leakage and irreproducibility across quantitative fields
Survey of 17 fields (329 papers affected); taxonomy of leakage types; civil war prediction reproducibility study; model info sheets proposal.
Kapoor, S., & Narayanan, A. (2022). Leakage and the Reproducibility Crisis in ML-based Science. ICML 2022; arXiv:2207.07048.
Why we cite it
Empirical lower bound on the ML reproducibility crisis: not anecdote but a structured survey showing leakage in every reviewed field and wildly overoptimistic claims when train/test discipline fails. Aligns with this corpus’s diagnosis — broken causal chain and missing commit boundaries between data prep, modelling, and evaluation — in Event Sourced Science.
Claims we use
- Leakage taxonomy (L1–L3) — textbook train/test errors through temporal and distribution-shift leakage; useful vocabulary alongside facet-based “where the chain breaks.”
- Scale — 20 survey papers across 17 fields, 329 downstream papers affected (lower bound).
- Science vs product ML — competition and production mitigations do not substitute for provenance on scientific claims.
- Model info sheets — reporting analogue to our L0 config + dataset + environment commits before bind.
Authors
Sayash Kapoor · Arvind Narayanan
Corpus stance
Context — empirical and methodological reference for the crisis; we adopt the architectural remedy (event-sourced provenance, pinned L1, committed labels) rather than their checklist alone.