Improving Reproducibility in Machine Learning Research
2021 — NeurIPS reproducibility program report and checklist
Report from NeurIPS 2019 reproducibility challenge; checklist for code, data, hyperparameters, and compute in ML papers.
Pineau, J., et al. (2021). Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program). Transactions on Machine Learning Research; arXiv:2003.12206.
Why we cite it
Foundational computational reproducibility reference for the ML reproducibility crisis — complements Kapoor & Narayanan (2022) (leakage and scientific claims) with checklist and community process for code, data, and environment. Maps to our L0/L1 commit requirements before L2 training runs (Event Sourced Science).
Claims we use
- Reproducibility checklist — hyperparameters, seeds, compute, code/data availability as first-class publication requirements.
- Program evidence — reproducibility reviews at scale are feasible and change practice.
- Gap — checklists help but do not alone fix provenance architecture (mutable data, metric–run divorce); see experiment architecture and event-sourced ML lifecycle.
Author (primary)
Joelle Pineau — full author list in original paper.
Corpus stance
Context — adopt checklist spirit; extend with facet-based event sourcing as structural fix.