Leakage and the Reproducibility Crisis in ML-based Science (Kapoor & Narayanan, 2023)¶
Reference
Citation: Kapoor, S., Narayanan, A. "Leakage and the reproducibility crisis in machine-learning-based science." Patterns 4(9), 100804 (2023). Type: paper. Link: doi.org/10.1016/j.patter.2023.100804.
What it is¶
A survey of data leakage as a failure mode in ML-based science, found across 294 papers in 17 fields. Leakage produces over-optimistic performance that does not reproduce.
Role in the record¶
- Grounds BP01: reaching for a model where the method does not fit the task produces non-reproducible, over-optimistic results, one of the avoidable failure modes the practice warns against.
Atom-level for/against detail and quotes are in the provenance data
(assets/provenance.yml), keyed by practice atom.