Elamur Research⌕
☰
home / papers / 2023 / Reproducibility in Machine Learning-Driven Research

Reproducibility in Machine Learning-Driven Research

Paper · 2023 · Harald Semmelrock, Simone Kopeinik, Dieter Theiler et al.

Summary

This mini survey examines why so many machine learning (ML) results cannot be reproduced, and what might help. The authors review existing literature with three goals: describe the state of ML reproducibility across disciplines, catalogue the barriers, and collect the drivers (tools, practices, interventions) that support it. They adopt a three-level framework: experiment reproducibility (R1), data reproducibility (R2) and method reproducibility (R3). Each step up trades exact repeatability for broader generalizability. The review covers computer science (deep learning, reinforcement learning, NLP, recommender systems) and health/life science. Common barriers include unshared code and data, nondeterministic training, sensitivity to library versions, and data leakage. Medical data adds privacy constraints. The surveyed remedies include containerized environments, reproducibility checklists, model info sheets, awareness initiatives and stricter journal policies. The paper offers no new experiments. Its value lies in a shared vocabulary and an organized map of barriers and drivers, which can help researchers choose suitable reproducibility measures.

Key points

  • Three degrees of reproducibility:
    • R1 requires the same code and the same data.
    • R2 requires a different implementation on the same data.
    • R3 requires a different implementation on different data.
    • The authors relate these levels to the ACM's distinction between reproducibility and replicability.
  • ML platforms: Running one experiment on different platforms (e.g., OpenML, Kaggle, Azure ML) can produce different outcomes.
  • NLP findings:
    • Simply rerunning code can change results.
    • Minor settings such as rare-word thresholds can shift performance strongly.
    • When a reproduction deviated from the original, it scored lower.
  • Checklists: A reproducibility checklist proved useful in the NeurIPS 2019 challenge. GPT-4 checked whether checklist items were fulfilled with 86.6% accuracy.
  • Other drivers:
    • Docker and Code Ocean for standardized environments.
    • Model info sheets to detect data leakage.
    • ReScience and ReproducedPapers.org to raise awareness.
    • Preregistration and reproducibility papers in ACM TORS.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Graz University of TechnologyGraz · Austria5 works · 108 citations🏢Know Center Research GmbHGraz · Austria3 works · 104 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksInformation Systems · 1,888 worksDecision Sciences · 306 worksInformation Systems and Management · 96 works

Cite

Cite as Harald Semmelrock, Simone Kopeinik, Dieter Theiler, Tony Ross-Hellauer and Dominik Kowald. 2023. Reproducibility in Machine Learning-Driven Research. arXiv:2307.10320.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.10320 · arXiv 2307.10320

@misc{semmelrock2023reproducibility,
  title = {{Reproducibility in Machine Learning-Driven Research}},
  author = {Harald Semmelrock, Simone Kopeinik, Dieter Theiler, Tony Ross-Hellauer and Dominik Kowald},
  year = {2023},
  doi = {10.48550/arxiv.2307.10320},
  eprint = {2307.10320},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.10320}
}

References

12 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 1. Abel, D.: simple rl: Reproducible Reinforcement Learning in Python. ICLR (2019) 2. Ahmed, H., Lofstead, J.: Managing Randomness to Enable Reproducible Machine Learning. In: Proceedings of the 5th International Workshop on Practical Reproducible Evaluation of Computer Systems. pp. 15–20. find
  2. ACM, Minneapolis MN USA (2022). https://doi.org/10.1145/3526062.3536353, https://dl.acm.org/doi/ 10.1145/3526062.3536353 3. Artrith, N., Butler, K.T., Coudert, F.X., Han, S., Isayev, O., Jain, A., Walsh, A.: Best practices in machine learning for chemistry. Nature Chemistry 13(6), 505–508 (2021). ht… ↗
  3. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: OpenAI Gym (2016). https://doi.org/10.48550/arXiv.1606. 01540, http://arxiv.org/abs/1606.01540, arXiv:1606.01540 [cs] 11. Clyburne-Sherin, A., Fei, X., Green, S.A.: Computational Reproducibility via Contain… ↗
  4. Reproducibility in Machine Learning-Driven Research 14. Dror, R., Baumer, G., Bogomolov, M., Reichart, R.: Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets. Transactions of the Association for Computational Linguistics 5, 471–486 (2017) 15. Ferigo,…
  5. Science (2018). https://doi.org/10.1126/science.aat3298 — [doi:10.1126/science.aat3298](https://doi.org/10.1126/science.aat3298) ↗
  6. Semmelrock, Kopeinik, Theiler, Ross-Hellauer, and Kowald 26. Kapoor, S., Narayanan, A.: Leakage and the Reproducibility Crisis in ML-based Science (2022), http://arxiv.org/abs/2207.07048, arXiv:2207.07048 [cs, stat] 27. Khetarpal, K., Ahmed, Z., Cianflone, A., Islam, R., Pineau, J.: RE-EVALUATE: Rep… ↗
  7. ICML (2018), https://openreview.net/forum?id=HJgAmITcgm 28. Kowald, D., Schedl, M., Lex, E.: The unfairness of popularity bias in music recommendation: A reproducibility study. In: Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2… ↗
  8. Springer (2020) 29. Liu, R., Shah, N.B.: ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing (2023). https://doi.org/10.48550/arXiv. 2306.00622, http://arxiv.org/abs/2306.00622, arXiv:2306.00622 [cs] 30. McDermott, M.B.A., Wang, S., Marinsek, N., Ranganath, R., Ghass… ↗
  9. Springer (2021) 32. Nagarajan, P., Warnell, G., Stone, P.: The Impact of Nondeterminism on Reproducibility in Deep Reinforcement Learning. find
  10. ICML (2018), https://openreview.net/forum?id=S1e-OsZ4e7 33. Nagarajan, P., Warnell, G., Stone, P.: Deterministic Implementations for Reproducibility in Deep Reinforcement Learning (2019). https://doi.org/10.48550/ arXiv.1809.05676, http://arxiv.org/abs/1809.05676, arXiv:1809.05676 [cs] 34. Nalepa,… ↗
  11. Reproducibility in Machine Learning-Driven Research 39. Pouchard, L., Lin, Y., Van Dam, H.: Replicating Machine Learning Experiments in Materials Science. In: Parallel Computing: Technology Trends, pp. 743–755. IOS Press (2020). https://doi.org/10.3233/APC200105, https://ebooks.iospress. nl/doi/10.3… — [doi:10.3233/apc200105](https://doi.org/10.3233/apc200105) ↗
  12. College & Research Libraries (2018). https://doi.org/https://doi.org/10.5860/crl.79.1.2, https://journals.acrl.org/index.php/crl/article/view/16846 42. Shahriari, M., Ramler, R., Fischer, L.: How Do Deep-Learning Framework Versions Affect the Reproducibility of Neural Network Models? Machine Learni… — [doi:10.5860/crl.79.1.2](https://doi.org/10.5860/crl.79.1.2) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/semmelrock-2023-reproducibility-machine-learning-driven/