Elamur Research⌕
☰
home / papers / 2024 / Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers

Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers

Paper · 2024 · Harald Semmelrock, Tony Ross-Hellauer, Simone Kopeinik et al.

Summary

The authors argue that ML-based research has a serious reproducibility problem and that the community still underestimates it. This applies both to computer science and to fields that use ML as a tool, such as biomedicine. The paper is a structured overview rather than an empirical study. It adopts the framework of Gundersen et al., which defines four levels of shared material: R1 Description, R2 Code, R3 Data and R4 Experiment. It also distinguishes outcome, analysis and interpretation reproducibility. Within this framework, the authors sort nine barriers by level. They then review drivers, meaning technical tools, procedural measures, and awareness or education, and weigh each driver's strengths against its limitations. The central contribution is a Drivers-Barriers-Matrix that shows which remedies address which obstacles. The authors conclude that the deepest cause is cultural: open, reproducible work is poorly rewarded, underfunded and rarely taught. This makes the paper a practical checklist for planning ML projects that others can verify and rerun.

Key points

  • Shared-material levels: R1 means only a textual description is shared and R4 means text, code and data are all shared. Moving toward R4 makes exact reproduction easier, but it weakens evidence that the findings generalize beyond the original setup.
  • R1 barriers: Reporting problems include underspecified models or training procedures, undefined evaluation metrics, and selective reporting (for example, showing only the best run instead of means and variances).
  • R4 barriers: Even when everything is shared, three obstacles remain:
    • inherent nondeterminism, partly manageable with fixed seeds and deterministic implementations;
    • software and hardware differences between environments;
    • limited compute, which is especially acute for LLMs.
  • Drivers compared: The paper weighs tools such as virtualization, hosting services, checklists, model cards, ROBUST-ML guidelines, standardized datasets and publication policies, noting their trade-offs. Examples include the time needed for compliance and slow adoption.
  • Future work: The authors flag AutoML and no/low-code tools used by domain non-experts as an emerging risk to reproducibility. The work was funded by the Horizon Europe project TIER2 and the FFG COMET program.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Graz University of TechnologyGraz · Austria5 works · 108 citations🏢Know Center Research GmbHGraz · Austria3 works · 104 citations🎓University of GrazGraz · Austria3 works · 92 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 worksDecision Sciences · 306 worksInformation Systems and Management · 96 works

Cite

Cite as Harald Semmelrock, Tony Ross-Hellauer, Simone Kopeinik, Dieter Theiler, Armin Haberl, Stefan Thalmann and Dominik Kowald. 2024. Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers. arXiv:2406.14325.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.14325 · arXiv 2406.14325

@misc{semmelrock2024reproducibility,
  title = {{Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers}},
  author = {Harald Semmelrock, Tony Ross-Hellauer, Simone Kopeinik, Dieter Theiler, Armin Haberl, Stefan Thalmann and Dominik Kowald},
  year = {2024},
  doi = {10.48550/arxiv.2406.14325},
  eprint = {2406.14325},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.14325}
}

References

18 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 1. Ahmed, H., Lofstead, J.: Managing Randomness to Enable Reproducible Machine Learning. In: Proceedings of the 5th International Workshop on Practical Reproducible Evaluation of Computer Systems. pp. 15–20. ACM, Minneapolis MN USA (2022) 2. Ahmed, H., Tchoua, R., Lofstead, J.: Measuring Reproducibl… — [arXiv:2302.12691](https://arxiv.org/abs/2302.12691) find
  2. Aslanyan, Z., Vasilikos, P.: Privacy-Preserving Machine Learning (2020) 9. Aumjaud, P., McAuliffe, D., Lera, F.J.R., Cardiff, P.: rl_reach: Reproducible reinforcement learning experiments for robotic reaching tasks. Software Impacts 8, 100061 (2021) find
  3. Semmelrock, Ross-Hellauer, Kopeinik, Theiler, Haberl, Thalmann, & Kowald 10. Baker, M.: 1,500 scientists lift the lid on reproducibility. Nature 533(7604), 452– 454 (2016) 11. Banks, G.C., Rogelberg, S.G., Woznyj, H.M., Landis, R.S., Rupp, D.E.: Evidence on questionable research practices: The good,… find
  4. Mit Press (2020) 16. Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A.F., Ammanamanchi, P.S., Black, S., Clive, J., et al.: Lessons from the trenches on reproducible evaluation of language models. arXiv preprint arXiv:2405.14782 (2024) 17. Boettiger, C.: An introducti… find
  5. Knowledge Exchange (2021) 23. Chirigati, F., Rampin, R., Shasha, D., Freire, J.: ReproZip: Computational Reproducibility With Ease. In: Proceedings of the 2016 International Conference on Management of Data. pp. 2085–2088. SIGMOD ’16, ACM, New York, NY, USA (2016) 24. Choi, Y.D., Roy, B., Nguyen, J.… find
  6. Reproducibility in Machine Learning-based Research 27. Collaboration, O.S.: Estimating the reproducibility of psychological science. Science 349(6251), aac4716 (2015) 28. Collins, G.S., Reitsma, J.B., Altman, D.G., Moons, K.G.: Transparent Reporting of a Multivariable Prediction Model for Individual… find
  7. Science (2018) 59. Ioannidis, J.P.: Why most published research findings are false. PLoS medicine 2(8), e124 (2005) find
  8. Reproducibility in Machine Learning-based Research 60. Iqbal, S.A., Wallach, J.D., Khoury, M.J., Schully, S.D., Ioannidis, J.P.: Reproducible research practices and transparency across the biomedical literature. PLoS biology 14(1), e1002333 (2016) 61. Ivanescu, A.E., Li, P., George, B., Brown, A.W.,… — MIMIC-IV, a freely accessible electronic health record dataset
  9. Springer (2022) 69. Kowald, D., Schedl, M., Lex, E.: The unfairness of popularity bias in music recommendation: A reproducibility study. In: 42nd European Conference on IR Research, ECIR 2020. pp. 35–42. find
  10. Springer (2020) 70. Kowald, D., Scher, S., Pammer-Schindler, V., Müllner, P., Waxnegger, K., Demelius, L., Fessl, A., Toller, M., Mendoza Estrada, I.G., Šimić, I., et al.: Establishing and evaluating trustworthy ai: overview and research challenges. Frontiers in Big Data 7, 1467222 (2024) 71. Kweon,… find
  11. Springer (2021) 84. Munafò, M.R., Nosek, B.A., Bishop, D.V.M., Button, K.S., Chambers, C.D., Per-cie du Sert, N., Simonsohn, U., Wagenmakers, E.J., Ware, J.J., Ioannidis, J.P.A.: A manifesto for reproducible science. Nature Human Behaviour 1(1), 1–9 (2017), publisher: Nature Publishing Group 85. Nag… — [arXiv:1809.05676](https://arxiv.org/abs/1809.05676) find
  12. ACM, Lausanne Switzerland (2020) 87. Nosek, B.A., Beck, E.D., Campbell, L., Flake, J.K., Hardwicke, T.E., Mellor, D.T., van ’t Veer, A.E., Vazire, S.: Preregistration Is Hard, And Worthwhile. Trends in Cognitive Sciences 23(10), 815–818 (2019) 88. Nosek, B.A., Hardwicke, T.E., Moshontz, H., Allard,… find
  13. IOS Press (2020) 97. Provenzano, D., Rao, Y.J., Goyal, S., Haji-Momenian, S., Lichtenberger, J., Loew, M.: Radiologist vs Machine Learning: A Comparison of Performance in Cancer Imaging. In: 2021 IEEE Applied Imagery Pattern Recognition Workshop (AIPR). pp. 1–10 (2021), iSSN: 2332-5615 98. Pushkarna… find
  14. CEUR Workshop Proceedings (2022) 100. Raff, E.: A Step Toward Quantifying Independently Reproducible Machine Learning Research. In: Advances in Neural Information Processing Systems. vol. 32. find
  15. Curran Associates, Inc. (2019) 101. Raste, S., Singh, R., Vaughan, J., Nair, V.N.: Quantifying Inherent Randomness in Machine Learning Algorithms. find
  16. SSRN Electronic Journal (2022) 102. Rijcke, S.d., Wouters, P.F., Rushforth, A.D., Franssen, T.P., Hammarfelt, B.: Evaluation practices and effects of indicator use—a literature review. Research evaluation 25(2), 161–169 (2016) 103. Rougier, N.P., Hinsen, K., Alexandre, F., Arildsen, T., Barba, L.A.,… — [arXiv:2405.14422](https://arxiv.org/abs/2405.14422) find
  17. Semmelrock, Ross-Hellauer, Kopeinik, Theiler, Haberl, Thalmann, & Kowald 106. Schacherer, D.P., Herrmann, M.D., Clunie, D.A., Höfener, H., Clifford, W., Longabaugh, W.J., Pieper, S., Kikinis, R., Fedorov, A., Homeyer, A.: The nci imaging data commons as a platform for reproducible research in comput… — Reproducibility in Machine Learning-Driven Research
  18. CEUR Workshop Proceedings (2018) 115. Trosten, D.J.: Questionable practices in methodological deep learning research. In: Proceedings of the Northern Lights Deep Learning Workshop. vol. 4 (2023) 116. Walonoski, J., Kramer, M., Nichols, J., Quina, A., Moesel, C., Hall, D., Duffett, C., Dube, K., Gall… — [arXiv:2108.04417](https://arxiv.org/abs/2108.04417) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/semmelrock-2024-reproducibility-machine-learning-based/