Elamur Research⌕
☰
home / papers / 2023 / REFORMS: Reporting Standards for Machine Learning Based Science

REFORMS: Reporting Standards for Machine Learning Based Science

Paper · 2023 · Sayash Kapoor, Emily Cantrell, Kenny Peng et al.

Summary

Machine learning is spreading quickly across scientific disciplines, and so are problems with validity, reproducibility, and generalizability. Different fields tend to repeat the same mistakes, and most have weak or no reporting norms for ML-driven work. The authors respond with REFORMS, a field-independent set of reporting standards for what they call ML-based science: research in which a model's performance is used to support a scientific claim. This category excludes ML methods development, commercial predictive analytics, and classical explanatory modeling. The checklist was built from a broad literature review of best practices and recurring pitfalls. It follows the stages of a typical ML pipeline, borrows from checklists in other domains, and was refined through several feedback rounds among the authors. Each module has three aims: tie the scientific claim to the modeling process, steer researchers away from execution errors, and make independent verification possible. The resulting tool can help authors design studies, reviewers assess papers, and journals enforce transparency.

Key points

  • REFORMS contains 32 items grouped into 8 modules that track the ML pipeline. Each item comes with explanatory guidelines (Appendix B).
  • The checklist was agreed on by 19 researchers from computer science, data science, mathematics, the social sciences, and the biomedical sciences.
  • The authors position it against earlier checklists:
    • CLAIM (AI in medical imaging) is specific to one field.
    • Pineau et al.'s checklist targets ML methods research and does not ask about the distribution a claim refers to.
  • The authors expect it to be most useful for predictive modeling studies. They see it as less useful for search tasks (e.g., materials discovery) or cases where lab verification is easy.
  • Supporting context: the EQUATOR network lists over 500 health-research reporting checklists, and the 2014 NIH principles promoted journal checklists. All materials are hosted at reforms.cs.princeton.edu.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Social Sciences · 2,009 worksSafety Research · 665 worksDecision Sciences · 306 worksInformation Systems and Management · 96 works

Cite

Cite as Sayash Kapoor, Emily Cantrell, Kenny Peng, Thanh Hien Pham, Christopher A. Bail, Odd Erik Gundersen, Jake M. Hofman, Jessica Hullman, Michael A. Lones, Momin M. Malik, Priyanka Nanayakkara, Russell A. Poldrack, Inioluwa Deborah Raji, Michael Roberts, Matthew J. Salganik, Marta Serra-Garcia, Brandon M. Stewart, Gilles Vandewiele and Arvind Narayanan. 2023. REFORMS: Reporting Standards for Machine Learning Based Science. arXiv:2308.07832.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2308.07832 · arXiv 2308.07832

@misc{kapoor2023reforms,
  title = {{REFORMS: Reporting Standards for Machine Learning Based Science}},
  author = {Sayash Kapoor, Emily Cantrell, Kenny Peng, Thanh Hien Pham, Christopher A. Bail, Odd Erik Gundersen, Jake M. Hofman, Jessica Hullman, Michael A. Lones, Momin M. Malik, Priyanka Nanayakkara, Russell A. Poldrack, Inioluwa Deborah Raji, Michael Roberts, Matthew J. Salganik, Marta Serra-Garcia, Brandon M. Stewart, Gilles Vandewiele and Arvind Narayanan},
  year = {2023},
  doi = {10.48550/arxiv.2308.07832},
  eprint = {2308.07832},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.07832}
}

References

187 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Susan Athey and Guido W. Imbens. Machine learning methods that economists should know about. Annual Review of Economics, 11(1):685–725, 2019. _eprint: https://doi.org/10.1146/annurev-economics-080217-053433. — [doi:10.1146/annurev-economics-080217-053433](https://doi.org/10.1146/annurev-economics-080217-053433) ↗
  2. Daniel R. Schrider and Andrew D. Kern. Supervised machine learning for population genetics: A new paradigm. Trends in Genetics, 34(4):301–312, April 2018. find
  3. John Joseph Valletta, Colin Torney, Michael Kings, Alex Thornton, and Joah Madden. Applications of machine learning in animal behaviour studies. Animal Behaviour, 124:203–220, February 2017. find
  4. R. Iniesta, D. Stahl, and P. McGuffin. Machine learning, statistical learning and the future of biological research in psychiatry. Psychological Medicine, 46(12):2455–2465, September 2016. Publisher: Cambridge University Press. find
  5. Scott Tonidandel, Eden B. King, and Jose M. Cortina. Big data methods: Leveraging modern data analytic techniques to build organizational science. Organizational Research Methods, 21(3):525–547, July 2018. Publisher: SAGE Publications Inc. find
  6. Tal Yarkoni and Jacob Westfall. Choosing prediction Over explanation in psychology: Lessons from machine learning. Perspectives on Psychological Science, 12(6):1100–1122, November 2017. Publisher: SAGE Publications Inc. find
  7. Justin Grimmer, Margaret E. Roberts, and Brandon M. Stewart. Machine learning for social science: An agnostic approach. Annual Review of Political Science, 24(1):395–419, 2021. _eprint: https://doi.org/10.1146/annurev-polisci-053119-015921. — [doi:10.1146/annurev-polisci-053119-015921](https://doi.org/10.1146/annurev-polisci-053119-015921) ↗
  8. Anja K. Leist, Matthias Klee, Jung Hyun Kim, David H. Rehkopf, Stéphane P. A. Bordas, Graciela Muniz-Terrera, and Sara Wade. Mapping of machine learning approaches for description, prediction, and causal inference in the social and health sciences. Science Advances, 8(42):eabk1942, October 2022. find
  9. Timothy L. Wiemken and Robert R. Kelley. Machine learning in epidemiology and health outcomes research. Annual Review of Public Health, 41(1):21–36, April 2020. find
  10. Hal R. Varian. Big data: New tricks for econometrics. Journal of Economic Perspectives, 28(2):3–28, May 2014. find
  11. Sendhil Mullainathan and Jann Spiess. Machine learning: An applied econometric approach. Journal of Economic Perspectives, 31(2):87– 106, May 2017. find
  12. Leo Breiman. Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical Science, 16(3):199–231, August 2001. Publisher: Institute of Mathematical Statistics. find
  13. Jessica Hullman, Sayash Kapoor, Priyanka Nanayakkara, Andrew Gelman, and Arvind Narayanan. The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pages 335–348, Ox… find
  14. Andrew L. Beam, Arjun K. Manrai, and Marzyeh Ghassemi. Challenges to the reproducibility of machine learning models in health care. JAMA, 323(4):305, January 2020. find
  15. Michael Roberts, Derek Driggs, Matthew Thorpe, Julian Gilbey, Michael Yeung, Stephan Ursprung, Angelica I. Aviles-Rivero, Christian Etmann, Cathal McCague, Lucian Beer, Jonathan R. Weir-McCall, Zhongzhao Teng, Effrossyni Gkrania-Klotsas, James H. F. Rudd, Evis Sala, and Carola-Bibiane Schönlieb. Com… find
  16. Xavier Bouthillier, César Laurent, and Pascal Vincent. Unreproducible research is reproducible. In International Conference on Machine Learning, pages 725–734. PMLR, May 2019. ISSN: 2640-3498. find
  17. Matthew B. A. McDermott, Shirly Wang, Nikki Marinsek, Rajesh Ranganath, Luca Foschini, and Marzyeh Ghassemi. Reproducibility in machine learning for health research: Still a ways to go. Science Translational Medicine, 13(586), March 2021. Publisher: American Association for the Advancement of Scienc… find
  18. Odd Erik Gundersen and Sigbjørn Kjensmo. State of the art: Reproducibility in artificial intelligence. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), April 2018. find
  19. Gaël Varoquaux and Veronika Cheplygina. Machine learning for medical imaging: Methodological failures and recommendations for the future. NPJ digital medicine, 5(1):48, April 2022. find
  20. Robin M. Schmidt, Frank Schneider, and Philipp Hennig. Descending through a crowded valley - benchmarking deep learning optimizers. In Proceedings of the 38th International Conference on Machine Learning, pages 9367–9376. PMLR, July 2021. ISSN: 2640-3498. find
  21. Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Naz Sepah, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Dmitriy Serdyuk, Tal Arbel, Chris Pal, Gaël Varoquaux, and Pascal Vincent. Accounting for variance in ma… find
  22. Orianna DeMasi, Konrad Kording, and Benjamin Recht. Meaningless comparisons lead to false optimism in medical machine learning. PLOS ONE, 12(9):e0184604, September 2017. find
  23. D. Sculley, Jasper Snoek, Alex Wiltschko, and Ali Rahimi. Winner’s curse? on pace, progress, and empirical rigor. June 2018. find
  24. David M. Liu and Matthew J. Salganik. Successes and struggles with computational reproducibility: Lessons from the fragile families challenge. Socius: Sociological Research for a Dynamic World, 5:237802311984980, January 2019. find
  25. Sören Dittmer, Michael Roberts, Julian Gilbey, Ander Biguri, AIX-COVNET Collaboration, Ian Selby, Anna Breger, Matthew Thorpe, Jonathan R. Weir-McCall, Effrossyni Gkrania-Klotsas, Anna Korhonen, Emily Jefferson, Georg Langs, Guang Yang, Helmut Prosch, Jan Stanczuk, Jing Tang, Judith Babar, Lorena Es… find
  26. Odd Erik Gundersen, Saeid Shamsaliei, and Richard Juul Isdahl. Do machine learning platforms provide out-of-the-box reproducibility? Future Generation Computer Systems, 126:34–47, January 2022. find
  27. Jake M. Hofman, Amit Sharma, and Duncan J. Watts. Prediction and explanation in social systems. Science, 355(6324):486–488, February 2017. find
  28. John Banja. AI hype and radiology: A plea for realism and accuracy. Radiology: Artificial Intelligence, 2(4):e190223, July 2020. find
  29. Valen E. Johnson, Richard D. Payne, Tianying Wang, Alex Asher, and Soutrik Mandal. On the reproducibility of psychological science. Journal of the American Statistical Association, 112(517):1–10, January 2017. find
  30. Samuel J. Bell and Onno P. Kampman. Perspectives on machine learning from psychology’s reproducibility crisis, April 2021. arXiv:2104.08878 [cs]. — [arXiv:2104.08878](https://arxiv.org/abs/2104.08878) find
  31. Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research (a report from the NeurIPS 2019 reproducibility program). The Journal of Machine Learning Researc… find
  32. Marta Serra-Garcia and Uri Gneezy. Nonreplicable publications are cited more than replicable ones. Science Advances, 7(21):eabd1705, May 2021. find
  33. Matthew J Salganik, Ian Lundberg, Alexander T Kindel, et al. Measuring the predictability of life outcomes with a scientific mass collaboration. Proceedings of the National Academy of Sciences, 117(15):8398–8403, 2020. find
  34. John Mongan, Linda Moy, and Charles E. Kahn. Checklist for artificial intelligence in medical imaging (CLAIM): A guide for authors and reviewers. Radiology: Artificial Intelligence, 2(2):e200029, March 2020. Publisher: Radiological Society of North America. find
  35. Patrick M Bossuyt, Johannes B Reitsma, David E Bruns, Constantine A Gatsonis, Paul P Glasziou, Les Irwig, Jeroen G Lijmer, David Moher, Drummond Rennie, Henrica C W de Vet, Herbert Y Kres-sel, Nader Rifai, Robert M Golub, Douglas G Altman, Lotty Hooft, Daniël A Korevaar, and Jérémie F Cohen. STARD 2… find
  36. Richard G. White, Avi J. Hakim, Matthew J. Salganik, Michael W. Spiller, Lisa G. Johnston, Ligia Kerr, Carl Kendall, Amy Drake, David Wilson, Kate Orroth, Matthias Egger, and Wolfgang Hladik. Strengthening the Reporting of Observational Studies in Epidemiology for respondent-driven sampling studies:… find
  37. Emily G. Simmonds, Kwaku Peprah Adjei, Christoffer Wold Andersen, Janne Cathrin Hetle Aspheim, Claudia Battistin, Nicola Bulso, Hannah Christensen, Benjamin Cretois, Ryan Cubero, Ivan A. Davidovich, Lisa Dickel, Benjamin Dunn, Etienne Dunn-Sigouin, Karin Dyrstad, Sigurd Einum, Donata Giglio, Haakon… find
  38. Sayash Kapoor and Arvind Narayanan. Leakage and the reproducibility crisis in ML-based science, July 2022. arXiv:2207.07048 [cs, stat]. find
  39. Gary S. Collins, Johannes B. Reitsma, Douglas G. Altman, and Karel GM Moons. Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): The TRIPOD statement. BMC Medicine, 13(1):1, January 2015. find
  40. Amy C. Plint, David Moher, Andra Morrison, Kenneth Schulz, Douglas G. Altman, Catherine Hill, and Isabelle Gaboury. Does the CON-SORT checklist improve the quality of reports of randomised controlled trials? A systematic review. The Medical Journal of Australia, 185(5):263–267, September 2006. find
  41. SeungHye Han, Tolani F. Olonisakin, John P. Pribis, Jill Zupetic, Joo Heung Yoon, Kyle M. Holleran, Kwonho Jeong, Nader Shaikh, Doris M. Rubio, and Janet S. Lee. A checklist is associated with increased quality of reporting preclinical biomedical research: A systematic review. PLoS ONE, 12(9):e01835… find
  42. Principles and guidelines for reporting preclinical research. https://www.nih.gov/research-training/rigor-reproducibility/principles-guidelines-reporting-preclinical-research, August 2015. ↗
  43. Reporting guidelines. The EQUATOR Network. https://www.equator-network.org/reporting-guidelines/. ↗
  44. Anna Rogers, Timothy Baldwin, and Kobi Leins. ‘just what do you think you’re doing, Dave?’ A checklist for responsible data use in NLP. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4821–4833, Punta Cana, Dominican Republic, November 2021. Association for Computatio… find
  45. Odd Erik Gundersen, Yolanda Gil, and David W. Aha. On reproducible AI: Towards reproducible research, open science, and digital scholarship in AI publications. AI Magazine, 39(3):56–68, September 2018. find
  46. David Donoho. 50 years of data science. Journal of Computational and Graphical Statistics, 26(4):745–766, October 2017. find
  47. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, et al. On the opportunities and risks of foundation models, 2022.
  48. Sayash Kapoor and Arvind Narayanan. OpenAI’s policies hinder reproducible research on language models, 2022. find
  49. Jake M. Hofman, Duncan J. Watts, Susan Athey, Filiz Garip, Thomas L. Griffiths, Jon Kleinberg, Helen Margetts, Sendhil Mullainathan, Matthew J. Salganik, Simine Vazire, Alessandro Vespignani, and Tal Yarkoni. Integrating explanation and prediction in computational social science. Nature, 595(7866):1… find
  50. Eric Winsberg. Science in the age of computer simulation. University of Chicago Press, 2010. find
  51. Jürgen Pfeffer and Momin M. Malik. Simulating the dynamics of socio-economic systems. In Betina Hollstein, Wenzel Matiaske, and Kai-Uwe Schnapp, editors, Networked governance: New research perspectives, pages 143–161. Springer International Publishing, 2017. find
  52. Juan Carrasquilla and Roger G. Melko. Machine learning phases of matter. Nature Physics, 13(5):431–434, May 2017. find
  53. Nature. Reporting standards and availability of data, materials, code and protocols. find
  54. Malcolm Macleod, Andrew M. Collings, Chris Graf, Veronique Kier-mer, David Mellor, Sowmya Swaminathan, Deborah Sweet, and Valda Vinson. The MDAR (Materials Design Analysis Reporting) framework for transparent reporting in the life sciences. Proceedings of the National Academy of Sciences, 118(17):e2… find
  55. Ian Lundberg, Rebecca Johnson, and Brandon M. Stewart. What is your estimand? Defining the target quantity connects statistical evidence to theory. American Sociological Review, 86(3):532–565, June 2021. find
  56. Jake M. Hofman, Amit Sharma, and Duncan J. Watts. Prediction and explanation in social systems. Science, 355(6324):486–488, February 2017. find
  57. Leland Wilkinson. Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54:594–604, 1999. find
  58. Alex Casteel and Nancy Bridier. Describing populations and samples in doctoral student research. International Journal of Doctoral Studies, 16:339–362, 2021. find
  59. Leigh Tooth, Robert Ware, Chris Bain, David M. Purdie, and Annette Dobson. Quality of reporting of observational longitudinal research. American Journal of Epidemiology, 161(3):280–288, February 2005. find
  60. Daniel J. Simons, Yuichi Shoda, and D. Stephen Lindsay. Constraints on generality (COG): A proposed addition to all empirical papers. Perspectives on Psychological Science, 12(6):1123–1128, August 2017. find
  61. Justin Grimmer, Margaret E. Roberts, and Brandon Stewart. Text as data: A new framework for Machine Learning and the Social Sciences. Princeton University Press, 2022. find
  62. Leo Breiman. Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical Science, 16(3):199 – 231, 2001. find
  63. Ian Lundberg, Jennie E. Brand, and Nanum Jeon. Researcher reasoning meets computational capacity: Machine learning for social science. Social Science Research, 108:102807, 2022. find
  64. Anna Neufeld and Daniela Witten. Discussion of breiman's "two cultures": From two cultures to one. Observational Studies, 7(1):171– 174, 2021. find
  65. Galit Shmueli. Comment on breiman's "two cultures" (2002): From two cultures to multicultural. Observational Studies, 7(1):197–201, 2021. find
  66. Michael Baiocchi and Jordan Rodu. Reasoning using data: Two old ways and one new. Observational Studies, 7(1):3–12, 2021. find
  67. Elizabeth L. Ogburn and Ilya Shpitser. Causal modelling: The two cultures. Observational Studies, 7(1):179–183, 2021. find
  68. Mario Molina and Filiz Garip. Machine learning for sociology. Annual Review of Sociology, 45(1):27–45, 2019. find
  69. Justin Grimmer, Margaret Roberts, and Brandon Stewart. Machine learning for social science: An agnostic approach. Annual Review of Political Science, 24, 05 2021. find
  70. Hooman H. Rashidi, Nam Tran, Samer Albahra, and Luke T. Dang. Machine learning in health care and laboratory medicine: General overview of supervised learning and auto-ML. International Journal of Laboratory Hematology, 43(S1):15–22, July 2021. find
  71. Andrew L. Beam and Isaac S. Kohane. Big data and machine learning in health care. JAMA, 319(13):1317, April 2018. find
  72. Liam G. McCoy, Connor T.A. Brenna, Stacy S. Chen, Karina Vold, and Sunit Das. Believing in black boxes: machine learning for healthcare does not need explainability to be evidence-based. Journal of Clinical Epidemiology, 142:252–257, 2022. find
  73. Adi L Tarca, Vincent J Carey, Xue wen Chen, Roberto Romero, and Sorin Drăghici. Machine learning and its applications to biology. PLoS Computational Biology, 3(6):e116, June 2007. find
  74. Victoria Stodden. Reproducing statistical results. Annual Review of Statistics and Its Application, 2(1):1–19, April 2015. find
  75. T. Herndon, M. Ash, and R. Pollin. Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff. Cambridge Journal of Economics, 38(2):257–279, March 2014. find
  76. Rubén Herzog, Pedro A. M. Mediano, Fernando E. Rosas, Robin Carhart-Harris, Yonatan Sanz Perl, Enzo Tagliazucchi, and Rodrigo Cofre. Retraction note: A mechanistic model of the neural entropy increase elicited by psychedelic drugs. Scientific Reports, 12(1):15500, September 2022. find
  77. May R. Berenbaum. Retraction for Shu et al., Signing at the beginning makes ethics salient and decreases dishonest self-reports in comparison to signing at the end. Proceedings of the National Academy of Sciences, 118(38):e2115397118, September 2021. find
  78. Marcel Neunhoeffer and Sebastian Sternberg. How cross-validation can go wrong and what to do about it. Political Analysis, 27(1):101– 106, January 2019. find
  79. Jake M. Hofman, Daniel G. Goldstein, Siddhartha Sen, Forough Poursabzi-Sangdeh, Jennifer Allen, Ling Liang Dong, Brenda Fried, Harpreet Gaur, Adnan Hoq, Emeka Mbazor, Naomi Moreira, Cindy Muso, Etta Rapp, and Roymil Terrero. Expanding the scope of reproducibility research through data analysis repli… find
  80. Gilles Vandewiele, Isabelle Dehaene, György Kovács, Lucas Sterckx, Olivier Janssens, Femke Ongenae, Femke De Backere, Filip De Turck, Kristien Roelens, Johan Decruyenaere, Sofie Van Hoecke, and Thomas Demeester. Overly optimistic prediction results on imbalanced data: A case study of flaws and benef… find
  81. John P. A. Ioannidis, David B. Allison, Catherine A. Ball, Issa Coulibaly, Xiangqin Cui, Aedín C. Culhane, Mario Falchi, Cesare Furlanello, Laurence Game, Giuseppe Jurman, Jon Mangion, Tapan Mehta, Michael Nitzberg, Grier P. Page, Enrico Petretto, and Vera van Noort. Repeatability of published micro… find
  82. Timothy Verstynen and Konrad Paul Kording. Overfitting to ‘predict’ suicidal ideation. Nature Human Behaviour, pages 1–2, April 2023. find
  83. Benjamin Haibe-Kains, George Alexandru Adam, Ahmed Hosny, Farnoosh Khodakarami, Levi Waldron, Bo Wang, Chris McIntosh, Anna Goldenberg, Anshul Kundaje, Casey S. Greene, Tamara Broderick, Michael M. Hoffman, Jeffrey T. Leek, Keegan Korthauer, Wolfgang Huber, Alvis Brazma, Joelle Pineau, Robert Tibshi… find
  84. Victoria Stodden, Jennifer Seiler, and Zhaokun Ma. An empirical analysis of journal policy effectiveness for computational reproducibility. Proceedings of the National Academy of Sciences, 115(11):2584–2589, March 2018. find
  85. Mirko Gabelica, Ružica Bojčić, and Livia Puljak. Many researchers were not compliant with their published data sharing statement: A mixed-methods study. Journal of Clinical Epidemiology, 150:33–41, October 2022. find
  86. Nicole A. Vasilevsky, Jessica Minnier, Melissa A. Haendel, and Robin E. Champieux. Reproducible and reusable research: Are journal data sharing policies meeting the mark? PeerJ, 5:e3208, April 2017. find
  87. Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep Reinforcement Learning That Matters. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), April 2018. Number: 1. find
  88. Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A Metric Learning Reality Check. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, Lecture Notes in Computer Science, pages 681–699, Cham, 2020. Springer International Publishing. find
  89. AAAI reproducibility checklist. https://aaai.org/conference/aaai/aaai-23/reproducibility-checklist/. ↗
  90. NeurIPS 2023 paper guidelines. https://neurips.cc/public/guides/PaperChecklist. ↗
  91. ICML 2023 paper guidelines. https://icml.cc/Conferences/2023/PaperGuidelines. ↗
  92. The Journal of Politics: Guidelines for data replication https://www.journals.uchicago.edu/journals/jop/data-replication. ↗
  93. B. A. Nosek, G. Alter, G. C. Banks, D. Borsboom, S. D. Bowman, S. J. Breckler, S. Buck, C. D. Chambers, G. Chin, G. Christensen, M. Con-testabile, A. Dafoe, E. Eich, J. Freese, R. Glennerster, D. Goroff, D. P. Green, B. Hesse, M. Humphreys, J. Ishiyama, D. Karlan, A. Kraut, A. Lupia, P. Mabry, T. Ma… find
  94. Koren, Miklós, Connolly, Marie, Lull, Joan, and Vilhuber, Lars. Data and code availability standard. December 2022. find
  95. Reviewing computational methods. Nature Methods, 12(12):1099– 1099, December 2015. find
  96. Kenneth Peng, Arunesh Mathur, and Arvind Narayanan. Mitigating dataset harms requires stewardship: Lessons from 1000 papers. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 1, December 2021. find
  97. Data dictionaries, U.S. Geological Survey. https://www.usgs.gov/data-management/data-dictionaries. ↗
  98. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12):86–92, December 2021. find
  99. Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464):447–453, October 2019. find
  100. Beata Nowok, Gillian M. Raab, and Chris Dibben. synthpop : Bespoke creation of synthetic data in R. Journal of Statistical Software, 74(11), 2016. find
  101. Geir Kjetil Sandve, Anton Nekrutenko, James Taylor, and Eivind Hovig. Ten simple rules for reproducible computational research. PLoS Computational Biology, 9(10):e1003285, October 2013. find
  102. Timothy H. Vines, Arianne Y.K. Albert, Rose L. Andrew, Florence Débarre, Dan G. Bock, Michelle T. Franklin, Kimberly J. Gilbert, Jean-Sébastien Moore, Sébastien Renaut, and Diana J. Rennison. The availability of research data declines rapidly with article age. Current Biology, 24(1):94–97, 2014. find
  103. Elizabeth Gibney and Richard Van Noorden. Scientists losing data at a rapid rate. Nature, December 2013. find
  104. Victoria Stodden and Sheila Miguez. Best practices for computational science: Software infrastructure and environments for reproducible and extensible research. 2(1):e21, July 2014. find
  105. Vilhuber, Lars, Connolly, Marie, Koren, Miklós, Llull, Joan, and Morrow, Peter. A template README for social science replication packages. December 2020. find
  106. Harbert. Bash scripting, October 2018. find
  107. April Clyburne-Sherin, Xu Fei, and Seth Ariel Green. Computational reproducibility via containers in psychology. Meta-Psychology, 3, November 2019. find
  108. Constanza L. Andaur Navarro, Johanna A. A. Damen, Toshihiko Takada, Steven W. J. Nijman, Paula Dhiman, Jie Ma, Gary S. Collins, Ram Bajpai, Richard D. Riley, Karel G. M. Moons, and Lotty Hooft. Completeness of reporting of clinical prediction models developed using supervised machine learning: A sys… find
  109. Mohamed Yusuf, Ignacio Atal, Jacques Li, Philip Smith, Philippe Ravaud, Martin Fergie, Michael Callaghan, and James Selfe. Reporting quality of studies using machine learning models for medical diagnosis: A systematic review. BMJ Open, 10(3), 2020. find
  110. Yoonsang Kim, Jidong Huang, and Sherry Emery. Garbage in, garbage out: Data collection, quality assessment and reporting standards for social media data use in health research, infodemiology and digital disease detection. Journal of Medical Internet Research, 18(2):e41, February 2016. find
  111. R. Stuart Geiger, Kevin Yu, Yanlai Yang, Mindy Dai, Jie Qiu, Rebekah Tang, and Jenny Huang. Garbage in, garbage out? In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. ACM, January 2020. find
  112. Franz Porzsolt, Felicitas Wiedemann, Susanne I. Becker, and C. J. Rhoads. Inclusion and exclusion criteria and the problem of describing homogeneity of study populations in clinical trials. BMJ Evidence - Based Medicine, 24(3):92, 06 2019. find
  113. Matthew Salganik. Bit by bit: Social research in the Digital age. Princeton University Press, 2019. find
  114. Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and Machine Learning: Limitations and Opportunities. fairmlbook.org, 2019. http://www.fairmlbook.org. ↗
  115. Abigail Z. Jacobs and Hanna Wallach. Measurement and fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 375–385, New York, NY, USA, 2021. Association for Computing Machinery. find
  116. Marcus Credé and P. D. Harms. Three cheers for descriptive statistics—and five more reasons why they matter. Industrial and Organizational Psychology, 14(4):486–488, 2021. find
  117. Jenifer Larson-Hall and Luke Plonsky. Reporting and interpreting quantitative research findings: What gets reported and recommendations for the field. Language Learning, 65(S1):127–159, May 2015. find
  118. Valerie C. Bradley, Shiro Kuriwaki, Michael Isakov, Dino Sejdinovic, Xiao-Li Meng, and Seth Flaxman. Unrepresentative big surveys significantly overestimated US vaccine uptake. Nature, 600(7890):695– 700, December 2021. find
  119. Luke Plonsky. Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition, 35(4):655–687, 2013. find
  120. Patrick McKnight, Katherine McKnight, Souraya Sidani, and Aurelio José Figueredo. Missing Data : A Gentle Introduction. Guilford Publications, 2007. find
  121. James L. Peugh and Craig K. Enders. Missing data in educational research: A review of reporting practices and suggestions for improvement. Review of Educational Research, 74(4):525–556, 2004. find
  122. Matthew J. Salganik, Ian Lundberg, Alexander T. Kindel, et al. Supplementary information for: Measuring the predictability of life outcomes with a scientific mass collaboration. Proceedings of the National Academy of Sciences, 117(15):8398–8403, March 2020. find
  123. Christina Mack, Zhaohui Su, and Daniel Westreich. Managing missing data in patient registries: Addendum to registries for evaluating patient outcomes: A user’s guide (Third edition). 2018. https://www.ncbi.nlm.nih.gov/books/NBK493611/. ↗
  124. SWJ Nijman, AM Leeuwenberg, I Beekers, I Verkouter, JJL Jacobs, ML Bots, FW Asselbergs, KGM Moons, and TPA Debray. Missing data is poorly handled and reported in prediction model studies using machine learning: A literature review. Journal of Clinical Epidemiology, 142:218–229, February 2022. find
  125. Todd D. Little, Terrence D. Jorgensen, Kyle M. Lang, and E. Whitney G. Moore. On the joys of missing data. Journal of Pediatric Psychology, 39(2):151–162, July 2013. find
  126. Jody S. Nicholson, Pascal R. Deboeck, and Waylon Howard. Attrition in developmental psychology. International Journal of Behavioral Development, 41(1):143–153, July 2016. find
  127. William R. Sterner. What is missing in counseling research? reporting missing data. Journal of Counseling and Development, 89(1):56– 62, January 2011. find
  128. Jamilla A. Hussain, Martin Bland, Dean Langan, Miriam J. Johnson, David C. Currow, and Ian R. White. Quality of missing data reporting and handling in palliative care trials demonstrates that further development of the CONSORT statement is required: a systematic review. Journal of Clinical Epidemiol… find
  129. Jake M Hofman, Amit Sharma, and Duncan J Watts. Prediction and explanation in social systems. Science, 355(6324):486–488, 2017. find
  130. Xu Chu, Ihab F Ilyas, Sanjay Krishnan, and Jiannan Wang. Data cleaning: Overview and emerging challenges. In Proceedings of the 2016 international conference on management of data, pages 2201– 2206, 2016. find
  131. Erin M Buchanan and John E Scofield. Methods to detect low quality data and its implication for psychological research. Behavior research methods, 50:2586–2596, 2018. find
  132. Gilles Vandewiele, Isabelle Dehaene, György Kovács, Lucas Sterckx, Olivier Janssens, Femke Ongenae, Femke De Backere, Filip De Turck, Kristien Roelens, Johan Decruyenaere, et al. Overly optimistic prediction results on imbalanced data: A case study of flaws and benefits when applying over-sampling.… find
  133. Tolou Shadbahr, Michael Roberts, Jan Stanczuk, Julian Gilbey, Philip Teare, Sören Dittmer, Matthew Thorpe, Ramon Vinas Torne, Evis Sala, Pietro Lio, et al. Classification of datasets with imputed missing values: Does imputation quality matter? arXiv preprint arXiv:2206.08478, 2022. — [arXiv:2206.08478](https://arxiv.org/abs/2206.08478) find
  134. Emilia Gryska, Isabella Björkman-Burtscher, Asgeir Store Jakola, Tora Dunås, Justin Schneiderman, and Rolf A Heckemann. Deep learning for automatic brain tumour segmentation on mri: Evaluation of recommended reporting criteria via a reproduction and replication study. BMJ open, 12(7):e059000, 2022. find
  135. Edward Raff. A step toward quantifying independently reproducible machine learning research. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. find
  136. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pages 220–229, 2019. find
  137. Jon Kleinberg, Annie Liang, and Sendhil Mullainathan. The theory is predictive, but is it complete? An application to human perception of randomness. In Proceedings of the 2017 ACM Conference on Economics and Computation, pages 125–126, 2017. find
  138. Ribana Roscher, Bastian Bohn, Marco F Duarte, and Jochen Garcke. Explainable machine learning for scientific insights and discoveries. Ieee Access, 8:42200–42216, 2020. find
  139. Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206–215, 2019.
  140. Sebastian Raschka. Model evaluation, model selection, and algorithm selection in machine learning, November 2020. arXiv:1811.12808 [cs, stat]. — [arXiv:1811.12808](https://arxiv.org/abs/1811.12808) find
  141. Gavin C. Cawley and Nicola L. C. Talbot. On over-fitting in model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research, 11(70):2079–2107, 2010. find
  142. Charles Marx, Flavio Calmon, and Berk Ustun. Predictive Multiplicity in Classification. In Proceedings of the 37th International Conference on Machine Learning, pages 6765–6774. PMLR, November 2020. ISSN: 2640-3498. find
  143. Jamelle Watson-Daniels, David C. Parkes, and Berk Ustun. Predictive Multiplicity in Probabilistic Classification \| Proceedings of the AAAI Conference on Artificial Intelligence. 2023. find
  144. Emily Black, Manish Raghavan, and Solon Barocas. Model multiplicity: Opportunities, concerns, and solutions. In 2022 ACM Conference on Fairness, Accountability, and Transparency. ACM, June 2022. find
  145. Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A Smith. Show your work: Improved reporting of experimental results. arXiv preprint arXiv:1909.03004, 2019. find
  146. Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. arXiv preprint arXiv:1708.04133, 2017. — [arXiv:1708.04133](https://arxiv.org/abs/1708.04133) find
  147. A. Feder Cooper, Yucheng Lu, Jessica Forde, and Christopher M De Sa. Hyperparameter Optimization Is Deceiving Us, and How to Stop It. In Advances in Neural Information Processing Systems, volume 34, pages 3081–3095. Curran Associates, Inc., 2021. find
  148. Prabhu Teja Sivaprasad, Florian Mai, Thijs Vogels, Martin Jaggi, and François Fleuret. Optimizer Benchmarking Needs to Account for Hyperparameter Tuning. In Proceedings of the 37th International Conference on Machine Learning, pages 9036–9045. PMLR, November 2020. ISSN: 2640-3498. find
  149. George E. Dahl, Frank Schneider, Zachary Nado, Naman Agarwal, Chandramouli Shama Sastry, Philipp Hennig, Sourabh Medapati, Runa Eschenhagen, Priya Kasimbeg, Daniel Suo, Juhan Bae, Justin Gilmer, Abel L. Peirson, Bilal Khan, Rohan Anil, Mike Rabbat, Shankar Krishnan, Daniel Snider, Ehsan Amid, Kongta… — [arXiv:2306.07179](https://arxiv.org/abs/2306.07179) find
  150. Philipp Probst, Anne-Laure Boulesteix, and Bernd Bischl. Tunability: Importance of hyperparameters of machine learning algorithms. The Journal of Machine Learning Research, 20(1):1934–1965, 2019. find
  151. Jimmy Lin. The neural hype and comparisons against weak baselines. ACM SIGIR Forum, 52(2):40–51, January 2019. find
  152. Michael A. Lones. How to avoid machine learning pitfalls: A guide for academic researchers, February 2023. arXiv:2108.02497 [cs]. — [arXiv:2108.02497](https://arxiv.org/abs/2108.02497) find
  153. Shachar Kaufman, Saharon Rosset, Claudia Perlich, and Ori Stitel-man. Leakage in data mining: Formulation, detection, and avoidance. ACM Transactions on Knowledge Discovery from Data, 6(4):15:1– 15:21, December 2012. find
  154. Casey Ross. Epic’s sepsis algorithm is going off the rails in the real world. The use of these variables may explain why, September 2021. find
  155. Max Kuhn and Kjell Johnson. Applied Predictive Modeling. Springer-Verlag, New York, 2013. find
  156. Russell A. Poldrack, Grace Huckins, and Gael Varoquaux. Establishment of best practices for evidence for prediction: A review. JAMA psychiatry, 77(5):534–540, May 2020. find
  157. Mustafa Umit Oner, Yi-Chih Cheng, Hwee Kuan Lee, and Wing-Kin Sung. Training machine learning models on patient level data segregation is crucial in practical clinical applications, April 2020. find
  158. Momin M. Malik. A hierarchy of limitations in machine learning, February 2020. arXiv:2002.05193 [cs, econ, math, stat]. — [arXiv:2002.05193](https://arxiv.org/abs/2002.05193) find
  159. Christoph Bergmeir and José M. Benítez. On the use of cross-validation for time series predictor evaluation. Information Sciences, 191:192–213, May 2012. find
  160. Nils Y. Hammerla and Thomas Plötz. Let’s (not) stick together: Pairwise similarity biases cross-validation in activity recognition. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pages 1041–1051, Osaka Japan, September 2015. ACM. find
  161. David R. Roberts, Volker Bahn, Simone Ciuti, Mark S. Boyce, Jane Elith, Gurutzeta Guillera-Arroita, Severin Hauenstein, José J. Lahoz-Monfort, Boris Schröder, Wilfried Thuiller, David I. Warton, Brendan A. Wintle, Florian Hartig, and Carsten F. Dormann. Cross-validation strategies for data with temp… find
  162. Alexandre Chiavegatto Filho, André Filipe De Moraes Batista, and Hellen Geremias Dos Santos. Data leakage in health outcomes prediction with machine learning. Comment on “Prediction of incident hypertension within the next year: Prospective study using statewide electronic health records and machine… find
  163. Chengyin Ye, Tianyun Fu, Shiying Hao, Yan Zhang, Oliver Wang, Bo Jin, Minjie Xia, Modi Liu, Xin Zhou, Qian Wu, Yanting Guo, Chunqing Zhu, Yu-Ming Li, Devore S Culver, Shaun T Alfreds, Frank Stearns, Karl G Sylvester, Eric Widen, Doff McElhinney, and Xuefeng Ling. Prediction of incident hypertension… find
  164. Jessica Zosa Forde, A. Feder Cooper, Kweku Kwegyir-Aggrey, Chris De Sa, and Michael Littman. Model Selection’s Disparate Impact in Real-World Deep Learning Applications, September 2021. arXiv:2104.00606 [cs]. — [arXiv:2104.00606](https://arxiv.org/abs/2104.00606) find
  165. Umang Bhatt, Javier Antorán, Yunfeng Zhang, Q. Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Melançon, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, Lama Nachman, Rumi Chunara, Madhulika Srikumar, Adrian Weller, and Alice Xiang. Uncertainty as a Form of Transparency: Measuring, Comm… find
  166. A. Feder Cooper, Katherine Lee, Madiha Zahrah Choksi, Solon Barocas, Christopher De Sa, James Grimmelmann, Jon Kleinberg, Siddhartha Sen, and Baobao Zhang. Is My Prediction Arbitrary? The Confounding Effects of Variance in Fair Classification Benchmarks, August 2023. arXiv:2301.11562 [cs, stat]. — [arXiv:2301.11562](https://arxiv.org/abs/2301.11562) find
  167. Shangshu Qian, Viet Hung Pham, Thibaud Lutellier, Zeou Hu, Jungwon Kim, Lin Tan, Yaoliang Yu, Jiahao Chen, and Sameena Shah. Are My Deep Learning Systems Fair? An Empirical Study of Fixed-Seed Training. In Advances in Neural Information Processing Systems, volume 34, pages 30211–30227. Curran Associ… find
  168. Cristobal Young. Model Uncertainty and the Crisis in Science. Socius, 4:2378023117737206, January 2018. Publisher: SAGE Publications. find
  169. Maria Carolina Monard and GEAPA Batista. Learning with skewed class distributions. Advances in Logic, Artificial Intelligence and Robotics, 85:173–180, 2002. find
  170. Jorge M Lobo, Alberto Jiménez-Valverde, and Raimundo Real. Auc: A misleading measure of the performance of predictive distribution models. Global ecology and Biogeography, 17(2):145–151, 2008. find
  171. Alexy Bhowmick and Shyamanta M Hazarika. E-mail spam filtering: A review of techniques and trends. Advances in Electronics, Communication and Computing: ETAEERE-2016, pages 583–590, 2018. find
  172. Emily G Simmonds, Kwaku Peprah Adjei, Christoffer Wold Andersen, Janne Cathrin Hetle Aspheim, Claudia Battistin, Nicola Bulso, Hannah Christensen, Benjamin Cretois, Ryan Cubero, Ivan A Davidovich, et al. How is model-related uncertainty quantified and reported in different disciplines? arXiv preprin… — [arXiv:2206.12179](https://arxiv.org/abs/2206.12179) find
  173. Valentin Amrhein, Sander Greenland, and Blake McShane. Scientists rise up against statistical significance. Nature, 567(7748):305–307, 2019. find
  174. William R. Shadish, Thomas D. Cook, and Donald T. Campbell. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin, 2 edition, 2001. find
  175. Naoki Egami and Erin Hartman. Elements of external validity: Framework, design, and analysis. American Political Science Review, 117(3):1070–1088, October 2022. find
  176. Inioluwa Deborah Raji, I. Elizabeth Kumar, Aaron Horowitz, and Andrew Selbst. The fallacy of AI functionality. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 959–972, Seoul Republic of Korea, June 2022. ACM. find
  177. Thomas Liao, Rohan Taori, Deborah Raji, and Ludwig Schmidt. Are we learning yet? A meta review of evaluation failures across machine learning. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 1, December 2021. find
  178. Arunesh Mathur, Angelina Wang, Carsten Schwemmer, Maia Hamin, Brandon M Stewart, and Arvind Narayanan. Manipulative tactics are the norm in political emails: Evidence from 300k emails from the 2020 US election cycle. Big Data & Society, 10(1):205395172211453, January 2023. find
  179. Jason Brownlee. Difference between algorithm and model in machine learning, April 2020. https://machinelearningmastery.com/difference-between-algorithm-and-model-in-machine-learning/. ↗
  180. Andrea Freyer Dugas, Mehdi Jalalpour, Yulia Gel, Scott Levin, Fred Torcaso, Takeru Igusa, and Richard E. Rothman. Influenza forecasting with Google Flu trends. PLoS ONE, 8(2):e56176, February 2013. find
  181. Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre Alvise-Rebuffi, Ira Ktena, Krishnamurthy Dvijotham, and Taylan Cemgil. A fine-grained analysis on distribution shift, 2021. https://arxiv.org/abs/2110.11328. — [arXiv:2110.11328](https://arxiv.org/abs/2110.11328) ↗
  182. Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pie… find
  183. Kirk Bansak, Jeremy Ferwerda, Jens Hainmueller, Andrea Dillon, Dominik Hangartner, Duncan Lawrence, and Jeremy Weinstein. Improving refugee integration through data-driven algorithmic assignment. Science, 359(6373):325–329, January 2018. find
  184. Selen Bozkurt, Eli M Cahan, Martin G Seneviratne, Ran Sun, Juan A Lossio-Ventura, John P A Ioannidis, and Tina Hernandez-Boussard. Reporting of demographic data and representativeness in machine learning models using electronic health records. Journal of the American Medical Informatics Association,… find
  185. Constanza L. Andaur Navarro, Johanna A.A. Damen, Toshihiko Takada, Steven W.J. Nijman, Paula Dhiman, Jie Ma, Gary S. Collins, Ram Bajpai, Richard D. Riley, Karel G.M. Moons, and Lotty Hooft. Systematic review finds “spin” practices and poor reporting standards in studies on machine learning-based pr… find
  186. Samuel G. Finlayson, Adarsh Subbaswamy, Karandeep Singh, John Bowers, Annabel Kupke, Jonathan Zittrain, Isaac S. Kohane, and Suchi Saria. The clinician and dataset shift in artificial intelligence. New England Journal of Medicine, 385(3):283–286, July 2021. find
  187. Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, November 2020. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kapoor-2023-reforms-reporting-standards-machine/