Elamur Research
☰
home / papers / 2024 / Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs

Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2024.eacl-long.5.pdf

Summary

The authors study how test data ends up in closed LLMs (data contamination). They separately examine a phenomenon they call indirect data leakage. When researchers evaluate GPT-3.5 and GPT-4 through ChatGPT, benchmark examples may become training material for subsequent versions of the model. Existing contamination detection methods check whether the model reproduces examples verbatim. As a result, they detect only extreme memorization and miss ordinary fine-tuning on leaked data. The team from Charles University systematically reviewed 255 papers published during the first year after the release of ChatGPT. For each paper, the authors estimated how many examples could have reached the models in such a way that OpenAI, under its own policy, was entitled to use them for training. Leakage was found in approximately 42% of the papers: in total, about 4.7 million examples from 263 benchmarks. The authors also describe typical flaws in evaluation: unfair or missing comparisons with baselines, differing test set sizes, and non-reproducible results. This work shows that comparisons of closed models with other models cannot be trusted without caveats. Finally, the authors propose guidelines for more rigorous evaluation.

Key points

  • The review covered 255 papers that evaluated GPT-3.5 and GPT-4. Leakage was found in approximately 42% of them: about 4.7 million examples from 263 benchmarks.
  • Access through the ChatGPT web interface is considered risky, because this data may be used for fine-tuning. Data submitted via the API, under OpenAI's policy in effect since March 1, 2023, is not used for training. It was possible to opt out of data sharing (opt-out), but none of the reviewed papers did so.
  • Flawed practices include the following: open models are tested on the entire benchmark, while closed models are tested only on a small subset. Authors do not publish their code and data, and they take SOTA results from other papers without rerunning them.
  • The authors' recommendations: whenever possible, choose open LLMs, and access closed ones through the API. Use the same subset for all models being compared. Publish the prompts, model version, experiment period, and the data itself. Report any detected leakage.
  • Resource: a shared list of datasets with estimates of the extent of leakage at leak-llm.github.io, to which other researchers can add new cases.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

117 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 116 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.eacl-long.5 · arXiv 2402.03927

@inproceedings{balloccu2024leak,
  title = {{Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs}},
  author = {Balloccu, Simone and Schmidtová, Patrícia and Lango, Mateusz and Dusek, Ondrej},
  booktitle = {Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.eacl-long.5},
  eprint = {2402.03927},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.eacl-long.5}
}

References

27 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. We are aware the list of contaminated datasets we compiled in our work is not fully conclusive for one of several reasons: Rachith Aiyappa, Jisun An, Haewoon Kwak, and Yong-yeol Ahn. 2023. Can we trust the evaluation on Chat-GPT? In Proceedings of the 3rd Workshop on Trustworthy Natural Language Pro… find
  2. (1) We review the information that has been publicly revealed via articles. We postulate more Anya Belz, Craig Thomson, Ehud Reiter, and Simon Mille. 2023. Non-repeatable experiments and non-reproducible results: The reproducibility crisis in \#\# Limitations human evaluation in NLP. In Findings of th… find
  3. Shahriar Golchin and Mihai Surdeanu. 2023. Time travel in LLMs: Tracing data contamination in large language models. arXiv preprint arXiv:2308.08493. find our card
  4. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. A… find
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  6. Daphne Ippolito, Florian Tramer, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher Choquette Choo, and Nicholas Carlini. 2023. Preventing generation of verbatim memorization in language models gives a false sense of privacy. In Proceedings of the 16th International Natural Lan… find
  7. Zhenguang G Cai, David A Haslett, Xufeng Duan, Shuqi Wang, and Martin J Pickering. 2023. Does ChatGPT resemble humans in language use? arXiv preprint arXiv:2303.08014. find
  8. Khalid S Khan, Regina Kunz, Jos Kleijnen, and Gerd Antes. 2003. Five steps to conducting a systematic review. Journal of the Royal Society of Medicine, 96(3):118–121. find
  9. Lingjiao Chen, Matei Zaharia, and James Zou. 2023. How is ChatGPT’s behavior changing over time? arXiv preprint arXiv:2307.09009. — [arXiv:2307.09009](https://arxiv.org/abs/2307.09009) find
  10. Jan Kocoń, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, Anna Kocoń, Bartłomiej Koptyra, Wiktoria Mieleszczenko-Kowszewicz, Piotr Miłkowski, Marcin Oleksy, Maciej Piasecki, Łukasz Radliński, Konr… find
  11. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… find
  12. TH Kung, M Cheatham, A Medenilla, C Sillos, L De Leon, C Elepaño, M Madriaga, R Aggabao, G Diaz-Candido, J Maningo, et al. 2023. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. Plos Digit Health, 2:000198. find
  13. Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning. find our card
  14. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p… find our card
  15. GitHub. 2022. About github copilot. https://github. com/features/copilot. find
  16. Inbal Magar and Roy Schwartz. 2022. Data contamination: From memorization to exploitation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 157–165, Dublin, Ireland. Association for Computational Linguistics. find
  17. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79, Hong Kong, China. Association for Computational Linguistics. find
  18. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. find
  19. SemEval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31– 41, San Diego, California. Association for Computational Linguistics. find
  20. Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022. LaMD… find our card
  21. OpenAI. 2023. GPT-4 technical report. find our card
  22. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Tra… find
  23. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. find
  24. Fanghua Ye, Jarana Manotumruksa, and Emine Yilmaz. 2022. MultiWOZ 2.4: A multi-domain task-oriented dialogue dataset with essential annotation corrections to improve state tracking evaluation. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 35… find
  25. Aleksandra Piktus, Christopher Akiki, Paulo Villegas, Hugo Laurençon, Gérard Dupont, Sasha Luccioni, Yacine Jernite, and Anna Rogers. 2023. The ROOTS search tool: Data transparency for LLMs. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System… find
  26. Bowen Zhang, Daijun Ding, and Liwen Jing. 2022. How would stance detection techniques evolve after the launch of ChatGPT? arXiv preprint arXiv:2212.14548. find
  27. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1). find