Elamur Research⌕
☰
home / papers / 2023 / Can we trust the evaluation on ChatGPT?

Can we trust the evaluation on ChatGPT?

Paper · 2023 · Rachith Aiyappa, Jisun An, Haewoon Kwak et al.

Summary

The paper examines whether published evaluations of ChatGPT can be trusted. The model is closed, its training data are undisclosed, and it is repeatedly updated through RLHF using user queries. Benchmark test sets that researchers submit could therefore leak into later versions and inflate scores.

As a case study, the authors take zero-shot stance detection on tweets. They replicate the prompt of Zhang et al. (2022) on the SemEval 2016 Task 6 and P-Stance test sets and compare several ChatGPT releases. Those releases are the early version presumably used by Zhang et al. (V1), the Jan 30 version and the Feb 13 ChatGPT plus version. Stance labels are extracted by hand from the model's often wordy answers.

Overall, scores rose after the earlier evaluation study appeared, which fits with possible contamination. However, ordinary model improvement explains the trend equally well. The authors stress that, without access to the model, leakage can be neither confirmed nor excluded. This matters for anyone using closed LLMs as research instruments or as benchmark baselines.

Key points

  • Data: 860 SemEval 2016 Task 6 test tweets for the targets Hillary Clinton, Feminist Movement and Legalization of Abortion, plus 2,157 P-Stance test tweets about Trump, Biden and Bernie Sanders.
  • P-Stance results (Micro-F1/Macro-F1, V1 → Feb 13 ChatGPT plus):
    • Trump: 82.8/83.2 → 83.76/83.09
    • Biden: 82.3/82.0 → 83.07/82.69
    • Bernie: 79.4/79.4 → 79.7/79.6
  • SemEval drop and possible causes: On SemEval, Feb 13 performed below Jan 30. The authors offer several possible explanations:
    • shared chat context in the manual Jan 30 sessions;
    • catastrophic forgetting;
    • a more "diplomatic", restricted model.
  • Practical constraints: No access to retired versions, a cap of roughly 40 queries per hour and no public API at the time. These also meant uncertainty across repeated queries could not be estimated.
  • Recommendations: Model creators should document possible contamination, allow leakage audits and block contamination coming from user inputs. The BIG-bench "canary" string is cited as one existing safeguard.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indiana University BloomingtonBloomington · United States37 works · 3,685 citations🎓Indiana UniversityBloomington · United States41 works · 3,663 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Rachith Aiyappa, Jisun An, Haewoon Kwak and Yong-yeol Ahn. 2023. Can we trust the evaluation on ChatGPT?. Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.trustnlp-1.5

@inproceedings{aiyappa2023can,
  title = {{Can we trust the evaluation on ChatGPT?}},
  author = {Rachith Aiyappa, Jisun An, Haewoon Kwak and Yong-yeol Ahn},
  booktitle = {Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023)},
  year = {2023},
  doi = {10.18653/v1/2023.trustnlp-1.5},
  url = {https://doi.org/10.18653/v1/2023.trustnlp-1.5}
}

References

32 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2023. Artificial hallucinations in chatgpt: Implications in scientific writing. Cureus, 15(2). find
  2. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623. find
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda sometimes factors like network errors which made Chat-GPT unresponsive forced us to open a new chat session in the same sitting. But for a major chunk, a… find
  4. Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  5. Dan Milmo. 2023b. Reinventing search with a new ai-powered microsoft bing and edge, your copilot for the web. The Official Microsoft Blog. https://archive.is/mCmFq. ↗
  6. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In USENIX Security Symposium, volume 6.
  7. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? Preprint arXiv:2202.12837.
  8. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. Preprint arXiv:2104.08758. find
  9. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016), pages 31–41.
  10. Michael Dowling and Brian Lucey. 2023. Chatgpt for (finance) research: The bananarama conjecture. Finance Research Letters, 53. find
  11. Saif M Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2017. Stance and sentiment in tweets. ACM Transactions on Internet Technology (TOIT), 17(3):1–23. find
  12. Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. 2023. Mathematical capabilities of chatgpt. Preprint arXiv:2301.13867.
  13. OpenAI. 2022. Chatgpt: Optimizing language models for dialogue. https://archive.is/4snnY. ↗
  14. OpenAI. 2023. Gpt-4 technical report. arXiv:2303.08774.
  15. Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. 2022. What can transformers learn in-context? a case study of simple function classes. Preprint arXiv:2208.01066. find
  16. Preprint Sajan B Patel and Kyle Lam. 2023. Chatgpt: the future of discharge summaries? The Lancet Digital Health, 5(3):e107–e108. find
  17. Jan Kocoń, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, et al. 2023. Chatgpt: Jack of all trades, master of none. Preprint arXiv:2302.10724.
  18. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  19. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. Preprint arXiv:1707.06347. find
  20. Dilek Küçük and Fazli Can. 2020. Stance detection: A survey. ACM Computing Surveys (CSUR), 53(1):1– 37. find
  21. Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. 2023. Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models. PLOS Digital H…
  22. Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. 2023. An analysis of the automatic bug fixing performance of chatgpt. Preprint arXiv:2301.08653. — [arXiv:2301.08653](https://arxiv.org/abs/2301.08653) find
  23. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Preprint arXiv:2206.04615.
  24. Yingjie Li, Tiberiu Sosea, Aditya Sawant, Ajith Jayaraman Nair, Diana Inkpen, and Cornelia Caragea. 2021. P-stance: A large dataset for stance detection in political domain. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2355–2365.
  25. Chris Stokel-Walker and Richard Van Noorden. 2023. What chatgpt and generative ai mean for science. Nature, 614(7947):214–216. find
  26. Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier. find
  27. Viriya Taecharungroj. 2023. What can chatgpt do? analyzing early reactions to the innovative ai chatbot on twitter. Big Data and Cognitive Computing, 7(1):35. find
  28. Christian Terwiesch. 2023. Would chatgpt3 get a wharton mba. a prediction based on its performance in the operations management course. https://archive.is/HsSUf. ↗
  29. Dan Milmo. 2023a. Chatgpt reaches 100 million users two months after launch. The Guardian. https://archive.is/u3HLx. ↗
  30. Eva AM van Dis, Johan Bollen, Willem Zuidema, Robert van Rooij, and Claudi L Bockting. 2023. Chatgpt: five priorities for research. Nature, 614(7947):224–226. find
  31. Patrick Y Wu, Joshua A Tucker, Jonathan Nagler, and Solomon Messing. 2023. Large language models can be used to estimate the ideologies of politicians in a zero-shot learning setting. Preprint arXiv:2303.12057. — [arXiv:2303.12057](https://arxiv.org/abs/2303.12057) find
  32. Bowen Zhang, Daijun Ding, and Liwen Jing. 2022. How would stance detection techniques evolve after the launch of chatgpt? Preprint arXiv:2212.14548. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aiyappa-2023-can-we-trust-evaluation-chatgpt/