Elamur Research⌕
☰
home / papers / 2025 / ChatBench: From Static Benchmarks to Human-AI Evaluation

ChatBench: From Static Benchmarks to Human-AI Evaluation

Paper · 2025 · Serina Chang, Ashton Anderson and Jake M. Hofman

Summary

Benchmarks such as MMLU test LLMs in isolation, but people use chatbots through open-ended dialogue. This paper asks whether "AI-alone" scores reflect how well a person does when working with a model. The authors turned MMLU questions into conversations. Prolific workers received a question, answered it on their own, and then chatted freely with an unnamed chatbot (GPT-4o or Llama-3.1-8b) to reach an answer. Two interaction conditions were compared: answer-first and direct-to-AI. The resulting resource, ChatBench, pairs AI-alone, user-alone and user-AI results for identical questions. The main finding is that AI-alone accuracy is a poor predictor of user-AI accuracy, with significant gaps in math, physics and moral reasoning. Differences between models can also shrink sharply once users are involved. The authors also fine-tuned an LLM-based user simulator on part of the data, which made its accuracy estimates much closer to real users. This matters for choosing which models to deploy and for scaling interactive evaluation more cheaply.

Key points

  • ChatBench covers 396 MMLU questions and two LLMs (GPT-4o, Llama-3.1-8b), with about 144K answers and 7,336 user-AI conversations. It is released on Hugging Face as microsoft/ChatBench.
  • On free-text AI-alone evaluation, GPT-4o leads Llama-3.1-8b by about 25 percentage points. In the answer-first user-AI condition, this gap narrows to roughly 5 points.
  • Fine-tuning the user simulator on ChatBench raised its correlation with real user-AI accuracy by 22–26 points on held-out questions, though the best correlation is still only 0.63. Its generated utterances also gained 11–16 points in BLEU and ROUGE, and it beat the AI-alone baselines and IQA-EVAL.
  • In the study setup, each participant answered 13 questions in sequence and earned a $0.10 bonus per correct answer. The models ran at temperature 0.7, and prompting was left unguided apart from requiring at least one message.
  • Stated limitations include coverage of only MMLU and only QA tasks. Ecological validity is also limited, because users were assigned their questions rather than bringing their own.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations🎓University of California, BerkeleyBerkeley · United States175 works · 227,440 citations🎓University of TorontoToronto · Canada96 works · 142,007 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Serina Chang, Ashton Anderson and Jake M. Hofman. 2025. ChatBench: From Static Benchmarks to Human-AI Evaluation. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1262

@inproceedings{chang2025chatbench,
  title = {{ChatBench: From Static Benchmarks to Human-AI Evaluation}},
  author = {Serina Chang, Ashton Anderson and Jake M. Hofman},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1262},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1262}
}

References

44 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. Out of one, many: Using language models to simulate human samples. Political Analysis, 31:337– 351.
  2. Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In CHI ’21: Proceedings of the 2021 CHI Conference on Human Factors in Computi… find
  3. Alexander Bick, Adam Blandin, and David Deming. 2024. The rapid adoption of generative AI. Federal Reserve Bank of St. Louis. https://www.stlouisfed.org/on-the-economy/2024/ sep/rapid-adoption-generative-ai. ↗
  4. James Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M. Larson. 2024. Synthetic replacements for human survey data? the perils of large language models. Political Analysis, 32:401– 416. find
  5. Serina Chang, Alicja Chaszczewicz, Emma Wang, Maya Josifovska, Emma Pierson, and Jure Leskovec. 2024. LLMs generate structurally realistic social networks but overestimate political homophily. arXiv preprint arXiv:2408.16629. find
  6. Myra Cheng, Esin Durmus, and Dan Jurafsky. 2023a. Marked personas: Using natural language prompts to measure stereotypes in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL’23).
  7. Myra Cheng, Tiziano Piccardi, and Diyi Yang. 2023b. Compost: Characterizing and evaluating caricature in LLM simulations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP’23). find
  8. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot Arena: An open platform for evaluating LLMs by human preference. arXiv preprint arXiv:2403.04132.
  9. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  10. Katherine M. Collins, Albert Q. Jiang, Simon Frieder, Lionel Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Joshua B. Tenenbaum, William Hart, Timothy Gowers, Wenda Li, Adrian Weller, and Mateja Jamnik. 2024. Evaluating language models for mathematics through interactions. Proceedings… find
  11. Paramveer S. Dhillon, Somayeh Molaei, Jiaqi Li, Maximilian Golub, Shaochun Zheng, and Lionel Peter Robert. 2024. Shaping human-ai collaboration: Varied scaffolding levels in co-writing with language models. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI’24). find
  12. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacafarm: A simulation framework for methods that learn from human feedback. In Proceedings of the 37th International Conference on Neural Information Pro…
  13. Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, and Pasquale Minervini. 2024. Are we done wit… find
  14. Gemini Team Google. 2023. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  15. Amelia Hardy, Anka Reuel, Kiana Jafari Meimandi, Lisa Soder, Allie Griffith, Dylan M. Asmar, Sanmi Koyejo, Michael S. Bernstein, and Mykel J. Kochenderfer. 2024. More than marketing? on the information value of ai benchmarks for practitioners. arXiv preprint arXiv:2412.05520. find
  16. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In The Ninth International Conference on Learning Representations (ICLR’21).
  17. Luke Hewitt, Ashwini Ashokkumar, Isaias Ghezae1, and Robb Willer. 2024. Predicting results of social science experiments using large language models. Working Paper. find
  18. John J. Horton. 2023. Large language models as simulated economic agents: What can we learn from homo silicus? arXiv preprint arXiv:2301.07543. — [arXiv:2301.07543](https://arxiv.org/abs/2301.07543) find
  19. EunJeong Hwang, Bodhisattwa Majumder, and Niket Tandon. 2023. Aligning language models to user opinions. In Findings of the Association for Computational Linguistics: EMNLP 2023.
  20. Panos Ipeirotis. 2010. Demographics of mechanical turk. CeDER Working Papers. find
  21. Irina Jurenka, Markus Kunesch amd Kevin R. McKee, Daniel Gillick, Shaojian Zhu, Sara Wiltberger, Shubham Milind Phal, Katherine Hermann, Daniel Kasenberg, Avishkar Bhoopchand, et al. 2024. Towards responsible development of generative ai for education: An evaluation-driven approach. arXiv preprint a… find
  22. Chuyi Kong, Yaxin Fan, Xiang Wan, Feng Jiang, and Benyou Wang. 2024. PlatoLM: Teaching LLMs in multi-round dialogue via a user simulator. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL’24). find
  23. Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard-Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, Rose E. Wang, Minae Kwon, Joon Sung Park, Hancheng Cao, Tony Lee, Rishi Bommasani, Michael Bernstein, and Percy Liang. 2023. Evaluating human-languag… find
  24. Ruosen Li, Ruochen Li, Barry Wang, and Xinya Du. 2024a. IQA-EVAL: Automatic evaluation of human-model interactive question answering. In Proceedings of the 38th International Conference on Neural Information Processing Systems (NeurIPS’24). find
  25. Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan S. Ilgen, Emma Pierson, Pang Wei Koh, and Yulia Tsvetkov. 2024b. Mediq: Question-asking LLMs and a benchmark for reliable interactive clinical reasoning. In Proceedings of the 38th International Conference on Neural Information Process…
  26. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2024c. From live data to high-quality benchmarks: The arena-hard pipeline. https://lmsys.org/blog/2024-04-19-arena-hard/. ↗
  27. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2023. Holistic evaluation of language models. Transactions of Machine Learning Research. find
  28. Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. 2024. WildBench: Benchmarking LLMs with challenging tasks from real users in the wild. In arXiv preprint arXiv:2406.04770.
  29. Llama Team, AI@Meta. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  30. Evan Miller. 2024. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint arXiv:2411.00640. find
  31. OpenAI. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774.
  32. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST’23). find
  33. Lisa Posch, Arnim Bleier, Fabian Flöck, Clemens M. Lechner, Katharina Kinder-Kurlanda, Denis Helic, and Markus Strohmaier. 2022. Characterizing the global crowd workforce:a cross-country comparison of crowdworker demographics. Human Computation, 9(1):22–57. find
  34. Ruiyang Ren, Peng Qiu, Yingqi Qu, Jing Liu, Xin Zhao, Hua Wu, Ji-Rong Wen, and Haifeng Wang. 2024. BASES: Large-scale web search user simulation with large language model based agents. In Findings of the Association for Computational Linguistics: EMNLP 2024. find
  35. Yijia Shao, Vinay Samuel, Yucheng Jiang, John Yang, and Diyi Yang. 2024. Collaborative gym: A framework for enabling and evaluating human-agent collaboration. arXiv preprint arXiv:2412.15701. find
  36. Joseph Suh, Erfan Jahanparast, Suhong Moon, Minwoo Kang, and Serina Chang. 2025. Language model fine-tuning on scaled survey data for predicting distributions of public opinions. arXiv preprint arXiv:2502.16761.
  37. U.S. Bureau of Labor Statistics. 2025. Labor Force Statistics from the Current Population Survey. https://www.bls.gov/web/empsit/ cpseea06.htm. Accessed March 28, 2025. ↗
  38. U.S. Census Bureau. 2024. National Population by Characteristics: 2020-2024. https://www.census.gov/data/tables/time-series/ demo/popest/2020s-national-detail.html. Accessed March 28, 2025. ↗
  39. Michelle Vaccaro, Abdullah Almaatouq, and Thomas Malone. 2024. When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour, 8:2293–2303. find
  40. Angelina Wang, Jamie Morgenstern, and John P. Dickerson. 2025. Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence. find
  41. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP’1…
  42. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. WildChat: 1m chatgpt interaction logs in the wild. In Proceedings of the 12th International Conference on Learning Representations (ICLR’24).
  43. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. 2024. LMSYS-Chat-1M: A large-scale real-world LLM conversation dataset. In Proceedings of the 12th International Conferen…
  44. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chang-2025-chatbench-static-benchmarks-human-ai/