AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2024.findings-naacl.149.pdf
Summary
Most widely used NLP benchmarks are built from artificially constructed datasets. They therefore say little about how foundation models handle the problems that people actually face. Researchers at Microsoft address this with AGIEval, a bilingual (English and Chinese) benchmark built from official, high-stakes exams written for human candidates. These include the Gaokao, SAT, GRE and GMAT maths, LSAT, Chinese lawyer qualification and civil service tests, and maths competitions. The benchmark's 20 tasks are used to test text-davinci-003, ChatGPT, GPT-4 and the open-source Vicuna under zero-shot, few-shot and chain-of-thought prompting. Model scores are compared with both average and top-1% human test-takers. GPT-4 beats the typical human on SAT, LSAT and maths contests, but it still falls short of the best candidates. It also struggles with multi-step reasoning and specialised domain knowledge such as law and chemistry. A manual analysis along four abilities (understanding, knowledge, reasoning, calculation) finds reasoning the weakest. Unlike the exam results in OpenAI's GPT-4 report, the data, code and model outputs are all public. This makes AGIEval a reproducible yardstick tied to human performance.
Key points
- The exams have very large real-world pools of test-takers: about 12 million sit the Gaokao each year and about 1.7 million the SAT. Eight Gaokao subjects are included. Everything is released at github.com/ruixiangcui/AGIEval.
- With zero-shot chain-of-thought prompting, GPT-4 reaches 95% on SAT Math and 92.5% on Gaokao English.
- The hardest tasks are LSAT analytical reasoning, LogiQA and Gaokao physics. Calculation also drops in chemistry and biology, where questions require substituting symbols for chemical elements.
- A contamination check used Gaokao questions from 2022, which is later than GPT-4's September 2021 training cutoff. GPT-4's accuracy on full vs. uncontaminated subsets: geography 76.9→73%, biology 75.7→77.6%, chemistry 51.7→42.2%, physics 40→40%. Apart from chemistry, scores changed only slightly.
- Only the text-only version is released; multimodal exam items are left for future work.
Cited by
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
- L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning
- Large Language Models for Mathematical Reasoning: Progresses and Challenges
- The Falcon Series of Open Language Models
- Qwen Technical Report
- InternLM2 Technical Report
- A Survey on Evaluation of Large Language Models
- Benchmarking Large Language Models in Retrieval-Augmented Generation
- DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- LawBench: Benchmarking Legal Knowledge of Large Language Models
- Evaluating Large Language Models: A Comprehensive Survey
- C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
- Challenges and Applications of Large Language Models
- Prometheus: Inducing Fine-grained Evaluation Capability in Language Models
- Kimi K2: Open Agentic Intelligence
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training
- CMMLU: Measuring massive multitask language understanding in Chinese
- From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
- DataComp-LM: In search of the next generation of training sets for language models
- DeepSeek-VL: Towards Real-World Vision-Language Understanding
- Orca 2: Teaching Small Language Models How to Reason
- s1: Simple test-time scaling
- Orca: Progressive Learning from Complex Explanation Traces of GPT-4
- Humanity's Last Exam
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation
- To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning
- Aligning Large Language Models with Human: A Survey
- OpenChat: Advancing Open-source Language Models with Mixed-Quality Data
- SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models
- Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity
- MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
- Baichuan 2: Open Large-scale Language Models
- Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
- MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning
- MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
- SafetyBench: Evaluating the Safety of Large Language Models
- The Lessons of Developing Process Reward Models in Mathematical Reasoning
- A Survey of Large Language Models
- GPT-4V(ision) is a Generalist Web Agent, if Grounded
Cite
@inproceedings{zhong2024agieval,
title = {{AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models}},
author = {Zhong, Wanjun and Cui, Ruixiang and Guo, Yiduo and Liang, Yaobo and Lu, Shuai and Wang, Yanlin and Saied, Amin and Chen, Weizhu and Duan, Nan},
booktitle = {Findings of the Association for Computational Linguistics: NAACL 2024},
year = {2024},
doi = {10.18653/v1/2024.findings-naacl.149},
eprint = {2304.06364},
archivePrefix = {arXiv},
url = {https://doi.org/10.18653/v1/2024.findings-naacl.149}
}References
- Edward Beeching, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. 2023. Open llm leaderboard. https://huggingface.co/spaces/ HuggingFaceH4/open\_llm\_leaderboard. ↗
- Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati… find
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
- Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
- Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. find
- Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find our card
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gu… find
- Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. arXiv preprint arXiv:2203.09509. find our card
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. find our card
- Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Sort, 2(4):0–6. find our card
- Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving quantitative reasoning problems with language models. find our card
- Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158–167. find
- Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2021. Logiqa: a challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 3622… find our card
- Gary Marcus and Ernest Davis. 2019. Rebooting AI: Building artificial intelligence we can trust. Vintage. find
- Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730. find our card
- OpenAI. 2022. Chatgpt. https://chat.openai. com/chat. ↗
- OpenAI. 2023. Gpt-4 technical report. find our card
- Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031. find our card
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392. find our card
- James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo… find our card
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32. find our card
- Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,… find our card
- Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan. 2022. From lsat: The progress and challenges of complex reasoning. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:2201–2216. find
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. find our card
- Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022a. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. find
- Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022b. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493. find our card
- Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. Jec-qa: A legal-domain question answering dataset. In Proceedings of AAAI. find
- Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Yining Chen, Jiahai Wang, Jian Yin, Ming Zhou, and Nan Duan. 2022. Analytical reasoning of text. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2306–2319, Seattle, United States. Association for Computational… find