Measuring Massive Multitask Language Understanding

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

3197 papers of our library cite it (of the 140673 whose bibliographies we read); 179 of them have a page here, the first 100 below, 3018 not yet.

• Fanar: An Arabic-Centric Multimodal Generative AI Platform
• Phi-4 Technical Report
• Do Diacritics Matter? Evaluating the Impact of Arabic Diacritics on Tokenization and LLM Benchmarks
• gpt-oss-120b & gpt-oss-20b Model Card
• L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning
• Yi: Open Foundation Models by 01.AI
• SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
• The Falcon Series of Open Language Models
• L-Eval: Instituting Standardized Evaluation for Long Context Language Models
• Survey on Computational Approaches to Implicature
• Refusal in Language Models Is Mediated by a Single Direction
• Aya 23: Open Weight Releases to Further Multilingual Progress
• Llemma: An Open Language Model For Mathematics
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
• Qwen Technical Report
• GPT-NeoX-20B: An Open-Source Autoregressive Language Model
• Can GPT-3 Perform Statutory Reasoning?
• On the Opportunities and Risks of Foundation Models
• InternLM2 Technical Report
• A Survey on Evaluation of Large Language Models
• JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
• AlpaGasus: Training A Better Alpaca with Fewer Data
• Benchmarking Large Language Models in Retrieval-Augmented Generation
• MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
• Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier
• DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
• Moshi: a speech-text foundation model for real-time dialogue
• Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
• Investigating Data Contamination in Modern Benchmarks for Large Language Models
• LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
• Improving Factuality and Reasoning in Language Models through Multiagent Debate
• Nahw: A Comprehensive Benchmark of Arabic Grammar Understanding, Error Detection, Correction, and Explanation
• LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
• LawBench: Benchmarking Legal Knowledge of Large Language Models
• Mathematical Capabilities of ChatGPT
• RARR: Researching and Revising What Language Models Say, Using Language Models
• Retrieval-Augmented Generation for Large Language Models: A Survey
• Gemini: A Family of Highly Capable Multimodal Models
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• Improving alignment of dialogue agents via targeted human judgements
• ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
• The False Promise of Imitating Proprietary LLMs
• LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
• Evaluating Large Language Models: A Comprehensive Survey
• Large Language Model based Multi-Agents: A Survey of Progress and Challenges
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
• CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
• Measuring Mathematical Problem Solving With the MATH Dataset
• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
• Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting
• C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• TrustLLM: Trustworthiness in Large Language Models
• Qwen2.5-Coder Technical Report
• OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• SqueezeLLM: Dense-and-Sparse Quantization
• Kimi k1.5: Scaling Reinforcement Learning with LLMs
• Kimi K2: Open Agentic Intelligence
• ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic
• MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models
• BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains
• Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model
• Measuring Faithfulness in Chain-of-Thought Reasoning
• What matters when building vision-language models?
• Solving Quantitative Reasoning Problems with Language Models
• Do Large Language Models Understand Word Senses?
• CMMLU: Measuring massive multitask language understanding in Chinese
• Compressing Context to Enhance Inference Efficiency of Large Language Models
• Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
• From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning
• Self-Alignment with Instruction Backtranslation
• Textbooks Are All You Need II: phi-1.5 technical report
• From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
• DataComp-LM: In search of the next generation of training sets for language models
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• RA-DIT: Retrieval-Augmented Dual Instruction Tuning
• VILA: On Pre-training for Visual Language Models
• What Makes Good In-Context Examples for GPT-3?
• AgentBench: Evaluating LLMs as Agents
• What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning
• LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
• The Flan Collection: Designing Data and Methods for Effective Instruction Tuning
• DeepSeek-VL: Towards Real-World Vision-Language Understanding
• Query Rewriting in Retrieval-Augmented Large Language Models
• Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools
• ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
• Augmented Language Models: a Survey
• GAIA: a benchmark for General AI Assistants
• Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
• Orca 2: Teaching Small Language Models How to Reason
• A Comprehensive Overview of Large Language Models
• Large Language Diffusion Models
• Capabilities of GPT-4 on Medical Challenge Problems
• OpenAI o1 System Card

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2009.03300 · arXiv 2009.03300

@misc{hendrycks2020measuring,
  title = {{Measuring Massive Multitask Language Understanding}},
  author = {Hendrycks, Dan and Burns, Collin and Basart, Steven and Zou, Andy and Mazeika, Mantas and Song, Dawn and Steinhardt, Jacob},
  year = {2020},
  doi = {10.48550/arxiv.2009.03300},
  eprint = {2009.03300},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2009.03300}
}

References

32 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents (extended abstract). J. Artif. Intell. Res., 47:253–279, 2013. find
2. Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. Piqa: Reasoning about physical commonsense in natural language, 2019. find our card
3. Y. Bisk, A. Holtzman, J. Thomason, J. Andreas, Y. Bengio, J. Chai, M. Lapata, A. Lazaridou, J. May, A. Nisnevich, N. Pinto, and J. Turian. Experience grounds language, 2020. find our card
4. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Ber… find
5. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457, 2018. find
6. P. Clark, O. Etzioni, D. Khashabi, T. Khot, B. D. Mishra, K. Richardson, A. Sabharwal, C. Schoenick, O. Tafjord, N. Tandon, S. Bhakthavatsalam, D. Groeneveld, M. Guerquin, and M. Schmitz. From ’f’ to ’a’ on the n.y. regents science exams: An overview of the aristo project. ArXiv, abs/1909.01958, 201… find
7. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805, 2019. find
8. R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks, 2020. find
9. C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. ICML, 2017. find
10. D. Hendrycks, M. Mazeika, and T. Dietterich. Deep anomaly detection with outlier exposure. ICLR, 2019a. find
11. D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. ArXiv, abs/1907.07174, 2019b. — [arXiv:1907.07174](https://arxiv.org/abs/1907.07174) find
12. D. Hendrycks, C. Burns, S. Basart, A. Critch, J. Li, D. Song, and J. Steinhardt. Aligning ai with shared human values, 2020. find our card
13. L. Huang, R. L. Bras, C. Bhagavatula, and Y. Choi. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning, 2019. find our card
14. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models, 2020. find
15. D. Khashabi, T. Khot, A. Sabharwal, O. Tafjord, P. Clark, and H. Hajishirzi. Unifiedqa: Crossing format boundaries with a single qa system, 2020. find our card
16. T. Khot, P. Clark, M. Guerquin, P. Jansen, and A. Sabharwal. Qasc: A dataset for question answering via sentence composition, 2019. find
17. A. Kumar, P. Liang, and T. Ma. Verified uncertainty calibration, 2019. find
18. G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy. Race: Large-scale reading comprehension dataset from examinations, 2017. find our card
19. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. Albert: A lite bert for self-supervised learning of language representations. ArXiv, abs/1909.11942, 2020. find
20. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692, 2019. find
21. T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018. find
22. Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. V. Dillon, B. Lakshminarayanan, and J. Snoek. Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. NeurIPS, 2019. find
23. F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel. Language models as knowledge bases?, 2019. find our card
24. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019. find
25. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2019. find
26. M. Richardson, C. J. Burges, and E. Renshaw. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 193–203, Seattle, Washington, USA, Oct. 2013. Association for Computational Lin… find our card
27. A. B. Sai, A. K. Mohankumar, and M. M. Khapra. A survey of evaluation metrics used for nlg systems. 2020. find
28. A. Turing. Computing machinery and intelligence. 1950. find
29. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2018. find our card
30. A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems, 2019. find our card
31. R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can a machine really finish your sentence?, 2019. find
32. R. Zellers, A. Holtzman, E. Clark, L. Qin, A. Farhadi, and Y. Choi. Evaluating machines by their real-world language use, 2020. find