Elamur Research⌕
☰
home / papers / 2026 / OpenCompass: A Universal Evaluation Platform for Large Language Models

OpenCompass: A Universal Evaluation Platform for Large Language Models

Paper · 2026 · Maosong Cao, Kai Chen, Haodong Duan et al.

Summary

Judging an LLM's overall quality usually means running dozens or hundreds of static benchmarks. Each benchmark has its own scoring method (regex matching, ROUGE, model-based judging), its own sensitivity to prompts, and its own data preparation and output cleanup. This fragmentation makes large, cross-domain evaluation slow and hard to standardize.

The OpenCompass Team (Shanghai AI Laboratory) presents OpenCompass, an open-source framework that handles the whole pipeline in one modular system: data preparation, inference, scoring and reporting. It has five decoupled parts:

  • a configuration layer built on MMEngine, with prompt construction from OpenICL;
  • Partitioners, which split the workload into atomic tasks;
  • Runners, which send those tasks to local processes or cluster jobs;
  • task execution units;
  • Summarizers, which present the results.

Scoring can be rule-based, done by an LLM-as-a-Judge, or cascaded. The platform accepts HuggingFace, accelerated-engine, API and custom models, and it covers over 100 datasets. The paper is mainly a system description. Its contribution is a shared, reproducible harness that lets labs compare models across many capability areas without rebuilding evaluation code for each benchmark.

Key points

  • Workflow and prompts: Each run passes through configuration, inference, evaluation and visualization stages. Inference and evaluation both follow the same partition-then-dispatch scheme to allow parallel execution. Prompts can be few-shot (examples chosen by a Retriever), zero-shot, or static multi-turn structures mapped to ChatML.
  • Benchmarks by domain:
    • Knowledge and reasoning: HLE (2,500 questions, text-only subset)
    • Computation: MATH (12,500 problems), plus AIME, HMMT and AMO-Bench
    • Science: PHYSICS (1,297 questions), ClimaQA, SmolInstruct (14 tasks, more than 3M samples)
    • Code: LiveCodeBench, BigCodeBench
    • Long context: Ruler, LongBench (21 tasks)
    • Other: Arc-AGI, IFEval, IFBench
  • Multilingual coverage: MMMLU (human translations into 14 languages) and PMMEval (10 languages from 8 families).
  • Evaluation modes: Objective evaluation uses Accuracy, Exact Match or F1 against gold answers. Subjective evaluation uses LLM-as-a-Judge for open-ended generation. Base models are typically scored by perplexity and chat models by generation.
  • Planned work and code: The authors plan pipeline parallelism across datasets, since evaluation currently waits for all inference on a dataset to finish. They also plan multimodal and multi-turn dialogue evaluation via ChatML. Code is available at github.com/open-compass/opencompass.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Maosong Cao, Kai Chen, Haodong Duan, Yixiao Fang, Zhiwei Fei, Tong Gao, Ge Jiaye, Mo Li, Hongwei Liu, Junnan Liu, Yuan Liu, Chengqi Lyu, Han Lyu, Ningsheng Ma, Zerun Ma, Yu Sun, Zhiyong Wu, Linchen Xiao, Zhuozhi Xiong, Jun Xu, Haochen Ye, Zhaohui Yu, Yike Yuan, Songyang Zhang, Yufeng Zhao, Fengzhe Zhou, Peiheng Zhou, Dongsheng Zhu, Lin Zhu and Jingming Zhuo. 2026. OpenCompass: A Universal Evaluation Platform for Large Language Models. arXiv:2605.19276.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2605.19276 · arXiv 2605.19276

@misc{cao2026opencompass,
  title = {{OpenCompass: A Universal Evaluation Platform for Large Language Models}},
  author = {Maosong Cao, Kai Chen, Haodong Duan, Yixiao Fang, Zhiwei Fei, Tong Gao, Ge Jiaye, Mo Li, Hongwei Liu, Junnan Liu, Yuan Liu, Chengqi Lyu, Han Lyu, Ningsheng Ma, Zerun Ma, Yu Sun, Zhiyong Wu, Linchen Xiao, Zhuozhi Xiong, Jun Xu, Haochen Ye, Zhaohui Yu, Yike Yuan, Songyang Zhang, Yufeng Zhao, Fengzhe Zhou, Peiheng Zhou, Dongsheng Zhu, Lin Zhu and Jingming Zhuo},
  year = {2026},
  doi = {10.48550/arxiv.2605.19276},
  eprint = {2605.19276},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2605.19276}
}

References

23 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd annual meeting of the association for computational linguistics (vol…
  2. Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. Arc prize 2024: Technical report. arXiv preprint arXiv:2412.04604, 2024. 4.8 find
  3. LMDeploy Contributors. Lmdeploy: A toolkit for compressing, deploying, and serving llm. https://github.com/InternLM/lmdeploy , 2023. 3.3 ↗
  4. MMEngine Contributors. MMEngine: Openmmlab foundational library for training deep learning models. 2022. 3.1 find
  5. Kaiyue Feng, Yilun Zhao, Yixin Liu, Tianyu Yang, Chen Zhao, John Sous, and Arman Cohan. Physics: Benchmarking foundation models on university-level physics problem solving. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023. 4.4 find
  6. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. 1, 4.1
  7. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. 4.5
  8. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2021. 4.3
  9. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024. 4.7
  10. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. 4.6
  11. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. 3.3
  12. Veeramakali Vignesh Manivannan, Yasaman Jafari, Srikar Eranky, Spencer Ho, Rose Yu, Duncan Watson-Parris, Yian Ma, Leon Bergen, and Taylor Berg-Kirkpatrick. Climaqa: An automated evaluation framework for climate question answering models. arXiv preprint arXiv:2410.16701, 2024. 4.4 — [arXiv:2410.16701](https://arxiv.org/abs/2410.16701) find
  13. Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity’s last exam. arXiv preprint arXiv:2501.14249, 2025. 1, 4.2
  14. Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. Generalizing verifiable instruction following. arXiv preprint arXiv:2507.02833, 2025. 4.8
  15. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. 1, 4.1
  16. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pages…
  17. Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368, 2024. 4.1
  18. Zhenyu Wu, Yaoxiang Wang, Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Jingjing Xu, and Yu Qiao. Openicl: An open-source framework for in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 489–498, 2023. 3.… find
  19. Botao Yu, Frazier N Baker, Ziqi Chen, Xia Ning, and Huan Sun. Llasmol: Advancing large language models for chemistry with a large-scale, comprehensive, high-quality instruction tuning dataset. arXiv preprint arXiv:2402.09391, 2024. 4.4 find
  20. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2019. 4.2 find
  21. Yidan Zhang, Yu Wan, Boyi Deng, Baosong Yang, Hao-Ran Wei, Fei Huang, Bowen Yu, Dayiheng Liu, Junyang Lin, and Jingren Zhou. P-mmeval: A parallel multilingual multitask benchmark for consistent evaluation of llms. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Process… find
  22. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. 4.8
  23. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877, 2024. 4.6

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cao-2026-opencompass-universal-evaluation-platform/