Elamur Research⌕
☰
home / papers / 2026 / LLMEval-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models

LLMEval-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models

Paper · 2026 · Ming Zhang, Yujiong Shen, Jingyi Deng et al.

Summary

Fixed public test sets can leak into training corpora and be overfitted, so leaderboard scores may overstate real LLM ability. The authors present LLMEval-Fair, a dynamic evaluation framework built on more than 220k Chinese-language university and postgraduate exam questions across 13 disciplines. These come from 78,009 expert-checked originals, expanded through LLM-based reformatting. Each run draws a new random sample of 1,000 questions and delivers it through a two-layer anti-cheating setup. The outer layer handles authenticated JWT sessions with access isolation. The inner layer enforces quotas and strips answers from what the model receives. A calibrated LLM judge grades responses, and models are ranked relative to each other within a session rather than by absolute score. Over 30 months, the team tracked nearly 60 models and collected more than 180k evaluation data points. Top models plateau near 90%, with persistent weak spots in literature, medicine and military knowledge. Replay tests point to leakage in public benchmarks. Rankings proved stable under resampling, supporting dynamic testing as a more trustworthy alternative to static benchmarks.

Key points

  • Question bank: 78,009 original questions were each validated by two of 30+ graduate annotators, then expanded to over 220k items. For example, an n-option multiple-choice question becomes n fill-in-the-blank variants. Augmentation left model rankings unchanged (Spearman ρ = 1.0).
  • Discipline coverage: Evaluation uses the 10 best-populated disciplines; Agronomy, Arts and Philosophy are excluded for lack of data. Every model is pinned to a fixed API snapshot and evaluated at least three times.
  • Judge reliability: The LLM-as-a-judge reaches Cohen's κ = 0.907 with human experts. Grading with GPT-4o, Doubao-1.5-Thinking-Pro and Gemini-2.5-Pro gives consistent rankings (Spearman ρ ≥ 0.847) with no sign of family bias.
  • Contamination evidence: In fill-in-the-blank replay tests, AGIEval and C-Eval produced far more completions than the private bank, which indicates contamination.
  • Ranking stability: Relative ranking kept an identical model order across resampling at n = 1000, 2000 and 4000 (variance σ² ≤ 1.68) and beat an Elo-style baseline. Data and results are released at github.com/llmeval/LLMEval-Fair. The bank is currently limited to Chinese.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China202 works · 22,509 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Ming Zhang, Yujiong Shen, Jingyi Deng, Yuhui Wang, Huayu Sha, Kexin Tan, Qiyuan Peng, Yue Zhang, Junzhe Wang, Shichun Liu, Yueyuan Huang, Jingqi Tong, Changhao Jiang, Yilong Wu, Zhihao Zhang, Mingqi Wu, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang and Xuanjing Huang. 2026. LLMEval-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.972

@inproceedings{zhang2026llmeval,
  title = {{LLMEval-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models}},
  author = {Ming Zhang, Yujiong Shen, Jingyi Deng, Yuhui Wang, Huayu Sha, Kexin Tan, Qiyuan Peng, Yue Zhang, Junzhe Wang, Shichun Liu, Yueyuan Huang, Jingqi Tong, Changhao Jiang, Yilong Wu, Zhihao Zhang, Mingqi Wu, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang and Xuanjing Huang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.972},
  url = {https://doi.org/10.18653/v1/2026.acl-long.972}
}

References

90 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah I Abdin, Jyoti Aneja, Harkirat S. Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and… — Phi-4 Technical Report
  2. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2023. A survey on evaluation of large language models. CoRR, abs/2307.03109.
  3. Alibaba Cloud. 2024. qwen-turbo-2024-11-01 (qwen-turbo-1101). https://www.alibabacloud.com/ help/en/model-studio/models. Official model list. Accessed 2026-04-10. ↗
  4. Simin Chen, Pranav Pusarla, and Baishakhi Ray. 2025. Dynamic benchmarking of reasoning capabilities in code large language models under data contamination. CoRR, abs/2503.04149. — [arXiv:2503.04149](https://arxiv.org/abs/2503.04149) find
  5. Alibaba Cloud. 2026a. Qwen-plus. https://www.alibabacloud.com/help/en/model-studio/ Official model list / documentation. models. Accessed 2026-04-10. ↗
  6. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: An open platform for evaluating LLMs by human preference. In Forty-first International Conference on Mac…
  7. Alibaba Cloud. 2026b. Qwen-turbo. https://www.alibabacloud.com/help/en/model-studio/ Official model list / documentation. models. Accessed 2026-04-10. ↗
  8. Anthropic. 2024a. The claude 3 model family: Opus, sonnet, haiku. Technical report, Anthropic. find
  9. Anthropic. 2024b. Model card addendum: Claude 3.5 haiku and upgraded claude 3.5 sonnet. Technical report, Anthropic. find
  10. Anthropic. 2025a. Claude sonnet 4.5 system card. System card, Anthropic. find
  11. Anthropic. 2025b. System card: Claude opus 4 & claude sonnet 4. Technical Report / System Card. find
  12. Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, and Wanli Ouyang. 2024. Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. In Proceedings of the 62nd Annual Meeting of the Associatio…
  13. Baidu. 2023a. Baidu announces second quarter 2023 results. Investor relations press release. find
  14. Baidu. 2023b. Baidu announces third quarter 2023 results. Investor relations press release. find
  15. Sourav Banerjee, Ayushi Agarwal, and Eishkaran Singh. 2024. The vulnerability of language model benchmarks: Do they accurately reflect true LLM performance? CoRR, abs/2412.03597. — [arXiv:2412.03597](https://arxiv.org/abs/2412.03597) find
  16. BELLE Group. 2023. Belle-llama2-13b-chat-0.4m. https://huggingface.co/BELLE-2/ BELLE-Llama2-13B-chat-0.4M. Official model card. Accessed 2026-04-10. ↗
  17. ByteDance. 2025. Bytedance ai introduces doubao-1.5-pro language model with a deep thinking mode. MarkTechPost article. find
  18. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. CoRR, abs/2110.14168.
  19. DeepSeek-AI. 2025. Deepseek-v3.2: Pushing the frontier of open large language models. CoRR, abs/2512.02556.
  20. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 81 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llm…
  21. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 81 others. 2024. Deepseek-v3 technical report. CoRR, abs/2412.194… find
  22. Jasper Dekoninck, Mark Niklas Müller, Maximilian Baader, Marc Fischer, and Martin T. Vechev. 2024. Evading data contamination detection for language models is (too) easy. CoRR, abs/2402.02823. find
  23. Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2024a. Investigating data contamination in modern benchmarks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno…
  24. Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2024b. Investigating data contamination in modern benchmarks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno…
  25. Deming Ding, Shichun Liu, Enhui Yang, Jiahang Lin, Ziying Chen, Shihan Dou, Honglin Guo, Weiyu Cheng, Pengyu Zhao, Chengjun Xiao, Qunhong Zeng, Qi Zhang, Xuanjing Huang, Qidi Xu, and Tao Gui. 2026. Octobench: Benchmarking scaffold-aware instruction following in repository-grounded agentic coding. Co…
  26. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  27. Xuanwen Ding, Chengjun Pan, Zejun Li, Jiwen Zhang, Siyuan Wang, and Zhongyu Wei. 2025. Autojudger: An agent-driven framework for efficient benchmarking of mllms. arXiv preprint arXiv:2505.21389. find
  28. Shihan Dou, Ming Zhang, Chenhao Huang, Jiayi Chen, Feng Chen, Shichun Liu, Yan Liu, Chenxiao Liu, Cheng Zhong, Zongzhang Zhang, Tao Gui, Chao Xin, Wei Chengzhi, Lin Yan, Qi Zhang, Yonghui Wu, and Xuanjing Huang. 2025. Evalearn: Quantifying the learning capability and efficiency of llms via sequentia… — [arXiv:2506.02672](https://arxiv.org/abs/2506.02672) find
  29. Shihan Dou, Ming Zhang, Zhangyue Yin, Chenhao Huang, Yujiong Shen, Junzhe Wang, Jiayi Chen, Yuchen Ni, Junjie Ye, Cheng Zhang, Huaibing Xie, Jianglu Hu, Shaolei Wang, Weichao Wang, Yanling Xiao, Yiting Liu, Zenan Xu, Zhen Guo, Pluto Zhou, and 8 others. 2026. Cl-bench: A benchmark for context learnin… — [arXiv:2602.03587](https://arxiv.org/abs/2602.03587) find
  30. Gemini Team, Google DeepMind. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Technical report, Google DeepMind.
  31. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring massive multitask language understanding. In International Conference on Learning Representations.
  32. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets a…
  33. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, and 1 others. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in neural information processing systems, 36:62991– 6301…
  34. InternLM Team. 2023. Internlm-chat-7b. https://huggingface.co/internlm/internlm-chat-7b. Official model card. Accessed 2026-04-10. ↗
  35. Changhao Jiang, Ming Zhang, Yifei Cao, Junjie Ye, Xiaoran Fan, Shihan Dou, Zhiheng Xi, Jiajun Sun, Yi Dong, Yujiong Shen, and 1 others. 2025. Beyond scaling: Measuring and predicting the upper bound of knowledge retention in language model pre-training. arXiv preprint arXiv:2502.04066. find
  36. Michael B. Jones, John Bradley, and Nat Sakimura. 2015. JSON Web Token (JWT). RFC 7519. find
  37. Tatsu Lab. 2023. Alpacaeval: An automatic evaluator for instruction-following language models. https://github.com/tatsu-lab/alpaca_ eval. Accessed: 2025-07-31. ↗
  38. Md. Tahmid Rahman Laskar, Sawsan Alqahtani, M. Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee-Wei Tan, Md. Rizwan Parvez, Enamul Hoque, Shafiq Joty, and Jimmy Huang. 2024. A systematic survey and critical review on evaluating large language m…
  39. Boxun Li, Yadong Li, Zhiyuan Li, Congyi Liu, Weilin Liu, Guowei Niu, Zheyue Tan, Haiyang Xu, Zhuyu Yao, Tao Yuan, Dong Zhou, Yueqing Zhuang, Shengen Yan, Guohao Dai, and Yu Wang. 2025. Megrez-omni technical report. CoRR, abs/2502.15803. — [arXiv:2502.15803](https://arxiv.org/abs/2502.15803) find
  40. Linly-AI. 2023. Chinese-llama-2-13b. https://huggingface.co/Linly-AI/ Chinese-LLaMA-2-13B-hf. Official model card. Accessed 2026-04-10. ↗
  41. Songyang Liu, Chaozhuo Li, Jiameng Qiu, Xi Zhang, Feiran Huang, Litian Zhang, Yiming Hei, and Philip S. Yu. 2025. The scales of justitia: A comprehensive survey on safety evaluation of llms. CoRR, abs/2506.11094. — [arXiv:2506.11094](https://arxiv.org/abs/2506.11094) find
  42. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Lin…
  43. Meta. 2024a. Llama-3.1-8b. https://huggingface.co/meta-llama/Llama-3.1-8B. Official model card. Accessed 2026-04-10. ↗
  44. Meta. 2024b. Llama-3.2-90b-vision-instruct. https://huggingface.co/meta-llama/Llama-3. 2-90B-Vision-Instruct. Official model card. Accessed 2026-04-10. ↗
  45. Millican, Ethan Dyer, Mia Glaese, Thibault Sottiaux, Benjamin Lee, and 34 others. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. CoRR, abs/2403.05530.
  46. Yujiong Shen, Yajie Yang, Zhiheng Xi, Binze Hu, Huayu Sha, Jiazheng Zhang, Qiyuan Peng, Junlin Shang, Jixuan Huang, Yutao Fan, Jingqi Tong, Shihan Dou, Ming Zhang, Lei Bai, Zhenfei Yin, Tao Gui, Xingjun Ma, Qi Zhang, Xuanjing Huang, and Yu-Gang Jiang. 2026. Sciagentgym: Benchmarking multi-step scien… — [arXiv:2602.12984](https://arxiv.org/abs/2602.12984) find
  47. MiniMax. 2024. Tongyong da moxing abab6.5s xilie. Official release note. find
  48. YuHong Sun, Zhangyue Yin, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Hui Zhao. 2024. Benchmarking hallucination in large language models based on unanswerable math word problem. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LRE… find
  49. Moonshot AI. 2025. Kimi k2 technical report. Technical report, Moonshot AI. find
  50. Gemini Team. 2023. Gemini: A family of highly capable multimodal models. CoRR, abs/2312.11805.
  51. Nanbeige LLM Lab. 2023. Nanbeige-16b. https://github.com/Nanbeige/Nanbeige. Official repository for the Nanbeige-16B family. Accessed 2026-04-10. ↗
  52. Qwen Team. 2025. Qwen3 technical report. CoRR, abs/2505.09388.
  53. Meta. 2024c. Llama-3.3-70b-instruct. https://huggingface.co/meta-llama/Llama-3. 3-70B-Instruct. Official model card. Accessed 2026-04-10. ↗
  54. Nanbeige LLM Lab. 2024. Nanbeige-plus-chat-v0.1. https://huggingface.co/Nanbeige/ Nanbeige-Plus-Chat-v0.1. Official Hugging Face model page path indexed in public benchmark registry; verify accessibility before final submission. Accessed 2026-04-10. ↗
  55. OpenAI. 2023a. GPT-4 technical report. abs/2303.08774.
  56. CoRR, OpenAI. 2023b. Introducing chatgpt and whisper apis. OpenAI blog. find
  57. OpenAI. 2024. o1 system card. Technical report, OpenAI. find
  58. OpenAI. 2025a. Gpt-5 system card. System card, OpenAI. find
  59. OpenAI. 2025b. o3-mini system card. Technical report, OpenAI. find
  60. Qwen Team. 2024. Qwq-32b. https://huggingface.co/Qwen/QwQ-32B. Official model card. Accessed 2026-04-10. ↗
  61. Ravi Raju, Swayambhoo Jain, Bo Li, Jonathan Li, and Urmish Thakker. 2024. Constructing domain-specific evaluation sets for llm-as-a-judge. Preprint, arXiv:2408.08808. find
  62. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean-Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Hugo Touvron, Louis Martin, Kevin Stone, Peter A… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  63. Cheng Xu, Shuhao Guan, Derek Greene, and M-Tahar Kechadi. 2024a. Benchmark data contamination of large language models: A survey. Preprint, arXiv:2406.04244.
  64. Fangzhi Xu, Hang Yan, Qiushi Sun, Jinyang Wu, Zixian Huang, Muye Huang, Jingyang Gong, Zichen Ding, Kanzhi Cheng, Yian Wang, and 1 others. 2026. Odysseyarena: Benchmarking large language models for long-horizon, active and inductive interactions. arXiv preprint arXiv:2602.05843. — [arXiv:2602.05843](https://arxiv.org/abs/2602.05843) find
  65. Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. 2024b. Benchmarking benchmark leakage in large language models. Preprint, arXiv:2404.18824.
  66. Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, and 36 others. 2023. Baichuan 2: Open large-scale language models. CoRR, abs/2309.1030…
  67. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 43 others. 2024a. Qwen2 technical report. CoRR, abs/2407.106… find
  68. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 22 others. 2024b. Qwen2.5 technical report. CoRR, abs/2412.151… find
  69. Jie Yang, Jiajun Chen, Zhangyue Yin, Shuo Chen, Yuxin Wang, Yiran Guo, Yuan Li, Yining Zheng, Xuanjing Huang, and Xipeng Qiu. 2025. Vehicle-world: A highly integrated multi-device environment for intelligent vehicle interaction. arXiv preprint arXiv:2509.06736. — [arXiv:2509.06736](https://arxiv.org/abs/2509.06736) find
  70. Junjie Ye, Zhengyin Du, Xuesong Yao, Weijian Lin, Yufei Xu, Zehui Chen, Zaiyuan Wang, Sining Zhu, Zhiheng Xi, Siyu Yuan, Tao Gui, Qi Zhang, Xuanjing Huang, and Jiecao Chen. 2025a. ToolHop: A query-driven benchmark for evaluating large language models in multi-hop tool use. In Proceedings of the 63rd…
  71. Junjie Ye, Caishuang Huang, Zhuohan Chen, Wenjie Fu, Chenyuan Yang, Leyi Yang, Yilong Wu, Peng Wang, Meng Zhou, Xiaolong Yang, and 1 others. 2025b. A multi-dimensional constraint framework for evaluating and improving instruction following in large language models. arXiv preprint arXiv:2505.07591. — [arXiv:2505.07591](https://arxiv.org/abs/2505.07591) find
  72. Junjie Ye, Guanyu Li, SongYang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Tao Ji, Qi Zhang, Tao Gui, and Xuanjing Huang. 2025c. ToolEyes: Fine-grained evaluation for tool learning capabilities of large language models in real-world scenarios. In Proceedings of the 31st Inte…
  73. Junjie Ye, Sixian Li, Guanyu Li, Caishuang Huang, Songyang Gao, Yilong Wu, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024a. ToolSword: Unveiling safety issues of large language models in tool learning across three stages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Li… find
  74. Junjie Ye, Yilong Wu, Songyang Gao, Caishuang Huang, Sixian Li, Guanyu Li, Xiaoran Fan, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024b. RoTBench: A multi-level benchmark for evaluating the robustness of large language models in tool learning. In Proceedings of the 2024 Conference on Empirical Methods… find
  75. Junjie Ye, Guoqiang Zhang, Wenjie Fu, Tao Gui, Qi Zhang, and Xuanjing Huang. 2026. Cctu: A benchmark for tool use under complex constraints. arXiv preprint arXiv:2603.15309. — [arXiv:2603.15309](https://arxiv.org/abs/2603.15309) find
  76. Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023. Do large language models know what they don’t know? In Findings of the Association for Computational Linguistics: ACL 2023, pages 8653–8665, Toronto, Canada. Association for Computational Linguistics. find
  77. Zhangyue Yin, Qiushi Sun, Zhiyuan Zeng, Zhiyuan Yu, Qipeng Guo, Xuanjing Huang, and Xipeng Qiu. 2025. Arise: An adaptive resolution-aware metric for test-time scaling evaluation in large reasoning models. arXiv preprint arXiv:2510.06014.
  78. Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, and 11 others. 2024. Yi: Open foundation models by 01.ai. CoRR, abs/2403.046…
  79. Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, and 36 others. 2024. Chatglm: A family of large language models from GLM-130B t…
  80. Jiaxing Zhang, Ruyi Gan, Junjie Wang, Yuxiang Zhang, Lin Zhang, Ping Yang, Xinyu Gao, Ziwei Wu, Xiaoqun Dong, Junqing He, Jianheng Zhuo, Qi Yang, Yongfeng Huang, Xiayu Li, Yanghan Wu, Junyu Lu, Xinyu Zhu, Weifeng Chen, Ting Han, and 6 others. 2022. Fengshenbang 1.0: Being the foundation of chinese c… — [arXiv:2209.02970](https://arxiv.org/abs/2209.02970) find
  81. Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang, Zhiheng Xi, Shihan Dou, Rongxiang Weng, Jiahuan Li, Jingang Wang, Mingxu Chai, Shibo Hong, and 1 others. 2025a. Two minds better than one: Collaborative reward modeling for llm alignment. arXiv preprint arXiv:2505.10597. — [arXiv:2505.10597](https://arxiv.org/abs/2505.10597) find
  82. Ming Zhang, Yujiong Shen, Zelin Li, Huayu Sha, Binze Hu, Yuhui Wang, Chenhao Huang, Shichun Liu, Jingqi Tong, Changhao Jiang, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. 2025b. LLMEval-Med: A real-world clinical benchmark for medical LLMs with physician validation. In… find
  83. Ming Zhang, Yue Zhang, Shichun Liu, Haipeng Yuan, Junzhe Wang, Yurui Dong, Jingyi Deng, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. Llmeval-2. find
  84. Ming Zhang, Jiabao Zhuang, Wenqing Jing, Kexin Tan, Ziyu Kong, Jingyi Deng, Yujiong Shen, Yuhang Zhao, Ning Luo, Renzhe Zheng, Jiahui Lin, Mingqi Wu, Long Ma, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. 2026. Can deep research agents retrieve and organize? evaluating the synthesis gap with ex… — [arXiv:2601.12369](https://arxiv.org/abs/2601.12369) find
  85. Yue Zhang, Ming Zhang, Haipeng Yuan, Shichun Liu, Yongyao Shi, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. Llmeval: A preliminary study on how to evaluate large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19615–19622. find
  86. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. 2024. LMSYS-chat-1m: A large-scale real-world LLM conversation dataset. In The Twelfth International Conference on Learning…
  87. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing…
  88. Zhipu AI. 2025. Glm-4.6. Official documentation. find
  89. Zhipu AI and THUDM. 2026. Glm-4-32b-0414. https://huggingface.co/zai-org/ GLM-4-32B-0414. Official model card. Accessed 2026-04-10. ↗
  90. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2024. Agieval: A human-centric benchmark for evaluating foundation models. In Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-llmeval-fair-large-scale-longitudinal/