Elamur Research⌕
☰
home / papers / 2026 / Position: Scores Without Context? Rethinking the Role of Evaluation in the Era of LLMs

Position: Scores Without Context? Rethinking the Role of Evaluation in the Era of LLMs

Paper · 2026 · Jiawei Zhou

Summary

This position paper examines how NLP evaluation has changed since large language models became dominant. Benchmarks once tested generalization from known training data to held-out test sets. Now they mostly measure what a system can do, while training corpora stay large and often undisclosed. The author argues that a score detached from training conditions cannot support scientific conclusions. Strong results might come from generalization, memorization or distributional overlap, and weak results might simply reflect unrealistic expectations. To illustrate the shift, the paper uses two pieces of evidence: a keyword-based trend analysis of the ACL Anthology and a small manual audit of papers. It then proposes expectation-aware evaluation, in which a result's worth is the gap between observed performance and what the training data and architecture would predict. That prediction depends on the observer's knowledge at a given time. Finally, the paper separates white-box evaluation for scientific understanding from black-box capability tracking. It recommends that researchers state their assumptions explicitly and limit each kind of claim to what its regime can justify.

Key points

  • Trend analysis: title keyword matching over the ACL Anthology (2015–2025) shows evaluation-focused papers rising steadily, with a sharp jump after ChatGPT.
  • Manual audit: 20 ACL/EMNLP papers each from 2018 and 2025 were inspected.
    • Papers discussing the relation between training and evaluation data fell from 17/20 (85%) to 3/20 (15%).
    • Capability-focused papers grew from 0/20 to 13/20 (65%).
  • Proposed formalization: V_eval = P(D_eval) − E(D_train, A), where expectation is conditioned on knowledge K(t, p). This yields three cases:
    • P ≈ E: tracking, with low scientific value.
    • P ≫ E: scientific gain, e.g. GPT-3 few-shot learning.
    • P ≪ E: diagnostic insight, e.g. unexpected unsafe behavior in safety-aligned models.
  • GPT reports: discussion of benchmark overlap and memorization appears in the GPT-2 and GPT-3 reports but is absent from GPT-4.
  • OSWorld-Verified example: GPT-5.4 reports 75.0% against a human baseline of 72.4%. The paper uses this as an example of product capability rather than scientific insight, alongside capability benchmarks such as SWE-Bench and GPQA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stony Brook UniversityStony Brook · United States46 works · 3,024 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Jiawei Zhou. 2026. Position: Scores Without Context? Rethinking the Role of Evaluation in the Era of LLMs. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.gem-main.82

@inproceedings{zhou2026position,
  title = {{Position: Scores Without Context? Rethinking the Role of Evaluation in the Era of LLMs}},
  author = {Jiawei Zhou},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.82},
  url = {https://doi.org/10.18653/v1/2026.gem-main.82}
}

References

27 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Roee Aharoni and Yoav Goldberg. 2018. Split and rephrase: Better evaluation and stronger baselines. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 719–724, Melbourne, Australia. Association for Computational Linguistics. find
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  4. Mark Chen. 2025. On reasoning improvements in openai o-series models. https://x.com/markchen90/ status/1884303237186216272. Accessed: 2026-03-21. ↗
  5. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, and 1 others. 2024. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on Machin…
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,…
  7. Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, and 1 others. 2024. Alignment faking in large language models. arXiv preprint arXiv:2412.14093. find
  8. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, and 1 others. 2025. Deepseekr1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638. find
  9. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974.
  10. Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, and 1 others. 2025. Gdpval: Evaluating ai model performance on real-world economically valuable tasks. arXiv preprint arXiv:2510.04374. find
  11. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  12. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 2383–2392.
  13. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First conference on language modeling.
  14. Lucia Specia, Zhenhao Li, Juan Pino, Vishrav Chaudhary, Francisco Guzmán, Graham Neubig, Nadir Durrani, Yonatan Belinkov, Philipp Koehn, Hassan Sajjad, and 1 others. 2020. Findings of the wmt 2020 shared task on machine translation robustness. In Proceedings of the fifth conference on machine transl… find
  15. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770.
  16. Haoxiang Sun, Yingqian Min, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. 2025. Challenging the boundaries of reasoning: An olympiad-level math benchmark for large language models. arXiv preprint arXiv:2503.21380.
  17. Yanhong Li, Tianyang Xu, Kenan Tang, Karen Livescu, David McAllester, and Jiawei Zhou. 2025. Okbench: Democratizing llm evaluation with fully automated, on-demand, open knowledge benchmarking. arXiv preprint arXiv:2511.08598. — [arXiv:2511.08598](https://arxiv.org/abs/2511.08598) find
  18. Jiuding Sun, Chantal Shaib, and Byron Wallace. 2024. Evaluating the zero-shot robustness of instruction-tuned language models. In International Conference on Learning Representations, volume 2024, pages 48103–48141. find
  19. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In International Conference on Learning Representations.
  20. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  21. Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, and 1 others. 2026. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868. — [arXiv:2601.11868](https://arxiv.org/abs/2601.11868) find
  22. OpenAI. 2024. Introducing OpenAI o1. Accessed: 2026-03-21. find
  23. OpenAI. 2026. Introducing GPT-5.4. Accessed: 2026-03-20. find
  24. Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Simón Posada Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of…
  25. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.
  26. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, and 1 others. 2024. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37…
  27. A broadly, modern evaluation increasingly measures both learning and knowledge capacity, blurring a distinction that was central in earlier paradigms of machine learning. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2026-position-scores-without-context-rethinking/