Elamur Research⌕
☰
home / papers / 2026 / Graduating the Benchmark Scale: Lessons from Thermometry

Graduating the Benchmark Scale: Lessons from Thermometry

Paper · 2026 · Sean Trott and Oisín Parkinson-Coombs

Summary

This position paper from the EvalEval 2026 workshop looks at a neglected part of LLM benchmark validity. It asks whether a benchmark score can be treated as a graded measure of an underlying capability, and what justifies reading equal score gaps as equal differences in ability. The authors frame this with Hasok Chang's "problem of nomic measurement" from the history of thermometry. The problem is that the function linking an observable reading to a hidden quantity cannot be tested without already knowing that quantity.

The paper introduces no new data or models. Instead, it compares hypothetical score-to-capability mappings. It then shows how thermometry escaped this circularity through epistemic iteration: step-by-step, mutually correcting refinement of instruments, fixed points and theory. LLM capabilities are more abstract and lack stable anchors, so the analogy only partly holds.

The central claim is that the problem's severity depends on what a benchmark is for. Because the assumed mapping can change deployment and regulatory choices, researchers should state their goals and theoretical commitments explicitly.

Key points

  • Figure 1 sketches six possible links between measured and "true" capability: null, linear, categorical, logarithmic, sigmoidal and exponential.

  • The authors illustrate the issue with an imagined 100-item Python benchmark. A jump from 30 to 50 and a jump from 50 to 70 mean the same thing only under a linear mapping. A logarithmic mapping makes the first gap larger, and an exponential one makes the second gap larger.

  • Chang (2004) describes three stages in thermometry:

    • bodily sensation;
    • ordinal thermoscopes;
    • numerical thermometers built on fixed points and instrument comparison, with Henri Regnault's work on air thermometers as the example.

    Along the way, the boiling point of water was later replaced by steam temperature as a fixed point.

  • Three benchmark goals are distinguished:

    • ranking models, which works under every mapping except null;
    • detecting a behavior, which needs a meaningful threshold;
    • measuring degrees of a capability, where the problem is hardest.

  • Human baselines (METR, 2026) are suggested as possible anchors. However, they are judged less stable than physical fixed points and may not be comparable with LLM behavior.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Rutgers, The State University of New JerseyNew Brunswick · United States40 works · 12,486 citations🎓University of BergenBergen · Norway9 works · 583 citations

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 worksDecision Sciences · 249 worksManagement Science and Operations Research · 134 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 works

Cite

Cite as Sean Trott and Oisín Parkinson-Coombs. 2026. Graduating the Benchmark Scale: Lessons from Thermometry. Proceedings of the Workshop on Evaluating Evaluations (EvalEval).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.evaleval-1.21

@inproceedings{trott2026graduating,
  title = {{Graduating the Benchmark Scale: Lessons from Thermometry}},
  author = {Sean Trott and Oisín Parkinson-Coombs},
  booktitle = {Proceedings of the Workshop on Evaluating Evaluations (EvalEval)},
  year = {2026},
  doi = {10.18653/v1/2026.evaleval-1.21},
  url = {https://doi.org/10.18653/v1/2026.evaleval-1.21}
}

References

23 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Inioluwa Deborah Raji, Emily M Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. 2021. Ai and the everything in the whole wide world benchmark. arXiv preprint arXiv:2111.15366. find
  2. Anna Alexandrova and Daniel M. Haybron. 2016. Is Construct Validation Valid? Philosophy of Science, 83(5):1098–1109. find
  3. Andrew M Bean, Ryan Othniel Kearns, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, Jan Batzner, Negar Foroutan, Chris Schmitz, Karolina Korgul, Hunar Batra, and 1 others. 2025. Measuring what matters: Construct validity in large language model benchmarks. arXiv preprint arXiv:2511.04703. — [arXiv:2511.04703](https://arxiv.org/abs/2511.04703) find
  4. Denny Borsboom, Gideon J. Mellenbergh, and Jaap Van Heerden. 2004. The Concept of Validity. Psychological Review, 111(4):1061–1071. find
  5. Hasok Chang. 2004. Inventing Temperature: Measurement and Scientific Progress. Oxford University Press. Google-Books-ID: yVOuV8qJkxMC. find
  6. Hasok Chang. 2015. The rising of chemical natural kinds through epistemic iteration. In Natural kinds and classification in scientific practice, pages 33–46. Routledge. find
  7. Hasok Chang. 2017. Epistemic iteration and natural kinds: Realism and pluralism in taxonomy. Philosophical issues in psychiatry IV: Psychiatric nosology, pages 229–245. find
  8. Jennifer Hu, Felix Sosa, and Tomer Ullman. 2025. Re-evaluating Theory of Mind evaluation in large language models. Philosophical Transactions of the Royal Society B: Biological Sciences, 380(1932):20230499. find
  9. Anna A. Ivanova. 2025. How to evaluate the cognitive abilities of LLMs. Nature Human Behaviour, 9(2):230–233. find
  10. Cameron Jones and Benjamin Bergen. 2026. Lies, damned lies, and language statistics: a comprehensive review of risks from manipulation, persuasion, and deception with large language models. Artificial Intelligence Review. find
  11. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
  12. Tom Kouwenhoven, Michiel van der Meer, and Max van Duijn. 2026. Traces of social competence in large language models. arXiv preprint arXiv:2603.04161. find
  13. Cristian Larroulet Philippi. 2021. Valid for What? On the Very Idea of Unconditional Validity. Philosophy of the Social Sciences, 51(2):151–175. Publisher: SAGE Publications Inc. find
  14. METR. 2026. Task-completion time horizons of frontier ai models. https://metr.org/ time-horizons/. ↗
  15. Olawale Salaudeen, Anka Reuel, Ahmed Ahmed, Suhana Bedi, Zachary Robertson, Sudharsan Sundar, Ben Domingue, Angelina Wang, and Sanmi Koyejo. 2025. Measurement to meaning: A validity-centered framework for ai evaluation. arXiv preprint arXiv:2505.10573. — [arXiv:2505.10573](https://arxiv.org/abs/2505.10573) find
  16. Michael Saxon, Ari Holtzman, Peter West, William Yang Wang, and Naomi Saphra. 2024. Benchmarks as Microscopes: A Call for Model Metrology. arXiv preprint. ArXiv:2407.16711 [cs]. — [arXiv:2407.16711](https://arxiv.org/abs/2407.16711) find
  17. Rylan Schaeffer, Punit Singh Koura, Binh Tang, Ranjan Subramanian, Aaditya K Singh, Todor Mihaylov, Prajjwal Bhargava, Lovish Madaan, Niladri S Chatterji, Vedanuj Goswami, and 1 others. 2025. Correlating and predicting human evaluations of language models from natural language processing benchmarks.… — [arXiv:2502.18339](https://arxiv.org/abs/2502.18339) find
  18. Sean Trott, Cameron Jones, Tyler Chang, James Michaelov, and Benjamin Bergen. 2023. Do Large Language Models Know What Humans Know? Cognitive Science, 47(7):e13309. find
  19. Sean Trott, Samuel Taylor, Cameron Jones, James A. Michaelov, and Pamela D. Rivière. 2026. Language Statistics and False Belief Reasoning: Evidence from 41 Open-Weight LMs. arXiv preprint. ArXiv:2602.16085 [cs].
  20. Tomer Ullman. 2023. Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks. arXiv preprint. ArXiv:2302.08399 [cs]. find
  21. Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alexandra Chouldechova, Emily Corvi, P. Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nicholas J Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Han… find
  22. Laura Weidinger, Inioluwa Deborah Raji, Hanna Wallach, Margaret Mitchell, Angelina Wang, Olawale Salaudeen, Rishi Bommasani, Deep Ganguli, Sanmi Koyejo, and William Isaac. 2025. Toward an evaluation science for generative ai systems. arXiv preprint arXiv:2503.05336. — [arXiv:2503.05336](https://arxiv.org/abs/2503.05336) find
  23. Tal Yarkoni. 2022. The generalizability crisis. Behavioral and Brain Sciences, 45:e1. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/trott-2026-graduating-benchmark-scale-lessons-thermometry/