Elamur Research⌕
☰
home / papers / 2025 / Is That Your Final Answer? Test-Time Scaling Improves Selective Question Answering

Is That Your Final Answer? Test-Time Scaling Improves Selective Question Answering

Paper · 2025 · William Jurayj, Jeffrey Cheng and Benjamin Van Durme

Summary

Benchmarks of test-time scaling for reasoning LLMs usually assume the model must always answer, so wrong answers cost nothing. Jurayj, Cheng and Van Durme ask what happens when a model is allowed to abstain. They control compute with budget forcing from s1: if the model tries to stop reasoning early, its end-of-thinking delimiter is suppressed and "Wait" is appended, and the delimiter is forced once the token budget runs out. They then add a simple selection function that accepts an answer only if the summed log-probability of its answer tokens passes a threshold.

They test DeepSeek-R1-32B and S1-32B on AIME 2024/2025, with GPQA in an appendix. Longer reasoning yields more correct answers and also raises confidence in the correct ones. This separation is clear for R1-32B and weaker for S1-32B. The authors also define a utility in which correct answers earn +1, abstentions earn 0 and errors are penalized. Under this utility, the two models compare differently than in the standard always-answer view. The work adds a confidence dimension to test-time scaling evaluation, which matters wherever mistakes are costly.

Key points

  • Setup: 60 AIME problems, each with a 3-digit answer (000–999), so all answers have equal token length.
    • Confidence thresholds: {0.0, 0.5, 0.95}.
    • Budgets: 500 to 8000 reasoning tokens in steps of 100.
    • Decoding at temperature 0 with vLLM and the Language Model Evaluation Harness, on 4 H100 GPUs.
  • Risk settings: Exam Odds (error penalty 0), Jeopardy Odds (−1) and High-Stakes Odds (−20). The authors focus on Jeopardy Odds, since abstaining is pointless under Exam Odds.
  • Model comparison: With no threshold, R1-32B and S1-32B scale similarly. Under Jeopardy Odds with a 0.95 threshold, R1-32B scales better.
  • Accuracy dips at high thresholds: More compute can lower the accuracy of answered questions. This happens because the newly answered questions are harder, not because the model performs worse.
  • Recommendations and limitations: The authors ask the community to report results under both Exam and Jeopardy Odds, and the code is on GitHub (wjurayj/final_answer). Limitations include likelihood-only confidence, possible harm from budget forcing, and English-only evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States240 works · 96,487 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksQuestion answering · 441 works

Cite

Cite as William Jurayj, Jeffrey Cheng and Benjamin Van Durme. 2025. Is That Your Final Answer? Test-Time Scaling Improves Selective Question Answering. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-short.50

@inproceedings{jurayj2025is,
  title = {{Is That Your Final Answer? Test-Time Scaling Improves Selective Question Answering}},
  author = {William Jurayj, Jeffrey Cheng and Benjamin {Van Durme}},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-short.50},
  url = {https://doi.org/10.18653/v1/2025.acl-short.50}
}

References

33 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pranjal Aggarwal and Sean Welleck. 2025. L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning. arXiv preprint. ArXiv:2503.04697 [cs].
  2. Ekin Akyürek, Mehul Damani, Linlu Qiu, Han Guo, Yoon Kim, and Jacob Andreas. 2024. The Surprising Effectiveness of Test-Time Training for Abstract Reasoning. arXiv preprint. ArXiv:2411.07279 [cs]. find
  3. Jordan Boyd-Graber and Benjamin Börschinger. 2020. What question answering can learn from trivia nerds. find
  4. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7422– 7435, Online. Association for Computational Linguistics. find
  5. Lang Cao. 2024. Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanism. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3628–3646, Miami, Florida, USA. Association for Comp… find
  6. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LL…
  7. D. A. Ferrucci. 2012. Introduction to “this is watson”. IBM Journal of Research and Development, 56(3.4):1:1–1:15. find
  8. David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A. Kalyanpur, Adam Lally, J. William Murdock, Eric Nyberg, John Prager, Nico Schlaefer, and Chris Welty. 2010. Building watson: An overview of the deepqa project. AI Magazine, 31(3):59–79. find
  9. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… find
  10. Yonatan Geifman and Ran El-Yaniv. 2017. Selective classification for deep neural networks. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 4885–4894, Red Hook, NY, USA. Curran Associates Inc. find
  11. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — Training Compute-Optimal Large Language Models
  12. Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. 2025. ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning. arXiv preprint. ArXiv:2504.01296 [cs].
  13. Chengsong Huang, Langlin Huang, Jixuan Leng, Jiacheng Liu, and Jiaxin Huang. 2025. Efficient test-time scaling via self-calibration. Preprint, arXiv:2503.00031. — [arXiv:2503.00031](https://arxiv.org/abs/2503.00031) find
  14. Amita Kamath, Robin Jia, and Percy Liang. 2020. Selective question answering under domain shift. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5684– 5696, Online. Association for Computational Linguistics. find
  15. Salman H. Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A. Sohel, and Roberto Togneri. 2018. Cost-sensitive learning of deep feature representations from imbalanced data. IEEE Transactions on Neural Networks and Learning Systems, 29(8):3573–3587. find
  16. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
  17. Wen-Ding Li, Keya Hu, Carter Larsen, Yuqing Wu, Simon Alford, Caleb Woo, Spencer M. Dunn, Hao Tang, Wei-Long Zheng, Yewen Pu, and Kevin Ellis. 2025. Combining induction and transduction for abstract reasoning. In The Thirteenth International Conference on Learning Representations. find
  18. Ibomoiye Domor Mienye and Yanxia Sun. 2021. Performance analysis of cost-sensitive learning methods with application to imbalanced medical data. Informatics in Medicine Unlocked, 25:100690. find
  19. Hussein Mozannar and David Sontag. 2020. Consistent estimators for learning to defer to an expert. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org. find
  20. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv preprint. ArXiv:2501.19393 [cs].
  21. Northpointe. 2017. Practitioner’s guide to compas core. find
  22. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don‘t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Ling…
  23. Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter J Liu. 2023. Out-of-distribution detection and selective generation for conditional language models. In The Eleventh International Conference on Learning Representations. find
  24. Pedro Rodriguez, Shi Feng, Mohit Iyyer, He He, and Jordan Boyd-Graber. 2021. Quizbowl: The Case for Incremental Question Answering. arXiv preprint. ArXiv:1904.04792 [cs]. — [arXiv:1904.04792](https://arxiv.org/abs/1904.04792) find
  25. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. Preprint, arXiv:2408.03314.
  26. Rajeev Verma, Daniel Barrejón, and Eric Nalisnick. 2023. Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensembles. arXiv preprint. ArXiv:2210.16955 [stat]. — [arXiv:2210.16955](https://arxiv.org/abs/2210.16955) find
  27. Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah Goodman. 2024. Hypothesis search: Inductive reasoning with language models. In The Twelfth International Conference on Learning Representations. find
  28. Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024. Measuring short-form factuality in large language models. Preprint, arXiv:2411.04368.
  29. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transf…
  30. Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. 2025. Inference scaling laws: An empirical analysis of compute-optimal inference for LLM problem-solving. In The Thirteenth International Conference on Learning Representations. find
  31. Anqi Zhang, Yulin Chen, Jane Pan, Chen Zhao, Aurojit Panda, Jinyang Li, and He He. 2025. Reasoning models know when they’re right: Probing hidden states for self-verification. Preprint, arXiv:2504.05419. find
  32. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling. *Figure 7: Additional Model Comparisons. We additionally compar…
  33. Com 4000 2000 pute 1000 Budg 500 et B the same basic procedure described in Section 3, except that we stop evaluate token budgets in range [500, 4000]. 0.0 0.2 0.6 0.8 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jurayj-2025-is-that-your-final-answer/