Elamur Research⌕
☰
home / papers / 2026 / ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering

ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering

Paper · 2026 · Francesco Maria Molfese, Luca Moroni, Ciro Porcaro et al.

Summary

Commonsense benchmarks usually score Small Language Models (SLMs) only on whether the final answer matches the gold label. This hides cases where a model picks the right option through faulty logic. Existing resources for checking reasoning traces focus mostly on math and science, so the authors build ReTraceQA for process-level evaluation in commonsense QA. They draw questions from CommonsenseQA, OpenBookQA, QASC and StrategyQA. Seven open SLMs from the Qwen2.5, Llama-3.x and Phi-4-mini families generate zero-shot CoT traces, which are reformatted into numbered paragraph steps. Experts then mark the earliest faulty step, or none, and categorize the error. The annotations show that many correct answers rest on flawed reasoning. When strong LLMs act as reasoning-aware judges, SLM scores fall noticeably. The judges reliably separate valid from invalid traces but often misplace the exact error, and PRMs trained on math transfer poorly to this domain. The work shows that answer-only accuracy overstates SLM ability. It also provides a public resource for step-level evaluation outside STEM.

Key points

  • ReTraceQA contains 2,421 traces with step-level error annotations. They were filtered from 2,779 balanced instances labeled by 3 PhD-level annotators, with agreement of κ = 0.84.
  • Between 14% and 24% of instances (17.9% on average) reach the correct answer despite at least one significant reasoning error.
  • Reasoning-aware scoring with LLM judges such as o1-mini lowers SLM results by 18.6 points on average and by up to 25%.
  • Hallucinations, meaning factual grounding failures, account for 41.9–62.5% of errors. Logical coherence problems account for 27.9–35.4%.
  • The task follows the ProcessBench formulation, in which a model must output the position of the earliest incorrect step, or −1 when no step contains a mistake. Code and data are available at github.com/SapienzaNLP/ReTraceQA, and the benchmark is English-only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sapienza University of RomeRome · Italy60 works · 10,105 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksReasoning · 1,685 worksQuestion answering · 487 works

Cite

Cite as Francesco Maria Molfese, Luca Moroni, Ciro Porcaro, Simone Conia and Roberto Navigli. 2026. ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1798

@inproceedings{molfese2026retraceqa,
  title = {{ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering}},
  author = {Francesco Maria Molfese, Luca Moroni, Ciro Porcaro, Simone Conia and Roberto Navigli},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1798},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1798}
}

References

30 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 oth… — Phi-4 Technical Report
  2. Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composition. Preprint, arXiv:1910.11473. find
  3. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large language models are zero-shot reasoners. Preprint, arXiv:2205.11916.
  4. J.R. Landis and G.G. Koch. 1977. Measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. find
  5. Sarah Pan, Vladislav Lialin, Sherin Muckatira, and Anna Rumshisky. 2023. Let’s reinforce step by step. Preprint, arXiv:2311.05821. — [arXiv:2311.05821](https://arxiv.org/abs/2311.05821) find
  6. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2023. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5315–5333, Toronto…
  7. Raffaele Pisano and Roberto Navigli. 2026. Process reward models meet planning: Generating precise and scalable datasets for step-level rewards. In Association for Computational Linguistics: ACL 2026, San Diego, California. Association for Computational Linguistics.
  8. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. Preprint, arXiv:2305.20050.
  9. Jiacheng Liu, Ramakanth Pasunuru, Hannaneh Hajishirzi, Yejin Choi, and Asli Celikyilmaz. 2023. Crystal: Introspective reasoners reinforced with self-feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11557–11572, Singapore. Association for Comp… find
  10. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association fo… find
  11. Mistral. 2025. Mistral small 3. https://mistral.ai/ news/mistral-small-3. Accessed: 2025-05-10. ↗
  12. Francesco Maria Molfese, Simone Conia, Riccardo Orlando, and Roberto Navigli. 2024. ZEBRA: Zero-shot example-based retrieval augmentation for commonsense question answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22429–22444, Miami, Florida,… find
  13. Francesco Maria Molfese, Luca Moroni, Luca Gioffré, Alessandro Scirè, Simone Conia, and Roberto Navigli. 2025. Right answer, wrong score: Uncovering the inconsistencies of LLM evaluation in multiple-choice question answering. In Findings of the Association for Computational Linguistics: ACL 2025, pa… find
  14. OpenAI, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, and Akila Welihinda. 2024a. Gpt-4o system card. Preprint, arXiv:2410.21276.
  15. OpenAI, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, and Alex Karpenko and. 2024b. Openai o1 system card. Preprint, arXiv:2412.16720.
  16. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, and Dayiheng Liu and. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  17. Skywork. 2024. Skywork-o1 open series. https://huggingface.co/Skywork. ↗
  18. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  19. Gladys Tyen, Hassan Mansoor, Victor Carbune, Peter Chen, and Tony Mak. 2024. LLMs cannot find reasoning errors, but can correct them given the error location. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13894–13908, Bangkok, Thailand. Association for Computational L…
  20. Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process- and outcome-based feedback. Preprint, arXiv:2211.14275.
  21. Fali Wang, Zhiwei Zhang, Xianren Zhang, Zongyu Wu, Tzuhao Mo, Qiuhao Lu, Wanjing Wang, Rui Li, Junjie Xu, Xianfeng Tang, Qi He, Yao Ma, Ming Huang, and Suhang Wang. 2024a. A comprehensive survey of small language models in the era of large language models: Techniques, enhancements, applications, col…
  22. Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. 2024b. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. Preprint, arXiv:2312.08935.
  23. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903.
  24. Qingchen Yu, Zifan Zheng, Shichao Song, Zhiyu Li, Feiyu Xiong, Bo Tang, and Ding Chen. 2025. xfinder: Large language models as automated evaluators for reliable evaluation. Preprint, arXiv:2405.11874. — [arXiv:2405.11874](https://arxiv.org/abs/2405.11874) find
  25. SYSTEM You are an expert in commonsense question answering. You are given as input a question and a set of choices. First, provide the reasoning process to answer the question. Finally, provide your final answer. find
  26. Zhongshen Zeng, Pengguang Chen, Shu Liu, Haiyun Jiang, and Jiaya Jia. 2024a. Mr-gsm8k: A meta-reasoning benchmark for large language model evaluation. Preprint, arXiv:2312.17080. find
  27. Zhongshen Zeng, Yinhong Liu, Yingjia Wan, Jingyao Li, Pengguang Chen, Jianbo Dai, Yuxuan Yao, Rongwu Xu, Zehan Qi, Wanru Zhao, Linling Shen, Jianqiao Lu, Haochen Tan, Yukang Chen, Hao Zhang, Zhan Shi, Bailin Wang, Zhijiang Guo, and Jiaya Jia. 2024b. Mr-ben: A meta-reasoning benchmark for evaluating… — [arXiv:2406.13975](https://arxiv.org/abs/2406.13975) find
  28. Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. The lessons of developing process reward models in mathematical reasoning. Preprint, arXiv:2501.07301.
  29. Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2024. Processbench: Identifying process errors in mathematical reasoning. Preprint, arXiv:2412.06559.
  30. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/molfese-2026-retraceqa-evaluating-reasoning-traces-small/