Elamur Research⌕
☰
home / papers / 2025 / Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Sampling

Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Sampling

Paper · 2025 · Guangya Wan, Yuqi Wu, Jie Chen et al.

Summary

Self-Consistency (SC) improves LLM reasoning by drawing many chain-of-thought samples and taking a majority vote. It gives no principled way to decide how many samples are enough, and it cannot pick out the most trustworthy explanation. Earlier early-stopping variants, Adaptive Consistency (AC) and ESC, look only at agreement between final answers and give every sample equal weight. RASC (Reasoning-Aware Self-Consistency) also judges the reasoning itself. Each sample is described by hand-crafted answer-level and reasoning-level features. A trained classifier turns these features into a sufficiency score. Samples with high scores go into a buffer, and generation stops once the buffer is full. The final answer comes from score-weighted voting, and the highest-scoring chain of thought serves as the rationale. Across 10 commonsense, mathematical and symbolic reasoning datasets, RASC uses roughly 60–80% fewer samples than standard SC with equal or better accuracy. Its selected explanations are also judged more faithful. This matters wherever LLM inference budgets are tight.

Key points

  • Features: answer-level features are Local-Consistency, Global-Consistency and Parsing-Error. Reasoning-level features are RP-Length, Num-of-Steps, Step-Relevance, Question-Relevance, Error-Admitting, Local-Relevance and Global-Relevance.
  • GSM8K (100 questions, GPT-3.5 Turbo): under zero-shot, few-shot and least-to-most prompting, RASC matched SC's accuracy (69/75/85%). It needed about 5 samples on average, versus 40 for SC and roughly 7–9.5 for AC and ESC.
  • Out-of-distribution tests (BigBench, StrategyQA): RASC reached 0.581 accuracy versus 0.579 for SC, using 5.8 samples instead of 40 (about 14.5%).
  • Rationale quality (200 questions): RASC's selected chains of thought beat SC on BARTScore, CTC and BLURT. Human faithfulness ratings were 4.7 versus 4.0 on a 5-point scale.
  • Limitations: the manually designed features add processing overhead and may miss some aspects of reasoning quality. Human evaluation of faithfulness was limited in scale.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of VirginiaCharlottesville · United States36 works · 9,730 citations🎓University of AlbertaEdmonton · Canada44 works · 3,236 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Guangya Wan, Yuqi Wu, Jie Chen and Sheng Li. 2025. Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Sampling. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.naacl-long.184

@inproceedings{wan2025reasoning,
  title = {{Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Sampling}},
  author = {Guangya Wan, Yuqi Wu, Jie Chen and Sheng Li},
  booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.184},
  url = {https://doi.org/10.18653/v1/2025.naacl-long.184}
}

References

43 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pranjal Aggarwal, Aman Madaan, Yiming Yang, et al. 2023. Let’s sample step by step: Adaptive-consistency for efficient reasoning and coding with llms. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12375–12396. find
  2. Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024a. Large language models for mathematical reasoning: Progresses and challenges. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages…
  3. Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024b. Large language models for mathematical reasoning: Progresses and challenges. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics.
  4. Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. Available online. Accessed: 2024-06-01. find
  5. Fu Bang. 2023. Gptcache: An open-source semantic cache for llm applications enabling faster answers and cost savings. In Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), pages 212–218. find
  6. BIG bench authors. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research.
  7. Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. 2024. Universal self-consistency for large language models. In ICML 2024 Workshop on In-Context Learning. find
  8. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  9. Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. 2021. Compression, transduction, and creation: A unified framework for evaluating natural language generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7580–7605, Online and Pu…
  10. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346– 361. find
  11. Olga Golovneva, Moya Peng Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2023. ROSCOE: A suite of metrics for scoring step-by-step reasoning. In The Eleventh International Conference on Learning Representations.
  12. Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: Re-evaluating factual consistency evaluation. In Proceedings of the 2022 Conference of the North American Chapter of the Associa…
  13. Tom Hosking, Phil Blunsom, and Max Bartolo. 2024. Human feedback is not gold standard. In The Twelfth International Conference on Learning Representations. find
  14. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations.
  15. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst., 43(2).
  16. Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. 2024. The impact of reasoning step length on large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1830–1842, Bangkok, Thailand. Association for Comput…
  17. Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, and Tim Rocktäschel. 2024. Debating with more persuasive llms leads to more truthful answers. In Proceedings of the International Conference on Machine Learning (ICML). find
  18. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213.
  19. Junlong Li, Fan Zhou, Shichao Sun, Yikai Zhang, Hai Zhao, and Pengfei Liu. 2024a. Dissecting human and LLM preferences. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1790– 1811, Bangkok, Thailand. Association for Computation… find
  20. Yiwei Li, Peiwen Yuan, Shaoxiong Feng, Boyuan Pan, Xinglin Wang, Bin Sun, Heda Wang, and Kan Li. 2024b. Escape sky-high cost: Early-stopping self-consistency for multi-step reasoning. In The Twelfth International Conference on Learning Representations. find
  21. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.
  22. Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems. find
  23. Meta. 2024. Meetllama3. Blog post. Available: https://llama.meta.com/llama3/. ↗
  24. Ning Miao, Yee Whye Teh, and Tom Rainforth. 2024. Selfcheck: Using LLMs to zero-shot check their own step-by-step reasoning. In The Twelfth International Conference on Learning Representations. find
  25. OpenAI. 2024. Introducing gpt-3.5 turbo. Blog post. Available: https://platform.openai.com/docs/ models/gpt-3-5-turbo. ↗
  26. Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Sam McCandlish, Sheer El Showk, Tamera Lanham, Tim Maxwell, Venkatesa Chandras… find
  27. Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Proceedings of ACL.
  28. Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. 2023. Can chatgpt replace traditional kbqa models? an in-depth analysis of the question answering performance of the gpt llm family. In International Semantic Web Conference, pages 348–367. Springer. find
  29. Zhengkai Tu, Wei Yang, Zihang Fu, Yuqing Xie, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2020. Approximate nearest neighbor search and lightweight dense vector reranking in multi-stage retrieval architectures. In Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Informat… find
  30. Guangya Wan, Yuqi Wu, Jie Chen, and Sheng Li. 2024a. Cot rerailer: Enhancing the reliability of large language models in complex reasoning tasks through error detection and correction. Preprint, arXiv:2408.13940. — [arXiv:2408.13940](https://arxiv.org/abs/2408.13940) find
  31. Guangya Wan, Yuqi Wu, Mengxuan Hu, Zhixuan Chu, and Sheng Li. 2024b. Bridging causal discovery and large language models: A comprehensive survey of integrative approaches and future directions. Preprint, arXiv:2402.11068. — [arXiv:2402.11068](https://arxiv.org/abs/2402.11068) find
  32. Li Wang, Xi Chen, XiangWen Deng, Hao Wen, MingKe You, WeiZhi Liu, Qi Li, and Jian Li. 2024a. Prompt engineering in consistency and reliability with the evidence-based guideline for llms. npj Digital Medicine, 7(1):41. find
  33. Xinglin Wang, Yiwei Li, Shaoxiong Feng, Peiwen Yuan, Boyuan Pan, Heda Wang, Yao Hu, and Kan Li. 2024b. Integrate the essence and eliminate the dross: Fine-grained self-consistency for free-form language generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lingui… find
  34. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
  35. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
  36. Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2023. Large language models are better reasoners with self-verification. In The 2023 Conference on Empirical Methods in Natural Language Processing.
  37. Yuqi Wu, Kaining Mao, Yanbo Zhang, and Jie Chen. 2024. Callm: Enhancing clinical interview analysis through data augmentation with large language models. IEEE Journal of Biomedical and Health Informatics, 28(12):7531–7542. find
  38. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. In Advances in Neural Information Processing Systems, volume 34, pages 27263–27277. Curran Associates, Inc.
  39. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. STar: Bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems.
  40. Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. 2024a. ReST-MCTS*: LLM self-training via process reward guided tree search. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  41. Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah Smith. 2024b. How language model hallucinations can snowball. In Proceedings of the 41st International Conference on Machine Learning (ICML), Honolulu, Hawaii. PMLR.
  42. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023. Siren’s song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219.
  43. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. • CPU: 16 cores 11th Generation Intel Core i7 Processors Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. In The Ele…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wan-2025-reasoning-aware-self-consistency-leveraging/