Elamur Research⌕
☰
home / papers / 2026 / Boosting Self-Consistency with Ranking

Boosting Self-Consistency with Ranking

Paper · 2026 · Maria Marina, Daniil Moskovskiy, Sergey Pletenev et al.

Summary

Self-consistency samples several reasoning chains from an LLM and picks the most frequent answer. The correct answer is often among the samples but loses the vote, leaving a wide gap to oracle selection. The authors propose RISC (Ranking-Improved Self-Consistency), which treats choosing the final answer as a learning-to-rank problem over the sampled candidate set. Candidates are scored relative to one another rather than in isolation. A LightGBM LambdaRank model, trained with NDCG-weighted pairwise comparisons, combines five interpretable features based on answer frequency, semantic centrality and agreement between reasoning traces. No extra LLM calls or neural reward model are needed. Experiments on PopQA, HotpotQA and MATH500 use generations from Llama-3.1-8B-Instruct and Olmo-3-7B-Instruct. RISC gives a better accuracy-versus-budget curve than majority voting and confidence-based baselines. The gains are large on QA and modest on math. Ablations and SHAP show that the features complement each other, so a learned combination beats any single handcrafted signal. This makes test-time scaling cheaper.

Key points

  • Efficiency on PopQA: RISC with 18 samples already exceeds self-consistency with 99 samples. The headline figure reports roughly 82% fewer LLM calls and accuracy gains of about +4–6.5 points at equal budgets.
  • Baselines: RISC was compared with self-consistency, CISC, Stable Rank and ReASC. It wins on HotpotQA and PopQA at every budget and stays competitive on MATH500.
  • Features: The named features include Answer Length, Ratio to Best, Answer Centroid Dist and Shared Checkpoints. Removing Ratio to Best and Shared Checkpoints together on MATH500 costs 5.7 points, although each alone barely matters.
  • Transfer: A ranker trained on one dataset transfers well between the QA sets but degrades noticeably on MATH500.
  • Code: Released at github.com/s-nlp/RISC. The main limitations are that only small (7–8B) models were tested and a gap to oracle selection remains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓École Polytechnique Fédérale de LausanneLausanne · Switzerland44 works · 7,050 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Maria Marina, Daniil Moskovskiy, Sergey Pletenev, Mikhail Salnikov, Alexander Panchenko and Viktor Moskvoretskii. 2026. Boosting Self-Consistency with Ranking. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-srw.89

@inproceedings{marina2026boosting,
  title = {{Boosting Self-Consistency with Ranking}},
  author = {Maria Marina, Daniil Moskovskiy, Sergey Pletenev, Mikhail Salnikov, Alexander Panchenko and Viktor Moskvoretskii},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-srw.89},
  url = {https://doi.org/10.18653/v1/2026.acl-srw.89}
}

References

36 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Avinash Amballa, Aditya Parashar, Aditya Vikram Singh, Jinlin Lai, and Benjamin Rozonoyer. 2025. Quasi-random Multi-Sample Inference for Large Language Models. In Frontiers in Probabilistic Inference: Learning meets Sampling. find
  2. Christopher J. C. Burges, Robert Ragno, and Quoc Viet Le. 2006. Learning to Rank with Nonsmooth Cost Functions. In Advances in Neural Information Processing Systems 19, Proceedings of the Twentieth Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, Decem… find
  3. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to Answer Open-Domain Questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 -August 4, Volume 1: Long Papers, pages 1870–1879. Assoc…
  4. Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. 2023. Universal Self-Consistency for Large Language Model Generation. Preprint, arXiv:2311.17311.
  5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. CoRR, abs/2110.14168.
  6. Xingyu Dang, Christina Baek, Kaiyue Wen, Zico Kolter, and Aditi Raghunathan. 2025. Weight Ensembling Improves Reasoning in Language Models. CoRR, abs/2504.10478. find
  7. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through rein… find
  8. Kalervo Järvelin and Jaana Kekäläinen. 2017. IR evaluation methods for retrieving highly relevant documents. SIGIR Forum, 51(2):243–250. find
  9. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe…
  10. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others. 2022.… find
  11. Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. 2024. When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs. Trans. Assoc. Comput. Linguistics, 12:1417–1440.
  12. Junseok Kim, Nakyeong Yang, Kyungmin Min, and Kyomin Jung. 2026. Reliability-Aware Adaptive Self-Consistency for Efficient Sampling in LLM Reasoning. CoRR, abs/2601.02970. find
  13. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s Verify Step by Step. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRevi…
  14. Tie-Yan Liu. 2009. Learning to Rank for Information Retrieval. Found. Trends Inf. Retr., 3(3):225–331. find
  15. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V… find
  16. Maria Marina, Nikolay Ivanov, Sergey Pletenev, Mikhail Salnikov, Daria Galimzianova, Nikita Krayko, Vasily Konovalov, Alexander Panchenko, and Viktor Moskvoretskii. 2025. Llm-independent adaptive RAG: let the question speak for itself. In Proceedings of the 2025 Conference on Empirical Methods in Na… find
  17. Marcus J. Min, Yangruibo Ding, Luca Buratti, Saurabh Pujar, Gail E. Kaiser, Suman Jana, and Baishakhi Ray. 2024. Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Aust… find
  18. Viktor Moskvoretskii, Chris Biemann, and Irina Nikishina. 2025. Self-Taught Self-Correction for Small Language Models. CoRR, abs/2503.08681. find
  19. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel J. Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNL…
  20. Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh, and 49 others. 20… — Olmo 3
  21. OpenAI. 2024. OpenAI o1 System Card. abs/2412.16720.
  22. CoRR, Sergey Pletenev, Maria Marina, Nikolay Ivanov, Daria Galimzianova, Nikita Krayko, Mikhail Salnikov, Vasily Konovalov, Alexander Panchenko, and Viktor Moskvoretskii. 2025. Will it still be true tomorrow? multilingual evergreen question classification to improve trustworthy QA. In Proceedings of… find
  23. Yu Shi, Guolin Ke, Damien Soukhavong, James Lamb, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, Tie-Yan Liu, Nikita Titov, and David Cortes. 2026. lightgbm: Light Gradient Boosting Machine. R package version 4.6.0.99. find
  24. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. CoRR, abs/2408.03314.
  25. Richard S. Sutton. 2019. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/ BitterLesson.html. Accessed: 2026-03-15. ↗
  26. Yixuan Tang and Yi Yang. 2025. SR-GRPO: Stable Rank as an Intrinsic Geometric Reward for Large Language Model Alignment. Preprint, arXiv:2512.02807. — [arXiv:2512.02807](https://arxiv.org/abs/2512.02807) find
  27. Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, and Gal Yona. 2025. Confidence Improves Self-Consistency in LLMs. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, volume ACL 2025 of Findings of ACL… find
  28. Llama Team. 2024. The Llama 3 Herd of Models. CoRR, abs/2407.21783.
  29. Weiqin Wang, Yile Wang, and Hui Huang. 2025. Ranked Voting based Self-Consistency of Large Language Models. Preprint, arXiv:2505.10772. find
  30. Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2021. MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pre-trained Transformers. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL-IJCNLP 20… find
  31. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,…
  32. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processi…
  33. Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. 2025. Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.ne… find
  34. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussel…
  35. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023,… find
  36. Zhan Zhuang, Xiequn Wang, Zebin Chen, Feiyang Ye, Ying Wei, Kede Ma, and Yu Zhang. 2026. One-Token Verification for Reasoning Correctness Estimation. CoRR, abs/2603.01025. — [arXiv:2603.01025](https://arxiv.org/abs/2603.01025) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/marina-2026-boosting-self-consistency-ranking/