Elamur Research⌕
☰
home / papers / 2026 / VecCISC: Improving Confidence-Informed Self-Consistency with Reasoning Trace Clustering and Candidate Answer Selection

VecCISC: Improving Confidence-Informed Self-Consistency with Reasoning Trace Clustering and Candidate Answer Selection

Paper · 2026 · James Petullo, Sonny George, Dylan Cashman et al.

Summary

Self-Consistency samples many answers from an LLM and takes a majority vote. CISC raises accuracy by weighting each vote with a confidence score from a critic LLM. The cost is an extra critic call for every sample, and these calls often use long prompts. VecCISC is designed to cut that cost.

The pipeline works in four steps:

  • It embeds each sampled reasoning trace with OpenAI's text-embedding-3-small.
  • It groups the traces by the final answer they reach.
  • Within each group, it forms up to K clusters using KMeans or HAC.
  • Only the trace closest to each cluster centroid (by cosine similarity) goes to the critic, which returns a verbal confidence between 0 and 1.

These scores are softmax-normalized with a temperature T and then combined in a weighted majority vote. Clustering removes redundant traces and helps isolate degenerate or hallucinated ones.

The authors tested five LLMs on AQuA_Rat, CommonsenseQA, ARC-Challenging, MMLU-Pro and GPQA. VecCISC lowers total token usage by roughly 47% and matches or slightly exceeds CISC accuracy. This makes confidence-weighted "think twice" voting much cheaper at the price of one inexpensive embedding step.

Key points

  • Data: 1000 sampled questions each from AQuA_Rat (train), CommonsenseQA (validation), ARC-Challenging (test) and MMLU-Pro (test), plus all 546 GPQA questions. All prompts were zero-shot, and each (dataset, model) setting was run 10 times.
  • Models: GPT-4o mini, Llama 3.1 8B, Llama 3.3 70B Versatile, Qwen 2.5 7B and Mistral 7B v0.1.
  • Efficiency: total token usage fell by about 47% on average. The HAC variant alone cut tokens spent on critic calls by 31.69%.
  • Accuracy: VecCISC + HAC gave the most consistent results, equal to or above CISC. For example, Qwen2.5 7B on MMLU-Pro reached 60.9 versus 60.2 for CISC. A random-selection ablation that keeps K traces per answer group often lost a lot of accuracy, e.g. Llama3.1 8B on CommonsenseQA fell to about 55–57 versus 77.3.
  • Design choices:
    • K and T were tuned by grid search on holdout data.
    • DBSCAN was rejected because its distance threshold is fragile in high-dimensional embedding spaces.
    • The authors say they will release their code.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksQuestion answering · 441 works

Cite

Cite as James Petullo, Sonny George, Dylan Cashman and Nianwen Xue. 2026. VecCISC: Improving Confidence-Informed Self-Consistency with Reasoning Trace Clustering and Candidate Answer Selection. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1305

@inproceedings{petullo2026veccisc,
  title = {{VecCISC: Improving Confidence-Informed Self-Consistency with Reasoning Trace Clustering and Candidate Answer Selection}},
  author = {James Petullo, Sonny George, Dylan Cashman and Nianwen Xue},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1305},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1305}
}

References

39 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samir Abdaljalil, Hasan Kurban, Parichit Sharma, Erchin Serpedin, and Rachad Atat. 2025. Sindex: Semantic inconsistency index for hallucination detection in llms. find
  2. Pranjal Aggarwal, Aman Madaan, Yiming Yang, and Mausam. 2023. Let’s sample step by step: Adaptive-consistency for efficient reasoning and coding with llms. find
  3. Meta AI. 2024. Llama 3.1 8b: Multilingual instruction-tuned large language model. https://ai.meta.com/blog/meta-llama-3-1/. Model card retrieved: Accessed October 20, 2025. ↗
  4. Mistral AI. 2023. Mistral 7b v0.1: A 7-billion-parameter large language model. https://huggingface.co/mistralai/Mistral-7B-v0.1. Accessed: 2025-10-20. ↗
  5. Jianhao Chen, Zishuo Xun, Bocheng Zhou, Han Qi, Hangfan Zhang, Qiaosheng Zhang, Yang Chen, Wei Hu, Yuzhong Qu, Wanli Ouyang, and Shuyue Hu. 2025. Do we truly need so many samples? multi-llm repeated sampling efficiently scales test-time compute. find
  6. Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou. 2024. Are more llm calls all you need? towards scaling laws of compound inference systems. find
  7. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1.
  8. Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (KDD-96), pages 226–231. AAAI Press. find
  9. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration.
  10. Chengsong Huang, Langlin Huang, Jixuan Leng, Jiacheng Liu, and Jiaxin Huang. 2025. Efficient test-time scaling via self-calibration. find
  11. Anil K. Jain and Richard C. Dubes. 1988. Algorithms for Clustering Data. Prentice Hall. find
  12. Tim Knappe, Ryan Li, Ayush Chauhan, Kaylee Chhua, Kevin Zhu, and Sean O’Brien. 2024. Semantic self-consistency: Enhancing language model reasoning via semantic weighting. find
  13. Sungjae Lee, Hoyoung Kim, Jeongyeon Hwang, Eunhyeok Park, and Jungseul Ok. 2025. Efficient latent semantic clustering for scaling test-time computation of llms. find
  14. Huayang Li, Tian Lan, Zihao Fu, Deng Cai, Lemao Liu, Nigel Collier, Taro Watanabe, and Yixuan Su. 2023. Repetition in repetition out: Towards understanding neural text degeneration from the data perspective. find
  15. Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, and Tat-Seng Chua. 2024a. Think twice before trusting: Self-detection for large language models through comprehensive answer reflection. find
  16. Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, and Tat-Seng Chua. 2024b. Think twice before trusting: Self-detection for large language models through comprehensive answer reflection. find
  17. Yiwei Li, Peiwen Yuan, Shaoxiong Feng, Boyuan Pan, Xinglin Wang, Bin Sun, Heda Wang, and Kan Li. 2024c. Escape sky-high cost: Early-stopping self-consistency for multi-step reasoning. find
  18. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. ACL. find
  19. Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. 2025. Can 1b llm surpass 405b llm? rethinking compute-optimal test-time scaling.
  20. Inc. Meta Platforms. 2024. Llama 3.3 70b versatile. https://huggingface.co/meta-llama/ Llama-3.3-70B-Instruct. Large language model. ↗
  21. Daniel Müllner. 2011. Modern hierarchical, agglomerative clustering algorithms. arXiv preprint arXiv:1109.2378. — [arXiv:1109.2378](https://arxiv.org/abs/1109.2378) find
  22. Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. 2024. Routellm: Learning to route llms with preference data. find
  23. OpenAI. 2024a. Gpt-4o mini: Advancing cost-efficient intelligence. https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence/. Accessed: 2025-10-20. ↗
  24. OpenAI. 2024b. text-embedding-3-small: A highly efficient embedding model. https://platform.openai.com/docs/models/ text-embedding-3-small. Accessed: 2025-10-20. ↗
  25. Shubham Parashar, Blake Olson, Sambhav Khurana, Eric Li, Hongyi Ling, James Caverlee, and Shuiwang Ji. 2025. Inference-time computations for llm reasoning and planning: A benchmark and insights. find
  26. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:282… find
  27. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2024. Qwen2.5 technical report. find
  28. Yutong Wang, Pengliang Ji, Chaoqun Yang, Kaixin Li, Ming Hu, Jiaoyang Li, and Guillaume Sartoretti. 2025. Mcts-judge: Test-time scaling in llm-as-a-judge for code correctness evaluation. find
  29. Junchi Yao, Shu Yang, Jianhua Xu, Lijie Hu, Mengdi Li, and Di Wang. 2025. Understanding the repeat curse in large language models from a feature perspective. find
  30. Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, Irwin King, Xue Liu, and Chen Ma. 2025a. A survey on test-time scaling in large language models: What, how, where, and how well? find
  31. Yiqun Zhang, Hao Li, Chenxu Wang, Linyao Chen, Qiaosheng Zhang, Peng Ye, Shi Feng, Daling Wang, Zhen Wang, Xinrun Wang, Jia Xu, Lei Bai, Wanli Ouyang, and Shuyue Hu. 2025b. The avengers: A simple recipe for uniting smaller language models to challenge proprietary giants. find
  32. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. Gpqa: A graduate-level google-proof q&a benchmark.
  33. Lior Rokach and Oded Maimon. 2005. Clustering Methods. Springer. find
  34. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters.
  35. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  36. Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, and Gal Yona. 2025. Confidence improves self-consistency in llms. find
  37. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models.
  38. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark…
  39. Prompt for Candidate Answer Sampling from the Base LLM (LLM gen ) \\\#\# Instructions Given a multiple-choice question and set of answer options for the question, choose the correct answer from the list of options. \\\#\# Question and Answer Options {question} {options} \\\#\# Output Produce the correct answ… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/petullo-2026-veccisc-improving-confidence-informed-self/