Elamur Research⌕
☰
home / papers / 2025 / Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs

Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs

Paper · 2025 · Hexiang Tan, Fei Sun, Sha Liu et al.

Summary

LLM error detectors often treat agreement across sampled answers as a sign of correctness. This paper identifies a blind spot in that assumption: self-consistent errors, where a model gives the same wrong answer, up to semantic equivalence, in its greedy output and in all stochastic samples. The authors define these errors formally and test them on TriviaQA and SciQ with Qwen2.5 and Llama models of several sizes. As models get larger, inconsistent errors become much less frequent, but self-consistent errors hold steady or grow slightly, so scaling alone does not solve them.

The authors compared four families of detectors: token probability, P(True), Semantic Entropy (SE) and hidden-state probes. Every family loses much of its accuracy on self-consistent errors. Consistency-based SE drops to about chance level.

The authors also noticed that different LLMs rarely share the same self-consistent mistakes. Based on this, they propose a cross-model probe:

  • An external verifier LLM reads the original model's answer.
  • A separate probe is trained on the verifier's hidden states.
  • Its score is blended with the original model's probe score.

The method clearly improves detection of these errors without hurting performance on ordinary errors.

Key points

  • How errors are labelled: each wrong greedy answer gets 15 extra samples (T = 0.5). Answers are grouped using the bidirectional NLI entailment test from Kuhn et al. (2023). The error counts as self-consistent only if all 16 answers are equivalent.
  • Evaluation setup: detectors are scored by AUROC on matched CE and IE subsets, which contain equal numbers of negative examples.
  • Models evaluated: Qwen2.5-7b/14b, Llama3.1-8b and Mistral-12b.
  • Main failure: SE falls to roughly random level on CE subsets. In the Llama3.1-8b SciQ setting, its AUROC is about 0.46 on CE versus 0.88 on IE.
  • Cross-model probe details:
    • The final score is a weighted mix of the two probe scores, controlled by λ.
    • The default verifier is Qwen2.5-14B; Llama3.1-70b serves as the verifier when the original model is Qwen2.5-14B itself.
    • Setting λ = 0.5 already gives solid gains without tuning.
  • Choosing a verifier: verifiers from a different model family beat same-family ones, and bigger verifiers help more. Even a 3B verifier still beats the baseline probe.
  • Possible causes (case studies): popular misconceptions found online, such as the claim that aluminum is the lightest structural metal. Another source is confusion between closely related concepts, such as new moon versus full moon. Code: github.com/Tan-Hexiang/Too-Consistent-to-Detect.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations🔬Institute of Computing TechnologyBeijing · China51 works · 3,411 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Hexiang Tan, Fei Sun, Sha Liu, Du Su, Qi Cao, Xin Chen, Jingang Wang, Xunliang Cai, Yuanzhuo Wang, Huawei Shen and Xueqi Cheng. 2025. Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.238

@inproceedings{tan2025too,
  title = {{Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs}},
  author = {Hexiang Tan, Fei Sun, Sha Liu, Du Su, Qi Cao, Xin Chen, Jingang Wang, Xunliang Cai, Yuanzhuo Wang, Huawei Shen and Xueqi Cheng},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.238},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.238}
}

References

32 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amos Azaria and Tom Mitchell. 2023. The internal state of an LLM knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore. Association for Computational Linguistics.
  2. Mohammad Beigi, Ying Shen, Runing Yang, Zihao Lin, Qifan Wang, Ankith Mohan, Jianfeng He, Ming Jin, Chang-Tien Lu, and Lifu Huang. 2024. InternalIn-spector i 2 : Robust confidence estimation in LLMs through internal states. In Findings of the Association for Computational Linguistics: EMNLP 2024, pa… find
  3. Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2023. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations. find
  4. Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. 2024a. INSIDE: LLMs’ internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations. find
  5. Zhiyu Chen, Jing Ma, Xinlu Zhang, Nan Hao, An Yan, Armineh Nourbakhsh, Xianjun Yang, Julian McAuley, Linda Ruth Petzold, and William Yang Wang. 2024b. A survey on large language models for critical societal domains: Finance, healthcare, and law. Transactions on Machine Learning Research. Survey Cert… find
  6. Xuefeng Du, Chaowei Xiao, and Yixuan Li. 2024. Haloscope: Harnessing unlabeled LLM generations for hallucination detection. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  7. Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2024. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Com… find
  8. Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630. find
  9. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst., 43(2).
  10. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611.
  11. Saurav Kadavath, Tom Conerly, and et al. 2022. Language models (mostly) know what they know. Preprint, arXiv:2207.05221. find
  12. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations. find
  13. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023. Inference-time intervention: Eliciting truthful answers from a language model. In Thirty-seventh Conference on Neural Information Processing Systems.
  14. Stephanie C. Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. find
  15. Trans. Mach. Learn. Res., 2022. find
  16. Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024. Generating with confidence: Uncertainty quantification for black-box large language models. Transactions on Machine Learning Research. find
  17. Matéo Mahaut, Laura Aina, Paula Czarnowska, Momchil Hardalov, Thomas Müller, and Lluis Marquez. 2024. Factual confidence of LLMs: on reliability and robustness of current estimators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p… find
  18. Andrey Malinin and Mark Gales. 2021. Uncertainty estimation in autoregressive structured prediction. In The Ninth International Conference on Learning Representations. find
  19. Potsawee Manakul, Adian Liusie, Mark JF Poon, Yun-Sung Chuang, and Philip HS Torr. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1516–1540. find
  20. Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024. Measuring short-form factuality in large language models. Preprint, arXiv:2411.04368.
  21. Johannes Welbl, Nelson F Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106.
  22. Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. In The Twelfth International Conference on Learning Representations. find
  23. Yihao Xue, Kristjan Greenewald, Youssef Mroueh, and Baharan Mirzasoleiman. 2025. Verify when uncertain: Beyond self-consistency in black box hallucination detection. arXiv preprint arXiv:2502.15845. — [arXiv:2502.15845](https://arxiv.org/abs/2502.15845) find
  24. Samuel Marks and Max Tegmark. 2024. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In First Conference on Language Modeling.
  25. An Yang, Baosong Yang, Beichen Zhang, and et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. find
  26. Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. 2025. LLMs know more than they show: On the intrinsic representation of LLM hallucinations. In The Thirteenth International Conference on Learning Representations. find
  27. Jiaxin Zhang, Zhuohang Li, Kamalika Das, Bradley Malin, and Sricharan Kumar. 2023. SAC 3 : Reliable hallucination detection in black-box language models via semantic-aware cross-check consistency. In Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational… find
  28. Weihang Su, Changyue Wang, Qingyao Ai, Yiran Hu, Zhijing Wu, Yujia Zhou, and Yiqun Liu. 2024. Unsupervised real-time hallucination detection based on the internal states of large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14379–14391, Bangkok, Thai…
  29. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conferen… find
  30. Chaojun Wang and Rico Sennrich. 2020. On exposure bias, hallucination and domain shift in neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3544–3552, Online. Association for Computational Linguistics.
  31. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
  32. Derui Zhu, Dingfan Chen, Qing Li, Zongxiong Chen, Lei Ma, Jens Grossklags, and Mario Fritz. 2024. PoLLMgraph: Unraveling hallucinations in large language models via state transition dynamics. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4737–4751, Mexico City, Mexi… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tan-2025-too-consistent-detect-study-self/