Elamur Research⌕
☰
home / papers / 2025 / Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models

Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models

Paper · 2025 · Nishanth Madhusudhan, Sathwik Tejaswi Madhusudhan, Vikas Yadav et al.

Summary

The paper examines whether large language models can recognise when they should decline to answer, which the authors call Abstention Ability (AA). Earlier methods usually depend on token probabilities, so they cannot be applied to closed commercial systems, and they lack a common benchmark. To address this, the authors propose a fully black-box evaluation that relies on a single generation per query. They also release Abstain-QA, a multiple-choice benchmark in which every question offers an explicit "I Don't Know/None of the Above" choice. Outcomes are scored with a new Answerable-Unanswerable Confusion Matrix (AUCM). They test GPT-3.5, GPT-4, Mistral 7B, Mixtral 8x7B and Mixtral 8x22B under three prompting strategies: strict abstain clauses, verbal confidence thresholds and Chain-of-Thought. Even strong models abstain poorly on reasoning, conceptual and niche-domain questions. Combining strict prompting with CoT improves both abstention and plain QA accuracy, though smaller models gain less. The work offers a reproducible way to compare reliability across open and proprietary LLMs.

Key points

  • Abstain-QA dataset: 2900 English MCQA items, each with five options. Answerable and unanswerable questions are split 50/50. Unanswerable items are created by replacing the correct option with a plausible distractor. Sources are Pop-QA, MMLU and the new Carnatic-QA.
  • Carnatic-QA (CQA): 900 questions from an under-represented domain, Carnatic music, organised as 9 templates × 100 questions.
    • Experts narrowed 930 scraped ragas down to 272: all 72 Melakarta ragas plus 200 Janya ragas, split evenly into popular and unpopular.
    • Data quality was checked by three expert musicians using majority voting.
  • AUCM: crosses the model's action (answered vs. abstained) with question type (answerable vs. unanswerable). It serves as the basis for the abstention metrics.
  • Prompting results:
    • Abstain/Extreme Abstain clauses combined with CoT gave the best setup for GPT-4 Turbo.
    • Five-shot ICL on MMLU helped GPT-4 Turbo but lowered Mistral 7b's abstention scores.
    • CoT reduced the small positional bias in where the correct option appears.
  • Limitations: the benchmark is MCQA-only, monolingual (English) and modest in size. ICL was studied only in a preliminary way.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Nishanth Madhusudhan, Sathwik Tejaswi Madhusudhan, Vikas Yadav and Masoud Hashemi. 2025. Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{madhusudhan2025do,
  title = {{Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models}},
  author = {Nishanth Madhusudhan, Sathwik Tejaswi Madhusudhan, Vikas Yadav and Masoud Hashemi},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.627/}
}

References

32 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when its lying. arXiv preprint arXiv:2304.13734.
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  3. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  4. Yu Gui, Ying Jin, and Zhimei Ren. 2024. Conformal alignment: Knowing when to trust foundation models with guarantees. arXiv preprint arXiv:2405.10301. — [arXiv:2405.10301](https://arxiv.org/abs/2405.10301) find
  5. Sankalp Gulati, Joan Serra, Vignesh Ishwar, Sertan Sen-türk, and Xavier Serra. 2016. Phrase-based rāga recognition using vector space modeling. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 66–70. IEEE. find
  6. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  7. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  8. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611.
  9. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. Unifiedqa: Crossing format boundaries with a single qa system. Preprint, arXiv:2005.00700.
  10. Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. 2024. Semantic entropy probes: Robust and cheap hallucination detection in llms. arXiv preprint arXiv:2406.15927. — [arXiv:2406.15927](https://arxiv.org/abs/2406.15927) find
  11. TM Krishna and Vignesh Ishwar. 2012. Carnatic music: Svara, gamaka, motif and raga identity. In Serra X, Rao P, Murthy H, Bozkurt B, editors. Proceedings of the 2nd CompMusic Workshop; 2012 Jul 12-13; Istanbul, Turkey. find
  12. Barcelona: Universitat Pompeu Fabra; 2012. Universitat Pompeu Fabra. find
  13. Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie. 2023a. Large language models understand and can be enhanced by emotional stimuli. arXiv preprint arXiv:2307.11760.
  14. Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. 2023b. Large language models in finance: A survey. In Proceedings of the Fourth ACM International Conference on AI in Finance, pages 374–382. find
  15. Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958.
  16. McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan,… — GPT-4 Technical Report
  17. Sathwik Tejaswi Madhusudhan and Girish Chowdhary. 2019. Deepsrgm-sequence classification and ranking in indian classical music with deep learning. In Proceedings of the 20th International Society for Music Information Retrieval Conference, pages 533–540. find
  18. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V… find
  19. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… — Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions
  20. S Samsekai Manjabhat, Shashidhar G Koolagudi, KS Rao, and Pravin Bhaskar Ramteke. 2017. Raga and tonic identification in carnatic music. Journal of New Music Research, 46(3):229–245. find
  21. Rajeswari Sridhar and TV Geetha. 2009. Raga identification of carnatic music for music information retrieval. International Journal of recent trends in Engineering, 1(1):571. find
  22. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conferen… find
  23. Christian Tomani, Kamalika Chaudhuri, Ivan Evtimov, Daniel Cremers, and Mark Ibrahim. 2024. Uncertainty-based abstention in llms improves safety and reduces hallucinations. arXiv preprint arXiv:2404.10960. — [arXiv:2404.10960](https://arxiv.org/abs/2404.10960) find
  24. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  25. Dennis Ulmer, Jes Frellsen, and Christian Hardmeier. 2022. Exploring predictive uncertainty and calibration in NLP: A study on the impact of method & data scarcity. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2707–2735, Abu Dhabi, United Arab Emirates. Association… find
  26. Neeraj Varshney and Chitta Baral. 2023. Post-abstention: Towards reliably re-attempting the abstained instances in qa. arXiv preprint arXiv:2305.01812. find
  27. Johannes Welbl, Nelson F Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106.
  28. Lisa Wimmer, Yusuf Sale, Paul Hofman, Bernd Bischl, and Eyke Hüllermeier. 2023. Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures? In Uncertainty in Artificial Intelligence, pages 2282–2292. PMLR. find
  29. Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In The Twelfth International Conference on Learning Representations. find
  30. HanXiang Xu, ShenAo Wang, Ningke Li, Yanjie Zhao, Kai Chen, Kailong Wang, Yang Liu, Ting Yu, and HaoYu Wang. 2024a. Large language models for cyber security: A systematic literature review. arXiv preprint arXiv:2405.04760. — [arXiv:2405.04760](https://arxiv.org/abs/2405.04760) find
  31. Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, and Jing Gao Gao. 2024b. Sayself: Teaching llms to express confidence with self-reflective rationales. arXiv preprint arXiv:2405.20974v. find
  32. Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek F Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. 2024. Benchmarking llms via uncertainty quantification. arXiv preprint arXiv:2401.12794. — [arXiv:2401.12794](https://arxiv.org/abs/2401.12794) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/madhusudhan-2025-do-llms-know-when-not/