Elamur Research⌕
☰
home / papers / 2026 / Knowing When Not to Answer: Lightweight KB-Aligned OOD Detection for Safe RAG

Knowing When Not to Answer: Lightweight KB-Aligned OOD Detection for Safe RAG

Paper · 2026 · Ilias Triantafyllopoulos, Renyi Qu, Salvatore Giorgi et al.

Summary

RAG assistants usually assume every incoming question can be answered from their knowledge base (KB). When a query falls outside that scope, retrieval returns loosely related passages and the LLM may still produce a confident but unsupported answer. This is especially risky in health settings. The authors propose a cheap, always-on gate that decides whether a query belongs to the KB's domain. They embed documents with a BERT-based bi-encoder and run PCA separately for each KB. They then keep a few components, chosen either by explained variance (EVR) or by t-test p-values that rank how well each component separates in-domain from OOD query projections. In this reduced space, they compare geometric neighbourhood rules with simple supervised classifiers. The evaluation covers 16 domains, including real COVID-19 and Substance Use KBs, and stress tests with LLM-generated and real 4chan attacks. The compact detectors rival prompted LLM judges while being faster, nearly free per query, and easier to interpret. Human and LLM-judge evaluations show that OOD queries mainly hurt answer relevance, which argues for an external abstention filter.

Key points

  • The paper splits "when not to answer" into two axes: KB answerability, which this work addresses, and intent/policy risk, which is left to separate moderation layers.
  • Three geometric rules are tested: ϵ-ball, ϵ-cube and ϵ-rect. Each labels a query by majority vote of nearby training queries, and a query with no neighbours defaults to OOD. Three lightweight ML classifiers are also tested.
  • The data spans four datasets and 16 domains, with two high-stakes real-world KBs (COVID-19 and Substance Use). Robustness is checked against synthetic LLM attacks and an in-the-wild 4chan attack set. Experiments use English only.
  • P-value component selection particularly helps the geometric methods. The top-ranked PCs map to coherent themes, such as vaccine eligibility and risk perception in COVID-19, or treatment and concealment queries in Substance Use.
  • Code is available at github.com/chateval/rag_safety_pca. The authors caution that synthetic OOD queries may be easier to separate than real borderline cases.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations🎓University of PennsylvaniaPhiladelphia · United States221 works · 63,234 citations🎓New York UniversityNew York · United States201 works · 100,180 citations

Also on the byline: National Institute on Drug Abuse.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Ilias Triantafyllopoulos, Renyi Qu, Salvatore Giorgi, Brenda Curtis, Lyle Ungar and João Sedoc. 2026. Knowing When Not to Answer: Lightweight KB-Aligned OOD Detection for Safe RAG. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.740

@inproceedings{triantafyllopoulos2026knowing,
  title = {{Knowing When Not to Answer: Lightweight KB-Aligned OOD Detection for Safe RAG}},
  author = {Ilias Triantafyllopoulos, Renyi Qu, Salvatore Giorgi, Brenda Curtis, Lyle Ungar and João Sedoc},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.740},
  url = {https://doi.org/10.18653/v1/2026.acl-long.740}
}

References

55 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yeonchan Ahn, Sang-Goo Lee, Junho Shim, and Jaehui Park. 2022. Retrieval-augmented response generation for knowledge-grounded conversation in the wild. IEEE Access, 10:131374–131385. find
  2. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268.
  3. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. In Proceedings of…
  4. Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, and Mohamed Abdelrazek. 2024. Seven failure points when engineering a retrieval augmented generation system. In Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering-Software Engineering for AI, pages 194– 199. find
  5. Joseph Berkson. 1944. Application of the logistic function to bio-assay. Journal of the American statistical association, 39(227):357–365. find
  6. Deng Cai, Yan Wang, Wei Bi, Zhaopeng Tu, Xiaojiang Liu, and Shuming Shi. 2019. Retrieval-guided dialogue response generation via a matching-to-generation framework. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on… find
  7. Dhivya Chandrasekaran and Vijay Mago. 2021. Evolution of semantic similarity—a survey. ACM Computing Surveys (CSUR), 54(2):1–37. find
  8. Harsh Chaudhari, Giorgio Severi, John Abascal, Anshuman Suri, Matthew Jagielski, Christopher A Choquette-Choo, Milad Nasr, Cristina Nita-Rotaru, and Alina Oprea. 2024. Phantom: General backdoor attacks on retrieval augmented language generation. ACM Transactions on AI Security and Privacy. find
  9. Xin Cheng, Di Luo, Xiuying Chen, Lemao Liu, Dongyan Zhao, and Rui Yan. 2024. Lift yourself up: Retrieval-augmented text generation with self-memory. Advances in Neural Information Processing Systems, 36.
  10. Sukmin Cho, Soyeong Jeong, Jeongyeon Seo, Taeho Hwang, and Jong C. Park. 2024. Typos that broke the RAG‘s back: Genetic attack on RAG pipeline by simulating documents in the wild via low-level perturbations. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2826–2844, M… find
  11. Antonia Creswell, Murray Shanahan, and Irina Higgins. 2022. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712. find
  12. Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The power of noise: Redefining retrieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Developm…
  13. Yi Dong, Ronghui Mu, Gaojie Jin, Yi Qi, Jinwei Hu, Xingyu Zhao, Jie Meng, Wenjie Ruan, and Xiaowei Huang. 2024. Building guardrails for large language models. arXiv preprint arXiv:2402.01822. — [arXiv:2402.01822](https://arxiv.org/abs/2402.01822) find
  14. Salvatore Giorgi, Kelsey Isman, Tingting Liu, Zachary Fried, João Sedoc, and Brenda Curtis. 2024. Evaluating generative ai responses to real-world drug-related questions. Psychiatry Research, 339:116058. find
  15. Shai Gretz, Assaf Toledo, Roni Friedman, Dan Lahav, Rose Weeks, Naor Bar-Zeev, João Sedoc, Pooja Sangha, Yoav Katz, and Noam Slonim. 2023. Benchmark data and evaluation framework for intent discovery around COVID-19 vaccine hesitancy. In Findings of the Association for Computational Linguistics: EAC… find
  16. Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. 1998. Support vector machines. IEEE Intelligent Systems and their applications, 13(4):18–28. find
  17. David W Hosmer Jr, Stanley Lemeshow, and Rodney X Sturdivant. 2013. Applied logistic regression. John Wiley & Sons. find
  18. Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251):1–43.
  19. Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969–7992.
  20. Alene Kennedy-Hendricks, Catherine K Ettman, Sarah E Gollust, Sachini N Bandara, Salma M Abdalla, Brian C Castrucci, and Sandro Galea. 2024. Experience of personal loss due to drug overdose among us adults. In JAMA Health Forum, volume 5, pages e241262–e241262. American Medical Association. find
  21. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213.
  22. Udo Kruschwitz and Maximilian Schmidhuber. 2024. LLM-based synthetic datasets: Applications and limitations in toxicity detection. In Proceedings of the Fourth Workshop on Threat, Aggression & Cyberbullying @ LREC-COLING-2024, pages 37–51, Torino, Italia. ELRA and ICCL. find
  23. Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31. find
  24. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929–3938. PMLR.
  25. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.
  26. Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. ToxiGen: A large-scale machine-generated dataset Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. 2023. Large language models with contro…
  27. Huayang Li, Yixuan Su, Deng Cai, Yan Wang, and Lemao Liu. 2022. A survey on retrieval-augmented text generation. arXiv preprint arXiv:2202.01110. — [arXiv:2202.01110](https://arxiv.org/abs/2202.01110) find
  28. Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. 2017. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.02690. — [arXiv:1706.02690](https://arxiv.org/abs/1706.02690) find
  29. Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7052–7063, Online and Punta Cana, Dominican Re…
  30. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V… find
  31. Peter McCullagh. 2019. Generalized linear models. Routledge. find
  32. Zhiyuan Peng, Jinming Nian, Alexandre Evfimievski, and Yi Fang. 2025. Eloq: Resources for enhancing llm detection of out-of-scope questions. find
  33. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784–3803.
  34. Claire Snell-Rood, Robin A Pollini, and Cathleen Willging. 2021. Barriers to integrated medication-assisted treatment for rural patients with co-occurring disorders: The gap in managing addiction. Psychiatric Services, 72(8):935–942. find
  35. Joe Stacey, Lisa Alazraki, Aran Ubhi, Beyza Ermis, Aaron Mueller, and Marek Rei. 2026. Improving the ood performance of closed-source llms on nli through strategic data selection. In Findings of the Association for Computational Linguistics: EACL 2026, pages 5378–5404.
  36. Substance Abuse and Mental Health Services Administration. 2023. Highlights for the 2022 national survey on drug use and health. https://www.samhsa.gov/data/sites/default/files/reports/ rpt42731/2022-nsduh-main-highlights.pdf. [Accessed 08-01-2025]. ↗
  37. Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. 2022. Out-of-distribution detection with deep nearest neighbors. In International conference on machine learning, pages 20827–20840. PMLR. find
  38. Flax Sentence Embeddings Team. 2021. Stack exchange question pairs. https://huggingface.co/datasets/flax-sentence-embeddings/. ↗
  39. Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11:1316–1331.
  40. Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. 2022. Vim: Out-of-distribution with virtual-logit matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4921–4930. find
  41. Benjamin Reichman and Larry Heck. 2024. Dense passage retrieval: Is it retrieving? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13540–13553, Miami, Florida, USA. Association for Computational Linguistics. find
  42. Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. 2024. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817.
  43. Douglas A Reynolds et al. 2009. Gaussian mixture models. Encyclopedia of biometrics, 741(659-663). find
  44. Jiaqi Xue, Mengxin Zheng, Yebowen Hu, Fei Liu, Xun Chen, and Qian Lou. 2024. Badrag: Identifying vulnerabilities in retrieval augmented generation of large language models. arXiv preprint arXiv:2406.00083. — [arXiv:2406.00083](https://arxiv.org/abs/2406.00083) find
  45. Avital Shafran, Roei Schuster, and Vitaly Shmatikov. 2025. Machine against the {RAG}: Jamming {Retrieval-Augmented} generation with blocker documents. In 34th USENIX Security Symposium (USENIX Security 25), pages 3787–3806. find
  46. Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2023. Making retrieval-augmented language models robust to irrelevant context. arXiv preprint arXiv:2310.01558.
  47. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 1671–1685.
  48. Wenhao Yu, Hongming Zhang, Xiaoman Pan, Peixin Cao, Kaixin Ma, Jian Li, Hongwei Wang, and Dong Yu. 2024. Chain-of-note: Enhancing robustness in retrieval-augmented language models. In Proceedings of the 2024 conference on empirical methods in natural language processing, pages 14672–14685.
  49. Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. 2025. Optimizing generative ai by backpropagating language model feedback. Nature, 639(8055):609–616. find
  50. Cyril Zakka, Rohan Shad, Akash Chaurasia, Alex R Dalal, Jennifer L Kim, Michael Moor, Robyn Fong, Curran Phillips, Kevin Alexander, Euan Ashley, et al. 2024. Almanac—retrieval-augmented language models for clinical medicine. NEJM AI, 1(2):AIoa2300068. find
  51. Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, and Chenliang Li. 2020. Adversarial attacks on deep-learning models in natural language processing: A survey. ACM Transactions on Intelligent Systems and Technology (TIST), 11(3):1–41. find
  52. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2025. Siren’s song in the ai ocean: A survey on hallucination in large language models. Computational Linguistics, 51(4):1373–1418.
  53. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.
  54. Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.
  55. Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. 2025. {PoisonedRAG}: Knowledge corruption attacks to {Retrieval-Augmented} generation of large language models. In 34th USENIX Security Symposium (USENIX Security 25), pages 3827–3844. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/triantafyllopoulos-2026-knowing-when-not-answer-lightweight/