Elamur Research⌕
☰
home / papers / 2025 / Prompt-Guided Internal States for Hallucination Detection of Large Language Models

Prompt-Guided Internal States for Hallucination Detection of Large Language Models

Paper · 2025 · Fujie Zhang, Peiqi Yu, Biao Yi et al.

Summary

Supervised hallucination detectors that read an LLM's hidden representations often fail when used outside the domain they were trained on. Fixing this usually requires costly multi-domain training data. The authors ask whether cross-domain robustness can be achieved with in-domain data alone. Their framework, PRISM, wraps each text in a prompt template that asks the model whether the statement is correct, and only then extracts the internal states. They also give a simple procedure for generating and selecting such prompts. Analyses on LLaMA2-7B-Chat, using PCA projections and variance ratios, show that this prompting makes the truthfulness-related geometry in the embeddings more prominent and more uniform across topics. PRISM is combined with existing probing detectors (MM, SAPLMA) and tested on the True-False and LogicStruct collections, with TruthfulQA as a realistic scenario. Prompt-guided features clearly improve transfer to unseen domains, and on TruthfulQA they also outperform the unsupervised SelfCheckGPT. The result matters because it offers a cheap, data-light way to make probe-based hallucination detection more transferable. Code and data are released on GitHub.

Key points

  • Data: The True-False dataset (Azaria and Mitchell, 2023) has 6 topical sub-datasets, including animals, cities, companies and inventions. LogicStruct (from Bürger et al., 2024) has 24 sub-datasets, covering 6 topics × 4 grammatical forms: affirmative, negated, conjunction and disjunction.
  • Middle-layer features of LLaMA2-7B-Chat (True-False, average accuracy):
    • Without PRISM: MM reaches 0.5166 and SAPLMA 0.5766.
    • With PRISM: PRISM-MM reaches 0.7924 and PRISM-SAPLMA 0.7938.
    • Simply reading the model's Yes/No answer gives only 0.6166.
  • TruthfulQA setup: 790 QA pairs spanning 37 categories, labeled by BLEURT thresholds of 0, 0.25 and 0.5. Detectors were trained on the first 20% and tested on the remaining 80%.
  • TruthfulQA result (AUROC at threshold 0.5): PRISM-SAPLMA scores 0.7248, compared with 0.6654 for SAPLMA and 0.6357 for SelfCheckGPT (GPT-4-based, 20 samples).
  • Limitations: The method requires access to the model's internal states, so a proxy model may be needed. It also ignores other generation signals, such as token probabilities and the output text itself.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Fujie Zhang, Peiqi Yu, Biao Yi, Baolei Zhang, Tong Li and Zheli Liu. 2025. Prompt-Guided Internal States for Hallucination Detection of Large Language Models. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1058

@inproceedings{zhang2025prompt,
  title = {{Prompt-Guided Internal States for Hallucination Detection of Large Language Models}},
  author = {Fujie Zhang, Peiqi Yu, Biao Yi, Baolei Zhang, Tong Li and Zheli Liu},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1058},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1058}
}

References

23 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amos Azaria and Tom Mitchell. 2023. The internal state of an LLM knows when it‘s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 967–976, Singapore. Association for Computational Linguistics.
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  3. Lennart Bürger, Fred A. Hamprecht, and Boaz Nadler. 2024. Truth is universal: Robust detection of lies in LLMs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  4. Sky CH-Wang, Benjamin Van Durme, Jason Eisner, and Chris Kedzie. 2024. Do androids know they‘re only dreaming of electric sheep? In Findings of the Association for Computational Linguistics: ACL 2024, pages 4401–4420, Bangkok, Thailand. Association for Computational Linguistics. find
  5. Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. 2024. INSIDE: LLMs’ internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations. find
  6. Yuyan Chen, Qiang Fu, Yichen Yuan, Zhihao Wen, Ge Fan, Dayiheng Liu, Dongmei Zhang, Zhixu Li, and Yanghua Xiao. 2023. Hallucination detection: Robustly discerning reliable answers in large language models. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Managemen… find
  7. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113.
  8. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powered conversational agents. In International Conference on Learning Representations.
  9. Bairu Hou, Yang Zhang, Jacob Andreas, and Shiyu Chang. 2024. A probabilistic framework for llm hallucination detection via belief tree propagation. arXiv preprint arXiv:2406.06950.
  10. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12).
  11. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. find
  12. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics…
  13. Junteng Liu, Shiqi Chen, Yu Cheng, and Junxian He. 2024. On the universal truthfulness hyperplane inside LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18199–18224, Miami, Florida, USA. Association for Computational Linguistics. find
  14. Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004–9017, Singapore. Association for Computation… find
  15. Samuel Marks and Max Tegmark. 2023. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824.
  16. Ernesto Quevedo, Jorge Yero, Rachel Koerner, Pablo Rivas, and Tomas Cerny. 2024. Detecting hallucinations in large language model generation: A token probability approach. arXiv preprint arXiv:2405.19648. — [arXiv:2405.19648](https://arxiv.org/abs/2405.19648) find
  17. Weihang Su, Changyue Wang, Qingyao Ai, Yiran Hu, Zhijing Wu, Yujia Zhou, and Yiqun Liu. 2024. Unsupervised real-time hallucination detection based on the internal states of large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14379–14391, Bangkok, Thai…
  18. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  19. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  20. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.
  21. Tianhang Zhang, Lin Qiu, Qipeng Guo, Cheng Deng, Yue Zhang, Zheng Zhang, Chenghu Zhou, Xinbing Wang, and Luoyi Fu. 2023. Enhancing uncertainty-based hallucination detection with stronger focus. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 915–932,… find
  22. Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. 2021. Detecting hallucinated content in conditional neural sequence generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1393–1404, Online. As…
  23. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. 2023. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405. *Table 9: The information of the True-False…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2025-prompt-guided-internal-states-hallucination/