Elamur Research⌕
☰
home / papers / 2023 / Measuring and Modifying Factual Knowledge in Large Language Models

Measuring and Modifying Factual Knowledge in Large Language Models

Paper · 2023 · Pouya Pezeshkpour

Summary

Fact probing of LLMs usually checks how highly the correct answer ranks in fill-in-the-blank predictions. This treats knowledge as yes/no, depends heavily on prompt wording, and ignores most of the output distribution. Pezeshkpour proposes information-theoretic scores instead. A model is considered to know a fact if adding that fact barely changes its uncertainty about the blank. Knowledge is measured either as the change in entropy or as the KL-divergence between the vocabulary distributions before and after the fact is instilled. For APIs that return only top-k tokens, the distribution is approximated over the union of returned tokens plus an oov bucket.

Facts are instilled in two ways: explicitly, by stating them in the prompt, or implicitly, by fine-tuning on them. In a synthetic test with BERT and T5 on LAMA/T-REx, both new metrics judge knowledge levels much more accurately than ranking. Explicit instillation proves unreliable mostly for location- and language-type relations. The metrics are also applied to factual alignment and hallucination analysis in in-context learning models, offering a cheaper alternative to fine-tuning where that is impractical.

Key points

  • Synthetic accuracy (Table 1), ranking / entropy / KL-divergence: BERT 51.6 / 72.2 / 74.5; T5 30.9 / 66.4 / 67.8. The paper reports the gain as about 30% in one place and over 35% in another.
  • Data: T-REx and LAMA, with 100 T-REx facts randomly sampled per LAMA relation for comparing instillation methods. Models: BERT and T5, plus InstructGPT (text-davinci-003) and FLAN-T5 (XL) for in-context experiments.
  • In InstructGPT and FLAN-T5, roughly 80% of samples still mispredicted after explicit instillation (gold label outside the top 5) involved location or language queries.
  • For relations with a location object, the model showed more knowledge of hallucinated facts than of correctly generated ones. Facts that did not appear in the output generally scored lowest.
  • Code and data are available at github.com/rit-git/lm-know.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Pouya Pezeshkpour. 2023. Measuring and Modifying Factual Knowledge in Large Language Models. arXiv:2306.06264.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.06264 · arXiv 2306.06264

@misc{pezeshkpour2023measuring,
  title = {{Measuring and Modifying Factual Knowledge in Large Language Models}},
  author = {Pouya Pezeshkpour},
  year = {2023},
  doi = {10.48550/arxiv.2306.06264},
  eprint = {2306.06264},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.06264}
}

References

22 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. 2022. A review on language models as knowledge bases. arXiv preprint arXiv:2204.06031.
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  3. Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021. Knowledgeable or educated guess? revisiting language models as knowledge bases. arXiv preprint arXiv:2106.09231.
  4. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. find
  5. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334. find
  6. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  7. OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  8. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
  9. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  10. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. find
  11. Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.12813.
  12. Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. 2023. Crawling the internal knowledge-base of language models. arXiv preprint arXiv:2301.12810.
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  14. Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluati…
  15. Ben Goodrich, Vinay Rao, Peter J Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 166–175. find
  16. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38.
  17. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? arXiv preprint arXiv:1909.01066.
  18. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
  19. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567.
  20. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  21. Neeraj Varshney, Swaroop Mishra, and Chitta Baral. 2022. Investigating selective prediction approaches across several tasks in iid, ood, and adversarial settings. arXiv preprint arXiv:2203.00211. find
  22. Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. arXiv preprint arXiv:2010.05731.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pezeshkpour-2023-measuring-modifying-factual-knowledge-large/