Elamur Research⌕
☰
home / papers / 2025 / Language Models Fail to Introspect About Their Knowledge of Language

Language Models Fail to Introspect About Their Knowledge of Language

Paper · 2025 · Siyuan Song, Jennifer Hu and Kyle Mahowald

Summary

This work tests whether large language models can introspect, meaning whether they can accurately report on their own linguistic knowledge without fine-tuning. The question matters for interpretability. It also matters for linguists who evaluate models with human-style questions, such as asking whether a sentence is grammatical.

The authors examined 21 open-source LLMs on two tasks: judging grammaticality in minimal sentence pairs (Exp. 1) and choosing the better next word after a prefix (Exp. 2). For each item, they computed two scores:

  • a Direct score, from string log probabilities, treated as the ground truth of what the model knows;
  • a Meta score, from the probabilities the model assigns to answering "1" or "2" to an explicit prompt.

Introspection was defined as a model's Meta scores tracking its own Direct scores better than those of a highly similar model, with similarity controlled in a regression.

Meta–Direct alignment grew as models became more similar, but there was no extra "same model" advantage. The authors conclude that prompted judgments carry real information but are separate from the knowledge that drives string probabilities, so they should not be used as a substitute for it.

Key points

  • Model similarity was defined in two ways. The feature-based version ranks pairs as self > seed variant > base/instruct > same family > other. The empirical version is the correlation between two models' Direct scores.
  • Both methods reached high task accuracy. Direct beat Meta for smaller models, while larger models were near ceiling on both. Larger models' Meta scores also predicted probabilities better. Consistency between methods was measured with Cohen's κ.
  • Exp. 2 used four datasets: wikipedia, news, nonsense and randomseq. Models were near ceiling on the first two and responded at chance on the last two, as expected.
  • In the regression, Direct–Direct similarity had a strong effect (p < .0001), but the self pairing showed no advantage. The same family and other pairings actually scored higher relative to self than expected. These results held when only large models were analysed.
  • Code and data are released at github.com/SiyuanSong2004/language-introspection, and all models are available on Huggingface.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Siyuan Song, Jennifer Hu and Kyle Mahowald. 2025. Language Models Fail to Introspect About Their Knowledge of Language. arXiv:2503.07513.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2503.07513 · arXiv 2503.07513

@misc{song2025language,
  title = {{Language Models Fail to Introspect About Their Knowledge of Language}},
  author = {Siyuan Song, Jennifer Hu and Kyle Mahowald},
  year = {2025},
  doi = {10.48550/arxiv.2503.07513},
  eprint = {2503.07513},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.07513}
}

References

48 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI@Meta. Llama 3 Model Card. 2024. URL https://github.com/meta-llama/llama3/blob/ main/MODEL_CARD.md. ↗
  2. Nishant Balepur, Rachel Rudinger, and Jordan Lee Boyd-Graber. Which of these best describes multiple choice evaluation with LLMs? a) forced B) flawed C) fixable D) all of the above. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual…
  3. Christopher C. Berger, Tara C. Dennehy, John A. Bargh, and Ezequiel Morsella. Nisbett and Wilson (1977) Revisited: The Little That We Can Know and Can Tell. Social Cognition, 34(3):167–195, June 2016. ISSN 0278-016X. doi: 10.1521/soco.2016.34.3.167. URL https://doi.org/10.1521/soco.2016.34.3.167. P… — [doi:10.1521/soco.2016.34.3.167](https://doi.org/10.1521/soco.2016.34.3.167) ↗
  4. Jan Betley, Xuchan Bao, Martín Soto, Anna Sztyber-Betley, James Chua, and Owain Evans. Tell me about yourself: LLMs are aware of their learned behaviors. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/ forum?id=IjQ2Jtemzy. ↗
  5. Felix Jedidja Binder, James Chua, Tomek Korbak, Henry Sleight, John Hughes, Robert Long, Ethan Perez, Miles Turpin, and Owain Evans. Looking inward: Language models can learn about themselves by introspection. In The Thirteenth International Conference on Learning Representations, 2025. URL https://… find
  6. Edwin G Boring. A history of introspection. Psychological Bulletin, 50(3):169, 1953. find
  7. Alex Byrne. Introspection. Philosophical Topics, 33(1):79–104, 2005. find
  8. N. Chomsky. Syntactic Structures. The Hague: Mouton, 1957. find
  9. Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, and Prithviraj Sen. A survey of the state of explainable ai for natural language processing. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th Internati…
  10. Vittoria Dentella, Fritz Günther, and Evelina Leivada. Systematic testing of three language models reveals low language accuracy, absence of response stability, and a yes-response bias. Proceedings of the National Academy of Sciences, 120(51):e2309583120, 2023. find
  11. Rudresh Dwivedi, Devam Dave, Het Naik, Smiti Singhal, Rana Omer, Pankesh Patel, Bin Qian, Zhenyu Wen, Tejal Shah, Graham Morgan, et al. Explainable ai (xai): Core ideas, techniques, and solutions. ACM computing surveys, 55(9):1–33, 2023. find
  12. W Nelson Francis and Henry Kucera. Brown corpus manual. Letters to the Editor, 5(2):7, 1979. find
  13. Richard Futrell and Kyle Mahowald. How linguistics learned to stop worrying and love the language models. Behavioral and Brain Sciences, pp. 1–98, 2025. doi: 10.1017/ S0140525X2510112X. find
  14. Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of NAACL-HLT 2019, pp. 32–42, Minneapolis, Minnesota, 2019. Association for Computational Linguistics.
  15. E. Gibson and E. Fedorenko. The need for quantitative methods in syntax and semantics research. Language and Cognitive Processes, 2010. find
  16. Jennifer Hu and Michael Frank. Auxiliary task demands mask the capabilities of smaller language models. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=U5BUzSn4tD. ↗
  17. Jennifer Hu and Roger Levy. Prompting is not a substitute for probability measurements in large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5040–5060, Singapore, December 2023. Assoc…
  18. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. A Systematic Assessment of Syntactic Generalization in Neural Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1725–1744, Online, July 2020. Association for Computation… find
  19. Jennifer Hu, Kyle Mahowald, Gary Lupyan, Anna Ivanova, and Roger Levy. Language models align with human judgments on key grammatical constructions. Proceedings of the National Academy of Sciences, 121(36):e2400917121, 2024. find
  20. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv:2310.06825, 2023. find
  21. Roni Katzir. Why Large Language Models Are Poor Theories of Human Linguistic Cognition: A Reply to Piantadosi. Biolinguistics, 17, 2023. URL https://doi.org/10.5964/bioling. 13153. — [doi:10.5964/bioling](https://doi.org/10.5964/bioling) ↗
  22. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking Cognitive Biases in Large Language Models as Evaluators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 517–545, Bangko… find
  23. Evelina Leivada, Fritz Günther, and Vittoria Dentella. Reply to Hu et al.: Applying different evaluation standards to humans vs. Large Language Models overestimates AI performance. Proceedings of the National Academy of Sciences, 121(36):e2406752121, September 2024. doi: 10.1073/pnas.2406752121. URL… — [doi:10.1073/pnas.2406752121](https://doi.org/10.1073/pnas.2406752121) find
  24. Bo Li, Peng Qi, Bo Liu, Shuai Di, Jingen Liu, Jiquan Pei, Jinfeng Yi, and Bowen Zhou. Trustworthy ai: From principles to practices. ACM Computing Surveys, 55(9):1–46, 2023. find
  25. David A Lieberman. Behaviorism and the mind: A (limited) call for a return to introspection. American Psychologist, 34(4):319, 1979. find
  26. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535, 2016.
  27. Kyle Mahowald. A discerning several thousand judgments: GPT-3 rates the article + adjective + numeral + noun construction. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 265–273, Dubr… find
  28. Kyle Mahowald, Peter Graff, Jeremy Hartman, and Edward Gibson. Snap judgments: A small n acceptability paradigm (snap) for linguistic acceptability judgments. Language, 92 (3):619–635, 2016. find
  29. Rebecca Marvin and Tal Linzen. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1192–1202, 2018. find
  30. Kanishka Misra. minicons: Enabling flexible behavioral and representational analyses of transformer language models. arXiv:2203.13112, 2022. URL https://arxiv.org/abs/ 2203.13112. ↗
  31. Richard E. Nisbett and Timothy D. Wilson. Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84:231–259, 1977. doi: 10.1037/0033-295X.84.3.231. — [doi:10.1037/0033-295x.84.3.231](https://doi.org/10.1037/0033-295x.84.3.231) find
  32. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656, 2024. find
  33. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM Evaluators Recognize and Favor Their Own Generations. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=4NJBV6Wp0h. ↗
  34. Steven Piantadosi. Modern language models refute chomsky’s approach to language. Lingbuzz Preprint, lingbuzz, 7180, 2023. find
  35. Eric Schwitzgebel. The unreliability of naive introspection. Philosophical Review, 117(2): 245–273, 2008. find
  36. Eric Schwitzgebel. Introspection. In Edward N. Zalta and Uri Nodelman (eds.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Fall 2024 edition, 2024. find
  37. BF Skinner. Representations and misrepresentations. Behavioral and Brain Sciences, 7(4): 655–667, 1984. find
  38. Jon Sprouse. A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory. Behavior Research Methods, 43(1):155–167, March 2011. ISSN 1554-3528. doi: 10.3758/s13428-010-0039-7. URL http://www.springerlink.com/index/ 10.3758/s13428-010-0039-7. — [doi:10.3758/s13428-010-0039-7](https://doi.org/10.3758/s13428-010-0039-7) ↗
  39. Jon Sprouse, Carson T Schütze, and Diogo Almeida. A comparison of informal and formal acceptability judgments using a random sample from linguistic inquiry 2001–2010. Lingua, 134:219–248, 2013. find
  40. Leonard Talmy. Introspection as a methodology in linguistics. In Ten lectures on cognitive semantics, pp. 218–262. Brill, 2018. find
  41. Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/blog/qwen2.5/. ↗
  42. Tristan Thrush, Jared Moore, Miguel Monares, Christopher Potts, and Douwe Kiela. I am a Strange Dataset: Metalinguistic Tests for Language Models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume… find
  43. Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. “my answer is C”: First-token probabilities do not match text answers in instruction-tuned language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Ass… — “My Answer is C”: First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models
  44. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the Association for Computational Linguistics, 8, 2020. URL https://doi.org/ 10.1162/tacl_a_00321. Publisher: MIT P…
  45. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In Alex Warstadt, Aaron Mue… find
  46. Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. What do rnn language models learn about filler–gap dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 211–221, 2018.
  47. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… — Transformers: State-of-the-Art Natural Language Processing
  48. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai D… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/song-2025-language-models-fail-introspect-about/