Elamur Research⌕
☰
home / papers / 2026 / Do Factual Recall Mechanisms Carry over from Text to Speech in Multimodal Language Models?

Do Factual Recall Mechanisms Carry over from Text to Speech in Multimodal Language Models?

Summary

Speech language models built on top of text LLMs can answer factual questions from audio. It is unclear whether they do this by reusing the knowledge-retrieval circuitry of their text backbone. This paper tests that question on SpiritLM, which feeds interleaved HuBERT speech tokens and Llama 2 text tokens into a Llama 2 backbone. The authors apply Causal Tracing, a form of causal mediation analysis, in two settings: text-to-text (T→T) and speech-to-text (S→T). For the speech setting, they synthesized audio for the Known factual-prompt dataset. They then located the subject span inside the audio with CTC forced alignment, so that noise could be injected there and speech tokens mapped back to text tokens.

With text input, SpiritLM shows the same pattern as plain Llama 2: strong causal effects concentrated on subject tokens in mid-layer MLPs, plus effects at the final token. With speech input, the effects are much weaker and more spread out, though still visible around the subject. This suggests that fact-recall mechanisms transfer from text to speech only partially. The finding matters for understanding, editing and improving speech-enabled LLMs.

Key points

  • Model studied: SpiritLM, which uses HuBERT for speech tokenization, a Llama 2 backbone with "T"/"S" modality tokens, and HiFi-GAN for speech decoding. A text-only Llama 2 run serves as the baseline.
  • Data: The Known dataset (Meng et al., 2022), roughly 1000 factual prompts, was converted to speech with MeloTTS.
    • Audio quality was checked by manual inspection and by Whisper-small transcription, which gave a WER of about 19%.
    • The prompts were then filtered into two subsets, Known-t2t and Known-s2t, keeping only items the model answers correctly in each modality.
  • Method: Causal Tracing uses clean, corrupted (noised subject) and restored runs. It measures the Average Indirect Effect (AIE) of hidden states, MLP and attention layers, grouped by token position. CTC-based forced alignment locates the subject in the speech input.
  • Results:
    • T→T in SpiritLM matches the Llama 2 baseline, indicating that its text pathways survived speech fine-tuning.
    • S→T shows a drastically lower and more diffuse AIE, with only a residual effect near the subject tokens.
  • Interpretation and limits: The authors hypothesize a semantic gap between the modalities: representations may align in direction but differ in magnitude.
    • The study covers only one dataset and one model, and discrete speech tokens may limit semantic understanding.
    • No new models or datasets are released.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.starsem-conference.28

@inproceedings{modica2026do,
  title = {{Do Factual Recall Mechanisms Carry over from Text to Speech in Multimodal Language Models?}},
  author = {Modica, Luca and Landin, Filip and Farahani, Mehrdad and Qian, Livia and Skantze, Gabriel and Johansson, Richard},
  booktitle = {Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.starsem-conference.28},
  url = {https://doi.org/10.18653/v1/2026.starsem-conference.28}
}

References

35 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samyadeep Basu, Martin Grayson, Cecily Morrison, Besmira Nushi, Soheil Feizi, and Daniela Massiceti. 2024. Understanding information storage and transfer in multi-modal large language models. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red… find
  2. Santiago Cuervo, Skyler Seto, Maureen de Seyssel, Richard He Bai, Zijin Gu, Tatiana Likhomanenko, Navdeep Jaitly, and Zakaria Aldeneh. 2026. Closing the gap between text and speech understanding in LLMs. In Proceedings of the The Fourteenth International Conference on Learning Representations, Rio d… find
  3. Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12216–12235, Singapore. Association for Computational… find
  4. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational… find
  5. Neta Glazer, Yael Segal-Feldman, Hilit Segev, Aviv Shamsian, Asaf Buchnick, Gill Hetz, Ethan Fetaya, Joseph Keshet, and Aviv Navon. 2026. Beyond transcription: Mechanistic interpretability in asr. In The Fortieth AAAI Conference on Artificial Intelligence (AAAI-26), Singapore. find
  6. Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Defossez, Gabriel Synnaeve, Emmanuel Dupoux, Roy Schwartz, and Yossi Adi. 2023. Textually pre-trained speech language models. In Proceedings of the 37th International Conference on Neural Informati… find
  7. Stefan Heimersheim and Neel Nanda. 2024. How to use and interpret activation patching. arXiv preprint arXiv:2404.15255. — [arXiv:2404.15255](https://arxiv.org/abs/2404.15255) find
  8. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 29:3451–3460. find our card
  9. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  10. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In Proceedings of the 40th International Conference on Machine Learning, ICML’23, Honolulu, Hawaii, USA. JMLR.org. find our card
  11. Jaehyeon Kim, Jungil Kong, and Juhee Son. 2021. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In Proceedings of the Thirty-Eighth International Conference on Machine Learning. find
  12. Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. HiFi-GAN: generative adversarial networks for efficient and high fidelity speech synthesis. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc. find
  13. Jungil Kong, Jihoon Park, Beomjeong Kim, Jeongmin Kim, Dohee Kong, and Sangjin Kim. 2023. Vits2: Improving quality and efficiency of single-stage text-to-speech with adversarial learning and architecture design. In Proceedings of INTERSPEECH, Dublin, Ireland. find
  14. Ludwig Kürzinger, Dominik Winkelbauer, Lujun Li, Tobias Watzel, and Gerhard Rigoll. 2020. CTC-Segmentation of Large Corpora for German End-to-End Speech Recognition, page 267–278. Springer International Publishing. find
  15. Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9… find
  16. Chia-Hsuan Li, Szu-Lin Wu, Chi-Liang Liu, and Hung yi Lee. 2018. Spoken SQuAD: A study of mitigating the impact of speech recognition errors on listening comprehension. In Proceedings of INTERSPEECH, Hyderabad, India. find
  17. Zihao Lin, Samyadeep Basu, Mohammad Beigi, Varun Manjunatha, Ryan A. Rossi, Zichao Wang, Yufan Zhou, Sriram Balasubramanian, Arman Zarei, Keivan Rezaei, Ying Shen, Barry Menglong Yao, Zhiyang Xu, Qin Liu, Yuxiang Zhang, Yan Sun, Shilong Liu, Li Shen, Hongxuan Li, and 2 others. 2025. A survey on mech… — [arXiv:2502.17516](https://arxiv.org/abs/2502.17516) find
  18. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V… find
  19. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, volume 35, pages 17359–17372. Curran Associates, Inc. find
  20. Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2023. Mass editing memory in a transformer. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR), Kigali, Rwanda. find
  21. Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino, Benoît Sagot, and Emmanuel Dupoux. 2025. SpiRit-LM: Interleaved spoken and… find
  22. Ankita Pasad, Chung-Ming Chien, Shane Settle, and Karen Livescu. 2024. What do self-supervised speech models know about words? Transactions of the Association for Computational Linguistics, 12:372–391. find
  23. Ankita Pasad, Ju-Chieh Chou, and Karen Livescu. 2021. Layer-wise analysis of a self-supervised speech representation model. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 914–921. find
  24. Judea Pearl. 2001. Direct and indirect effects. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, UAI’01, page 411–420, Seattle, Washington. Morgan Kaufmann Publishers Inc. find
  25. Jing Peng, Yucheng Wang, Yu Xi, Xu Li, Xizhuo Zhang, and Kai Yu. 2026. A survey on speech large language models. IEEE Journal of Selected Topics in Signal Processing, 20(1). find
  26. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural… find our card
  27. Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, Hannah Muckenhirn, Dirk Padfield, James Qin, Danny Rozenberg, Tara Sainath, Johan Schalkwyk, Matt Sharifi, Michelle Tadmor Rama… find our card
  28. Bajian Xiang, Shuaijiang Zhao, Tingwei Guo, and Wei Zou. 2025. Understanding the modality gap: An empirical study on the speech-text alignment mechanism of large speech language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 5187–5202, Suzhou… find
  29. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. find our card
  30. Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023. SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15757–15773, Singapore. Associa… find our card
  31. Wenliang Zhao, Xumin Yu, and Zengyi Qin. 2023. MeloTTS: High-quality multi-lingual multi-accent text-to-speech. find
  32. Gaofei Shen, Michaela Watkins, Afra Alishahi, Arianna Bisazza, and Grzegorz Chrupała. 2024. Encoding of lexical tone in self-supervised models of spoken language. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech… find
  33. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2024. SALMONN: Towards generic hearing abilities for large language models. In Proceedings of the Twelfth International Conference on Learning Representations, Vienna, Austria. find our card
  34. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber. 2020. Causal mediation analysis for interpreting neural NLP: The case of gender bias. Preprint, arXiv:2004.12265. — [arXiv:2004.12265](https://arxiv.org/abs/2004.12265) find
  35. Dingdong Wang, Junan Li, Mingyu Cui, Dongchao Yang, Xueyuan Chen, and Helen M. Meng. 2025. Speech discrete tokens or continuous features? A comparative analysis for spoken language understanding in SpeechLLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/modica-2026-do-factual-recall-mechanisms-carry/