Elamur Research⌕
☰
home / papers / 2025 / Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline

Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline

Paper · 2025 · Meng Lu, Ruochen Zhang, Carsten Eickhoff et al.

Summary

Multilingual LLMs often answer the same factual question differently depending on the language, and they tend to do worse outside high-resource languages. This paper uses mechanistic interpretability, mainly the logit lens applied to Llama-3.2-3B, to explain why. The authors built a manually validated dataset of parallel facts in six languages. Their analysis shows the model handles a non-English query in three stages: it retrieves the fact through an English-centric mechanism in the middle layers, then converts the English answer into the target language near the final layers.

Errors come from two places. In some cases the English answer is found but the conversion back to the target language fails. In most cases the English recall step is never properly engaged. The authors propose two steering-vector interventions, neither tied to a specific language or dataset:

  • one strengthens the late translation step,
  • the other, derived from in-context learning runs, reinforces the English recall stage.

Used together, they substantially raise factual accuracy, especially for the weakest languages. This shows that mechanistic findings can unlock multilingual abilities the model already has.

Key points

  • Dataset: 2,862 manually checked (subject, relation, answer) triples, parallel across English, Chinese, Japanese, Korean, French and Spanish. It covers ten relation types, such as currencies, religions and musicians' instruments.
  • Baseline and error breakdown: Only 21.3% of multilingual queries were answered correctly. Among the failures, 21.7% had the correct English answer appear around layer 21 that was then lost in translation. The other 78.3% never retrieved the English answer at all.
  • Translation gap: When explicitly prompted to translate the expected English answer, the model reached 56.1 accuracy, versus 21.3 in factual recall. It can translate, but it does not use this ability fully during recall.
  • Combined interventions: Accuracy rose by up to 37.6 percentage points for the lowest-performing language, with a mean gain of 19.04 points. On held-out tasks the method beat baselines such as explicit translation. Gains were largest for non-Latin-script languages (Chinese, Japanese, Korean).
  • Limitations: Standard 5-shot ICL still outperforms the recall vector. The study covers only one model, five non-English languages and ten relations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 works

Cite

Cite as Meng Lu, Ruochen Zhang, Carsten Eickhoff and Ellie Pavlick. 2025. Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.762

@inproceedings{lu2025paths,
  title = {{Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline}},
  author = {Meng Lu, Ruochen Zhang, Carsten Eickhoff and Ellie Pavlick},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.762},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.762}
}

References

52 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Explosion AI. 2020. spacy: Industrial-strength natural language processing in python. Accessed: 2025-05-20. find
  2. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. Preprint, arXiv:2406.11717.
  3. Nora Belrose. 2023. Diff-in-means concept editing is worst-case optimal: Explaining a result by sam marks and max tegmark. https://blog.eleuther.ai/diff-in-means/. Accessed on: May 20, 2025. ↗
  4. Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Lev McKinney, Igor Ostrovsky, Stella Biderman, and Jacob Steinhardt. 2023. Eliciting latent predictions from transformers with the tuned lens. to appear.
  5. Bilal Chughtai, Alan Cooney, and Neel Nanda. 2024. Summing up the facts: Additive mechanisms behind factual recall in llms. arXiv preprint arXiv:2402.07321. — [arXiv:2402.07321](https://arxiv.org/abs/2402.07321) find
  6. Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Emerging cross-lingual structure in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6022– 6034, Online. Association for Computational Ling…
  7. Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2023. Multilingual jailbreak challenges in large language models. arXiv preprint arXiv:2310.06474.
  8. Clément Dumas, Veniamin Veselovsky, Giovanni Monea, Robert West, and Chris Wendler. 2024. How do llamas process multilingual text? a latent exploration through activation patching. In ICML 2024 Workshop on Mechanistic Interpretability. find
  9. Javier Ferrando and Marta R Costa-jussà. 2024. On the similarity of circuits across languages: a case study on the subject-verb agreement task. arXiv preprint arXiv:2410.06496. find
  10. Constanza Fierro, Negar Foroutan, Desmond Elliott, and Anders Søgaard. 2025. How do multilingual language models remember facts? Preprint, arXiv:2410.14387. find
  11. Weihao Liu, Ning Wu, Wenbiao Ding, Shining Liang, Ming Gong, and Dongmei Zhang. 2025. Selected languages are all you need for cross-lingual truthfulness transfer. Preprint, arXiv:2406.14434. find
  12. Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associations in auto-regressive language models. Preprint, arXiv:2304.14767. find
  13. Samuel Marks and Max Tegmark. 2024a. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. Preprint, arXiv:2310.06824.
  14. Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. 2024. Patchscopes: A unifying framework for inspecting hidden representations of language models. arXiv preprint arXiv:2401.06102. find
  15. Samuel Marks and Max Tegmark. 2024b. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. Preprint, arXiv:2310.06824.
  16. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  17. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610. find
  18. Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. 2023. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. Advances in Neural Information Processing Systems, 36:17643– 17668. find
  19. Roee Hendel, Mor Geva, and Amir Globerson. 2023. In-context learning creates task vectors. Preprint, arXiv:2310.15916.
  20. Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting. Preprint, arXiv:2305.07004.
  21. Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020. X-FACTR: Multilingual factual knowledge retrieval from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5943–5959, Online. Ass… find
  22. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36:41451–41530.
  23. Zheng Wei Lim, Alham Fikri Aji, and Trevor Cohn. 2025. Language-specific latent process hinders cross-lingual performance. Preprint, arXiv:2505.13141. — [arXiv:2505.13141](https://arxiv.org/abs/2505.13141) find
  24. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359–17372. find
  25. Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. 2023. Language models implement simple word2vec-style vector arithmetic. arXiv preprint arXiv:2305.16130. find
  26. Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. 2024. Circuit component reuse across tasks in transformer language models. Preprint, arXiv:2310.08744. find
  27. George A. Miller and the Princeton WordNet Group. 1995. Wordnet: A lexical database for english. Accessed: 2025-05-20.
  28. Benjamin Muller, Yanai Elazar, Benoît Sagot, and Djamé Seddah. 2021. First align, then predict: Understanding the cross-lingual ability of multilingual BERT. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2214–2231,…
  29. Neel Nanda and Joseph Bloom. 2022. Transformerlens. https://github.com/neelnanda-io/ TransformerLens. Accessed: 2025-05-19. ↗
  30. Ercong Nie, Helmut Schmid, and Hinrich Schütze. 2025. Mechanistic understanding and mitigation of language confusion in english-centric large language models. Preprint, arXiv:2505.16538. find
  31. Nostalgebraist. 2020. Interpreting gpt: the logit lens. find
  32. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, and 7 others. 2022. In-con…
  33. Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. 2023. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681. find
  34. Jirui Qi, Raquel Fernández, and Arianna Bisazza. 2023. Cross-lingual consistency of factual knowledge in multilingual language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, page 10650–10666. Association for Computational Linguistics.
  35. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  36. Lisa Schut, Yarin Gal, and Sebastian Farquhar. 2025. Do multilingual llms think in english? arXiv preprint arXiv:2502.15603. — [arXiv:2502.15603](https://arxiv.org/abs/2502.15603) find
  37. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2022. Language models are multilingual chain-of-thought reasoners. Preprint, arXiv:2210.03057.
  38. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in english? on the latent language of multilingual transformers. Preprint, arXiv:2402.10588.
  39. Bryan Wilie, Samuel Cahyawijaya, Junxian He, and Pascale Fung. 2025. High-dimensional interlingual representations of large language models. arXiv preprint arXiv:2503.11280. find
  40. Zhaofeng Wu, Xinyan Velocity Yu, Dani Yogatama, Jiasen Lu, and Yoon Kim. 2024. The semantic hub hypothesis: Language models share semantic representations across languages and modalities. arXiv preprint arXiv:2411.04986. find
  41. Yunzhi Yao, Ningyu Zhang, Zekun Xi, Mengru Wang, Ziwen Xu, Shumin Deng, and Huajun Chen. 2024. Knowledge circuits in pretrained transformers. arXiv preprint arXiv:2405.17969. — [arXiv:2405.17969](https://arxiv.org/abs/2405.17969) find
  42. Suzanna Sia, David Mueller, and Kevin Duh. 2024. Where does in-context translation happen in large language models. Preprint, arXiv:2403.04510. — [arXiv:2403.04510](https://arxiv.org/abs/2403.04510) find
  43. Zheng-Xin Yong, Cristina Menghini, and Stephen H Bach. 2023. Low-resource languages jailbreak gpt-4. arXiv preprint arXiv:2310.02446.
  44. Nishant Subramani, Nivedita Suresh, and Matthew E Peters. 2022. Extracting latent steering vectors from pretrained language models. arXiv preprint arXiv:2205.05124.
  45. Ruochen Zhang, Qinan Yu, Matianyu Zang, Carsten Eickhoff, and Ellie Pavlick. 2024. The same but different: Structural similarities and differences in multilingual language modeling. Preprint, arXiv:2410.09223. find
  46. Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. Language-specific neurons: The key to multilingual capabilities in large language models. arXiv preprint arXiv:2402.16438.
  47. Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024. How do large language models handle multilingualism? Preprint, arXiv:2402.18815.
  48. Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. 2023. Linear representations of sentiment in large language models. Preprint, arXiv:2310.15154. find
  49. Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. 2024. Function vectors in large language models. Preprint, arXiv:2310.15213. find
  50. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248.
  51. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems, volume 33, pages 12388–12401. Curran Associates, Inc. find
  52. Mingyang Wang, Heike Adel, Lukas Lange, Yihong Liu, Ercong Nie, Jannik Strötgen, and Hinrich Schütze. 2025. Lost in multilinguality: Dissecting cross-lingual factual inconsistency in transformer language models. arXiv preprint arXiv:2504.04264.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lu-2025-paths-not-taken-understanding-mending/