Elamur Research⌕
☰
home / papers / 2025 / Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Models

Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Models

Paper · 2025 · Hongchuan Zeng, Senyu Han, Lu Chen et al.

Summary

Multilingual LLMs can carry skills across languages, but how they do this internally is poorly understood. This COLING 2025 study examines neuron activations in BLOOM and LLaMA2. The authors feed a parallel corpus to each model, mean-pool MLP neuron outputs over tokens, concatenate them across layers, and compare samples using cosine-similarity maps. They introduce three metrics: LRDS for language-specific clustering, SADS for meaning-based clustering, and SKLR for the total size of key linguistic regions. Released BLOOM training checkpoints let them follow these patterns over the course of training. Each language turns out to have its own set of key neurons, and switching them off damages that language while leaving the others mostly intact. Sentences that mean the same thing in different languages produce similar activations, which points to a shared semantic space the authors call a "Lingua Franca." This alignment grows stronger with more training and with larger models. These results help explain cross-lingual transfer and suggest where interventions could improve it.

Key points

  • Experiments cover the BLOOM family, using intermediate checkpoints and BLOOM-7b1, and the LLaMA2 family, including LLaMA2-7B. Code is available at github.com/X-LANCE/LinguaFranca.
  • The analysis spans nine languages: English (en), Chinese (zh), French (fr), Spanish (es), Portuguese (pt), Arabic (ar), Vietnamese (vi), Hindi (hi) and Indonesian (id).
  • Key linguistic neurons sit mainly in the first and last layers. With more training and larger scale, they become more concentrated in the early layers, while the overall linguistic regions shrink.
  • In LLaMA2-7B, the key neurons for a language make up a few percent of all neurons, from 4.8% for en up to 16.5% for vi. Deactivating them raises that language's perplexity far more than deactivating a random 17% of neurons. In this condition the model's outputs become degenerate.
  • LRDS in LLaMA2 fluctuates more than in BLOOM, which the authors link to LLaMA2's English-centric training data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China149 works · 16,161 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksNeuroscience · 165 worksCognitive Neuroscience · 164 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Hongchuan Zeng, Senyu Han, Lu Chen and Kai Yu. 2025. Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Models. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zeng2025converging,
  title = {{Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Models}},
  author = {Hongchuan Zeng, Senyu Han, Lu Chen and Kai Yu},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.707/}
}

References

42 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Terra Blevins and Luke Zettlemoyer. 2022. Language contamination helps explain the cross-lingual capabilities of english pretrained models. Preprint, arXiv:2204.08110. find
  2. Eleftheria Briakou, Colin Cherry, and George Foster. 2023. Searching for needles in a haystack: On the role of incidental bilingualism in palm’s translation capability. Preprint, arXiv:2305.10266. find
  3. Nadezhda Chirkova and Vassilina Nikoulina. 2024. Zero-shot cross-lingual transfer in instruction tuning of large language models. Preprint, arXiv:2402.14778. find
  4. Christos Christodouloupoulos and Mark Steedman. 2015. A massively parallel corpus: the bible in 100 languages. Language resources and evaluation, 49:375–395.
  5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. Preprint, arXiv:1911.02116. find
  6. Yiming Cui, Ziqing Yang, and Xin Yao. 2024. Efficient and effective text encoding for chinese llama and alpaca. Preprint, arXiv:2304.08177.
  7. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. Preprint, arXiv:2104.08696. find
  8. Kunbo Ding, Weijie Liu, Yuejian Fang, Weiquan Mao, Zhe Zhao, Tao Zhu, Haoyan Liu, Rong Tian, and Yiren Chen. 2022. A simple and effective method to improve zero-shot cross-lingual transfer learning. Preprint, arXiv:2210.09934. find
  9. Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. 2023. Steerlm: Attribute conditioned sft as an (user-steerable) alternative to rlhf. Preprint, arXiv:2310.05344. find
  10. Jinlan Fu, See-Kiong Ng, and Pengfei Liu. 2022. Polyglot prompt: Multilingual multitask promptraining. Preprint, arXiv:2204.14264. — [arXiv:2204.14264](https://arxiv.org/abs/2204.14264) find
  11. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Ani… find
  12. Pengzhi Gao, Zhongjun He, Hua Wu, and Haifeng Wang. 2024. Towards boosting many-to-many multilingual machine translation with large language models. Preprint, arXiv:2401.05861. — [arXiv:2401.05861](https://arxiv.org/abs/2401.05861) find
  13. Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pa… find
  14. Dan Hendrycks and Kevin Gimpel. 2023. Gaussian error linear units (gelus). Preprint, arXiv:1606.08415. find
  15. Oskar Holmström, Jenny Kunz, and Marco Kuhlmann. 2023. Bridging the resource gap: Exploring the efficacy of English and multilingual LLMs for Swedish. In Proceedings of the Second Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2023), pages 92–110, Tó…
  16. Guanlin Li, Xuechen Zhao, Amir Jafari, Wenhao Shao, Reza Farahbakhsh, and Noel Crespi. 2024. Improving cross-lingual transfer with contrastive negative learning and self-training. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluatio… find
  17. Huiling Li, Jing Qu, Chuansheng Chen, Yanjun Chen, Gui Xue, Lei Zhang, Chengrou Lu, and Leilei Mei. 2019. Lexical learning in a new language leads to neural pattern similarity with word reading in native language. Human Brain Mapping, 40(1):98–109. find
  18. Jindřich Libovický, Rudolf Rosa, and Alexander Fraser. 2020. On the language neutrality of pre-trained multilingual representations. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1663–1674, Online. Association for Computational Linguistics.
  19. Saima Malik-Moraleda, Olessia Jouravlev, Maya Taliaferro, Zachary Mineroff, Theodore Cucu, Kyle Mahowald, Idan A Blank, and Evelina Fedorenko. 2024. Functional characterization of the language network of polyglots and hyperpolyglots with precision fMRI. Cerebral Cortex, 34(3):bhae049. find
  20. Emily S Nichols, Yue Gao, Sofia Fregni, Li Liu, and Marc F Joanisse. 2021. Individual differences in representational similarity of first and second languages in the bilingual brain. Human Brain Mapping, 42(16):5433–5445. find
  21. OpenAI. 2024. Gpt-4 technical report. arXiv:2303.08774.
  22. Preprint, Michel Paradis. 1985. On the representation of two languages in one brain. Language Sciences, 7(1):1– 39. Towards a Neurology of Language. find
  23. Michel Paradis. 2000. Generalizable outcomes of bilingual aphasia research. Folia phoniatrica et logopaed-ica : official organ of the International Association of Logopedics and Phoniatrics (IALP), 52:54–64. find
  24. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001, Florence, Italy. Association for Computational Linguistics.
  25. Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. 2023. Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages. Preprint, arXiv:2310.14799.
  26. Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S. Yu. 2024. Multilingual large language model: A survey of resources, taxonomy and frontiers. Preprint, arXiv:2404.04925. find
  27. Prajit Ramachandran, Barret Zoph, and Quoc V. Le. 2017. Searching for activation functions. Preprint, arXiv:1710.05941. find
  28. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. Preprint, arXiv:1908.10084.
  29. Andy Rosenbaum, Saleh Soltan, Wael Hamza, Yannick Versley, and Markus Boese. 2022. LINGUIST: Language model instruction tuning to generate annotated utterances for intent classification and slot tagging. In Proceedings of the 29th International Conference on Computational Linguistics, pages 218–241,… find
  30. Uri Shaham, Jonathan Herzig, Roee Aharoni, Idan Szpektor, Reut Tsarfaty, and Matan Eyal. 2024. Multilingual instruction tuning with just a pinch of multilinguality. Preprint, arXiv:2401.01854.
  31. Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. 2024. Salmon: Self-alignment with instructable reward models. Preprint, arXiv:2310.05910.
  32. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  33. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention is all you need. Preprint, arXiv:1706.03762.
  34. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in english? on the latent language of multilingual transformers. Preprint, arXiv:2402.10588.
  35. BigScience Workshop. 2023. Bloom: A 176b-parameter open-access multilingual language model. Preprint, arXiv:2211.05100. find
  36. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844,…
  37. Min Xu, Duo Li, and Ping Li. 2021. Brain decoding in multiple languages: Can cross-language brain decoding work? Brain and Language, 215:104922. find
  38. Dongkeun Yoon, Joel Jang, Sungdong Kim, Seungone Kim, Sheikh Shafayat, and Minjoon Seo. 2024. Langbridge: Multilingual reasoning without multilingual supervision. Preprint, arXiv:2401.10695.
  39. Hongchuan Zeng, Hongshen Xu, Lu Chen, and Kai Yu. 2024a. Multilingual brain surgeon: Large language models can be compressed leaving no language behind. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages… find
  40. Jiali Zeng, Fandong Meng, Yongjing Yin, and Jie Zhou. 2024b. Tim: Teaching large language models to translate with comparison. Preprint, arXiv:2307.04408. — [arXiv:2307.04408](https://arxiv.org/abs/2307.04408) find
  41. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trai…
  42. Zhihao Zhang, Jun Zhao, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Unveiling linguistic regions in large language models. Preprint, arXiv:2402.14700.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zeng-2025-converging-lingua-franca-evolution-linguistic/