Elamur Research⌕
☰
home / papers / 2026 / Multilingual Language Models Encode Script Over Linguistic Structure

Multilingual Language Models Encode Script Over Linguistic Structure

Paper · 2026 · Aastha A K Verma, Anwoy Chatterjee, Mehak Gupta et al.

Summary

Earlier studies found that multilingual LMs contain small groups of neurons that respond selectively to particular languages and can be manipulated to switch the output language. It remained unclear what these units actually capture. This paper asks whether they represent an abstract notion of language or surface cues such as the writing system.

The authors use LAPE, an entropy-based selectivity score, on two kinds of units: MLP neurons and SAE features (SAE-LAPE). They test Llama-3.2-1B, Llama-3-8B, Gemma-2-2B and Gemma-2-9B on languages written in Latin, Cyrillic, Devanagari, Perso-Arabic and logographic scripts. The experiments are:

  • romanizing non-Latin text,
  • shuffling word order,
  • probing units against lang2vec typological features,
  • causal interventions on selected units.

Romanized text activates units almost entirely separate from both the native-script units and the English units, even though the models translate romanized input well. Shuffling barely changes which units are selected. Typological information becomes easier to decode in deeper layers. However, generation relies most on units that stay stable under surface changes.

These results suggest that transliteration alone is a weak bridge for cross-lingual transfer. They may also explain why switching scripts can alter model behaviour, including safety responses.

Key points

  • A "unit" is either an MLP neuron or an SAE feature. Top-K SAEs from EleutherAI are used for Llama and JumpReLU SAEs (Lieberum et al., 2024) for Gemma. Main results are reported for Llama-3.2-1B, with Gemma-2-2B in the appendix and Llama-3-8B and Gemma-2-9B as scale checks.
  • Units are bound to script, not language. Native-script and romanized versions of the same non-Latin language (e.g., Hindi, Chinese) share very few language-associated units. Romanized inputs do not move toward English either, and this pattern persists in the 8B/9B models.
  • Shuffling word order has only a small effect on which units are identified. This points to lexical or orthographic statistics rather than syntax.
  • Typological structure from lang2vec becomes more decodable with depth. Still, no small set of neurons selectively controls it. Causal importance follows robustness to script and word-order perturbations, so the paper finds no single interlingua module.
  • Code is available at github.com/loadthecode0/multilingual-interpretability, together with romanized and shuffled versions of the evaluation sets. The analysis is limited to MLP activations, does not study training dynamics, and covers models of up to 9B parameters.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indian Institute of Technology DelhiNew Delhi · India22 works · 1,036 citations

Also on the byline: Indian Institute of Technology Delhi Abu Dhabi.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksNeuroscience · 165 worksCognitive Neuroscience · 164 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Aastha A K Verma, Anwoy Chatterjee, Mehak Gupta and Tanmoy Chakraborty. 2026. Multilingual Language Models Encode Script Over Linguistic Structure. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1554

@inproceedings{verma2026multilingual,
  title = {{Multilingual Language Models Encode Script Over Linguistic Structure}},
  author = {Aastha A K Verma, Anwoy Chatterjee, Mehak Gupta and Tanmoy Chakraborty},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1554},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1554}
}

References

54 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lyzander Marciano Andrylie, Inaya Rahmanisa, Mahardika Krisna Ihsani, Alfan Farizki Wicaksono, Haryo Akbarianto Wibowo, and Alham Fikri Aji. 2025. Sparse autoencoders can capture language-specific concepts across diverse languages. Preprint, arXiv:2507.11230. — [arXiv:2507.11230](https://arxiv.org/abs/2507.11230) find
  2. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics.
  3. David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Network dissection: Quantifying interpretability of deep visual representations. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 3319–3327. IEEE Com… find
  4. A.C. Baugh and T. Cable. 2002. A History of the English Language. Prentice Hall. find
  5. Clay Beckner, Richard Blythe, Joan Bybee, Morten Christiansen, William Croft, Nick Ellis, John Holland, Jinyun Ke, Diane Larsen-Freeman, and Tom Schoenemann. 2009. Language is a complex adaptive system: Position paper. Language Learning -LANG LEARN, 59:1–26. find
  6. Yonatan Belinkov. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219.
  7. Jannik Brinkmann, Chris Wendler, Christian Bartelt, and Aaron Mueller. 2025. Large language models share representations of latent grammatical concepts across typologically diverse languages. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computat… find
  8. Augusto Buchweitz, Svetlana V Shinkareva, Robert A Mason, Tom M Mitchell, and Marcel Adam Just. 2011. Identifying bilingual semantic neural representations across languages. Brain Lang, 120(3):282– 289. find
  9. Albert Costa and Núria Sebastián-Gallés. 2014. How does the bilingual experience sculpt the brain? Nat Rev Neurosci, 15(5):336–345. find
  10. D. Crystal. 2003. English as a Global Language. Canto (Cambridge University Press). Cambridge University Press. find
  11. Boyi Deng, Yu Wan, Baosong Yang, Yidan Zhang, and Fuli Feng. 2025. Unveiling language-specific features in large language models via sparse autoencoders. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4563–4608, Vienna, Austr…
  12. Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2024. Multilingual jailbreak challenges in large language models. In The Twelfth International Conference on Learning Representations.
  13. Nicholas Evans and Stephen C. Levinson. 2009. The myth of language universals: Language diversity and its importance for cognitive science. Behavioral and Brain Sciences, 32(5):429–448. find
  14. Constanza Fierro, Negar Foroutan, Desmond Elliott, and Anders Søgaard. 2025. How do multilingual language models remember facts? In Findings of the Association for Computational Linguistics: ACL 2025, pages 16052–16106, Vienna, Austria. Association for Computational Linguistics. find
  15. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao,… — The Llama 3 Herd of Models
  16. Daniil Gurgurov, Katharina Trinley, Yusser Al Ghussin, Tanja Baeumel, Josef Van Genabith, and Simon Ostermann. 2025. Language arithmetics: Towards systematic language neuron identification and manipulation. In Proceedings of the 14th International Joint Conference on Natural Language Processing and… find
  17. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743, Hong Kong, Chi… find
  18. Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. 2024. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations. find
  19. Go Inoue, Bashar Alhafni, Nizar Habash, and Timothy Baldwin. 2026. Do diacritics matter? evaluating the impact of Arabic diacritics on tokenization and LLM benchmarks. In Findings of the Association for Computational Linguistics: EACL 2026, pages 426– 442, Rabat, Morocco. Association for Computation…
  20. Tommi Jauhiainen, Heidi Jauhiainen, Tero Alstola, and Krister Lindén. 2019. Language and dialect identification of cuneiform texts. In Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 89–98, Ann Arbor, Michigan. Association for Computational Linguistics. find
  21. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Ass…
  22. Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka, and Yutaka Matsuo. 2024. On the multilingual ability of decoder-based pre-trained language models: Finding and controlling language-specific neurons. In Proceedings of the 2024 Conference of the North American Chapter of the Association…
  23. Sneha Kudugunta, Ankur Bapna, Isaac Caswell, and Orhan Firat. 2019. Investigating multilingual NMT representations at scale. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCN…
  24. Jindřich Libovický, Rudolf Rosa, and Alexander Fraser. 2020. On the language neutrality of pre-trained multilingual representations. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1663–1674, Online. Association for Computational Linguistics.
  25. Jindřich Libovický, Rudolf Rosa, and Alexander Fraser. 2019. How language-neutral is multilingual bert? Preprint, arXiv:1911.03310. — [arXiv:1911.03310](https://arxiv.org/abs/1911.03310) find
  26. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Inte… find
  27. Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational…
  28. Meng Lu, Ruochen Zhang, Carsten Eickhoff, and Ellie Pavlick. 2025. Paths not taken: Understanding and mending the multilingual factual recall pipeline. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15077–15107, Suzhou, China. Association for Computa…
  29. Viorica Marian, Michael Spivey, and Joy Hirsch. 2003. Shared and separate systems in bilingual language processing: Converging evidence from eyetracking and brain imaging. Brain and Language, 86(1):70– 82. Understanding Language. find
  30. Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. 2025. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. In The Thirteenth International Conference on Learning Representations. find
  31. Jérôme Michaud. 2024. A complex systems perspective on language evolution. In The Evolution of Language : Proceedings of the 15th International Conference (EVOLANG XV), The Evolution of Language Conferences, pages 374–382. find
  32. Michele Miozzo, Albert Costa, Mireia Hernández, and Brenda Rapp. 2010. Lexical processing in the bilingual brain: Evidence from grammatical/morphological deficits. Aphasiology, 24(2):262– 287. find
  33. Ibraheem Muhammad Moosa, Mahmud Elahi Akhter, and Ashfia Binte Habib. 2023. Does transliteration help multilingual language modeling? In Findings of the Association for Computational Linguistics: EACL 2023, pages 670–685, Dubrovnik, Croatia. Association for Computational Linguistics. find
  34. Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lingu…
  35. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hof… find
  36. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001, Florence, Italy. Association for Computational Linguistics.
  37. Inaya Rahmanisa, Lyzander Marciano Andrylie, Mahardika Krisna Ihsani, Alfan Farizki Wicaksono, Haryo Akbarianto Wibowo, and Alham Fikri Aji. 2025. Unveiling the influence of amplifying language-specific neurons. In Proceedings of the 14th International Joint Conference on Natural Language Processing… find
  38. Taraka Rama, Lisa Beinborn, and Steffen Eger. 2020. Probing multilingual BERT for genetic and typological signals. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1214–1228, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  39. Leah Estel Reppucci. 2017. Speaking denglish: Exploring the impact of denglish and anglicisms in german culture and identity. find
  40. Alan Saji, Jaavid Aktar Husain, Thanmay Jayakumar, Raj Dabre, Anoop Kunchukuttan, and Ratish Puduppully. 2025. RomanLens: The role of latent Romanization in multilinguality in LLMs. In Findings of the Association for Computational Linguistics: ACL 2025, pages 26410–26429, Vienna, Austria. Associatio… find
  41. Lisa Schut, Yarin Gal, and Sebastian Farquhar. 2025. Do multilingual LLMs think in english? In ICLR 2025 Workshop on Building Trust in Language Models and Applications. find
  42. Wei Shi, Sihang Li, Tao Liang, Mingyang Wan, Guojun Ma, Xiang Wang, and Xiangnan He. 2025. Route sparse autoencoder to interpret large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 6801–6815, Suzhou, China. Association for Computati…
  43. Kenny Smith and Simon Kirby. 2008. Cultural evolution: implications for understanding the human language faculty and its evolution. Philosophical Transactions of the Royal Society B: Biological Sciences, 363(1509):3591–3603. find
  44. Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. Language-specific neurons: The key to multilingual capabilities in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu…
  45. Gemma Team et al. 2024. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118. units, sparse autoencoders, typology, and script effects. find
  46. The Unicode Consortium. 2024. ICU: International components for unicode. Version 78.1. find
  47. Appendix C: Identifying Language-Associated Units with LAPE and SAE-LAPE. Explains the identification frameworks and provides the hyperparameter thresholds used for both neurons and sparse features. find
  48. Sarah Grey Thomason and Terrence Kaufman. 1988. Language Contact, Creolization, and Genetic Linguistics, 1 edition. University of California Press. find
  49. Joseph C. Toscano, Lynn K. Perry, Kathryn L. Mueller, Allison F. Bean, Marcus E. Galle, and Larissa K. Samuelson. 2008. Language as shaped by the brain; the brain as shaped by development. Behavioral and Brain Sciences, 31(5):535–536. find
  50. Katharina A. T. T. Trinley, Toshiki Nakai, Tatiana Anikina, and Tanja Baeumel. 2025. What language(s) does aya-23 think in? how multilinguality affects internal language representations. In Proceedings of the Workshop on Beyond English: Natural Language Processing for all Languages in an Era of Larg… find
  51. R. Wardhaugh and J.M. Fuller. 2014. An Introduction to Sociolinguistics. Blackwell Textbooks in Linguistics. Wiley. find
  52. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in English? on the latent language of multilingual transformers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15366–15394, Bangkok, T…
  53. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120–130, Online. Association for Computational Linguistics.
  54. Zheng Xin Yong, Cristina Menghini, and Stephen Bach. 2023. Low-resource languages jailbreak GPT-4. In Socially Responsible Language Modelling Research.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/verma-2026-multilingual-language-models-encode-script/