Elamur Research⌕
☰
home / papers / 2026 / Lost in the Mix: Evaluating LLM Understanding of Code-Switched Text

Lost in the Mix: Evaluating LLM Understanding of Code-Switched Text

Paper · 2026 · Amr Mohamed, Yang Zhang, Michalis Vazirgiannis et al.

Summary

The paper asks whether large language models still comprehend and reason well when a text mixes languages (code-switching, CSW). Existing CSW benchmarks mostly test shallow skills such as language identification or POS tagging. The authors instead build mixed-language versions of the established benchmarks Belebele, MMLU and XNLI, covering English, Arabic, German, French and Chinese. A two-step LLM pipeline first marks switch points and then fills them with aligned words from a parallel sentence. One variant replaces only nouns, drawing on two linguistic frameworks: the Matrix Language Frame model and Equivalence Constraint Theory. The other replaces about 20% of tokens at random.

Across several model families, the effect depends on direction:

  • Foreign words in English text reliably lower accuracy.
  • English words in other languages often raise accuracy, sometimes above the monolingual baseline.

Grammatically constrained and random switching hurt about equally. The authors read this as a representational bias toward English rather than a reaction to ungrammatical input.

For mitigation, in-context cues give mixed results, while fine-tuning on CSW data gives small but consistent gains. The findings matter for fair handling of multilingual users and for evaluating non-English-centric models.

Key points

  • Benchmarks and languages: CSW variants of Belebele, MMLU and XNLI, with English, Arabic, German, French and Chinese as matrix or embedded languages. Code and data are released at github.com/amr-mohamedd/Lost-in-the-Mix.
  • Pipeline choice: The LLM-centric generation pipeline was compared with an alignment-first baseline (AWESOME + LaBSE + Stanza POS-guided substitution). Bilingual annotators preferred the LLM outputs, and LLM-as-a-Judge scores agreed.
  • Metrics: accuracy, size-weighted average accuracy across benchmarks, and the accuracy change (ΔAcc) relative to the monolingual baseline.
  • In-context learning: ICL hints sometimes help broadly multilingual models (Qwen, Allam) but fail or backfire for Llama and Mistral.
  • Fine-tuning: Llama 8B was instruction-tuned on about 14.6k noun-token CSW generation examples, producing CSW-Llama 8B.
    • It improved on all embedded languages, most strongly for EN→AR.
    • English-only accuracy stayed stable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓École PolytechniquePalaiseau · France14 works · 898 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Amr Mohamed, Yang Zhang, Michalis Vazirgiannis and Guokan Shang. 2026. Lost in the Mix: Evaluating LLM Understanding of Code-Switched Text. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2080

@inproceedings{mohamed2026lost,
  title = {{Lost in the Mix: Evaluating LLM Understanding of Code-Switched Text}},
  author = {Amr Mohamed, Yang Zhang, Michalis Vazirgiannis and Guokan Shang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2080},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2080}
}

References

48 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gustavo Aguilar, Sudipta Kar, and Thamar Solorio. 2020. LinCE: A centralized benchmark for linguistic code-switching evaluation. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1803–1813, Marseille, France. European Language Resources Association.
  2. Q Jiang Albert, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, and Devendra Singh Chaplot. 2023. Mistral 7b. arXiv. find
  3. Li Nguyen and. 2018. Borrowing or code-switching? traces of community norms in vietnamese-english speech. Australian Journal of Linguistics, 38(4):443– 466. find
  4. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2023. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. arXiv preprint arXiv:2308.16884.
  5. M Saiful Bari, Yazeed Alnumay, Norah A Alzahrani, Nouf M Alotaibi, Hisham A Alyahya, Sultan Al-Rashed, Faisal A Mirza, Shaykhah Z Alsubaie, Hassan A Alahmed, Ghadah Alabduljabbar, et al. 2024. Allam: Large language models for arabic and english. arXiv preprint arXiv:2407.15390.
  6. Barbara E. Bullock and Almeida Jacqueline Toribio. 2009. The Cambridge Handbook of Linguistic Code-switching. Cambridge Handbooks in Language and Linguistics. Cambridge University Press. find
  7. Kelvin Wey Han Chan, Christopher Bryant, Li Nguyen, Andrew Caines, and Zheng Yuan. 2024. Grammatical error correction for code-switched sentences by learners of English. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-CO… find
  8. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  9. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Bel… find
  10. Richeek Das, Sahasra Ranjan, Shreya Pathak, and Preethi Jyothi. 2023. Improving pretraining techniques for code-switched NLP. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1176–1191, Toronto, Canada. Association for Computat… find
  11. Sourya Dipta Das, Ayan Basak, Soumil Mandal, and Dipankar Das. 2022. Advcodemix: Adversarial attack on code-mixed data. In Proceedings of the 5th Joint International Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD), pages 125–129. find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  13. A. Seza Doğruöz, Sunayana Sitaram, Barbara E. Bullock, and Almeida Jacqueline Toribio. 2021. A survey of code-switching: Linguistic and social perspectives for language technologies. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internationa…
  14. Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2112–2128, Online. Association for Computational Linguistics. find
  15. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891, Dublin, Ireland. Association for Computation…
  16. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Ani… find
  17. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  18. Nicolas Guerin, Shane Steinert-Threlkeld, and Emmanuel Chemla. 2024. The impact of syntactic and semantic proximity on machine translation with back-translation. arXiv preprint arXiv:2403.18031. find
  19. Ayushman Gupta, Akhil Bhogal, and Kripabandhu Ghosh. 2024. Code-mixer ya nahi: Novel approaches to measuring multilingual llms’ code-mixing capabilities. arXiv preprint arXiv:2410.11079. — [arXiv:2410.11079](https://arxiv.org/abs/2410.11079) find
  20. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  21. Maite Heredia, Gorka Labaka, Jeremy Barnes, and Aitor Soroa. 2025. Conditioning llms to generate code-switched text: A methodology grounded in naturally occurring data. arXiv preprint arXiv:2502.12924. find
  22. Muhammad Huzaifah, Weihua Zheng, Nattapol Chanpaisit, and Kui Wu. 2024. Evaluating code-switching translation with large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 6381– 6394, To… find
  23. Pranjal Khanuja et al. 2020. Improving code-switched nlp using data augmentation. In Proceedings of ACL 2020, pages 1860–1871. find
  24. Prashant Kodali, Anmol Goel, Likhith Asapu, Vamshi Krishna Bonagiri, Anirudh Govil, Monojit Choudhury, Manish Shrivastava, and Ponnurangam Kumaraguru. 2024. From human judgements to predictive models: Unravelling acceptability in code-mixed sentences. arXiv preprint arXiv:2405.05572.
  25. Garry Kuwanto, Chaitanya Agarwal, Genta Indra Winata, and Derry Tanti Wijaya. 2024. Linguistics theory meets llm: Code-switched text generation via equivalence constrained large language models. arXiv preprint arXiv:2410.22660. find
  26. Zihan Liu, Genta Indra Winata, Zhaojiang Lin, Peng Xu, and Pascale Fung. 2020. Attention-informed mixed-language training for zero-shot cross-lingual task-oriented dialogue systems. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8433– 8440. find
  27. Amr Mohamed, Mingmeng Geng, Michalis Vazirgiannis, and Guokan Shang. 2025. Llm as a broken telephone: Iterative generation distorts information. arXiv preprint arXiv:2502.20258. find
  28. R. Myers-Scotton. 1993. Social Motivations for Code-Switching: Evidence from Africa. Oxford University Press. find
  29. Mark Myslín. 2014. Codeswitching and predictability of meaning in discourse. In Codeswitching and predictability of meaning in discourse. find
  30. Lynnette Hui Xian Ng and Luo Qi Chan. 2024. What talking you?: Translating code-mixed messaging texts to english. arXiv preprint arXiv:2411.05253. — [arXiv:2411.05253](https://arxiv.org/abs/2411.05253) find
  31. Millicent Ochieng, Varun Gumma, Sunayana Sitaram, Jindong Wang, Vishrav Chaudhary, Keshet Ronen, Kalika Bali, and Jacki O’Neill. 2024. Beyond metrics: evaluating llms’ effectiveness in culturally nuanced, low-resource real-world scenarios. arXiv preprint arXiv:2406.00343. find
  32. Tanmay Parekh, Emily Ahn, Yulia Tsvetkov, and Alan W Black. 2020. Understanding linguistic accommodation in code-switched human-machine dialogues. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 565–577, Online. Association for Computational Linguistics. find
  33. Parth Patwa, Gustavo Aguilar, Sudipta Kar, Suraj Pandey, Srinivas PYKL, Björn Gambäck, Tanmoy Chakraborty, Thamar Solorio, and Amitava Das. 2020. SemEval-2020 task 9: Overview of sentiment analysis of code-mixed tweets. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 774–790,…
  34. Susan Poplack. 1978. Sometimes i’ll start a sentence in spanish y termino en español: Toward a typology of code-switching. Linguistics, 16(7-8):581–618. find
  35. Tom Potter and Zheng Yuan. 2024. LLM-based code-switched text generation for grammatical error correction. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16957–16965, Miami, Florida, USA. Association for Computational Linguistics. find
  36. Adithya Pratapa, Gayatri Bhat, Monojit Choudhury, Sunayana Sitaram, Sandipan Dandapat, and Kalika Bali. 2018. Language modeling for code-mixing: The role of linguistic theory based synthetic data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: L…
  37. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101–108, Online.…
  38. Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua…
  39. Ruochen Zhang, Samuel Cahyawijaya, Jan Christian Blaise Cruz, Genta Winata, and Alham Fikri Aji. 2023. Multilingual large language models are not (yet) code-switchers. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12567–12582, Singapore. Association…
  40. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2).
  41. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.
  42. Zhihong Zhu, Xuxin Cheng, Zhiqi Huang, Dongsheng Chen, and Yuexian Zou. 2023. Enhancing code-switching for cross-lingual SLU: A unified view of semantic and grammatical coherence. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7849–7856, Singapore. A… find
  43. Samson Tan and Shafiq Joty. 2021. Code-mixing on sesame street: Dawn of the adversarial polyglots. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3596–3616, Online. Association for Computational… find
  44. Genta Winata et al. 2021a. Multilingual pretrained models are effective for code-switching nlp. In Proceedings of EMNLP 2021, pages 2345–2356. find
  45. Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, and Pascale Fung. 2021b. Language models are few-shot multilingual learners. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 1–15, Punta Cana, Dominican Republic. Association for Computa…
  46. Genta Indra Winata, Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2019. Code-switched language models using neural based synthetic data from parallel sentences. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 271–280, Hong Kong, China. Association… find
  47. Anjali Yadav, Tanya Garg, Matej Klemen, Matej Ulcar, Basant Agarwal, and Marko Robnik Sikonja. 2024. Code-mixed sentiment and hate-speech prediction. arXiv preprint arXiv:2405.12929. — [arXiv:2405.12929](https://arxiv.org/abs/2405.12929) find
  48. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mohamed-2026-lost-mix-evaluating-llm-understanding/