Elamur Research⌕
☰
home / papers / 2025 / Language Mixing in Reasoning Language Models: Patterns, Impact, and Internal Causes

Language Mixing in Reasoning Language Models: Patterns, Impact, and Internal Causes

Paper · 2025 · Mingyang Wang, Lukas Lange, Heike Adel et al.

Summary

Reasoning language models (RLMs) such as DeepSeek-R1 sometimes switch into other languages within their chain of thought. Earlier reports disagreed on whether this switching helps or hurts accuracy. The authors present the first systematic study of the behaviour, asking when it occurs, how it affects results and what drives it internally. They test DeepSeek-R1 and its Qwen- and Llama-based distilled versions, plus QwQ-32B, Qwen3 and Gemini 2.0 Flash Thinking. The benchmarks are the Knights-and-Knaves (K&K) logic puzzles and multilingual MMLU. Mixing is measured as the entropy of the language distribution inside reasoning traces. It is strongest for prompts outside English and Chinese, grows as puzzles get harder, and is highest in STEM subjects. A script-control technique based on constrained decoding shows that forcing reasoning into Latin or Han script can substantially raise accuracy for inputs written in other scripts. A script-level Logit Lens analysis finds that hidden states match the script makeup of the visible reasoning and consistently lean toward Latin. For users of languages other than English and Chinese, the choice of reasoning language is therefore a practical lever for both quality and interpretability.

Key points

  • Coverage: 15 input languages, 7 difficulty levels (K&K puzzles with 2–8 characters) and 18 m-MMLU subjects grouped into Humanities, Social Science, STEM and Other.
  • K&K translation: The original English K&K set was translated with gpt-4o-mini into Arabic, French, Hindi, Japanese and Chinese, giving six languages in total.
  • Script control: Restricting reasoning to Latin or Han script via constrained decoding improved accuracy by up to 110% in some settings. The gains were especially clear for Arabic, Hindi and Japanese inputs.
  • Effect of distillation: For inputs other than English or Chinese, distilled models showed higher mixing entropy than their backbones. The pairs compared were DeepSeek-R1-Distill-Qwen-14B/32B against Qwen2.5-14B/32B, and DeepSeek-R1-Distill-Llama-70B against Llama3.3-70B-Instruct. DeepSeek-R1 itself still mixes languages despite its language consistency reward.
  • Internal causes: Script-level Logit Lens shows that internal representations mirror the script composition of the reasoning traces, with a steady Latin bias. The authors release their data and code.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Ludwig-Maximilians-Universität MünchenMunich · Germany107 works · 56,478 citations🎓Munich Center for Machine LearningMunich · Germany57 works · 1,094 citations🏢Robert BoschStuttgart · Germany11 works · 2,306 citations🎓Karlsruhe University of Applied SciencesKarlsruhe · Germany4 works · 47 citations🎓Stuttgart Media UniversityStuttgart · Germany2 works · 47 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Mingyang Wang, Lukas Lange, Heike Adel, Yunpu Ma, Jannik Strötgen and Hinrich Schuetze. 2025. Language Mixing in Reasoning Language Models: Patterns, Impact, and Internal Causes. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.132

@inproceedings{wang2025language,
  title = {{Language Mixing in Reasoning Language Models: Patterns, Impact, and Internal Causes}},
  author = {Mingyang Wang, Lukas Lange, Heike Adel, Yunpu Ma, Jannik Strötgen and Hinrich Schuetze},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.132},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.132}
}

References

32 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. 2025. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. arXiv preprint arXiv:2503.09567.
  2. Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, and 1 others. 2025. The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks. arXiv preprint arXiv:2502.08235. find
  3. A. Seza Doğruöz, Sunayana Sitaram, Barbara E. Bullock, and Almeida Jacqueline Toribio. 2021. A survey of code-switching: Linguistic and social perspectives for language technologies. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internationa…
  4. Yihong Liu, Mingyang Wang, Amir Hossein Kargaran, Felicia Körner, Ercong Nie, Barbara Plank, François Yvon, and Hinrich Schütze. 2025. Tracing multilingual factual knowledge acquisition in pretraining. arXiv preprint arXiv:2505.14824. find
  5. Clément Dumas, Veniamin Veselovsky, Giovanni Monea, Robert West, and Chris Wendler. 2024. How do llamas process multilingual text? a latent exploration through activation patching. In ICML 2024 Workshop on Mechanistic Interpretability. find
  6. Kelly Marchisio, Wei-Yin Ko, Alexandre Berard, Théo Dehaze, and Sebastian Ruder. 2024. Understanding and mitigating language confusion in LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6653– 6677, Miami, Florida, USA. Association for Computatio…
  7. Google. 2025. Gemini 2.0 flash thinking. find
  8. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  9. Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. 2024. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769. find
  10. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  11. Ercong Nie, Helmut Schmid, and Hinrich Schütze. 2025. Mechanistic understanding and mitigation of language confusion in english-centric large language models. arXiv preprint arXiv:2505.16538. find
  12. Nostalgebraist. 2022. Interpreting gpt: The logit lens. find
  13. OpenAI. 2025. Openai o3 and o4-mini system card. find
  14. QwenTeam. 2024. Qwq: Reflect deeply on the boundaries of the unknown. find
  15. QwenTeam. 2025a. Qwen3: Think deeper, act faster. find
  16. QwenTeam. 2025b. Qwq-32b: Embracing the power of reinforcement learning. find
  17. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720.
  18. Mingyang Wang, Heike Adel, Lukas Lange, Yihong Liu, Ercong Nie, Jannik Strötgen, and Hinrich Schuetze. 2025. Lost in multilinguality: Dissecting cross-lingual factual inconsistency in transformer language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguist…
  19. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, Valencia, Spain. Associatio… find
  20. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in English? on the latent language of multilingual transformers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15366–15394, Bangkok, T…
  21. Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, and Monojit Choudhury. 2020. GLUECoS: An evaluation benchmark for code-switched NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3575–3585, Online. Association for Computati…
  22. Genta Winata, Alham Fikri Aji, Zheng Xin Yong, and Thamar Solorio. 2023. The decades progress on code-switching research in NLP: A systematic survey on trends and challenges. In Findings of the Association for Computational Linguistics: ACL 2023, pages 2936–2978, Toronto, Canada. Association for Com… find
  23. Bin Li, Bin Sun, Shutao Li, Encheng Chen, Hongru Liu, Yixuan Weng, Yongping Bai, and Meiling Hu. 2024a. Distinct but correct: generating diversified and entity-revised medical response. Science China Information Sciences, 67(3):132106. find
  24. Bin Li, Yixuan Weng, Fei Xia, and Hanjun Deng. 2024b. Towards better chinese-centric neural machine translation for low-resource languages. Computer Speech & Language, 84:101566. find
  25. Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. 2024. On memorization of large language models in logical reasoning. arXiv preprint arXiv:2410.23123.
  26. Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. 2025. Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning. Preprint, arXiv:2502.14768.
  27. Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, and 1 others. 2025. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprint arXiv:2501.09686. find
  28. Sikuan Yan, Xiufeng Yang, Zuchao Huang, Ercong Nie, Zifeng Ding, Zonggen Li, Xiaowen Ma, Hinrich Schütze, Volker Tresp, and Yunpu Ma. 2025. Memory-r1: Enhancing large language model agents to manage and utilize memories via reinforcement learning. arXiv preprint arXiv:2508.19828.
  29. Zheng-Xin Yong, M Farid Adilazuarda, Jonibek Mansurov, Ruochen Zhang, Niklas Muennighoff, Carsten Eickhoff, Genta Indra Winata, Julia Kreutzer, Stephen H Bach, and Alham Fikri Aji. 2025. Crosslingual reasoning through test-time scaling. arXiv preprint arXiv:2505.05408. — [arXiv:2505.05408](https://arxiv.org/abs/2505.05408) find
  30. Zheng Xin Yong, Ruochen Zhang, Jessica Forde, Skyler Wang, Arjun Subramonian, Holy Lovenia, Samuel Cahyawijaya, Genta Winata, Lintang Sutawika, Jan Christian Blaise Cruz, Yin Lin Tan, Long Phan, Long Phan, Rowena Garcia, Thamar Solorio, and Alham Fikri Aji. 2023. Prompting multilingual large languag…
  31. Ruochen Zhang, Samuel Cahyawijaya, Jan Christian Blaise Cruz, Genta Winata, and Alham Fikri Aji. 2023. Multilingual large language models are not (yet) code-switchers. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12567–12582, Singapore. Association…
  32. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, and 1 others. 2023. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2025-language-mixing-reasoning-language-models/