Elamur Research⌕
☰
home / papers / 2026 / OLA: Output Language Alignment in Code-Switched LLM Interactions

OLA: Output Language Alignment in Code-Switched LLM Interactions

Paper · 2026 · Juhyun Oh, Haneul Yoo, Faiz Ghifari Haznitrama et al.

Summary

Multilingual users often mix languages in prompts without saying which language they want the answer in. Humans infer this easily, but LLMs often reply in the wrong language. The authors introduce OLA, a Korean–English benchmark with two settings:

  • Simple: intra-sentential noun-phrase mixing, where the expected reply language is the matrix language.
  • Complex: the instruction and the content are in different languages, so the right language depends on what the task means.

Prompts were synthesized with GPT-4o from monolingual seeds and checked by native-speaker annotators. Even frontier models fail often. They show a consistent bias toward answering in the non-English language, and the same bias appears for Chinese and Indonesian pairs. Errors also include language switches partway through a response. Chain-of-Thought prompting tends to make results worse rather than better.

The authors' fix, Code-Switching Aware DPO (CS-DPO), uses about 1K synthetic preference pairs. It raises alignment noticeably and transfers to unseen pairs and to the Complex setting. This suggests the failures come from missing alignment, not from a basic limitation of the models. The work matters for any bilingual user base where mixing languages is routine.

Key points

  • How often users state the language: in 335 Korean–English code-switched conversations from WildChat-4.8M, only 45 (about 13%) named the desired output language, and many of those were translation requests.

  • Simple-setting construction:

    • 299 seed prompts: 199 English from the Language Confusion Benchmark and 100 Korean from WildChat-1M.
    • Code-switching was done by noun-phrase insertion that keeps the matrix-language word order.
    • Six native annotators (3 English, 3 Korean) labeled the expected language and how severe a wrong-language reply would be.
    • After filtering for agreement, 279 KO Matrix and 258 EN Matrix prompts remained.
    • Over 92% of retained prompts were rated at least "Uncomfortable" in severity (the extracted text is cut off at this point).

  • CS-DPO training: 1,079 preference pairs built from LIMA, trained on top of OLMo 3.1 DPO.

  • Pass-rate changes (Table 3):

    SettingBeforeAfter
    EN Matrix–KO Embed57.36%66.28%
    EN Matrix–ZH Embed (unseen pair)32.47%49.07%
    KO Instr–EN Content (Complex)77.84%83.96%
    EN Instr–KO Content (Complex)47.90%43.77%

    The body text gives slightly different numbers for the first two rows: 54.27→62.11 and 47.97. The EN Instr–KO Content case is the one setting that got worse.

  • No loss of general ability: KMMLU accuracy changed by −1.1%p, which is not significant (p = 0.63). There was also no blanket shift toward preferring English.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Korea Advanced Institute of Science and TechnologyDaejeon · South Korea51 works · 3,167 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Juhyun Oh, Haneul Yoo, Faiz Ghifari Haznitrama and Alice Oh. 2026. OLA: Output Language Alignment in Code-Switched LLM Interactions. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2162

@inproceedings{oh2026ola,
  title = {{OLA: Output Language Alignment in Code-Switched LLM Interactions}},
  author = {Juhyun Oh, Haneul Yoo, Faiz Ghifari Haznitrama and Alice Oh},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2162},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2162}
}

References

42 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Peter Auer. 1998. Code-switching in conversation. Routledge, London, England. find
  2. Niyati Bafna, Tianjian Li, Kenton Murray, David R. Mortensen, David Yarowsky, Hale Sirin, and Daniel Khashabi. 2025. The translation barrier hypothesis: Multilingual generation with large language models suffers from implicit translation failure. arXiv preprint arXiv:2506.22724.
  3. Anshul Bawa, Pranav Khadpe, Pratik Joshi, Kalika Bali, and Monojit Choudhury. 2020. Do multilingual users prefer chat-bots that code-mix? let’s nudge and find out! Proc. ACM Hum.-Comput. Interact., 4(CSCW1). find
  4. Yik Siu Chan, Zheng-Xin Yong, and Stephen H. Bach. 2025. Can we predict alignment before models finish thinking? towards monitoring misaligned reasoning models. arXiv preprint arXiv:2507.12428. — [arXiv:2507.12428](https://arxiv.org/abs/2507.12428) find
  5. Liang Chen, Shuming Ma, Dongdong Zhang, Furu Wei, and Baobao Chang. 2023. On the off-target problem of zero-shot multilingual neural machine translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9542–9558, Toronto, Canada. Association for Computational Linguistic… find
  6. Yunjae J. Choi, Minha Lee, and Sangsu Lee. 2023. Toward a multilingual conversational agent: Challenges and expectations of code-mixing multilingual users. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, New York, NY, USA. Association for Computing Machinery… find
  7. A. Seza Doğruöz, Sunayana Sitaram, Barbara E. Bullock, and Almeida Jacqueline Toribio. 2021. A survey of code-switching: Linguistic and social perspectives for language technologies. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internationa…
  8. Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, and Mikel Artetxe. 2024. Do multilingual language models think better in English? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume…
  9. Google. 2025. Gemini-3-Pro model card. Accessed: 2026-01-01. find
  10. Emily Halpin and Gigliana Melzi. 2021. Code-switching in the narratives of dual-language Latino preschoolers. International Journal of Bilingual Education and Bilingualism, 24(9):1271–1287. find
  11. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, Valencia, Spain. Associatio… find
  12. Haeji Jung, Changdae Oh, Jooeon Kang, Jimin Sohn, Kyungwoo Song, Jinkyu Kim, and David R Mortensen. 2024. Mitigating the linguistic gap with phonemic representations for robust cross-lingual transfer. In Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024), pages 200… find
  13. Seoyeon Kim, Huiseo Kim, Chanjun Park, Jinyoung Yeo, and Dongha Lee. 2025. Can code-switched texts activate a knowledge switch in LLMs? a case study on English-Korean code-switching. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 22303–22327, Suzhou, China. Associati… find
  14. Garry Kuwanto, Chaitanya Agarwal, Genta Indra Winata, and Derry Tanti Wijaya. 2024. Linguistics theory meets LLM: Code-switched text generation via equivalence constrained large language models. arXiv preprint arXiv:2410.22660. find
  15. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Validad Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Christopher Wilhelm, Luca Soldaini,… — Tulu 3: Pushing Frontiers in Open Language Model Post-Training
  16. Yihao Li, Jiayi Xin, Miranda Muqing Miao, Qi Long, and Lyle Ungar. 2025. The impact of language mixing on bilingual LLM reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 32519–32536, Suzhou, China. Association for Computational Linguistics. find
  17. John M Lipski. 2014. Spanish-English code-switching among low-fluency bilinguals: Towards an expanded typology. Sociolinguistic Studies, 8(1):23–55. find
  18. Kelly Marchisio, Wei-Yin Ko, Alexandre Berard, Théo Dehaze, and Sebastian Ruder. 2024. Understanding and mitigating language confusion in LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6653– 6677, Miami, Florida, USA. Association for Computatio…
  19. Carol Myers-Scotton. 1997. Duelling languages: Grammatical structure in codeswitching. Oxford University Press. find
  20. Ercong Nie, Helmut Schmid, and Hinrich Schuetze. 2025. Mechanistic understanding and mitigation of language confusion in English-centric large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 690–706, Suzhou, China. Association for Computational Lingui… find
  21. Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh, and 49 others. 20… — Olmo 3
  22. OpenAI. 2025. GPT-5 system card. Accessed: 2026-01-01. find
  23. OpenAI, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, and 400 others. 2024. GPT-4o sys…
  24. Debjit Paul, Robert West, Antoine Bosselut, and Boi Faltings. 2024. Making reasoning matter: Measuring and improving faithfulness of chain-of-thought reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15012– 15032, Miami, Florida, USA. Association for Computat…
  25. Liwei Wu, Shanbo Cheng, Mingxuan Wang, and Lei Li. 2021. Language tags matter for zero-shot neural machine translation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3001–3007, Online. Association for Computational Linguistics. find
  26. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2025. Qwen2.5 technical report. arXiv preprint arXiv:24… find
  27. Peng Xie, Xingyuan Liu, Tsz Wai Chan, Yequan Bie, Yangqiu Song, Yang Wang, Hao Chen, and Kani Chen. 2025. SwitchLingua: The first large-scale multilingual and multi-ethnic code-switching dataset. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks T… find
  28. Qwen Team. 2025. QwQ-32B: Embracing the power of reinforcement learning. Accessed: 2026-01-01. find
  29. Haneul Yoo, Cheonbok Park, Sangdoo Yun, Alice Oh, and Hwaran Lee. 2025. Code-switching curriculum learning for multilingual transfer in LLMs. In Findings of the Association for Computational Linguistics: ACL 2025, pages 7816–7836, Vienna, Austria. Association for Computational Linguistics. find
  30. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pages 53728–53741. Curran Associates, Inc.
  31. LG AI Research, Kyunghoon Bae, Eunbi Choi, Kibong Choi, Stanley Jungkyu Choi, Yemuk Choi, Kyubeen Han, Seokhee Hong, Junwon Hwang, Taewan Hwang, Joonwon Jang, Hyojin Jeon, Kijeong Jeon, Gerrard Jeongwon Jo, Hyunjik Jo, Jiyeon Jung, Euisoon Kim, Hyosang Kim, Jihoon Kim, and 22 others. 2025. EXAONE 4.… — [arXiv:2507.11407](https://arxiv.org/abs/2507.11407) find
  32. Lisa Schut, Yarin Gal, and Sebastian Farquhar. 2025. Do multilingual LLMs think in English? arXiv preprint arXiv:2502.15603. — [arXiv:2502.15603](https://arxiv.org/abs/2502.15603) find
  33. Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020. Improving massively multilingual neural machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1628– 1639, Online. Association for Computational…
  34. Ruochen Zhang, Samuel Cahyawijaya, Jan Christian Blaise Cruz, Genta Winata, and Alham Fikri Aji. 2023. Multilingual large language models are not (yet) code-switchers. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12567–12582, Singapore. Association…
  35. Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. 2025. KMMLU: Measuring massive multitask language understanding in Korean. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Ass…
  36. Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024a. LLaMA beyond English: An empirical study on language capability transfer. arXiv preprint arXiv:2401.01055. find
  37. Mingyang Wang, Lukas Lange, Heike Adel, Yunpu Ma, Jannik Strötgen, and Hinrich Schuetze. 2025. Language mixing in reasoning language models: Patterns, impact, and internal causes. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2637–2665, Suzhou, Chin…
  38. Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024c. How do large language models handle multilingualism? In Advances in Neural Information Processing Systems, volume 37, pages 15296–15319. Curran Associates, Inc.
  39. Genta Winata, Alham Fikri Aji, Zheng Xin Yong, and Thamar Solorio. 2023. The decades progress on code-switching research in NLP: A systematic survey on Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024b. WildChat: 1m ChatGPT interaction logs in the wild. In The…
  40. Chengzhi Zhong, Qianying Liu, Fei Cheng, Junfeng Jiang, Zhen Wan, Chenhui Chu, Yugo Murawaki, and Sadao Kurohashi. 2025. What language do non-English-centric large language models think in? In Findings of the Association for Computational Linguistics: ACL 2025, pages 26333–26346, Vienna, Austria. As… find
  41. Xinyi Zhong, Lay Hoon Ang, and Sharon Sharmini. 2024. Why do Mandarin speakers code-switch? a case study of conversational code-switching in China. Humanities and Social Sciences Communications, 11(1):1–10. find
  42. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. LIMA: Less is more for alignment. In Advances in Neural Information Processing Systems, volume 36, pages… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/oh-2026-ola-output-language-alignment-code/