Elamur Research⌕
☰
home / papers / 2026 / HiKE: Hierarchical Evaluation Framework for Korean-English Code-Switching Speech Recognition

HiKE: Hierarchical Evaluation Framework for Korean-English Code-Switching Speech Recognition

Paper · 2026 · Gio Paik, Yongbeom Kim, Soungmin Lee et al.

Summary

Multilingual ASR systems now perform very well on monolingual benchmarks, but they handle utterances that mix languages poorly. The shortfall is most pronounced when the two languages are structurally very different, as with Korean and English, a combination for which no openly accessible real-speech test set existed. HiKE addresses this. The authors wrote seed scripts, created one LLM-generated variant per script, corrected the variants by hand, and had bilingual speakers record all of them. Every utterance carries one of three code-switching granularities: word, phrase or sentence. Loanwords are also marked, so that either a Hangul or a Latin spelling counts as correct.

Ten models were scored with MER and PIER. All of them degraded sharply on mixed speech, and switches within a sentence proved harder than switches at sentence boundaries. Fine-tuning Whisper-Medium improved results substantially. Natural intra-sentential data from AI-Hub gave the largest gains, but cheap synthetic data built by concatenating monolingual clips also helped. The benchmark therefore offers a fine-grained diagnostic tool and suggests a low-cost training route for under-resourced language pairs.

Key points

  • Dataset: 1,121 utterances (about 2.2 h) recorded by 13 bilingual speakers across 8 topics. They were built from 575 hand-written seed scripts, each cloned once with Claude-3.5-Sonnet.
  • Loanword labels: accepting both spellings of loanwords reduced measurement noise by 4.9% in MER and 7.9% in PIER on average.
  • Model evaluation: for the 10 tested models, MER on mixed speech was 3–13 times the monolingual (FLEURS) error. GPT-4o-transcribe performed best, with an overall MER of 21.8 and PIER of 28.8.
  • Fine-tuning Whisper-Medium:
    • Natural AI-Hub data cut overall MER from 31.3 to 9.0 and PIER from 41.3 to 19.5.
    • Synthetic FLEURS + Common Voice concatenations alone reached a MER of 22.1 and PIER of 34.5.
  • Common error types: phonetic transliteration into the wrong script, failure to follow the transcription instruction (notably in Audio-Flamingo-3), and hallucinated repetitive output.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgia Institute of TechnologyAtlanta · United States133 works · 24,497 citations🎓Seoul National UniversitySeoul · South Korea41 works · 45,671 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksMultilingual & cross-lingual NLP · 828 worksSpeech & spoken language · 480 works

Cite

Cite as Gio Paik, Yongbeom Kim, Soungmin Lee, Sangmin Ahn and Chan Woo Kim. 2026. HiKE: Hierarchical Evaluation Framework for Korean-English Code-Switching Speech Recognition. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.33

@inproceedings{paik2026hike,
  title = {{HiKE: Hierarchical Evaluation Framework for Korean-English Code-Switching Speech Recognition}},
  author = {Gio Paik, Yongbeom Kim, Soungmin Lee, Sangmin Ahn and Chan Woo Kim},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.33},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.33}
}

References

28 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Maha Tufail Agro, Atharva Kulkarni, Karima Kadaoui, Zeerak Talat, and Hanan Aldarmaki. 2025. Code-switching in end-to-end automatic speech recognition: A systematic literature review. arXiv preprint arXiv:2507.07741. find
  2. AI-Hub, S. Korea. Korean-english mixed speech recognition dataset. https://www.aihub.or.kr/ aihubdata/data/view.do?dataSetSn=71260. Keyu An, Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Yue Gu, Ting He, Hangrui Hu, Kai Hu, and 1 others. 2024. Funaudiollm: Voice understanding and gener… — [arXiv:2407.04051](https://arxiv.org/abs/2407.04051) ↗
  3. Anthropic. 2024. Claude 3.5 sonnet model card addendum. find
  4. Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. Common Voice: A Massively-Multilingual Speech Corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference (LREC), pa…
  5. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 12449–12460.
  6. Loïc Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, and 1 others. 2023. Seamless: Multilingual expressive and streaming speech translation. arXiv preprint arXiv:2312.05187.
  7. Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2022. FLEURS: FEW-Shot Learning Evaluation of Universal Representations of Speech. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 798–805.
  8. Gemma Team. 2025. Gemma 3n. find
  9. Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and 1 others. 2025. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models. arXiv preprint arXiv:2507.08128. — [arXiv:2507.08128](https://arxiv.org/abs/2507.08128) find
  10. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. In IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP), volume 29…
  11. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  12. Amir Hussein, Dorsa Zeinali, Ondřej Klejch, Matthew Wiesner, Brian Yan, Shammur Chowdhury, Ahmed Ali, Shinji Watanabe, and Sanjeev Khudanpur. 2024. Speech collage: code-switched audio generation by collaging monolingual corpora. In 2024 IEEE International Conference on Acoustics, Speech and Signal… find
  13. Chengfei Li, Shuhao Deng, Yaoping Wang, Guangjing Wang, Yaguang Gong, Changbin Chen, and Jinfeng Bai. 2022. TALCS: An open-source Mandarin-English code-switching corpus and a speech recognition baseline. In Interspeech 2022, pages 1741–1745. find
  14. Yupei Li, Zifan Wei, Heng Yu, Huichi Zhou, and Björn W Schuller. 2025. Dota-me-cs: Daily oriented text audio-mandarin english-code switching dataset. arXiv preprint arXiv:2501.12122. — [arXiv:2501.12122](https://arxiv.org/abs/2501.12122) find
  15. Holy Lovenia, Samuel Cahyawijaya, Genta Winata, Peng Xu, Yan Xu, Zihan Liu, Rita Frieske, Tiezheng Yu, Wenliang Dai, Elham J. Barezi, Qifeng Chen, Xiaojuan Ma, Bertram Shi, and Pascale Fung. 2022. ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversation. In Proceed… find
  16. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210. find
  17. Krishna C. Puvvada, Piotr Żelasko, He Huang, Oleksii Hrinchuk, Nithin Rao Koluguri, Kunal Dhawan, Somshubra Majumdar, Elena Rastorgueva, Zhehuai Chen, Vitaly Lavrukhin, Jagadeesh Balam, and Boris Ginsburg. 2024. Less is More: Accurate Speech Recognition & Translation without Web-Scale Data. In Inte… find
  18. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust Speech Recognition via Large-Scale Weak Supervision. In International Conference on Machine Learning (ICML), pages 28492–28518.
  19. Anthony Rousseau, Paul Deléglise, and Yannick Estève. 2012. TED-LIUM: an Automatic Speech Recognition dedicated corpus. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC), pages 125–129. find
  20. George Saon, Avihu Dekel, Alexander Brooks, Tohru Nagano, Abraham Daniels, Aharon Satt, Ashish Mittal, Brian Kingsbury, David Haws, Edmilson Morais, and 1 others. 2025. Granite-speech: open-source speech-aware llms with strong english asr capabilities. arXiv preprint arXiv:2505.08699. — [arXiv:2505.08699](https://arxiv.org/abs/2505.08699) find
  21. Hiroshi Seki, Shinji Watanabe, Takaaki Hori, Jonathan Le Roux, and John R. Hershey. 2018. An End-to-End Language-Tracking Speech Recognizer for Mixed-Language Speech. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4919–4923. find
  22. Yash Sharma, Basil Abraham, Karan Taneja, and Preethi Jyothi. 2020. Improving Low Resource Code-switched ASR using Augmented Code-switched TTS. In Interspeech 2020. find
  23. Xian Shi, Qiangze Feng, and Lei Xie. 2020. The asru 2019 mandarin-english code-switching speech recognition challenge: Open datasets, tracks, methods and results. arXiv preprint arXiv:2007.05916. *Figure 3: Screenshot of Recording Tool* Enes Yavuz Ugan, Ngoc-Quan Pham, Leonard Bärmann, and Alex Waib… — [arXiv:2007.05916](https://arxiv.org/abs/2007.05916) find
  24. Characteristic Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems (NeurIPS), volume 30.
  25. Educational Level Count Gender Male Female Brian Yan, Injy Hamed, Shuichiro Shimizu, Vasista Lodagala, William Chen, Olga Iakovenko, Bashar Talafha, Amir Hussein, Alexander Polok, Kalvin Chang, and 1 others. 2025. Cs-fleurs: A massively multilingual and code-switched speech dataset. arXiv preprint a… — [arXiv:2509.14161](https://arxiv.org/abs/2509.14161) find
  26. Haibin Yu, Yuxuan Hu, Yao Qian, Ma Jin, Linquan Liu, Shujie Liu, Yu Shi, Yanmin Qian, Edward Lin, and Michael Zeng. 2023. Code-switching text generation and injection in mandarin-english asr. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages… find
  27. Jiaming Zhou, Yujie Guo, Shiwan Zhao, Haoqin Sun, Hui Wang, Jiabei He, Aobo Kong, Shiyao Wang, Xi Yang, Yequan Wang, and 1 others. 2025. Cs-dialogue: A 104-hour dataset of spontaneous mandarin-english code-switching dialogues for speech recognition. arXiv preprint arXiv:2502.18913. 11 2 — [arXiv:2502.18913](https://arxiv.org/abs/2502.18913) find
  28. Undergraduate Bachelor’s 6 7 Total *Table 4: Demographics of Participants* find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/paik-2026-hike-hierarchical-evaluation-framework-korean/