Elamur Research⌕
☰
home / papers / 2026 / Benchmarking Large Language Models for Chinese and Japanese IMEs: Phonetic-to-Character Generation and Textual Error Correction

Benchmarking Large Language Models for Chinese and Japanese IMEs: Phonetic-to-Character Generation and Textual Error Correction

Summary

Chinese and Japanese cannot be typed directly on a keyboard, so users rely on Input Method Editors (IMEs). These convert phonetic input (Pinyin or Romaji) into characters. Two problems make this hard: widespread homophony, and user mistakes ranging from simple typos to choosing the wrong homophone. Traditional IMEs mostly rank candidates using local context, so they handle both problems poorly. The authors built a benchmark to test whether LLMs can serve as next-generation IMEs. It covers two tasks: phonetic-to-character generation and textual error correction, each at word and sentence level. Proprietary, open-source and Japanese-specific LLMs are compared with conventional baselines. Evaluation combines similarity, lexical and error-rate metrics with latency measures. The strongest LLMs clearly outperform the baselines at resolving ambiguity and fixing complex errors. They also stay robust as the number of errors grows. However, accuracy comes with a computational cost that matters for real-time typing. The released datasets, scripts and results give future IME research a common reference point.

Key points

  • Data: Chinese data comes from the Phrase Pinyin Data repository (words) and the SIGHAN 2015 bake-off corpus (sentences). Japanese tasks cover Kanji, Hiragana and Katakana output.
  • Metrics:
    • Quality: SimHash, BERTScore, BLEU, ROUGE and CER.
    • Efficiency: completion time, TTFT and TPS.
  • Models tested: The evaluation includes GPT-4/GPT-4o(-mini), the Gemini models (2.5 Flash-Lite and 2.0 Flash), Anthropic's Claude 3.5 Haiku and Claude 3 Sonnet, plus Llama3-8B, Llama3.2-3B, Mistral-7B, GLM4-9B, Baichuan2-7B, Qwen2-7B, Rinna Nekomata-7B, RakutenAI-7B, RakutenAI2.0-mini-1.5B and Swallow-8B.
  • Baselines: macOS Romaji, WanaKana and OpenNMT.
  • Robustness to errors:
    • GPT-4o and Gemini 2.0 Flash keep high SimHash scores even with 4–5 injected errors.
    • Smaller offline models such as Llama3-8B and Mistral-7B start lower and decline faster.
    • The OpenNMT baseline weakens as noise increases.
    • Completion time depends on sentence length rather than on the number of errors.
  • Japanese by script:
    • Kanji generation is the hardest sub-task, and the gap between models is widest there.
    • Hiragana and Katakana behave more like transliteration, where even small models and macOS Romaji perform well.
    • As a result, averaged "Japanese" scores can be misleading.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/42jiimjriyga

@inproceedings{zou2026benchmarking,
  title = {{Benchmarking Large Language Models for Chinese and Japanese IMEs: Phonetic-to-Character Generation and Textual Error Correction}},
  author = {Zou, Yuchun and Lee, Tedd and Fan, Xiaodi and Li, Jun},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/42jiimjriyga},
  url = {https://doi.org/10.63317/42jiimjriyga}
}

References

35 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Manar Alkhatib, Azza Abdel Monem, and Khaled Shaalan. 2020. Deep learning for Arabic error detection and correction. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 19(5):1–13. find
  2. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora De… find our card
  3. BoppreH. 2024. keyboard: Hook and simulate keyboard events on windows and linux. https://github.com/boppreh/keyboard. ↗
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  5. Pi-Chuan Chang, Michel Galley, and Christopher D. Manning. 2008. Optimizing Chinese word segmentation for machine translation performance. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 224– 232, Columbus, Ohio. Association for Computational Linguistics. find our card
  6. Shen-Yuan Chen, Rui Wang, and Hai Zhao. 2015. Neural network language model for chinese pinyin input method engine. In Proceedings of the 29th Pacific Asia conference on language, information and computation, pages 455–461. Waseda University. find
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  8. Seyed Mohammad Sadegh Dashti, Amid Khatibi Bardsiri, and Mehdi Jafari Shahbazzadeh. 2024. PERCORE: A deep learning-based framework for persian spelling correction with phonetic analysis. International Journal of Computational Intelligence Systems, 17(1):114. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  10. Daniel Jurafsky and James H. Martin. 2025. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, with Language Models, 3rd edition. Online manuscript released August 24, 2025. find
  11. Sa Kasmaiee, Si Kasmaiee, and M Homayounpour. 2023. Correcting spelling mistakes in persian texts with rules and deep learning methods. Scientific Reports, 13(1):19945. find
  12. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. Open-NMT: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pages 67–72, Vancouver, Canada. Association for Computational Linguistics. find
  13. Tian Le. 2023. pinyin2hanzi: Convert pinyin to chinese characters. https://github.com/l etiantian/Pinyin2Hanzi. ↗
  14. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
  15. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational… find our card
  16. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI. find
  17. Kei Sawada, Tianyu Zhao, Makoto Shing, Kentaro Mitsui, Akio Kaga, Yukiya Hono, Toshiaki Wakatsuki, and Koh Mitsuda. 2024. Release of pre-trained models for the Japanese language. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation… find
  18. Felix Stahlberg. 2020. Neural machine translation: A review. Journal of Artificial Intelligence Research, 69:343–418. find
  19. Zijun Sun, Xiaoya Li, Xiaofei Sun, Yuxian Meng, Xiang Ao, Qing He, Fei Wu, and Jiwei Li. 2021. ChineseBERT: Chinese pretraining enhanced by glyph and Pinyin information. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer… find
  20. Ashima Suvarna, Harshita Khandelwal, and Nanyun Peng. 2024. PhonologyBench: Evaluating phonological skills of large language models. In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pages 1–14, Bangkok, Thailand. Association for Computational Linguistics. find
  21. Minghuan Tan, Yong Dai, Duyu Tang, Zhangyin Feng, Guoping Huang, Jing Jiang, Jiwei Li, and Shuming Shi. 2022. Exploring and adapting Chinese GPT to Pinyin input method. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1899–1909… find
  22. Yuen-Hsien Tseng, Lung-Hao Lee, Li-Ping Chang, and Hsin-Hsi Chen. 2015. Introduction to SIGHAN 2015 bake-off for Chinese spelling check. In Proceedings of the Eighth SIGHAN Workshop on Chinese Language Processing, pages 32–37, Beijing, China. Association for Computational Linguistics. find
  23. Robert A Wagner and Michael J Fischer. 1974. The string-to-string correction problem. Journal of the ACM (JACM), 21(1):168–173. find
  24. Pak-kwong Wong and Chorkin Chan. 1996. Chinese word segmentation based on maximum matching and word binding force. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics. find
  25. Ming Xu. 2024. pycorrector: Chinese text error correction tool. https://github.com/shi bing624/pycorrector. ↗
  26. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with bert. In International Conference on Learning Representations (ICLR). find
  27. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A survey of l… find our card
  28. Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, Hao Peng, Jianxin Li, Jia Wu, Ziwei Liu, Pengtao Xie, Caiming Xiong, Jian Pei, Philip S. Yu, and Lichao Sun. 2023. A comprehensive survey on pretrained foundation models: A history from BERT to C… — [arXiv:2302.09419](https://arxiv.org/abs/2302.09419) find
  29. Language Resource References Albert Villanova del Moral. 2024. Snow simplified Japanese corpus. https://huggingface.co/datasets/snow_simplified_japane se_corpus/viewer/snow_t15. A dataset of simplified Japanese texts, useful for various NLP tasks. Hosted on Hugging Face. ↗
  30. Huang Huang. 2024. Phrase pinyin data. https://github.com/mozillazg/phrase-pin yin-data. ↗
  31. Nguyen Huy. 2024. CSCD-IME: Chinese sentence completion dataset and input method editor. https://github.com/nghuyong/cs cd-ime/. A repository for the Chinese Sentence Completion Dataset and Input Method Editor (CSCD-IME), including data and code for sentence completion tasks. ↗
  32. Stephen Kraus. 2024. Yomichan JLPT vocab. https://github.com/stephenmk/yomicha n-jlpt-vocab. JLPT level tags for words in Yomichan. The repository includes instructions for installation and building the dictionary from source, along with statistics on unique words per JLPT level. ↗
  33. Kyoto University. 2024. Japanese word text database (jwtd). https://nlp.ist.i.ky oto-u.ac.jp/EN/edit.php?JWTD. A comprehensive database of Japanese texts used for various natural language processing tasks. ↗
  34. Zou, Yuchun and Lee, Tedd and Fan, Xiaodi and Li, Jun. 2026a. Chinese Phonetic (Pinyin) Error Dataset. ISLRN 023-543-404-586-9. A newly created corpus of Chinese sentences with programmatically introduced phonetic and semantic errors for evaluating IME systems. find
  35. Zou, Yuchun and Lee, Tedd and Fan, Xiaodi and Li, Jun. 2026b. Japanese Phonetic (Romaji) Error Dataset. ISLRN 482-587-871-494-1. A newly created corpus of Japanese sentences with programmatically introduced phonetic errors for evaluating IME systems. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zou-2026-benchmarking-large-language-models-chinese/