Elamur Research⌕
☰
home / papers / 2026 / Benchmarking Multilingual LLM Translation Accuracy for Fuzhounese

Benchmarking Multilingual LLM Translation Accuracy for Fuzhounese

Summary

Fuzhounese is an Eastern Min Sinitic language with roughly 10 million speakers. It has no standard orthography, almost no digital text, and had received essentially no NLP attention. The authors measure how well current multilingual LLMs handle it through translation in two directions. Comprehension is tested by translating Fuzhounese into English, Mandarin and Cantonese. Generation is tested by translating English into Fuzhounese, Mandarin and Cantonese.

Borrowing design ideas from the HKCanto-Eval benchmark, they built a small parallel set of 305 sentences: 300 constructed English sentences plus 5 additional reference translations. A native speaker wrote the Fuzhounese gold versions in Bàng-uâ-cê (Foochow Romanized), and further native and heritage speakers checked them. Outputs were scored with BERTScore, chrF++ and human Likert ratings.

Every tested model performed poorly on Fuzhounese. Comprehension scores stayed low, and generation scores were usually less than half of those for Mandarin or Cantonese. The work provides the first Fuzhounese MT benchmark and shows that multilingual LLMs are skewed toward well-resourced languages.

Key points

  • Dataset and release: The benchmark has 305 sentences grouped by category (Everyday, Grammar, Idioms, Imperatives, Narrative) and by length (short, medium, long). Data, prompts and scripts are public at github.com/fzhkch/fuzhounese-translation-benchmark.
  • Models and prompts: Five models were evaluated: GPT-4o, GPT-4o mini, Gemini-2.5 Flash, DeepSeek-V3.2 and Qwen-2.5-72B. All received identical structured prompts that name the source and target languages and ask for the translation only.
  • Strongest model: Gemini-2.5 Flash scored best on Fuzhounese input, with BERTScore of about 0.48–0.56 and chrF++ of about 20–28. Its Likert scores for Fuzhounese output were around 2.7–2.9. DeepSeek-V3.2 ranked second.
  • Weakest models: GPT-4o mini and Qwen-2.5-72B scored close to the minimum, with Fuzhounese-output Likert scores of about 1.0–1.2. Qwen's BERTScore on Fuzhounese input was only about 0.12–0.25.
  • Research gap: A survey cited by the authors found no mention of Fuzhounese in the 73,285 ACL Anthology papers published through 2022. No LLM is known to have been trained or fine-tuned on Fuzhounese data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4mm9bs8yy4ie

@inproceedings{zheng2026benchmarking,
  title = {{Benchmarking Multilingual LLM Translation Accuracy for Fuzhounese}},
  author = {Zheng, Sue and Bloem, Jelke},
  booktitle = {Proceedings of the SIGUL 2026 Joint Workshop with ELE, EURALI, and DCLRL: Towards Inclusivity and Equality: Language Resources and Technologies for Under-Resourced and Endangered Languages},
  year = {2026},
  doi = {10.63317/4mm9bs8yy4ie},
  url = {https://doi.org/10.63317/4mm9bs8yy4ie}
}

References

33 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jordi Armengol-Estapé, Casimiro Pio Carrino, Carlos Rodriguez-Penagos, Ona de Gibert, Carme Armentano-Oller, Aitor Gonzalez-Agirre, Maite Melero, and Marta Villegas. 2021. Are multilingual models the best choice for moderately under-resourced languages? A comprehensive assessment for Catalan. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4933–4946. find
  2. Caleb C Baldwin. 1871. A manual of the Foochow dialect. Methodist Episcopal Mission Press. find
  3. David Prager Branner. 2000. Problems in comparative Chinese dialectology: The classification of Miin and Hakka, volume 123. Walter de Gruyter. find
  4. Matthew Y Chen. 2000. Tone sandhi: Patterns across Chinese dialects, volume 92. Cambridge University Press. find
  5. Tsz Chung Cheng, Chung Shing Cheng, Chaak-ming Lau, Eugene Lam, Wong Chun Yat, Hoi On Yu, and Cheuk Hei Chong. 2025. HKCanto-Eval: A benchmark for evaluating Cantonese language understanding and cultural comprehension in LLMs. In Proceedings of the 29th Conference on Computational Natural Language Learning, pages 1–11. ↗
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th annual meeting of the Association for Computational Linguistics, pages 8440–8451. find
  7. Sumanth Doddapaneni, Mohammed Safi Ur Rahman Khan, Dilip Venkatesh, Raj Dabre, Anoop Kunchukuttan, and Mitesh M Khapra. 2025. Cross-lingual auto evaluation for assessing multilingual LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29297–29329. ↗
  8. San Duanmu. 2007. The phonology of standard Chinese. Oxford University Press. ↗
  9. David M. Eberhard, Gary F. Simons, and Charles D. Fennig, editors. 2023. Ethnologue: Languages of the World, 26 edition. SIL International. find
  10. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th linguistic annotation workshop and interoperability with discourse, pages 33–41. find
  11. Pierre Isabelle, Colin Cherry, and George Foster. 2017. A challenge set approach to evaluating machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2486–2496. find our card
  12. Jiyue Jiang, Pengan Chen, Liheng Chen, Sheng Wang, Qinghang Bao, Lingpeng Kong, Yu Li, and Chuan Wu. 2025. How well do LLMs handle Cantonese? Benchmarking Cantonese capabilities of large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4464–4505. find
  13. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th annual meeting of the Association for Computational Linguistics, pages 6282–6293. find our card
  14. Klaus Krippendorff. 2004. Content analysis: An introduction to its methodology. Sage publications. find
  15. Deepak Kumar, Kshetrimayum Boynao Singh, and Asif Ekbal. 2025. Tackling low-resource NMT with instruction-tuned LLaMA: A study on Kokborok and Bodo. In Proceedings of the Tenth Conference on Machine Translation, pages 1215–1221. find
  16. Shuheng Liu and Michael Best. 2025. A survey of NLP progress in Sino-Tibetan low-resource languages. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7804–7825. ↗
  17. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62–90. find our card
  18. R. S. Maclay and C. C. Baldwin. 1929. Dictionary of the Foochow Dialect. Methodist Episcopal Mission Press. Accessed via Wikisource transcription. find
  19. Mukhammadsaid Mamasaidov, Azizullah Aral, Abror Shopulatov, and Mironshoh Inomjonov. 2025. Filling the gap for Uzbek: Creating translation resources for Southern Uzbek. In Proceedings of the Tenth Conference on Machine Translation, pages 1081–1087. find
  20. Jerry Norman. 1988. Chinese. Cambridge University Press, Cambridge. find
  21. Jerry Norman. 1991. The Mǐn dialects in historical perspective. Journal of Chinese Linguistics Monograph Series, (3):323–358. find
  22. Partha Pakray, Alexander Gelbukh, and Sivaji Bandyopadhyay. 2025. Natural language processing applications for low-resource languages. Natural Language Processing, 31(2):183–197. ↗
  23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. ↗ our card
  24. Maja Popović. 2015. chrF: character n-gram Fscore for automatic MT evaluation. In Proceedings of the tenth workshop on statistical machine translation, pages 392–395. find
  25. Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S Yu. 2025. A survey of multilingual large language models. Patterns, 6(1). ↗
  26. S Robert Ramsey. 1989. The languages of China. Princeton University Press. find
  27. Shaurya Rohatgi, Yanxia Qin, Benjamin Aw, Niranjana Unnithan, and Min-Yen Kan. 2023. The ACL OCL corpus: Advancing open science in computational linguistics. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10348–10361. find
  28. Nadav Shani and Ali Basirat. 2025. Language dominance in multilingual large language models. In Proceedings of the 8th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 137–148. find
  29. Chaoju Tang and Vincent J van Heuven. 2007. Mutual intelligibility and similarity of Chinese dialects: Predicting judgments from objective measures. Linguistics in the Netherlands, 24(1):223–234. ↗
  30. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120–130. find our card
  31. Yuemei Xu, Ling Hu, Jiayi Zhao, Zihan Qiu, Kexin Xu, Yuqi Ye, and Hanwen Gu. 2025. A survey on multilingual large language models: Corpora, alignment, and bias. Frontiers of Computer Science, 19(11):1911362. ↗
  32. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations. A. Prompt Design This appendix provides the full set of prompt templates used in the evaluation. The prompts specify the source and target languages, as well as output constraints, and are kept consistent across all models and translation directions to ensure comparability and minimize prompt-engineering effects. ↗
  33. HKCanto-Eval (Cheng et al., 2025) makes use of structured translation prompts that explicitly specify the source language, target language, and output constraints. Consequently, each prompt in this study likewise specifies these elements. This design ensures comparability across models, languages, and translation directions while minimizing prompt-engineering effects. A.1. Fuzhounese-to-Target Language Prompts For translations from Fuzhounese (in Bàng-uâ-cê) into English, Standard Mandarin, and Cantonese, the following prompt templates are used. <Text> represents the source sentence inserted into the prompt during evaluation: Fuzhounese to English Fuzhounese (Bàng-uâ-cê) Text to Translate: <Text> Ensure the translation is accurate and natural, preserving the original meaning and using concise and fluent English expression. ONLY RETURN THE TRANSLATION. DO NOT EXPLAIN. Fuzhounese to Standard Mandarin Fuzhounese (Bàng-uâ-cê) Text to Translate: <Text> Ensure the translation is accurate and natural, preserving the original meaning and using concise and fluent Standard Mandarin expression. ONLY RETURN THE TRANSLATION. DO NOT EXPLAIN. Fuzhounese to Written Cantonese Fuzhounese (Bàng-uâ-cê) Text to Translate: <Text> Ensure the translation is accurate and natural, preserving the original meaning and using concise and fluent written Cantonese expression. ONLY RETURN THE TRANSLATION. DO NOT EXPLAIN. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/zheng-2026-benchmarking-multilingual-llm-translation-accuracy/