Elamur Research⌕
☰
home / papers / 2025 / CUTE: A Multilingual Dataset for Enhancing Cross-Lingual Knowledge Transfer in Low-Resource Languages

CUTE: A Multilingual Dataset for Enhancing Cross-Lingual Knowledge Transfer in Low-Resource Languages

Paper · 2025 · Wenhao Zhuang and Yuan Sun

Summary

Large language models handle high-resource languages well but give weak support to Uyghur and Tibetan, minority languages of China with roughly 13 and 8 million speakers. The main obstacle is the lack of training text. Existing collections that include these languages, such as OSCAR, CulturaX, CC100 and MC2, are small, unevenly distributed or contain misidentified languages. The authors build CUTE, a four-language corpus (Chinese, Uyghur, Tibetan, English) created by machine-translating text from the resource-rich languages. Before building it, native speakers rated Chinese→Uyghur and Chinese→Tibetan output and judged its quality close to that of Chinese→English translation. CUTE consists of two equally sized collections: one in which all four languages are mutually parallel and one in which they are not. The authors then train two CUTE-Llama variants, one per corpus type, using vocabulary expansion and embedding initialization. Zero-shot tests on downstream tasks indicate that the parallel version moves knowledge from high-resource to low-resource languages more effectively. The work provides the largest open resource for these two languages and evidence that corpus parallelism matters for cross-lingual transfer.

Key points

  • CUTE contains two 25GB four-language sets, one parallel and one non-parallel, for a total of about 50GB.
  • Its Uyghur and Tibetan portions are about ten times larger than MC2, the previous largest open corpus for Chinese minority languages, which is under 3GB.
  • Generating the data by machine translation removes language misidentification and allows a more balanced spread across languages and content domains.
  • Two CUTE-Llama models, one trained on parallel data and one on non-parallel data, were built with vocabulary expansion and embedding initialization and released publicly.
  • The corpus and models are available at github.com/CMLI-NLP/CUTE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 882 worksLow-resource languages · 287 works

Cite

Cite as Wenhao Zhuang and Yuan Sun. 2025. CUTE: A Multilingual Dataset for Enhancing Cross-Lingual Knowledge Transfer in Low-Resource Languages. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zhuang2025cute,
  title = {{CUTE: A Multilingual Dataset for Enhancing Cross-Lingual Knowledge Transfer in Low-Resource Languages}},
  author = {Wenhao Zhuang and Yuan Sun},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.670/}
}

References

38 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. The CUTE dataset provides rich resources for low-resource language research; however, its large scale inevitably leads to some errors in the translation process, especially in sentences with com- References plex grammar or significant cultural differences. Julien Abadji, Pedro Ortiz Suarez, Laurent… — Towards a Cleaner Document-Oriented Multilingual Crawled Corpus
  2. ArXiv, BigScience, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. find
  3. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems,…
  4. Samuel Cahyawijaya, Holy Lovenia, and Pascale Fung. 2024. Llms are few-shot in-context low-resource language learners. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 405–…
  5. Samuel Cahyawijaya, Holy Lovenia, Tiezheng Yu, Willy Chung, and Pascale Fung. 2023. Instructalign: High-and-low resource language alignment via continual crosslingual instruction tuning. In Proceedings of the First Workshop in South East Asian Language Processing, pages 55–78.
  6. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113.
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. In Annual Meeting of the Association for Computational Linguist…
  8. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. find
  9. Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. 2019. A span-extraction dataset for chinese machine reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conf…
  10. Junjie Deng, Hanru Shi, Xinhe Yu, Wugedele Bao, Yuan Sun, and Xiaobing Zhao. 2023. Milmo: minority Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association fo…
  11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  12. Benedikt Ebing and Goran Glavas. 2023. To translate or not to translate: A systematic investigation of translation-based cross-lingual transfer to low-resource languages. ArXiv, abs/2311.09404. find
  13. Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John E. Ortega, Ricardo Ramos, Annette Rios Gonzales, Ivan Vladimir, Gustavo A. Gim’enez-Lugo, Elisabeth Mager, Graham Neubig, Alexis Palmer, Rolando A. Coto Solano, Ngoc Thang Vu, and Katharina Kann. 2021.… — AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages
  14. Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. 2022. Dataless knowledge fusion by merging weights of language models. ArXiv, abs/2212.09849.
  15. Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71.
  16. Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier García, Christopher A. Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2023. Madlad-400: A multilingual and document-level large audited dataset. ArXiv, abs/2309.04662.
  17. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. ArXiv, abs/1901.07291. find
  18. Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. 2022. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Proc… find
  19. Bin Li, Yixuan Weng, Bin Sun, and Shutao Li. 2022. A multi-tasking and multi-stage chinese minority pre-trained language model. In CCMT. find
  20. Chong Li, Wen Yang, Jiajun Zhang, Jinliang Lu, Shaonan Wang, and Chengqing Zong. 2024. X-instruction: Aligning language model in low-resource languages with self-curated cross-lingual instructions. ArXiv, abs/2405.19744. find
  21. Chen Cecilia Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2023. Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings. ArXiv, abs/2309.08591.
  22. Zhongtao Miao, Qiyu Wu, Kaiyan Zhao, Zilong Wu, and Yoshimasa Tsuruoka. 2024. Enhancing cross-lingual sentence embedding for low-resource languages with word alignment. ArXiv, abs/2404.02490. find
  23. Thut Nguyen, Chien Van Nguyen, Viet Dac Lai, Hiu Mn, Nghia Trung Ngo, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen. 2024. Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. In Proceedings of the 2024 Joint International Conference on Computa…
  24. Trinh Pham, Khoi M. Le, and Anh Tuan Luu. 2024. Unibridge: A unified approach to cross-lingual transfer learning for low-resource languages. In Annual Meeting of the Association for Computational Linguistics. find
  25. Nuo Qun, Xing Li, Xipeng Qiu, and Xuanjing Huang. 2017. End-to-end neural text classification for tibetan. In China National Conference on Chinese Computational Linguistics. find
  26. P Rajpurkar. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
  27. Y Sun, S Liu, C Chen, Z Dan, and X Zhao. 2021. Construction of high-quality tibetan dataset for machine reading comprehension. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 208–218. find
  28. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA). find
  29. Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, Chenxia Li, Liu Yang, Xilin Luo, Xuejie Wu, Lunan Liu, Wenjun Cheng, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Lei Lin, Xiaokun Wang, Yutuan Ma, Chuanhai Dong, Yanqi Sun, Yifu Chen, Yongy… — Skywork: A More Open Bilingual Foundation Model
  30. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. In North American Chapter of the Association for Computational Linguistics.
  31. Ziqing Yang, Zihang Xu, Yiming Cui, Baoxin Wang, Min Lin, Dayong Wu, and Zhigang Chen. 2022. Cino: A chinese minority pre-trained language model. In International Conference on Computational Linguistics. find
  32. Zheng-Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Indra Winata, Stella Biderman, Dragomir R. Radev, and Vassilina Nikoulina. 2022. Bloom+1: Adding language support to blo…
  33. Chen Zhang, Xiao Liu, Jiuheng Lin, and Yansong Feng. 2024a. Teaching large language models an unseen language on the fly. ArXiv, abs/2402.19167. find
  34. Chen Zhang, Mingxu Tao, Quzhe Huang, Jiuheng Lin, Zhibin Chen, and Yansong Feng. 2024b. Mc 2 : Towards transparent and culturally-aware nlp for minority languages in china. arXiv preprint arXiv:2311.08348. find
  35. Wenhao Zhuang, Dawa Cairen, and Yuan Sun. 2024a. Tifd: Tibetan instruction-following dataset for large language models supervised fine-tuning. Data Intelligence. find
  36. Wenhao Zhuang, Yuan Sun, and Xiaobing Zhao. 2024b. Tilamb (tilamb: A tibetan large language model based on incremental pre-training). In Proceedings of the 23th Chinese National Conference on Computational Linguistics. find
  37. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  38. Daan van Esch, Tamar Lucassen, Sebastian Ruder, Isaac Caswell, and Clara Rivera. 2022. Writing system and speaker metadata for 2,800+ language varieties. In International Conference on Language Resources and Evaluation. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhuang-2025-cute-multilingual-dataset-enhancing-cross/