Elamur Research⌕
☰
home / papers / 2025 / A Rigorous Evaluation of LLM Data Generation Strategies for Low-Resource Languages

A Rigorous Evaluation of LLM Data Generation Strategies for Low-Resource Languages

Paper · 2025 · Tatiana Anikina, Jan Cegin, Jakub Simko et al.

Summary

LLM-generated synthetic data is often used to train compact task models, but it is unclear which prompting strategies work best outside English, especially for under-resourced languages. The authors compare seven generation setups built from three ingredients: label descriptions summarised by Llama3-70b, ten demonstrations per label written in English or in the target language, and a revision step in which the same LLM discards weak samples. Four open-source LLMs from the Gemma-3 and Llama-3 families each produce 100 examples per label for intent recognition, topic classification and sentiment analysis. This is done in 11 languages, ranging from English and German down to Welsh, Telugu and Azerbaijani. XLM-R is fine-tuned on each synthetic set and evaluated on the original human-labelled test data. The most reliable recipe pairs native-language examples with LLM filtering. In some settings it comes within about 5% of training on the same amount of real data, and careful prompting shrinks the advantage of bigger generators. The work offers practical guidance for cheaply building training data in low-resource languages.

Key points

  • Languages and tasks: The study covers high-resource English and German; mid-resource Thai, Hebrew, Indonesian and Swahili; and low-resource Romanian, Azerbaijani, Slovenian, Telugu and Welsh. The tasks use MASSIVE (10 of its 60 intents, 1,000 samples), SIB-200 (7 topics, 700 samples) and binary sentiment (200 samples).
  • Downstream model: XLM-R (279M parameters, 94 languages) was chosen over mBERT after pilot runs. Training used early stopping with a patience of 5, and testing always used gold data.
  • Strategy findings: Revision helps consistently. For Llama models, English demonstrations perform nearly as well as target-language ones. Label summaries help intent recognition but hurt topic classification. Larger models generate better data, and Llama slightly outperforms Gemma.
  • Similarity and diversity: Cosine similarity to gold data correlates only weakly with F1 (Pearson 0.111). Data from demonstrations plus revision shows higher n-gram diversity than gold data for intent (+0.218) and sentiment (+0.581), but lower diversity for topic (−0.279). Gemma 27b produces the most diverse output.
  • Resources and limitations: Code is available at github.com/tanikina/multilingual-generation. The authors note that there was no human quality check, only two LLM families were tested, and possible pretraining contamination is unmeasured.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬German Research Centre for Artificial IntelligenceKaiserslautern · Germany79 works · 29,970 citations🎓Brno University of TechnologyBrno · Czechia11 works · 32,326 citations🤝Kempelen Institute of Intelligent TechnologiesBratislava · Slovakia11 works · 82 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEvaluation & benchmarks · 1,341 worksLow-resource languages · 287 works

Cite

Cite as Tatiana Anikina, Jan Cegin, Jakub Simko and Simon Ostermann. 2025. A Rigorous Evaluation of LLM Data Generation Strategies for Low-Resource Languages. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.418

@inproceedings{anikina2025rigorous,
  title = {{A Rigorous Evaluation of LLM Data Generation Strategies for Low-Resource Languages}},
  author = {Tatiana Anikina, Jan Cegin, Jakub Simko and Simon Ostermann},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.418},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.418}
}

References

42 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Louis Abraham, Charles Arnal, and Antoine Marie. 2025. Prompt selection matters: Enhancing text annotations for social sciences with large language models. Preprint, arXiv:2407.10645. — [arXiv:2407.10645](https://arxiv.org/abs/2407.10645) find
  2. David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, and En-Shiun Annie Lee. 2024. SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. In Proceedings of the 18th Conference of the Eu…
  3. AI@Meta. 2024. Llama 3 model card. find
  4. Jan Cegin, Branislav Pecher, Jakub Simko, Ivan Srba, Maria Bielikova, and Peter Brusilovsky. 2024a. Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Ling… find
  5. Jan Cegin, Branislav Pecher, Jakub Simko, Ivan Srba, Maria Bielikova, and Peter Brusilovsky. 2024b. Use random selection for now: Investigation of few-shot selection strategies in llm-based text augmentation for classification. Preprint, arXiv:2410.10756. find
  6. Jan Cegin, Jakub Simko, and Peter Brusilovsky. 2023. ChatGPT to replace crowdsourcing of paraphrases for intent classification: Higher diversity and comparable model robustness. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1889–1905, Singapore. Ass… find
  7. Jan Cegin, Jakub Simko, and Peter Brusilovsky. 2025. LLMs vs established text augmentation techniques for classification: When do the benefits outweight the costs? In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Lan… find
  8. Yi-Ling Chung, Aurora Cobo, and Pablo Serna. 2025. Beyond translation: Llm-based data generation for multilingual fact-checking. arXiv preprint arXiv:2502.15419. — [arXiv:2502.15419](https://arxiv.org/abs/2502.15419) find
  9. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116.
  10. Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu, Wei Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, Hongmin Cai, Lichao Sun, Quanzheng Li, Dinggang Shen, Tianming Liu, and Xiang Li. 2023. Auggpt: Leveraging chatgpt for text data augmentation. Preprint,…
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  12. Chandra Kiran Reddy Evuru, Sreyan Ghosh, Sonal Kumar, S Ramaneswaran, Utkarsh Tyagi, and Dinesh Manocha. 2024. Coda: Constrained generation based data augmentation for low-resource nlp. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3754–3769. find
  13. Luyang Fang, Gyeong-Geon Lee, and Xiaoming Zhai. 2023. Using gpt-4 to augment unbalanced data for automatic scoring. Preprint, arXiv:2310.18365. — [arXiv:2310.18365](https://arxiv.org/abs/2310.18365) find
  14. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891, Dublin, Ireland. Association for Computation…
  15. Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021. A survey of data augmentation approaches for NLP. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 968–988, Online. Association for Computational Lin…
  16. Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natarajan. 2023. MASSIVE: A 1M-example multilingual natural language… find
  17. Sreyan Ghosh, Chandra Kiran Evuru, Sonal Kumar, S Ramaneswaran, S Sakshi, Utkarsh Tyagi, and Dinesh Manocha. 2023. Dale: Generative data augmentation for low-resource legal nlp. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Sentosa, Singapore. find
  18. Parker Glenn, Alolika Gon, Nikhil Kohli, Sihan Zha, Parag Pravin Dakle, and Preethi Raghavan. 2023. Jetsons at the FinNLP-2023: Using synthetic data and transfer learning for multilingual ESG issue classification. In Proceedings of the Fifth Workshop on Financial Technology and Natural Language Proc… find
  19. Xu Gu, Xiaoliang Chen, Peng Lu, Zonggen Li, Yajun Du, and Xianyong Li. 2024. Agcvt-prompt for sentiment classification: Automatically generating chain of thought and verbalizer in prompt learning. Eng. Appl. Artif. Intell., 132(C). find
  20. Daniil Gurgurov, Mareike Hartmann, and Simon Ostermann. 2024. Adapting multilingual LLMs to low-resource languages with knowledge graphs via adapters. In Proceedings of the 1st Workshop on Knowledge Graphs and Large Language Models (KaLLM 2024), pages 63–74, Bangkok, Thailand. Association for Comput…
  21. Daniil Gurgurov, Rishu Kumar, and Simon Ostermann. 2025a. GrEmLIn: A repository of green baseline embeddings for 87 low-resource languages injected with multilingual graph knowledge. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1204–1221, Albuquerque, New Mexico. A… find
  22. Daniil Gurgurov, Ivan Vykopal, Josef van Genabith, and Simon Ostermann. 2025b. Small models, big impact: Efficient corpus and graph-based adaptation of small multilingual language models for low-resource languages. arXiv preprint arXiv:2502.10140. find
  23. Alina Kramchaninova and Arne Defauw. 2022. Synthetic data generation for multilingual domain-adaptable question answering systems. In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pages 151–160, Ghent, Belgium. European Association for Machine Transla… find
  24. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. find
  25. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
  26. Xiaonan Li, Changtai Zhu, Linyang Li, Zhangyue Yin, Tianxiang Sun, and Xipeng Qiu. 2024. LLatrieval: LLM-verified retrieval for verifiable generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vo… find
  27. Linlin Liu, Bosheng Ding, Lidong Bing, Shafiq Joty, Luo Si, and Chunyan Miao. 2021. MulDA: A multilingual data augmentation framework for low-resource cross-lingual NER. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer… find
  28. Qijiong Liu, Nuo Chen, Tetsuya Sakai, and Xiao-Ming Wu. 2024. Once: Boosting content-based recommendation with both open- and closed-source large language models. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, WSDM ’24, page 452–461, New York, NY, USA. Associa… find
  29. Sepideh Mollanorozy, Marc Tanti, and Malvina Nissim. 2023. Cross-lingual transfer learning with Persian. In Proceedings of the 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pages 89–95, Dubrovnik, Croatia. Association for Computational Linguistics. find
  30. Amani Namboori, Shivam Sadashiv Mangale, Andy Rosenbaum, and Saleh Soltan. 2023. Gemquad: Generating multilingual question answering datasets from large language models using few shot learning. In NeurIPS 2023 Workshop on Synthetic Data Generation with Generative AI. find
  31. Aytuğ Onan. 2023. Srl-aco: A text augmentation framework based on semantic role labeling and ant colony optimization. Journal of King Saud University - Computer and Information Sciences, 35(7):101611. find
  32. Frédéric Piedboeuf and Philippe Langlais. 2023. Is ChatGPT the ultimate data augmentation algorithm? In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15606–15615, Singapore. Association for Computational Linguistics. find
  33. Salsabila Zahirah Pranida, Rifo Ahmad Genadi, and Fajri Koto. 2025. Synthetic data generation for culturally nuanced commonsense reasoning in low-resource languages. Preprint, arXiv:2502.12932.
  34. Anand Rajaraman and Jeffrey David Ullman. 2011. Data Mining, page 1–17. Cambridge University Press. find
  35. Gaurav Sahu, Pau Rodriguez, Issam Laradji, Parmida Atighehchian, David Vazquez, and Dzmitry Bahdanau. 2022. Data augmentation for intent classification with off-the-shelf large language models. In Proceedings of the 4th Workshop on NLP for Conversational AI, pages 47–57, Dublin, Ireland. Association…
  36. Indira Sen, Dennis Assenmacher, Mattia Samory, Isabelle Augenstein, Wil Aalst, and Claudia Wagner. 2023. People make better edits: Measuring the efficacy of LLM-generated counterfactually augmented data for harmful language detection. In Proceedings of the 2023 Conference on Empirical Methods in Nat… find
  37. Tran Ngoc Son, Nguyen Anh Tu, and Nguyen Minh Tri. 2025. An efficient approach for machine translation on low-resource languages: A case study in vietnamese-chinese. arXiv preprint arXiv:2501.19314. — [arXiv:2501.19314](https://arxiv.org/abs/2501.19314) find
  38. Solomon Ubani, Suleyman Olcay Polat, and Rodney Nielsen. 2023. Zeroshotdataaug: Generating and augmenting training data with chatgpt. Preprint, arXiv:2304.14334. find
  39. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in English? on the latent language of multilingual transformers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15366–15394, Bangkok, T…
  40. Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. 2024. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116. — [arXiv:2402.13116](https://arxiv.org/abs/2402.13116) find
  41. Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woomyoung Park. 2021. GPT3Mix: Leveraging large-scale language models for text augmentation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2225–2239, Punta Cana, Dominican Republic. Association for Computati…
  42. Elena Zotova, Rodrigo Agerri, and German Rigau. 2021. Semi-automatic generation of multilingual datasets for stance detection in twitter. Expert Systems with Applications, 170:114547. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/anikina-2025-rigorous-evaluation-llm-data-generation/