Elamur Research⌕
☰
home / papers / 2024 / On the Utility of Pretraining Language Models on Synthetic Data

On the Utility of Pretraining Language Models on Synthetic Data

Paper · 2024 · Alcides Alcoba Inciarte, Sang Yun Kwon, El Moatez Billah Nagoudi et al.

Summary

Pretraining language models normally needs very large amounts of clean text, which low-resource languages often lack. The authors test whether Arabic models can be pretrained mostly or entirely on machine-produced text. They build four synthetic corpora:

  • MT: English Wikipedia translated into Arabic with NLLB.
  • OCR: Hindawi library books run through Tesseract.
  • ASR: transcripts from Arabic speech collections, some made with a Whisper-based Arabic model.
  • TG: sentence completions produced by the Arabic GPT model Jasmine.

Arabic Wikipedia serves as the gold baseline. They pretrain BERT Base, T5 and GPT-2 style models on each source and on stacked combinations, then evaluate on the ORCA benchmark for Arabic NLU against ARBERT v2. Synthetic-trained models come close to, and in some settings beat, gold-trained ones. A GPT-2 model trained on combined synthetic data is reported to exceed ARBERT v2. The authors also find that mixing heterogeneous sources helps more than relying on a single uniform corpus like Wikipedia. This matters for any language where clean web text is scarce but scans, audio or translation systems exist.

Key points

  • Corpus sizes are roughly matched at about 100M words each, except ASR:
    • Arabic Wikipedia (gold): 8.8M sentences / 102M words
    • Translated English Wikipedia: 6M / 103M
    • OCR: 11.6M / 101M
    • ASR: 1.9M / 25M
    • TG: 8.8M / 110M
  • OCR data comes from 3,000 Hindawi books, read with Google Tesseract at page level and then split into paragraphs.
  • ASR data draws on four collections:
    • MGB-2 (~1,200 h of Aljazeera)
    • Common Voice (all 11 versions, ~690 h)
    • Arabic Speech Corpus (3.7 h)
    • Massive Arabic Speech Corpus (1,000+ h from 700+ YouTube channels)
  • Mixtures C1–C4 add sources step by step: MT+OCR, then +ASR, then +TG, then +Gold.
  • Per-task ORCA scores track ARBERT v2 closely. The correlation is 0.85 for the decoder-only model, 0.728 for encoder-only and 0.725 for encoder-decoder.
  • The authors flag several limitations:
    • Only the ASR data contains dialectal Arabic.
    • Model sizes differ across architectures.
    • Evaluation covers NLU only.
    • The OCR text is academic in register.

Interesting ideas

  • I1. Keeping every data source at roughly the same word budget lets the effect of where text comes from be separated from how much there is.
  • I2. Besides the aggregate score, they correlate a model's per-task score profile with a strong reference model, which shows whether a cheap model fails in the same places or in different ones.
  • I3. Synthetic text is produced by letting a generator complete the opening of real sentences, with repetition prevention, so the output stays anchored to genuine content while adding novelty.
  • I4. Speech transcripts supply registers (dialect, informal broadcast talk) that clean encyclopedic text lacks, and this boosted the corresponding tasks.
  • I5. Machine-translated text is filtered by dropping outputs with degenerate repeated words, a cheap guard against a typical MT failure.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When comparing silver corpora from different witness parsers or LLM teacher tiers, cap each at an equal token count so gains reflect source quality rather than volume.
  • I2 (learn): Correlating per-category accuracy against a reference parser or reader would reveal whether a new training regime shifts error patterns, complementing negative controls.
  • I3 (back): Seeding generation from real sentence prefixes and filtering repeats could yield varied yet grounded paraphrases for testing the text → graph → text compressor.
  • I4 (tokens): Adding ASR-style transcribed English to tagger and lemmatizer training could harden them against disfluent, conversational input that Wikipedia-like text never shows.
  • I5 (tree): If translated sentences are ever used to grow parser training data, a simple repetition filter before silver parsing would remove broken inputs that corrupt treebank-style annotations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada98 works · 34,805 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksSpeech & spoken language · 480 works

Cite

Cite as Alcides Alcoba Inciarte, Sang Yun Kwon, El Moatez Billah Nagoudi and Muhammad Abdul-Mageed. 2024. On the Utility of Pretraining Language Models on Synthetic Data. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.23

@inproceedings{inciarte2024utility,
  title = {{On the Utility of Pretraining Language Models on Synthetic Data}},
  author = {Alcides Alcoba Inciarte, Sang Yun Kwon, El Moatez Billah Nagoudi and Muhammad Abdul-Mageed},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.23},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.23}
}

References

43 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020a. Arbert & marbert: Deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. ↗
  2. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020b. NADI 2020: The first nuanced Arabic dialect identification shared task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 97–110, Barcelona, Spain (Online). Association for Computational Linguistics. https://alliancecan.ca ↗
  3. Mohammad Al-Fetyani, Muhammad Al-Barham, Gheith Abandah, Adham Alsharkawi, and Maha Dawas. 2021. Masc: Massive arabic speech corpus. find
  4. Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G Baraniuk. 2023. Self-consuming generative models go mad. arXiv preprint arXiv:2307.01850. ↗
  5. Ahmed Ali, Peter Bell, James Glass, Yacine Messaoui, Hamdy Mubarak, Steve Renals, and Yifan Zhang. 2016. The mgb-2 challenge: Arabic multi-dialect broadcast media recognition. In 2016 IEEE Spoken Language Technology Workshop (SLT), pages 279– 284. IEEE.
  6. Elfilali Ali. 2023. Hindawi books dataset. Dataset. find
  7. Manel Aloui, Hasna Chouikhi, Ghaith Chaabane, Haithem Kchaou, and Chehir Dhaouadi. 2024. 101 billion arabic words dataset. arXiv preprint arXiv:2405.01590. ↗
  8. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. 2019. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670. ↗
  9. Giusepppe Attardi. 2015. Wikiextractor. https://github.com/attardi/wikiextractor . ↗
  10. Riadh Belkebir and Nizar Habash. 2021. Automatic error type annotation for arabic. arXiv preprint arXiv:2109.08068. ↗
  11. Arne Bewersdorff, Christian Hartmann, Marie Hornberger, Kathrin Seßler, Maria Bannert, Enkelejda Kasneci, Gjergji Kasneci, Xiaoming Zhai, and Claudia Nerdel. 2024. Taking the next step with generative artificial intelligence: The transformative role of multimodal large language models in science education. arXiv preprint arXiv:2401.00832. ↗
  12. Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. 2023. Audiolm: a language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing. ↗
  13. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. ↗
  14. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  15. Jiaao Chen, Derek Tam, Colin Raffel, Mohit Bansal, and Diyi Yang. 2023. An empirical survey of data augmentation for limited data learning in nlp. Transactions of the Association for Computational Linguistics, 11:191–211. ↗
  16. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. ↗
  17. Fahim Dalvi, Yifan Zhang, Sameer Khurana, Nadir Durrani, Hassan Sajjad, Ahmed Abdelali, Hamdy Mubarak, Ahmed Ali, and Stephan Vogel. 2017. QCRI live speech translation system. In Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 61–64, Valencia, Spain. Association for Computational Linguistics. find
  18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  19. Bosheng Ding, Linlin Liu, Lidong Bing, Canasai Kruengkrai, Thien Hai Nguyen, Shafiq Joty, Luo Si, and Chunyan Miao. 2020. Daga: Data augmentation with a generation approach for low-resource tagging tasks. arXiv preprint arXiv:2011.01549. ↗
  20. AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2022. Orca: A challenging benchmark for arabic language understanding. arXiv preprint arXiv:2212.10758. ↗
  21. Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021. A survey of data augmentation approaches for NLP. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 968–988, Online. Association for Computational Linguistics.
  22. Nawar Halabi. 2016. Modern standard Arabic phonetics for speech synthesis. Ph.D. thesis, University of Southampton. find
  23. Xiaoshuai Hao, Yi Zhu, Srikar Appalaraju, Aston Zhang, Wanqian Zhang, Bo Li, and Mu Li. 2023. Mixgen: A new multi-modal data augmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 379–389. ↗
  24. Alex Hernández-García and Peter König. 2018. Data augmentation instead of explicit regularization. arXiv preprint arXiv:1806.03852. ↗
  25. Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Gptaraeval: a comprehensive evaluation of chatgpt on arabic nlp. arXiv preprint arXiv:2305.14976. ↗
  26. Sang Kwon, Gagan Bhatia, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Beyond English: Evaluating LLMs for Arabic grammatical error correction. In Proceedings of ArabicNLP 2023, pages 101–119, Singapore (Hybrid). Association for Computational Linguistics. find
  27. Zichang Liu, Zhiqiang Tang, Xingjian Shi, Aston Zhang, Mu Li, Anshumali Shrivastava, and Andrew Gordon Wilson. 2022. Learning multimodal data augmentation in feature space. arXiv preprint arXiv:2212.14453. ↗
  28. Alexandre Magueresse, Vincent Carles, and Evan Heetderks. 2020. Low-resource languages: A review of past work and future challenges. arXiv preprint arXiv:2006.07264. ↗
  29. El Moatez Billah Nagoudi, Muhammad Abdul-Mageed, AbdelRahim Elmadany, Alcides Alcoba Inciarte, and Md Tawkat Islam Khondaker. 2022. Jasmine: Arabic gpt models for few-shot learning. arXiv preprint arXiv:2212.10755. ↗
  30. El Moatez Billah Nagoudi, Ahmed El-Shangiti, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2023. Dolphin: A challenging and diverse benchmark for arabic nlg. arXiv preprint arXiv:2305.14989. ↗
  31. Rasha Obeidat, Marwa Al-Harbi, Mahmoud Al-Ayyoub, and Luay Alawneh. 2024. Arquad: An expert-annotated arabic machine reading comprehension dataset. Cognitive Computation, pages 1–20. ↗
  32. Luis Perez and Jason Wang. 2017. The effectiveness of data augmentation in image classification using deep learning. arXiv preprint arXiv:1712.04621. ↗
  33. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  34. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67. ↗
  35. Dania Refai, Saleh Abu-Soud, and Mohammad J Abdel-Rahman. 2023. Data augmentation using transformers and similarity measures for improving arabic text classification. IEEE Access, 11:132516–132531. ↗
  36. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695. ↗
  37. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278–25294. ↗
  38. Connor Shorten and Taghi M. Khoshgoftaar. 2019. A survey on image data augmentation for deep learning. J. Big Data, 6:60. ↗
  39. R. Smith. 2007. An overview of the tesseract ocr engine. volume 2, pages 629 – 633. find
  40. Aiman Solyman, Wang Zhenyu, Tao Qian, Arafat Abdulgader Mohammed Elhag, Muhammad Toseef, and Zeinab Aleibeid. 2021. Synthetic data with neural machine translation for automatic correction in arabic grammar. Egyptian Informatics Journal, 22(3):303– 315. ↗
  41. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27. ↗
  42. Bashar Talafha, Abdul Waheed, and Muhammad Abdul-Mageed. 2023. N-shot benchmarking of whisper on diverse arabic speech recognition. arXiv preprint arXiv:2306.02902. ↗
  43. Terry Yue Zhuo, Zhou Yang, Zhensu Sun, Yufei Wang, Li Li, Xiaoning Du, Zhenchang Xing, and David Lo. 2023. Data augmentation approaches for source code models: A survey. arXiv preprint arXiv:2305.19915. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/inciarte-2024-utility-pretraining-language-models-synthetic/