Elamur Research⌕
☰
home / papers / 2022 / Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltese

Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltese

Paper · 2022 · Kurt Micallef, Albert Gatt, Marc Tanti et al.

Summary

Maltese is a low-resource language. It is absent from mBERT and XLM-R, and mBERT does poorly on it without adaptation. The authors address this in two ways. First, they build Korpus Malti v4.0 (KM), a curated corpus of a little under 500M tokens. Instead of scraping the web at random, they gather text from chosen sources, filter it by language, remove duplicates and sort it into 19 domains. Second, they train two BERT models on it: BERTu, a monolingual model built from scratch, and mBERTu, which continues pre-training mBERT on Maltese.

They test both on four tasks: dependency parsing, POS tagging, NER and sentiment analysis. Further experiments vary how much data is used and which domains it comes from.

Both models set new state-of-the-art results. BERTu generally matches or beats mBERTu, with its clearest gains on the higher-level task. Mixed-domain data tends to beat Wikipedia-only data of comparable size, and even a small part of KM produces large improvements. For other low-resource languages, the message is that a modest, carefully collected corpus can be enough.

Key points

  • Corpus: KM v4.0 is roughly double the size of Korpus Malti v3.0 and covers 19 domains, such as news, legal texts, parliament transcripts, blogs and theses.
    • Non-Maltese sentences were removed with LiD, and duplicates with Onion.
    • Wikipedia accounts for under 1% of tokens and sentences.
  • BERTu setup:
    • BERT BASE architecture (12 layers, hidden size 768, 12 heads) with a 52K vocabulary.
    • Trained with MLM only, since the authors drop NSP.
    • 1M steps at batch size 512: sequence length 128 for 90% of steps, then 512.
    • About 53 hours on 8 and then 16 A100 GPUs.
  • Data-size study: models were pre-trained on 10%, 20% … 100% of KM, and each result is the mean of 5 seeds.
    • With only 10% of the data, every configuration beats mBERT.
    • On NER, BERTu and mBERTu only get close to each other at around 70% of the data.
  • Diminishing returns: gains grow roughly linearly while data grows exponentially. The curves are not monotonic; for example, BERTu trained on 70% of the data beat the 100% model on sentiment.
  • Releases: the corpus and both models are on the Hugging Face Hub under MLRS (korpus_malti, BERTu, mBERTu), and the code is on GitHub.

Interesting ideas

  • I1. Every KM document records its source and domain, so users can pick subsets for domain adaptation, trace text back to where it came from, or remove a source if ethical problems surface later.
  • I2. The authors pick sources deliberately rather than crawling widely. They note that the web-sourced Maltese OSCAR data looks, to a native speaker, like low-quality machine translation.
  • I3. They compare mixed-domain and Wikipedia-only pre-training at similar small sizes. This separates the effect of domain variety from the effect of data volume.
  • I4. They draw learning curves at 10% steps, average 5 seeds and show the spread. This makes it visible that more data does not always help and that gains shrink.
  • I5. They compare a from-scratch monolingual model with continued pre-training of a multilingual one. The monolingual model's advantage was largest on the higher-level, meaning-oriented task.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Tag each silver sentence with its origin text, genre and agreeing witness parsers. A bad source or genre can then be dropped from training without rebuilding the whole set.
  • I2 (tokens): Build lexicons and lemma or morphology statistics from curated, language-checked and deduplicated text. This keeps machine-translated noise out of the tokenizer and tagger.
  • I3 (tree): When mixing or converting treebanks for gparse, compare a multi-genre mix with a single-genre set of equal size. This shows whether variety, not volume, is what helps.
  • I4 (learn): Use the same seeded learning-curve protocol to choose how much silver data each training round needs. Treat a single-run jump as unreliable unless it falls outside the seed variance.
  • I5 (answer): Evaluate encoder choices on the end tasks (fable QA, word problems), not only on tagging or LAS. The two kinds of score can rank encoders differently.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Utrecht UniversityUtrecht · Netherlands39 works · 1,603 citations🎓University of MaltaMsida · Malta29 works · 894 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksLow-resource languages · 267 works

Cite

Cite as Kurt Micallef, Albert Gatt, Marc Tanti, Lonneke van der Plas and Claudia Borg. 2022. Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltese. Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.deeplo-1.10

@inproceedings{micallef2022pre,
  title = {{Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltese}},
  author = {Kurt Micallef, Albert Gatt, Marc Tanti, Lonneke {van der Plas} and Claudia Borg},
  booktitle = {Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.deeplo-1.10},
  url = {https://doi.org/10.18653/v1/2022.deeplo-1.10}
}

References

39 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rodrigo Agerri, Iñaki San Vicente, Jon Ander Campos, Ander Barrena, Xabier Saralegi, Aitor Soroa, and Eneko Agirre. 2020. Give your text representation models some love: the case for Basque. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4781–4788, Marseille, France. European Language Resources Association. ↗
  2. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  3. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the fourth ACM Conference on Fairness, Accountability, and Transparency (FAccT’21), Online. Association for Computing Machinery. find
  4. Joseph Brincat. 2011. Maltese and other languages: A linguistic history of Malta. Midsea Books, Malta. find
  5. Slavomír Čéplö. 2018. Constituent order in Maltese: A quantitative analysis. Ph.D. thesis, Charles University, Prague. find
  6. Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020. Parsing with multilingual BERT, a small corpus, and a small treebank. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1324–1334, Online. Association for Computational Linguistics.
  7. and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. Association for Computational Linguistics. find
  8. Ethan C. Chau and Noah A. Smith. 2021. Specializing multilingual language models: An empirical study. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 51–61, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  9. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339–351. ↗
  10. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  11. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span-BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77.
  12. Dan Kondratyuk and Milan Straka. 2019. 75 languages, 1 model: Parsing Universal Dependencies universally. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2779–2795, Hong Kong, China. Association for Computational Linguistics. ↗
  13. Keith Cortis and Brian Davis. 2019. A social opinion gold standard for the Malta government budget 2018. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 364–369, Hong Kong, China. Association for Computational Linguistics. find
  14. Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. 2022. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics, 10:50–72. ↗
  15. Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. BERTje: A dutch BERT model. ↗
  16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  17. Alexei Dingli and Nicole Sant. 2016. Sentiment analysis on Maltese using machine learning. In The Tenth International Conference on Advances in Semantic Processing (SEMAPRO 2016), pages 21–25. find
  18. Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open-Review.net.
  19. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240. ↗
  20. Albert Gatt and Slavomír Čéplö. 2013. Digital Corpora and Other Electronic Resources for Maltese. In Proceedings of the International Conference on Corpus Linguistics, pages 96–97. UCREL, Lancaster, UK. find
  21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. ↗
  22. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, Marco Lui and Timothy Baldwin. 2014. Accurate language identification of Twitter messages. In Proceedings of the 5th Workshop on Language Analysis for Social Media (LASM), pages 17–25, Gothenburg, Sweden. Association for Computational Linguistics. find
  23. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203– 7219, Online. Association for Computational Linguistics.
  24. Antonio Martínez-García, Toni Badia, and Jeremy Barnes. 2021. Evaluating morphological typology in zero-shot cross-lingual transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3136–3153, Online. Association for Computational Linguistics. find
  25. Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 448–462, Online. Association for Computational Linguistics.
  26. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9–16, Mannheim. Leibniz-Institut für Deutsche Sprache. ↗
  27. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada. Association for Computational Linguistics.
  28. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2021. UNKs everywhere: Adapting multilingual language models to new scripts. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10186–10203, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  29. Marco Polignano, Pierpaolo Basile, Marco De Gemmis, Giovanni Semeraro, and Valerio Basile. 2019. Al-BERTo: Italian bert language understanding model for NLP challenging tasks based on tweets. In 6th Italian Conference on Computational Linguistics, CLiC-it 2019, volume 2481, pages 1–6. CEUR. find
  30. Jan Pomikálek. 2011. Removing boilerplate and duplicate content from web corpora. Ph.D. thesis, Masaryk university, Faculty of informatics, Brno, Czech Republic. find
  31. Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Massively multilingual transfer for NER. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151–164, Florence, Italy. Association for Computational Linguistics.
  32. Anna Rogers. 2021. Changing the world by changing the data. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2182–2194, Online. Association for Computational Linguistics. ↗
  33. Mike Rosner and Claudia Borg. 2022. Report on the Maltese Language. Language Technology Support of Europe’s Languages in 2020/2021. Maria Giagkou, Stelios Piperidis, Georg Rehm, Jane Dunne (Series Editors). Available online at https://european-language-equality.eu/deliverables/. ↗
  34. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3118–3135, Online. Association for Computational Linguistics.
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Llion Uszkoreit, Jakob an Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  36. Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: Bert for finnish. ↗
  37. Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. 2020. Extending multilingual BERT to low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2649–2656, Online. Association for Computational Linguistics. find
  38. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844, Hong Kong, China. Association for Computational Linguistics. ↗
  39. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120–130, Online. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/micallef-2022-pre-training-data-quality-quantity/