Elamur Research
☰
home / papers / 2020 / RobBERT: a Dutch RoBERTa-based Language Model

RobBERT: a Dutch RoBERTa-based Language Model

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2001.06286

Summary

Multilingual models like mBERT usually trail models trained on one language, so the authors built a dedicated Dutch encoder. RobBERT follows the RoBERTa base architecture and training recipe, which uses masked language modelling only and drops next-sentence prediction. It is pre-trained on the Dutch part of the OSCAR corpus, a Common Crawl derivative that is far larger than the data behind the competing Dutch models BERTje and BERT-NL. There are two variants. v1 keeps RoBERTa's original BPE vocabulary, and v2 uses a byte-level BPE tokenizer learned on Dutch text. Comparing them isolates the value of a language-specific tokenizer. Evaluation covers sentiment classification (DBRD), die/dat pronoun prediction (including zero-shot use), POS tagging, NER and a gender-fairness analysis. RobBERT v2 improves the state of the art on several tasks, and its lead is widest when fine-tuning data is scarce. All pre-trained and fine-tuned models are openly released, giving Dutch NLP a strong reusable foundation.

Key points

  • Pre-training data: Dutch OSCAR, about 39GB, 6.6 billion words in 126 million lines. BERTje used roughly 12GB and BERT-NL used SoNaR-500 (about 2.2GB).
  • Model and training: RoBERTa-base layout with 12 layers, 12 attention heads and about 117M parameters. The v2 Dutch vocabulary has 40k tokens, versus 50k in v1. Training used Fairseq with a fixed batch of 8192 sentences for two epochs (over 16k batches). It took about three days on up to 80 Nvidia P100 GPUs.
  • Sentiment results (DBRD accuracy): RobBERT v2 reaches 95.144% on the full training set, compared with 93.8% for the earlier ULMFiT result and 93.0% for BERTje. With only 10k training examples, v2 gets 94.379% while v1 gets 86.730%. This large gap points to the importance of the Dutch tokenizer.
  • Fairness on DBRD: The test compares male and female reviewers. For positive reviews, the demographic parity ratio is 70.2%, which falls below the common 80% threshold. However, the equal-opportunity difference is only 0.028. Gaps become larger for highly positive reviews.
  • Availability and future work: Code and models are published at github.com/iPieter/RobBERT. Proposed next steps include an unshuffled OSCAR, morpheme-aware tokens for Dutch compounds, newer pre-training objectives and fairness-aware training.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

111 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 105 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2001.06286 · arXiv 2001.06286

@misc{delobelle2020robbert,
  title = {{RobBERT: a Dutch RoBERTa-based Language Model}},
  author = {Delobelle, Pieter and Winters, Thomas and Berendt, Bettina},
  year = {2020},
  doi = {10.48550/arxiv.2001.06286},
  eprint = {2001.06286},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2001.06286}
}

References

49 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Liesbeth Allein, Artuur Leeuwenberg, and Marie-Francine Moens. 2020. Binary and Multitask Classification Model for Dutch Anaphora Resolution: Die/Dat Prediction. arXiv:2001.02943 [cs]. ↗
  2. Doris Allhutter, Florian Cech, Fabian Fischer, Gabriel Grill, and Astrid Mager. 2020. Algorithmic Profiling of Job Seekers in Austria: How Austerity Politics Are Made Effective. Front. Big Data, 3:5. find
  3. Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias. find
  4. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (Technology) is Power: A Critical Survey of ”Bias” in NLP. arXiv:2005.14050 [cs]. ↗ our card
  5. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems, pages 4349–4357. find our card
  6. Antal van den Bosch, Bertjan Busser, Sander Canisius, and Walter Daelemans. 2007. An efficient memory-based morphosyntactic tagger and parser for dutch. LOT Occasional Series, 7:191–206. find
  7. Alex Brandsen, Anne Dirkson, Suzan Verberne, Maya Sappelli, Dung Manh Chu, and Kimberly Stoutjes-dijk. 2019. BERT-NL a set of language models pre-trained on the Dutch SoNaR corpus. find
  8. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. find
  9. Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. BERTje: A Dutch BERT Model. arXiv:1912.09582 [cs]. ↗ our card
  10. Pieter Delobelle, Paul Temple, Gilles Perrouin, Benoı̂t Frénay, Patrick Heymans, and Bettina Berendt. 2020. Ethical Adversaries: Towards Mitigating Unfairness with Adversarial Machine Learning. arXiv:2005.06852 [cs, stat]. ↗
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  12. Dimitrios Papadimoulis. 2018. Genderneutraal taalgebruik in het Europees Parlement. Technical report, European Parlement. find
  13. Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness Through Awareness. In 3rd Innovations in Theoretical Computer Science Conference, pages 214–226. ACM. find
  14. Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 609–614, Minneapolis, Minnesota. Association for Computational Linguistics. find our card
  15. Sepand Haghighi, Masoomeh Jasemi, Shaahin Hessabi, and Alireza Zolanvari. 2018. PyCM: Multiclass confusion matrix library in Python. Journal of Open Source Software, 3(25):729. find
  16. Moritz Hardt, Eric Price, ecprice, and Nati Srebro. 2016. Equality of Opportunity in Supervised Learning. In NIPS, pages 3315–3323. Curran Associates. find our card
  17. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019a. Spanbert: Improving pre-training by representing and predicting spans. find our card
  18. Mandar Joshi, Omer Levy, Daniel S Weld, and Luke Zettlemoyer. 2019b. Bert for coreference resolution: Baselines and analysis. arXiv preprint arXiv:1908.09091. ↗ our card
  19. Diederik P. Kingma and Jimmy Ba. 2017. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs]. ↗
  20. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT Summit, volume 5, pages 79–86. [object Object]. find
  21. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166–172, Florence, Italy. Association for Computational Linguistics. find our card
  22. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. ↗
  23. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. find
  24. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. find our card
  25. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692 [cs]. ↗
  26. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, and Benoı̂t Sagot. 2019. CamemBERT: A Tasty French Language Model. arXiv:1911.03894 [cs]. ↗ our card
  27. Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On Measuring Social Biases in Sentence Encoders. arXiv:1903.10561 [cs]. ↗ our card
  28. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. ↗
  29. Nelleke Oostdijk, Martin Reynaert, Véronique Hoste, and Ineke Schuurman. 2013a. The construction of a 500-million-word reference corpus of contemporary written dutch. In Essential speech and language technology for Dutch, pages 219–247. Springer. find
  30. Nelleke Oostdijk, Martin Reynaert, Véronique Hoste, and Ineke Schuurman. 2013b. The construction of a 500-million-word reference corpus of contemporary written Dutch. In Essential Speech and Language Technology for Dutch: Results by the STEVIN-Programme, chapter 13. Springer Verlag. find
  31. Pedro Javier Ortiz Suárez, Benoı̂t Sagot, and Laurent Romary. 2019. Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7), Cardiff, United Kingdom. find
  32. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. Fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. find our card
  33. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pages 8024–8035. find our card
  34. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. ↗
  35. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. find
  36. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? arXiv preprint arXiv:1906.01502. ↗ our card
  37. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2.amazonaws. com/openai-assets/researchcovers/languageunsupervised/language Thomas Wolf, Lysandre Debut, Victor Sanh, Julien understanding paper. pdf. Chaumond, Clement Delangue, Anthony Moi, Pier-Alec Radford, Jeffrey Wu, Rewon Child, David Luan, ric Cistac, Tim Rault, R’emi Louf, Morgan Funtow-Dario Amodei, and Ilya Sutskever. 2019. Language icz, and Jamie Brew. 2019. HuggingFace’s trans-models are unsupervised multitask learners. OpenAI formers: State-of-the-art natural language process-Blog, 1(8). ing. ArXiv, abs/1910.03771. ↗ our card
  38. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv:1910.10683 [cs, stat]. ↗
  39. Samuel Rönnqvist, Jenna Kanerva, Tapio Salakoski, and Filip Ginter. 2019. Is multilingual BERT fluent in language generation? In Proceedings of the First NLPL Workshop on Deep Learning for Natural Language Processing, pages 29–36, Turku, Finland. Linköping University Electronic Press. find
  40. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958. find
  41. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune bert for text classification? In China National Conference on Chinese Computational Linguistics, pages 194–206. Springer. find our card
  42. Erik F. Tjong Kim Sang. 2002. Introduction to the conll-2002 shared task: Language-independent named entity recognition. In Proceedings of the 6th Conference on Natural Language Learning - Volume 20, COLING-02, page 14, USA. Association for Computational Linguistics. find
  43. Benjamin van der Burgh and Suzan Verberne. 2019. The merits of Universal Language Model Fine-tuning for Small Datasets – a case with Dutch book reviews. arXiv:1910.00896 [cs]. ↗
  44. Jeroen Van Hautte, Vincent Schelstraete, and Mikaël Wornoo. 2020. Leveraging the inherent hierarchy of vacancy titles for automated job ontology expansion. In Proceedings of the 6th International Workshop on Computational Terminology, pages 37–42, Marseille, France. European Language Resources Association. find
  45. Gertjan Van Noord, Gosse Bouma, Frank Van Eynde, Daniel De Kok, Jelmer Van der Linde, Ineke Schuurman, Erik Tjong Kim Sang, and Vincent Vandeghinste. 2013. Large scale syntactic annotation of written Dutch: Lassy, pages 147–164. Springer. find
  46. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc. find
  47. Shijie Wu and Mark Dredze. 2019. Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844, Hong Kong, China. Association for Computational Linguistics. find our card
  48. Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. 2013. Learning fair representations. In International Conference on Machine Learning, pages 325–333. find
  49. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender Bias in Contextualized Word Embeddings. arXiv:1904.03310 [cs]. ↗ our card