Elamur Research
☰
home / papers / 2019 / BERTje: A Dutch BERT Model

BERTje: A Dutch BERT Model

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/1912.09582

Summary

Multilingual BERT covers Dutch, but it was trained only on Wikipedia, which is a narrow domain, and it shares its capacity across 104 languages. The authors built BERTje, a monolingual Dutch model with the same architecture as BERT BASE. They pre-trained it on about 2.4B tokens (12GB) drawn from several genres: fiction, news, the SoNaR-500 reference corpus and Wikipedia. They removed documents that overlap with their evaluation sets. The pre-training recipe follows later work on BERT. It swaps next sentence prediction for sentence order prediction, and it masks every word piece of a selected word together rather than individual fragments. After fine-tuning, BERTje beats multilingual BERT on NER, POS tagging, semantic role labeling, spatio-temporal relations and book-review sentiment. Comparing checkpoints shows that low-level information such as POS stops improving earlier than entity knowledge, which keeps improving with longer training. The released model gives Dutch NLP a strong general-purpose base, and the work supports building monolingual models for other languages.

Key points

  • Pre-training data: Books (4.4GB), TwNC (2.4GB), SoNaR-500 (2.2GB, with chat, Twitter and Wikipedia documents removed), Web news from 2015–2019 (1.6GB) and the Wikipedia October 2019 dump (1.5GB). Texts that also appear in SoNaR-1 and Lassy Small were excluded.
  • Vocabulary and training: The vocabulary has 30K entries, built with SentencePiece and then converted to WordPiece format. Training uses the SOP objective with whole-word masking of 15% of tokens and runs for 1M iterations; a checkpoint at 850k iterations was also evaluated.
  • NER results: On CoNLL-2002, test F1 rose from 80.7 with multilingual BERT to 88.3 with BERTje. This is still below the 90.9 reported by Wu and Dredze (2019). On SoNaR-1, test F1 rose from 79.7 to 82.1.
  • POS tagging: On UD Lassy Small (UD v2.5), BERTje's accuracy exceeds the 95.98% of UDPipe 2.0. The 850k and 1M checkpoints differ by no more than 0.3 points.
  • Sentiment analysis: On the 110k Dutch Book Reviews Dataset, BERTje reaches 93.0% accuracy without hyperparameter tuning. Multilingual BERT gets 89.1%, and a manually tuned ULMFiT model gets 93.8%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

163 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 151 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1912.09582 · arXiv 1912.09582

@misc{vries2019bertje,
  title = {{BERTje: A Dutch BERT Model}},
  author = {Vries, Wietse de and Cranenburgh, Andreas van and Bisazza, Arianna and Caselli, Tommaso and Noord, Gertjan van and Nissim, Malvina},
  year = {2019},
  doi = {10.48550/arxiv.1912.09582},
  eprint = {1912.09582},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1912.09582}
}

References

21 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Benjamin van der Burgh and Suzan Verberne. 2019. The merits of universal language model fine-tuning for small datasets–a case with Dutch book reviews. arXiv preprint 1910.00896. ↗
  2. Isabelle Delaere, Veronique Hoste, and Paola Monachesi. 2009. Cultivating trees: Adding several semantic layers to the Lassy treebank in SoNaR. In 7th International workshop on Treebanks and Linguistic Theories (TLT-7), pages 135–146. LOT (Landelijke Onderzoekschool Taalwetenschap). find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL, pages 4171–4186. find
  4. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics. find
  5. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of ACL, pages 3651–3657. find our card
  6. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of EMNLP: System Demonstrations, pages 66–71. find our card
  7. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite BERT for self-supervised learning of language representations. arXiv preprint 1909.11942. ↗
  8. Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoît Crabbé, Laurent Besacier, and Didier Schwab. 2019. FlauBERT: Unsupervised language model pre-training for French. find our card
  9. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint 1907.11692. ↗
  10. Gertjan van Noord, Gosse Bouma, Frank Van Eynde, Daniël de Kok, Jelmer van der Linde, Ineke Schuurman, Erik Tjong Kim Sang, and Vincent Vandeghinste. 2013. Large scale syntactic annotation of written Dutch: Lassy. In Peter Spyns and Jan Odijk, editors, Essential Speech and Language Technology for Dutch: Results by the STEVIN programme, pages 147–164. Springer Berlin Heidelberg, Berlin, Heidelberg. find
  11. Nelleke Oostdijk, Martin Reynaert, Véronique Hoste, and Ineke Schuurman. 2013. The construction of a 500-million-word reference corpus of contemporary written Dutch. In Peter Spyns and Jan Odijk, editors, Essential Speech and Language Technology for Dutch: Results by the STEVIN programme, pages 219–247. Springer Berlin Heidelberg, Berlin, Heidelberg. find
  12. Roeland J.F. Ordelman, Franciska M.G. de Jong, Adrianus J. van Hessen, and G.H.W. Hondorp. 2007. TwNC: a multifaceted Dutch news corpus. ELRA Newsletter, 12(3-4). find
  13. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. find
  14. Marco Polignano, Pierpaolo Basile, Marco de Gemmis, Giovanni Semeraro, and Valerio Basile. 2019. Alberto: Italian BERT language understanding model for NLP challenging tasks based on tweets. In Proceedings of the Sixth Italian Conference on Computational Linguistics (CLiC-it 2019). CEUR. find
  15. Milan Straka. 2018. UDPipe 2.0 prototype at CoNLL 2018 UD shared task. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 197–207. find our card
  16. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of ACL, pages 4593–4601. find our card
  17. Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In Proceedings of CoNLL. find
  18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  19. Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: BERT for Finnish. arXiv preprint 1912.07076. ↗ our card
  20. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of EMNLP, pages 833–844. find our card
  21. Daniel Zeman, Joakim Nivre, et al. 2019. Universal dependencies 2.5. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find