Elamur Research
☰
home / papers / 2019 / FlauBERT: Unsupervised Language Model Pre-training for French

FlauBERT: Unsupervised Language Model Pre-training for French

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/1912.05372

Summary

Pre-trained contextual language models such as BERT have reshaped NLP. Most of the gains, however, were shown for English. French relied mainly on multilingual models (mBERT, XLM, XLM-R) and had no shared evaluation benchmark. The authors built FlauBERT, a monolingual French model that uses the BERT architecture and was trained on the CNRS Jean Zay supercomputer. The training data combine 24 freely available sub-corpora from WMT19 monolingual data, OPUS and Wikimedia projects, ranging from books and Wikipedia to Common Crawl web text. These total 270 GB of raw text, which filtering, Unicode normalization and Moses tokenization reduced to 71 GB. Training uses only the MLM objective with dynamic masking and a 50K BPE vocabulary, producing base and large variants. The authors also introduce FLUE, a GLUE-like French suite covering text classification, paraphrasing, NLI, parsing and word sense disambiguation. On most tasks, FlauBERT beats other pre-training approaches, including multilingual models in comparable setups. The public release of the models, pipeline and benchmark gives French NLP a reproducible baseline.

Key points

  • Corpus: 24 sub-corpora with 270 GB of raw text. Common Crawl, the largest source, was cut from 215 GB to 43.4 GB through aggressive cleaning. The final training set is 71 GB, tokenized with the Moses tokenizer.
  • Models: FlauBERT BASE has L=12, H=768, A=12 and about 138M parameters. FlauBERT LARGE has L=24, H=1024, A=16 and about 373M parameters. Both use MLM only (no NSP), BPE 50K and dynamic sub-word masking.
  • Concurrent model: CamemBERT is a parallel French BERT trained on 138 GB with SentencePiece 32K and whole-word masking. The authors view the two efforts as complementary because they differ in data, preprocessing and evaluation tasks.
  • FLUE benchmark: a unified French evaluation protocol spanning classification, paraphrase, natural language inference, parsing and WSD, designed for reproducible comparison.
  • Release: the models, data download and preprocessing code are available at github.com/getalp/Flaubert. Training ran on public CNRS infrastructure without assistance from private stakeholders.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Word sense disambiguation

Cited by

216 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 209 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1912.05372 · arXiv 1912.05372

@misc{le2019flaubert,
  title = {{FlauBERT: Unsupervised Language Model Pre-training for French}},
  author = {Le, Hang and Vial, Loïc and Frej, Jibril and Segonne, Vincent and Coavoux, Maximin and Lecouteux, Benjamin and Allauzen, Alexandre and Crabbé, Benoît and Besacier, Laurent and Schwab, Didier},
  year = {2019},
  doi = {10.48550/arxiv.1912.05372},
  eprint = {1912.05372},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1912.05372}
}

References

77 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abeillé, A., Clément, L., and Toussenel, F., (2003). Building a Treebank for French, pages 165–187. find
  2. Springer Netherlands, Dordrecht. Antoun, W., Baly, F., and Hajj, H. (2020). Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. find our card
  3. Artetxe, M. and Schwenk, H. (2019). Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597–610. find our card
  4. Bengio, Y., Ducharme, R., Vincent, P., and Jauvin, C. (2003). A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137–1155. find
  5. Blitzer, J., McDonald, R., and Pereira, F. (2006). Domain adaptation with structural correspondence learning. In Proceedings of the 2006 conference on empirical methods in natural language processing, pages 120–128. Association for Computational Linguistics. https://huggingface.co/transformers/ http… ↗
  6. Dataset Post-processed text size Number of Tokens (Moses) Number of Sentences CommonCrawl (Buck et al., 2014) \#\#\# 43.4 GB \#\#\# 7.85 B 293.37 M NewsCrawl (Li et al., 2019) \#\#\# 9.2 GB \#\#\# 1.69 B \#\#\# 63.05 M Wikipedia (Meta, 2019) \#\#\# 4.2 GB 750.76 M \#\#\# 31.00 M Wikisource (Meta, 2019) \#\#\# 2.4 GB 458.85… find
  7. M GIGA (Tiedemann, 2012) \#\#\# 2.0 GB 353.33 M \#\#\# 10.65 M PCT \#\#\# 1.2 GB 197.48 M \#\#\# 7.13 M Project Gutenberg \#\#\# 1.1 GB 219.73 M \#\#\# 8.23 M OpenSubtitles (Lison and Tiedemann, 2016) \#\#\# 1.1 GB 218.85 M \#\#\# 13.98 M Le Monde 664 MB 122.97 M \#\#\# 4.79 M DGT (Tiedemann, 2012) 311 MB \#\#\# 53.31 M \#\#\# 1.73… find
  8. Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., and Inkpen, D. (2017). Enhanced lstm for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1657–1668. find
  9. Collobert, R. and Weston, J. (2008). A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160– 167. find
  10. ACM. Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa, P. (2011). Natural language processing (almost) from scratch. Journal of machine learning research, 12(Aug):2493–2537. find our card
  11. Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S., Schwenk, H., and Stoyanov, V. (2018). Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485. Conneau, A., Khandelwal, K., Goyal… find
  12. Constant, M., Candito, M., and Seddah, D. (2013). The ligm-alpage architecture for the spmrl 2013 shared task: Multiword expression analysis and dependency parsing. In Proceedings of the EMNLP Workshop on Statistical Parsing of Morphologically Rich Languages (SPMRL 2013). find
  13. Dai, A. M. and Le, Q. V. (2015). Semi-supervised sequence learning. In Advances in neural information processing systems, pages 3079–3087. de Vries, W., van Cranenburgh, A., Bisazza, A., Caselli, T., van Noord, G., and Nissim, M. (2019). Bertje: A dutch bert model. arXiv preprint arXiv:1912.09582. find our card
  14. Delobelle, P., Winters, T., and Berendt, B. (2020). Robbert: a dutch roberta-based language model. arXiv preprint arXiv:2001.06286. find our card
  15. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (… find
  16. Dozat, T. and Manning, C. D. (2016). Deep biaffine attention for neural dependency parsing. In ICLR. find our card
  17. Eisele, A. and Chen, Y. (2010). Multiun: A multilingual corpus from united nation documents. In Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC’10). find
  18. Eisenschlos, J., Ruder, S., Czapla, P., Kardas, M., Gugger, S., and Howard, J. (2019). Multifit: Efficient multilingual language model fine-tuning. In Proceedings of the 2019 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  19. Fan, A., Grave, E., and Joulin, A. (2019). Reducing transformer depth on demand with structured dropout. In International Conference on Learning Representations. find our card
  20. Gong, L., He, D., Li, Z., Qin, T., Wang, L., and Liu, T. (2019). Efficient training of bert by progressively stacking. In International Conference on Machine Learning, pages 2337–2346. find
  21. Hadj Salah, M. (2018). Arabic word sense disambiguation for and by machine translation. Theses, Université Grenoble Alpes ; Université de Sfax (Tunisie). Faculté des Sciences économiques et de gestion, December. find
  22. Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8):1735–1780. find
  23. Howard, J. and Ruder, S. (2018). Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339. find
  24. Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger, K. Q. (2016). Deep networks with stochastic depth. In European conference on computer vision, pages 646– 661. find
  25. Springer. Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  26. Kitaev, N. and Klein, D. (2018). Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2676–2686, Melbourne, Australia, July. Association for Computational Linguistics. find our card
  27. Kitaev, N., Cao, S., and Klein, D. (2019). Multilingual constituency parsing with self-attention and pre-training. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3499–3505, Florence, Italy, July. Association for Computational Linguistics. find our card
  28. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., et al. (2007). Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the association for computational linguistics compani… find
  29. Koehn, P. (2005). Europarl: A parallel corpus for statistical machine translation. find
  30. Machine Translation Summit, 2005, pages 79–86. find
  31. Kuratov, Y. and Arkhipov, M. (2019). Adaptation of deep bidirectional multilingual transformers for russian language. arXiv preprint arXiv:1905.07213. find
  32. Lample, G. and Conneau, A. (2019). Cross-lingual language model pretraining. In Advances in neural information processing systems. find
  33. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2019). Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. find
  34. Li, X., Michel, P., Anastasopoulos, A., Belinkov, Y., Durrani, N., Firat, O., Koehn, P., Neubig, G., Pino, J., and Sajjad, H. (2019). Findings of the first shared task on machine translation robustness. find
  35. WMT 2019, page 91. Lison, P. and Tiedemann, J. (2016). Opensubtitles2015: Extracting large parallel corpora from movie and tv subtitles. In International Conference on Language Resources and Evaluation. find
  36. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Martin, L., Muller, B., Ortiz Suárez, P. J., Dupont, Y., Romary, L., Villemonte de la Cl… find
  37. McCann, B., Bradbury, J., Xiong, C., and Socher, R. (2017). Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pages 6294–6305. find our card
  38. Meta. (2019). Data dumps — meta, discussion about wikimedia projects. find
  39. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS’13, pages 3111–3119, USA. find
  40. Curran Associates Inc. Miller, G. A., Leacock, C., Tengi, R., and Bunker, R. T. (1993). A semantic concordance. In Proceedings of the workshop on Human Language Technology, HLT ’93, pages 303–308, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  41. Miller, G. A. (1995). Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41. find our card
  42. Navigli, R. and Ponzetto, S. P. (2010). Babelnet: Building a very large multilingual semantic network. In Proceedings of the 48th annual meeting of the association for computational linguistics, pages 216–225. Association for Computational Linguistics. find
  43. Navigli, R., Jurgens, D., and Vannella, D. (2013). SemEval-2013 Task 12: Multilingual Word Sense Disambiguation. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 222–231. find
  44. Navigli, R. (2009). Word sense disambiguation: A survey. ACM Computing Surveys, 41(2):10:1–10:69, feb. find
  45. Nguyen, D. Q. and Nguyen, A. T. (2020). Phobert: Pre-trained language models for vietnamese. arXiv preprint arXiv:2003.00744. find our card
  46. Nguyen, T. Q. and Salazar, J. (2019). Transformers without tears: Improving the normalization of self-attention. arXiv preprint arXiv:1910.05895. find
  47. Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., and Auli, M. (2019). fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. find our card
  48. Pennington, J., Socher, R., and Manning, C. D. (2014). Glove: Global vectors for word representation. In In EMNLP. find
  49. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227–2237. find
  50. Pham, N.-Q., Nguyen, T.-S., Niehues, J., Muller, M., and Waibel, A. (2019). Very deep self-attention networks for end-to-end speech recognition. arXiv preprint arXiv:1904.13377. — [arXiv:1904.13377](https://arxiv.org/abs/1904.13377) find
  51. Polignano, M., Basile, P., de Gemmis, M., Semeraro, G., and Basile, V. (2019). AlBERTo: Italian BERT Language Understanding Model for NLP Challenging Tasks Based on Tweets. In Proceedings of the Sixth Italian Conference on Computational Linguistics (CLiC-it 2019), volume 2481. find
  52. CEUR. Prettenhofer, P. and Stein, B. (2010). Cross-language text classification using structural correspondence learning. In Proceedings of the 48th annual meeting of the association for computational linguistics, pages 1118–1127. find
  53. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2018). Improving language understanding by generative pre-training. Technical report, OpenAI. find
  54. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. find
  55. Rajpurkar, P., Jia, R., and Liang, P. (2018). Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822. find our card
  56. Ramachandran, P., Liu, P., and Le, Q. (2017). Unsupervised pretraining for sequence to sequence learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 383–391. Seddah, D., Tsarfaty, R., Kübler, S., Candito, M., Choi, J. D., Farkas, R., Foster, J.,… find
  57. Sennrich, R., Haddow, B., and Birch, A. (2016). Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725. Sérasset, G. (2012). Dbnary: Wiktionary as a lmf based mu… find
  58. Skadins, R., Tiedemann, J., Rozis, R., and Deksne, D. (2014). Billions of parallel words for free: Building and using the EU bookshop corpus. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, Reykjavik, Iceland, May 26-31, 2014, pages 1850–1855. find
  59. Souza, F., Nogueira, R., and Lotufo, R. (2019). Portuguese named entity recognition using bert-crf. arXiv preprint arXiv:1909.10649. — [arXiv:1909.10649](https://arxiv.org/abs/1909.10649) find
  60. Styler, W. (2011). The enronsent corpus. Technical Report 01-2011. University of Colorado at Boulder Institute of Cognitive Science, Boulder, CO. find
  61. Sutskever, I., Vinyals, O., and Le, Q. V. (2014). Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104– 3112. find
  62. Tiedemann, J. (2012). Parallel data, tools and interfaces in opus. In Nicoletta Calzolari (Conference Chair), et al., editors, Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey, may. European Language Resources Association (ELRA). find
  63. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  64. Vaswani, A., Bengio, S., Brevdo, E., Chollet, F., Gomez, A., Gouws, S., Jones, L., Kaiser, Ł., Kalchbrenner, N., Parmar, N., et al. (2018). Tensor2tensor for neural machine translation. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Resear… find our card
  65. Vial, L., Lecouteux, B., and Schwab, D. (2018). UF-SAC: Unification of Sense Annotated Corpora and Tools. In Language Resources and Evaluation Conference (LREC), Miyazaki, Japan, May. find
  66. Vial, L., Lecouteux, B., and Schwab, D. (2019). Sense Vocabulary Compression through the Semantic Knowledge of WordNet for Neural Word Sense Disambiguation. In Proceedings of the 10th Global Wordnet Conference, Wroclaw, Poland. find
  67. Virtanen, A., Kanerva, J., Ilo, R., Luoma, J., Luotolahti, J., Salakoski, T., Ginter, F., and Pyysalo, S. (2019). Multilingual is not enough: Bert for finnish. arXiv preprint arXiv:1912.07076. find our card
  68. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. (2018). GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belg… find our card
  69. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. (2019a). Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537. find our card
  70. Wang, Q., Li, B., Xiao, T., Zhu, J., Li, C., Wong, D. F., and Chao, L. S. (2019b). Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1810–1822. find our card
  71. Williams, A., Nangia, N., and Bowman, S. (2018). A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers),… find
  72. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtow-icz, M., and Brew, J. (2019). Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. find our card
  73. Xu, H., Liu, Q., van Genabith, J., and Zhang, J. (2019). Why deep transformers are difficult to converge? from computation order to lipschitz restricted parameter initialization. arXiv preprint arXiv:1911.03179. — [arXiv:1911.03179](https://arxiv.org/abs/1911.03179) find
  74. Yang, Y., Zhang, Y., Tar, C., and Baldridge, J. (2019a). Paws-x: A cross-lingual adversarial dataset for paraphrase identification. arXiv preprint arXiv:1908.11828. find our card
  75. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., and Le, Q. V. (2019b). Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems. find our card
  76. Zhang, H., Dauphin, Y. N., and Ma, T. (2019a). Fixup initialization: Residual learning without normalization. arXiv preprint arXiv:1901.09321. find
  77. Zhang, Y., Baldridge, J., and He, L. (2019b). Paws: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298–1308. find our card