Elamur Research
☰
home / papers / 2020 / PhoBERT: Pre-trained language models for Vietnamese

PhoBERT: Pre-trained language models for Vietnamese

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.findings-emnlp.92.pdf

Summary

Before this work, Vietnamese had no publicly available large-scale monolingual BERT-style model. Existing monolingual and multilingual models had two weaknesses. They applied BPE directly to syllables, even though Vietnamese uses spaces between syllables and many words span several syllables. They also relied mainly on the small Vietnamese Wikipedia (about 1GB). The authors built PhoBERT in base and large sizes, following the BERT architectures and the RoBERTa pre-training recipe in fairseq. They first word-segmented a 20GB corpus of Wikipedia and deduplicated news text with RDRSegmenter, then applied fastBPE. The models were then fine-tuned and evaluated on four downstream problems: part-of-speech (POS) tagging, named-entity recognition (NER), natural language inference (NLI) and parsing of dependency structure. PhoBERT set new state-of-the-art results on all four and beat XLM-R on every task, even though XLM-R saw roughly seven times more Vietnamese text and has more parameters. This supports the view that dedicated single-language models can outperform multilingual ones. Because the models are released for fairseq and transformers, they serve as a strong baseline for Vietnamese NLP.

Key points

  • Pre-training data: about 1GB of Vietnamese Wikipedia plus about 19GB of news text, filtered for duplicates from a 50GB news corpus. After segmentation this gives about 145M sentences and about 3B word tokens, with a 64K subword vocabulary.
  • Training: both models were trained on 4 V100 GPUs. PhoBERT base took about 3 weeks (≈540K steps) and PhoBERT large about 5 weeks (≈1.08M steps).
  • Benchmarks: VLSP 2013 POS, VnDT treebank v1.1, VLSP 2016 NER and the Vietnamese portion of XNLI.
  • Best scores:
    • POS accuracy: 96.8 (PhoBERT large).
    • Dependency parsing: LAS 78.77 / UAS 85.22 (Biaffine parser with PhoBERT base).
    • NER F1: 94.7 (PhoBERT large).
    • NLI accuracy: 80.0 (PhoBERT large).
  • Model size and caveats: PhoBERT is smaller than XLM-R (135M vs 250M parameters for base, 370M vs 560M for large). However, the large model parsed slightly worse than the base model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Named entity recognition · Dependency parsing

Cited by

115 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 112 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.findings-emnlp.92

@inproceedings{nguyen2020phobert,
  title = {{PhoBERT: Pre-trained language models for Vietnamese}},
  author = {Nguyen, Dat Quoc and Nguyen, Anh Tuan},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.92},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.92}
}

References

34 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe and Holger Schwenk. 2019. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond. TACL, 7:597– 610. find our card
  2. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of ACL, pages 8440–8451. find
  3. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual Language Model Pretraining. In Proceedings of NeurIPS, pages 7059–7069. find
  4. Alexis Conneau, Ruty Rinott, Guillaume Lample, Holger Schwenk, Ves Stoyanov, Adina Williams, and Samuel R. Bowman. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of EMNLP, pages 2475–2485. find
  5. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019. Pre-Training with Whole Word Masking for Chinese BERT. arXiv preprint, arXiv:1906.08101. find our card
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL, pages 4171– 4186. find
  7. Timothy Dozat and Christopher D. Manning. 2017. Deep Biaffine Attention for Neural Dependency Parsing. In Proceedings of ICLR. find our card
  8. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of NAACL, pages 4129– 4138. find our card
  9. Ganesh Jawahar, Benoı̂t Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of ACL, pages 3651– 3657. find our card
  10. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv preprint, arXiv:1412.6980. find
  11. Taku Kudo and John Richardson. 2018. Sentence-Piece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In Proceedings of EMNLP: System Demonstrations, pages 66–71. find
  12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint, arXiv:1907.11692. find
  13. Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In Proceedings of ICLR. find
  14. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of ACL, pages 1064–1074. find our card
  15. Xuezhe Ma, Zecong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. 2018. Stack-Pointer Networks for Dependency Parsing. In Proceedings of ACL, pages 1403–1414. find
  16. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, and Benoı̂t Sagot. 2020. CamemBERT: a Tasty French Language Model. In Proceedings of ACL, pages 7203– 7219. find our card
  17. Dat Quoc Nguyen. 2019. A neural joint model for Vietnamese word segmentation, POS tagging and dependency parsing. In Proceedings of ALTA, pages 28– 34. find
  18. Dat Quoc Nguyen, Dai Quoc Nguyen, Dang Duc Pham, and Son Bao Pham. 2014a. RDRPOSTagger: A Ripple Down Rules-based Part-Of-Speech Tagger. In Proceedings of the Demonstrations at EACL, pages 17–20. find
  19. Dat Quoc Nguyen, Dai Quoc Nguyen, Son Bao Pham, Phuong-Thai Nguyen, and Minh Le Nguyen. 2014b. From Treebank Conversion to Automatic Dependency Parsing for Vietnamese. In Proceedings of NLDB, pages 196–207. find
  20. Dat Quoc Nguyen, Dai Quoc Nguyen, Thanh Vu, Mark Dras, and Mark Johnson. 2018. A Fast and Accurate Vietnamese Word Segmenter. In Proceedings of LREC, pages 2582–2587. find
  21. Dat Quoc Nguyen and Karin Verspoor. 2018. An improved neural network model for joint POS tagging and dependency parsing. In Proceedings of the CoNLL 2018 Shared Task, pages 81–91. find
  22. Dat Quoc Nguyen, Thanh Vu, Dai Quoc Nguyen, Mark Dras, and Mark Johnson. 2017. From word segmentation to POS tagging for Vietnamese. In Proceedings of ALTA, pages 108–113. find
  23. Huyen Nguyen, Quyen Ngo, Luong Vu, Vu Tran, and Hien Nguyen. 2019a. VLSP Shared Task: Named Entity Recognition. Journal of Computer Science and Cybernetics, 34(4):283–294. find
  24. Kim Anh Nguyen, Ngan Dong, and Cam-Tu Nguyen. 2019b. Attentive Neural Network for Named Entity Recognition in Vietnamese. In Proceedings of RIVF. find
  25. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, pages 48–53. find our card
  26. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of ACL, pages 1715–1725. find
  27. Dinh Quang Thang, Le Hong Phuong, Nguyen Thi Minh Huyen, Nguyen Cam Tu, Mathias Rossignol, and Vu Xuan Luong. 2008. Word segmentation of Vietnamese texts: a comparison of approaches. In Proceedings of LREC, pages 1933–1936. find
  28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems 30, pages 5998–6008. find
  29. Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. BERTje: A Dutch BERT Model. arXiv preprint, arXiv:1912.09582. find our card
  30. Thanh Vu, Dat Quoc Nguyen, Dai Quoc Nguyen, Mark Dras, and Mark Johnson. 2018. VnCoreNLP: A Vietnamese Natural Language Processing Toolkit. In Proceedings of NAACL: Demonstrations, pages 56–60. find
  31. Xuan-Son Vu, Thanh Vu, Son Tran, and Lili Jiang. 2019. ETNLP: A visual-aided systematic approach to select pre-trained embeddings for a downstream task. In Proceedings of RANLP, pages 1285–1294. find
  32. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of NAACL, pages 1112–1122. find
  33. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace’s Transformers: State-of-the-art Natural Language Processing. arXiv preprint, arXiv:1910.03771. find our card
  34. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of EMNLP-IJCNLP, pages 833–844. find our card