Elamur Research⌕
☰
home / papers / 2020 / Byte Pair Encoding is Suboptimal for Language Model Pretraining

Byte Pair Encoding is Suboptimal for Language Model Pretraining

Summary

The authors test whether the subword tokenization algorithm affects the quality of pretrained language models. Before this work, the question had not been studied directly. They compare two methods with open implementations. BPE greedily merges the most frequent pairs of tokens. Unigram LM, by contrast, starts from a large vocabulary and gradually discards the units least useful to the unigram model. The authors first analyze the segmentations themselves: unigram LM token boundaries align better with morphemes (the references are CELEX2 for English and MeCab with UniDic for Japanese). This method produces longer tokens and does not clutter the vocabulary with rare intermediate units. They then train four models with the RoBERTa-base architecture from scratch on English and Japanese Wikipedia. All models share the same vocabulary size of 20,000 tokens and differ only in tokenization. After fine-tuning, unigram LM matches or outperforms BPE on all tasks, with the largest gain on Japanese TyDi QA. Conclusion: tokenization noticeably affects what a model learns. Therefore, unigram LM should be preferred for new models, particularly for morphologically rich languages such as Ukrainian.

Key points

  • Experiment: 4 RoBERTa-base models were trained in fairseq: English Wikipedia (~3B tokens) and Japanese Wikipedia (~0.6B), each in a BPE variant and a unigram LM variant. Each model was pretrained once, and fine-tuning was repeated with 5 different seeds.
  • Alignment of token boundaries with morphemes (F1): English (CELEX2): 19.3% for BPE vs. 30.3% for unigram LM, precision 38.6% vs. 62.2%. Japanese (MeCab): 73.8% vs. 77.2%.
  • Fine-tuning results, BPE → unigram LM: SQuAD 1.1 EM 80.6 → 81.8, F1 88.2 → 89.3. MNLI (matched) 81.4 → 82.8. CoNLL 2003 NER, test F1 90.2 → 90.4. Japanese TyDi QA: EM 41.4 → 53.7, F1 42.1 → 54.4.
  • Why BPE is worse: because of its greedy construction, it attaches frequent affixes (-s, -ed, -ly) to neighboring units and leaves intermediate "junk" tokens in the vocabulary. As a result, its vocabulary contains roughly 1,500 more very rare entries.
  • Practical details: the methods do not differ substantially in speed. The SentencePiece library implements BPE and unigram LM, not WordPiece. The authors did not compare WordPiece itself, as its implementation is closed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Language models & pretraining Tokenization & segmentation

Cite

Cite as Kaj Bostrom and Greg Durrett. 2020. Byte Pair Encoding is Suboptimal for Language Model Pretraining. Findings of the Association for Computational Linguistics: EMNLP 2020.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.findings-emnlp.414 · arXiv 2004.03720

@inproceedings{bostrom2020byte,
  title = {{Byte Pair Encoding is Suboptimal for Language Model Pretraining}},
  author = {Kaj Bostrom and Greg Durrett},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.414},
  eprint = {2004.03720},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.414}
}

References

33 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Harald Baayen, Richard Piepenbrock, and Leon Gulikers. 1995. The CELEX lexical database (release 2). find
  2. Jonathan Clark, Jennimaria Palomaki, Vitaly Nikolaev, Eunsol Choi, Dan Garrette, Michael Collins, and Tom Kwiatkowski. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8(0):454– 47… our card
  3. Mathias Creutz and Krista Lagus. 2005. Unsupervised morpheme segmentation and morphology induction from text corpora using Morfessor 1.0. Helsinki University of Technology Helsinki. find
  4. Mathias Creutz and Krister Lindén. 2004. Morpheme segmentation gold standards for finnish and english. Report, Helsinki University of Technology. find
  5. Mathias Johan Philip Creutz and Bo Krister Johan Linden. 2004. Morpheme segmentation gold standards for Finnish and English. Publications in Computer and Information Science Report A77. find
  6. Taku Kudo. 2006. MeCab: Yet another part-of-speech and morphological analyzer. find
  7. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational… our card
  8. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ… our card
  9. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A lite BERT for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. find
  10. Yasuharu Den, Toshinobu Ogiso, Hideki Ogura, Atsushi Yamada, Nobuaki Minematsu, Kiyotaka Uchimoto, and Hanae Koiso. 2007. The development of an electronic dictionary for morphological analysis and its application to japanese corpus linguistics. Japanese Linguistics, 22:101–123. find
  11. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… our card
  13. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr… our card
  14. Miguel Domingo, Mercedes Garcıa-Martınez, Alexandre Helle, and Francisco Casacuberta. 2018. How much does tokenization affect in neural machine translation? arXiv preprint arXiv:1812.08621. — [arXiv:1812.08621](https://arxiv.org/abs/1812.08621) find
  15. Philip Gage. 1994. A new algorithm for data compression. C Users Journal, 12(2):23–38. find
  16. Matthias Gallé. 2019. Investigating the effectiveness of BPE: The power of shorter sequences. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1375–1381, Hong Kon… find
  17. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. find
  18. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  19. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. our card
  20. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. our card
  21. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and Korean voice search. 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152. find
  22. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li… our card
  23. A Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147. our card
  24. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. our card
  25. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… find
  26. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771. our card
  27. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. find
  28. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237. our card
  29. Hyperparameters Pretraining Model architecture R O BERT A - BASE (Liu et al., 2019) find
  30. Implementation fairseq (Ott et al., 2019) find
  31. Optimizer A DAM ,  = 1e-6 β = (0.9, 0.98) (Kingma and Ba, 2015) find
  32. Learning rate decay Peak learning rate Warmup steps Weight decay Batch size Sequence length Total updates MLP dropout Attention dropout Precision Polynomial 0.0005 10000 0.01 2048 512 125000 0.1 0.1 16-bit Fine-tuning Implementations transformers (Wolf et al., 2019) find
  33. Optimizer A DAM ,  = 1e-8 β = (0.9, 0.999) Learning rate decay Peak learning rate Warmup steps Weight decay Batch size Sequence length (SQuAD, TyDi QA) Passage stride (SQuAD, TyDi QA) Sequence length (MNLI, NER) Epochs Precision Linear 5e-5 0 0 32 3 16-bit Tokenization Implementations Vocabulary si… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bostrom-2020-byte-pair-encoding-is-suboptimal/