Elamur Research
☰
home / papers / 2020 / BPE-Dropout: Simple and Effective Subword Regularization

BPE-Dropout: Simple and Effective Subword Regularization

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.170.pdf

Summary

Byte Pair Encoding (BPE) is the usual way to split text into subword units for neural machine translation, but it always produces one fixed segmentation per word. The model therefore sees each word split in only one way, which limits how well it learns word structure and makes it vulnerable to segmentation errors. An earlier remedy, subword regularization by Kudo (2018), meant replacing BPE with a unigram language model trained with EM and sampled with Viterbi. The authors propose BPE-dropout instead. It keeps an ordinary BPE vocabulary and merge table, but during training it randomly skips each candidate merge with probability p (typically 0.1). At inference, standard BPE is used (p = 0). They test it with Transformer base on IWSLT, WMT14 and ASPEC translation directions. BPE-dropout outperforms standard BPE by as much as 2.3 BLEU and Kudo's method by as much as 0.9 BLEU. It also learns better embeddings for rare tokens and handles misspelled input more robustly. Because it needs no extra segmentation model, it is an easy drop-in upgrade.

Key points

  • Method: At every merge step, each applicable merge from the BPE table is independently discarded with probability p. Setting p = 0 recovers ordinary BPE, and p = 1 yields single characters. Most settings use p = 0.1. The Chinese and Japanese target sides use p = 0.6.
  • Data: IWSLT15 En↔Vi and En↔Zh, IWSLT17 En↔Fr and En↔Ar, WMT14 En↔De (4.5M pairs) and ASPEC En↔Ja (2M pairs). Random subsets of WMT14 En-Fr serve for extra analyses. Text is preprocessed with Moses, and vocabulary size (4k–32k) is matched to corpus size.
  • Vocabulary: The original BPE vocabulary is reused, so a small number of UNKs appear during training. For example, they make up about 0.6% of tokens on the English side of IWSLT15 En-Vi with a 32k vocabulary. Adding the extra tokens to the vocabulary made no measurable difference.
  • Robustness: On WMT14 test sets where 10% of source words receive a one-character edit, BPE-dropout gains 1.58–2.32 BLEU over BPE. On clean input the gains are 0.45–1.5 BLEU. For example, De-En improves from 29.71 to 32.03 on misspelled input.
  • Embeddings: In SVD visualizations, rare tokens learned with BPE form an isolated cluster. With BPE-dropout they mix with the rest of the embedding space.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

170 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 161 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.170 · arXiv 1910.13267

@inproceedings{provilkov2020bpe,
  title = {{BPE-Dropout: Simple and Effective Subword Regularization}},
  author = {Provilkov, Ivan and Emelianenko, Dmitrii and Voita, Elena},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.170},
  eprint = {1910.13267},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.acl-main.170}
}

References

28 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Farooq Ahmad and Grzegorz Kondrak. 2005. Learning a spelling error model from search query logs. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 955–962, Vancouver, British Columbia, Canada. Association for Computationa… find
  2. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. Unsupervised statistical machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3632–3642, Brussels, Belgium. Association for Computational Linguistics. find our card
  3. Tamali Banerjee and Pushpak Bhattacharyya. 2018. Meaningless yet meaningful: Morphology grounded subword-level NMT. In Proceedings of the Second Workshop on Subword/Character LEvel Models, pages 55–60, New Orleans. Association for Computational Linguistics. find
  4. Loı̈c Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translatio… find our card
  5. Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 272–303, Belgium, Brussels. A… find our card
  6. Eric Brill and Robert C. Moore. 2000. An improved error model for noisy channel spelling correction. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pages 286– 293, Hong Kong. Association for Computational Linguistics. find
  7. Colin Cherry, George Foster, Ankur Bapna, Orhan Firat, and Wolfgang Macherey. 2018. Revisiting character-based neural machine translation with capacity and compression. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4295–4305, Brussels, Belgium. Asso… find
  8. Rohan Chitnis and John DeNero. 2015. Variable-length word encodings for neural translation models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2088–2093, Lisbon, Portugal. Association for Computational Linguistics. find
  9. Mathias Creutz and Krista Lagus. 2006. Morfessor in the morpho challenge. In Proceedings of the PAS-CAL Challenge Workshop on Unsupervised Segmentation of Words into Morphemes, pages 12–17. Citeseer. find
  10. Shuoyang Ding, Adithya Renduchintala, and Kevin Duh. 2019. A call for prudent choice of subword merge operations in neural machine translation. In Proceedings of Machine Translation Summit XVII Volume 1: Research Track, pages 204–213, Dublin, Ireland. European Association for Machine Translation. find
  11. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 567– 573, Vancouver, Canada. Association for Computation… find our card
  12. Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2018. Frage: Frequency-agnostic word representation. In Advances in neural information processing systems, pages 1334–1345. find
  13. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural L… find
  14. Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 452–457, New Orle… find our card
  15. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. find
  16. Julia Kreutzer and Artem Sokolov. 2018. Learning to segment inputs for nmt favors character-level processing. In Proceedings of the 15th International Workshop on Spoken Language Translation. find
  17. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66– 75, Melbourne, Australia. Association for Computational… find
  18. Anoop Kunchukuttan and Pushpak Bhattacharyya. 2016. Orthographic syllable as basic unit for SMT between related languages. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1912–1917, Austin, Texas. Association for Computational Linguistics. find
  19. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representations. find our card
  20. Myle Ott, Michael Auli, David Grangier, and MarcAu-relio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In International Conference on Machine Learning, pages 3956–3965. find our card
  21. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational… find our card
  22. Yuval Pinter, Robert Guthrie, and Jacob Eisenstein. 2017. Mimicking word embeddings using subword RNNs. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 102–112, Copenhagen, Denmark. Association for Computational Linguistics. find
  23. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Belgium, Brussels. Association for Computational Linguistics. find our card
  24. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152. IEEE. find
  25. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li… find
  26. Rico Sennrich and Biao Zhang. 2019. Revisiting low-resource neural machine translation: A case study. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 211– 221, Florence, Italy. Association for Computational Linguistics. find
  27. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958. find
  28. Ziang Xie, Sida I. Wang, Jiwei Li, Daniel Lvy, Aiming Nie, Dan Jurafsky, and Andrew Y. Ng. 2017. Data noising as smoothing in neural network language models. In International Conference on Learning Representations. find