Elamur Research⌕
☰
home / papers / 2018 / Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

Summary

Neural machine translation (NMT) operates with a limited vocabulary, so rare words are usually split into subwords. The standard approach is BPE. It produces a single fixed segmentation for each sentence. However, with the same vocabulary a sentence can be segmented in different ways, and the model perceives these variants as entirely different inputs. The author proposes to exploit this ambiguity as useful noise. During training, the segmentation of each sentence is sampled anew each time from a probability distribution. This effectively augments the training data while leaving the network architecture unchanged. To obtain segmentation probabilities, the author introduces a new method for splitting text that relies on a unigram language model. It gradually prunes a large initial vocabulary using EM and can output multiple candidates together with their probabilities. Experiments on IWSLT15, IWSLT17, WMT14 and other corpora showed consistent BLEU improvements. Low-resource language pairs and out-of-domain texts benefit the most. The method is implemented in the open-source library SentencePiece, so it is easy to apply in other systems.

Key points

  • The subword regularization method has two hyperparameters: l, the number of candidates for sampling (e.g., 64 or ∞, i.e., the entire lattice), and α, a smoothing parameter. If α = 0, sampling becomes uniform and quality drops noticeably. Hence, the distribution provided by the language model really matters.
  • WMT14 en→de: word-level model 23.12 BLEU, BPE 24.53, unigram without regularization 24.50, unigram with regularization (l = 64, α = 0.1) 25.04. This is the best result among the compared segmentations.
  • The gain is largest on out-of-domain data. For WMT14 en→de patent texts, BLEU rose from 15.63 to 25.76; for IWSLT15 en→vi web texts, from 13.86 to 17.36.
  • Regularization applied only on the source side or only on the target side also helps, but it works best on both sides. For IWSLT15 en→vi, BLEU rose from 25.49 to 27.68. The author believes the method can be transferred to tasks that have only an encoder or only a decoder.
  • The implementation is publicly available: github.com/google/sentencepiece.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics: Machine translation · Tokenization & segmentation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1007 · arXiv 1804.10959

@inproceedings{kudo2018subword,
  title = {{Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates}},
  author = {Kudo, Taku},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1007},
  eprint = {1804.10959},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p18-1007}
}

References

33 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Toshiaki Nakazawa, Shohei Higashiyama, Chenchen Ding, Hideya Mino, Isao Goto, Hideto Kazawa, Yusuke Oda, Graham Neubig, and Sadao Kurohashi. 2017. Overview of the 4th workshop on asian translation. In Proceedings of the 4th Workshop on Asian Translation (WAT2017). pages 1–54. find
  2. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2017. Unsupervised neural machine translation. arXive preprint arXiv:1710.11041 . find our card
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 . find our card
  4. Ge Nong, Sen Zhang, and Wai Hong Chan. 2009. Linear suffix array construction by almost pure induced-sorting. In Proc. of DCC. find
  5. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proc. of ACL. find our card
  6. Yonatan Belinkov and Yonatan Bisk. 2017. Synthetic and natural noise both break neural machine translation. arXive preprint arXiv:1711.02173 . find
  7. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proc. of EMNLP. find
  8. William Chan, Yu Zhang, Quoc Le, and Navdeep Jaitly. 2016. Latent sequence decompositions. arXiv preprint arXiv:1610.03035 . — [arXiv:1610.03035](https://arxiv.org/abs/1610.03035) find
  9. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In Proc. of ICASSP. find
  10. Rohan Chitnis and John DeNero. 2015. Variable-length word encodings for neural translation models. In Proc. of EMNLP. pages 2088–2093. find
  11. Steven L Scott. 2002. Bayesian methods for hidden markov models: Recursive computing in the 21st century. Journal of the American Statistical Association . find
  12. Michael Denkowski and Graham Neubig. 2017. Stronger baselines for trustable results in neural machine translation. Proc. of Workshop on Neural Machine Translation . find our card
  13. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. of ACL. find our card
  14. Philip Gage. 1994. A new algorithm for data compression. C Users J. 12(2):23–38. find
  15. Matthias Sperber, Graham Neubig, Jan Niehues, and Alex Waibel. 2017. Neural lattice-to-sequence models for uncertain inputs. In Proc. of EMNLP. find
  16. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122 . find our card
  17. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. JMLR 15(1). find
  18. Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In Proc. of ICLR. find our card
  19. Jinsong Su, Zhixing Tan, De yi Xiong, Rongrong Ji, Xiaodong Shi, and Yang Liu. 2017. Lattice-based recurrent neural network encoders for neural machine translation. In AAAI. pages 3302–3308. find
  20. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proc. of ACL. find our card
  21. Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. Proc. of ACL . find
  22. Diederik P Kingma and Jimmy Ba Adam. 2014. A method for stochastic optimization. arXiv preprint arXiv:1412.6980 . find
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. arXive preprint arXiv:1706.03762 . find our card
  24. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proc. of EMNLP. find our card
  25. Guillaume Lample, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2017. Unsupervised machine translation using monolingual corpora only. arXive preprint arXiv:1711.00043 . find our card
  26. Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In Proc. of ICML. find
  27. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In Proc of EMNLP. find our card
  28. Oriol Vinyals and Quoc V. Le. 2015. A neural conversational model. In ICML Deep Learning Workshop. find our card
  29. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Computer Vision and Pattern Recognition. find our card
  30. Masaaki Nagata. 1994. A stochastic japanese morphological analyzer using a forward-dp backward-a* nbest search algorithm. In Proc. of COLING. find
  31. Andrew Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE transactions on Information Theory 13(2):260–269. find
  32. Yonghui Wu, Mike Schuster, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 . find
  33. Ziang Xie, Sida I. Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky, and Andrew Y. Ng. 2017. Data noising as smoothing in neural network language models. In Proc. of ICLR. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/