Elamur Research
☰
home / papers / 2016 / Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models

Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P16-1100.pdf

Summary

Neural machine translation systems usually work with a fixed, limited vocabulary. Words outside it are collapsed into a single <unk> token and later patched with dictionary lookup or copying. That approach ignores word morphology and breaks down when scripts or spellings differ across languages. Luong and Manning propose a hybrid NMT model to address this. A deep LSTM encoder-decoder with global attention handles translation of frequent words at the word level. Character-level LSTMs cover everything else. On the source side, a character LSTM builds an embedding for any rare word on the fly. On the target side, a separate character decoder spells out the actual word whenever the word-level model emits <unk>. Both parts are trained jointly, end-to-end, so no separate unk-replacement step is needed. On WMT'15 English→Czech, a hard case because Czech is heavily inflected, the hybrid system reaches 20.7 BLEU, a new state of the art. This shows that open-vocabulary translation can be both practical and efficient.

Key points

  • Task: WMT'15 English→Czech. Example translations are drawn from newstest2015.
  • Best hybrid model scores 20.7 BLEU, beating the strongest earlier word-based NMT and non-neural systems. Character components add +2.1 to +11.4 BLEU over models that already use unk replacement.
  • Training combines word-level and character-level losses (J = J_w + αJ_c). Source character LSTMs start from zero states, so rare-word embeddings can be precomputed once per word type for each mini-batch.
  • A Barnes-Hut-SNE visualization shows that character-built English representations group words by both form and meaning (e.g., "acceptable" vs. "unacceptable").
  • Purely character-based models also beat earlier word-level NMT. Code, data and models are released at nlp.stanford.edu/projects/nmt.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

124 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 119 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1100 · arXiv 1604.00788

@inproceedings{luong2016achieving,
  title = {{Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models}},
  author = {Luong, Minh-Thang and Manning, Christopher D.},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1100},
  eprint = {1604.00788},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p16-1100}
}

References

30 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. find our card
  2. Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio. 2016. End-to-end attention-based large vocabulary speech recognition. In ICASSP. find
  3. Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2015. Improved transition-based parsing by modeling characters instead of words with LSTMs. In EMNLP. find
  4. Ondr̆ej Bojar and Ales̆ Tamchyna. 2015. CUNI in WMT15: Chimera Strikes Again. In WMT. find
  5. William Chan, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals. 2016. Listen, attend and spell. In ICASSP. find
  6. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP. find
  7. Cícero Nogueira dos Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In ICML. find
  8. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. 9(8):1735–1780. find
  9. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015a. On using very large target vocabulary for neural machine translation. In ACL. find our card
  10. Sébastien Jean, Orhan Firat, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015b. Montreal neural machine translation systems for WMT’15. In WMT. find
  11. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. find
  12. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP. find
  13. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2016. Character-aware neural language models. In AAAI. find our card
  14. Jiwei Li, Minh-Thang Luong, and Dan Jurafsky. 2015. A hierarchical neural autoencoder for paragraphs and documents. In ACL. find
  15. Wang Ling, Isabel Trancoso, Chris Dyer, and Alan Black. 2015b. Character-based neural machine translation. find our card
  16. Minh-Thang Luong and Christopher D. Manning. 2015. Stanford neural machine translation systems for spoken language domain. In IWSLT. find
  17. Minh-Thang Luong, Richard Socher, and Christopher D. Manning. 2013. Better word representations with recursive neural networks for morphology. In CoNLL. find our card
  18. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015a. Effective approaches to attention-based neural machine translation. In EMNLP. find
  19. Minh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. 2015b. Addressing the rare word problem in neural machine translation. In ACL. find our card
  20. Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL. find our card
  21. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP. find
  22. Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour. 2014. Dropout improves recurrent neural networks for handwriting recognition. In ICFHR. find
  23. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In WMT. find our card
  24. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL. find
  25. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NIPS. find
  26. Laurens van der Maaten. 2013. Barnes-Hut-SNE. In ICLR. find
  27. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. abs/1409.2329. find our card
  28. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NIPS. find our card
  29. Percy Liang, Ben Taskar, and Dan Klein. 2006. Alignment by agreement. In NAACL. find
  30. Wang Ling, Chris Dyer, Alan W. Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luís Marujo, and Tiago Luís. 2015a. Finding function in form: Compositional character models for open vocabulary word representation. In EMNLP. find our card