Elamur Research⌕
☰
home / papers / 2016 / A Character-level Decoder without Explicit Segmentation for Neural Machine Translation

A Character-level Decoder without Explicit Segmentation for Neural Machine Translation

Summary

Machine translation systems, both phrase-based and neural, have traditionally worked on words or other explicitly segmented units. Chung, Cho and Bengio test whether a neural translator can instead generate its output one character at a time, with no segmentation on the target side. They build on the attention-based encoder–decoder of Bahdanau et al. (2015). The source is encoded as byte-pair-encoded subwords, and the decoder emits either subwords or raw characters. Besides a plain stacked RNN decoder, they propose a bi-scale recurrent network that models fast and slow timescales in the output sequence. Experiments cover four WMT'15 language pairs: En-Cs, En-De, En-Ru and En-Fi. Character-level decoders beat subword-level ones on every pair. Ensembles built on them outperform the best non-neural systems on En-Cs, En-De and En-Fi and roughly match them on En-Ru. The work shows that data sparsity and long-range dependencies do not prevent character-level generation. This makes it relevant for morphologically rich target languages such as Ukrainian.

Key points

  • Setup: BPE subword encoder (Sennrich et al., 2015) paired with a character-level decoder, either a base stacked RNN or the new bi-scale RNN. It is evaluated on WMT'15 parallel corpora for En-Cs, En-De, En-Ru and En-Fi.
  • Among single models, character-level decoders clearly beat subword decoders on En-Cs and En-Fi. On En-De, the base character decoder performed best. On En-Ru, all three variants were close.
  • Ensembles of eight independent models with character-level decoders beat subword-decoder ensembles in all cases. They also exceed state-of-the-art non-neural systems on three of the four pairs.
  • Analysis findings:
    • Character outputs are 5–10× longer but no less coherent.
    • The character-level advantage grows for rare words.
    • Feeding only the slower layer into attention helped.
    • Alignments capture compounds such as German "Lichtersets."
  • Decoding speed on newstest2013 (En-De, one Titan X, online setting) was 31.9 words/s for the subword decoder, 27.5 for the character base decoder and 25.6 for the bi-scale decoder. The main limitation is that the source side still uses subword segmentation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

98 papers of our library cite it (of the 140884 whose bibliographies we read); 9 of them have a page here, 89 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1160

@inproceedings{chung2016character,
  title = {{A Character-level Decoder without Explicit Segmentation for Neural Machine Translation}},
  author = {Chung, Junyoung and Cho, Kyunghyun and Bengio, Yoshua},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1160},
  url = {https://doi.org/10.18653/v1/p16-1160}
}

References

42 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations (ICLR). find our card
  2. Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157–166. find
  3. Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. 2001. A neural probabilistic language model. In Advances in Neural Information Processing Systems, pages 932–938. find
  4. Geert Booij. 2012. The grammar of words: An introduction to linguistic morphology. Oxford University Press. find
  5. Jan A Botha and Phil Blunsom. 2014. Compositional morphology for word representations and language modelling. In ICML 2014. find
  6. Rohan Chitnis and John DeNero. 2015. Variable-length word encodings for neural translation models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2088–2093. find
  7. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014), October. find
  8. Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2015. Gated feedback recurrent neural networks. In Proceedings of the 32nd International Conference on Machine Learning. find
  9. Marta R Costa-Jussà and José AR Fonollosa. 2016. Character-based neural machine translation. arXiv preprint arXiv:1603.00810. find our card
  10. Mathias Creutz and Krista Lagus. 2005. Unsupervised morpheme segmentation and morphology induction from text corpora using Morfessor 1.0. Helsinki University of Technology. find
  11. Nadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. 2014. Edinburgh’s phrase-based machine translation systems for wmt-14. In Proceedings of the ACL 2014 Ninth Workshop on Statistical Machine Translation, Baltimore, MD, USA, pages 97–104. find
  12. Markus Freitag, Stephan Peitz, Joern Wuebker, Hermann Ney, Matthias Huck, Rico Sennrich, Nadir Durrani, Maria Nadejde, Philip Williams, Philipp Koehn, et al. 2014. Eu-bridge mt: Combined machine translation. find
  13. Dan Gillick, Cliff Brunk, Oriol Vinyals, and Amarnag Subramanya. 2015. Multilingual language processing from bytes. arXiv preprint arXiv:1512.00103. find
  14. Barry Haddow, Matthias Huck, Alexandra Birch, Nikolay Bogoychev, and Philipp Koehn. 2015. The edinburgh/jhu phrase-based machine translation systems for wmt 2015. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 126– 133. find
  15. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  16. Sepp Hochreiter. 1998. The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 6(02):107–116. find
  17. Changning Huang and Hai Zhao. 2007. Chinese word segmentation: A decade review. Journal of Chinese Information Processing, 21(3):8–20. find
  18. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2. find our card
  19. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2015. Character-aware neural language models. arXiv preprint arXiv:1508.06615. find our card
  20. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  21. Hyoung-Gyu Lee, JaeSong Lee, Jun-Seok Kim, and Chang-Ki Lee. 2015. Naver machine translation system for wat 2015. In Proceedings of the 2nd Workshop on Asian Translation (WAT2015), pages 69–73. find
  22. Wang Ling, Tiago Luı́s, Luı́s Marujo, Ramón Fernandez Astudillo, Silvio Amir, Chris Dyer, Alan W Black, and Isabel Trancoso. 2015a. Finding function in form: Compositional character models for open vocabulary word representation. arXiv preprint arXiv:1508.02096. find our card
  23. Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W Black. 2015b. Character-based neural machine translation. arXiv preprint arXiv:1511.04586. find our card
  24. Thang Luong, Richard Socher, and Christopher D Manning. 2013. Better word representations with recursive neural networks for morphology. In CoNLL, pages 104–113. find our card
  25. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015a. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025. find
  26. Minh-Thang Luong, Ilya Sutskever, Quoc V Le, Oriol Vinyals, and Wojciech Zaremba. 2015b. Addressing the rare word problem in neural machine translation. arXiv preprint arXiv:1410.8206. find our card
  27. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In IN-TERSPEECH, volume 2, page 3. find
  28. Tomas Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le, Stefan Kombrink, and J Cernocky. 2012. Subword language modeling with neural networks. Preprint. find
  29. Graham Neubig, Taro Watanabe, Shinsuke Mori, and Tatsuya Kawahara. 2013. Substring-based machine translation. Machine translation, 27(2):139–166. find
  30. Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2013. How to construct deep recurrent neural networks. arXiv preprint arXiv:1312.6026. find
  31. Raphael Rubino, Tommi Pirinen, Miquel Espla-Gomis, N Ljubešic, Sergio Ortiz Rojas, Vassilis Papavassiliou, Prokopis Prokopidis, and Antonio Toral. 2015. Abu-matran at wmt 2015 translation task: Morphological segmentation and web crawling. In Proceedings of the Tenth Workshop on Statistical Machine… find
  32. Cicero D Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pages 1818–1826. find
  33. Holger Schwenk. 2007. Continuous space language models. Computer Speech & Language, 21(3):492– 518. find
  34. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. find
  35. Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Training very deep networks. In Advances in Neural Information Processing Systems, pages 2368–2376. find
  36. Ilya Sutskever, James Martens, and Geoffrey E Hinton. 2011. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML’11), pages 1017–1024. find
  37. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104–3112. find
  38. The Theano Development Team, Rami Al-Rfou, Guillaume Alain, Amjad Almahairi, Christof Angermueller, Dzmitry Bahdanau, Nicolas Ballas, Frédéric Bastien, Justin Bayer, Anatoly Belikov, et al. 2016. Theano: A python framework for fast computation of mathematical expressions. arXiv preprint arXiv:1605… find our card
  39. David Vilar, Jan-T Peter, and Hermann Ney. 2007. Can we translate letters? In Proceedings of the Second Workshop on Statistical Machine Translation, pages 33–39. Association for Computational Linguistics. find
  40. Philip Williams, Rico Sennrich, Maria Nadejde, Matthias Huck, and Philipp Koehn. 2015. Edinburgh’s syntax-based systems at wmt 2015. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 199–209. find
  41. Yijun Xiao and Kyunghyun Cho. 2016. Efficient character-level document classification by combining convolution and recurrent layers. arXiv preprint arXiv:1602.00367. — [arXiv:1602.00367](https://arxiv.org/abs/1602.00367) find
  42. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, pages 649–657. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chung-2016-character-level-decoder-without-explicit/