Elamur Research
☰
home / papers / 2017 / Fully Character-Level Neural Machine Translation without Explicit Segmentation

Fully Character-Level Neural Machine Translation without Explicit Segmentation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/Q17-1026.pdf

Summary

Word- and subword-level NMT systems rely on a separate segmentation step and handle rare or unseen word forms poorly. This is a serious weakness for morphologically rich languages such as Czech or Finnish. Lee, Cho and Hofmann propose char2char, an attentional NMT model that reads source characters and produces target characters with no tokenization at all. Character sequences are much longer than word sequences, so the encoder first shortens the input using convolution, max-pooling and highway layers and then passes it to a bidirectional GRU. This keeps training cost close to that of subword systems. The decoder also works purely on characters. On WMT'15 data, the bilingual model beats a bpe2char baseline (subword encoder, character decoder) on DE-EN and CS-EN and matches it on FI-EN and RU-EN. The authors also trained one shared character encoder on four source languages translating into English. It clearly outperforms its subword counterpart on every pair. For CS, FI and RU it even surpasses dedicated bilingual models in both BLEU and human evaluation. The work shows that explicit word boundaries are not needed and that character-level encoders make compact multilingual translation practical.

Key points

  • Languages and data: Source languages are German, Czech, Finnish and Russian, translated into English using WMT'15 data. In characters, sentences are about 6 times longer (DE, CS, RU) to 8 times longer (FI) than in words.
  • Encoder design: The encoder adapts the convolution/pooling/highway stack from the character language model of Kim et al. (2015). By comparison, the earlier purely character-level baseline of Luong and Manning (2016) needed roughly 3 months to train.
  • Multilingual model:
    • It is trained without any language identifier, yet copes with intra-sentence code-switching.
    • It overfits less on the low-resource FI-EN pair.
    • It covers four languages without increasing model size.
  • Speed (single Titan X GPU):
    • With the same batch size, char2char trains about 35% slower than bpe2char.
    • A bilingual model takes about two weeks to train.
    • Decoding 3,000 sentences takes 71.71 minutes with char2char versus 66.63 minutes with bpe2char.
  • Code release: Source code and pre-trained models are available at github.com/nyu-dl/dl4mt-c2c.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

151 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 140 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00067

@article{lee2017fully,
  title = {{Fully Character-Level Neural Machine Translation without Explicit Segmentation}},
  author = {Lee, Jason and Cho, Kyunghyun and Hofmann, Thomas},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2017},
  doi = {10.1162/tacl\_a\_00067},
  url = {https://doi.org/10.1162/tacl_a_00067}
}

References

23 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations. find our card
  2. Chris Callison-Burch. 2009. Fast, cheap, and creative: Evaluating translation quality using amazon’s mechanical turk. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 286–295. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2… find
  3. Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. A character-level decoder without explicit segmentation for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1693–1703. Marta R. Costa-Jussá and Josè A. R. Fonollosa… find
  4. Ferdinand de Saussure. 1916. Course in General Linguistics. find
  5. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016a. Multi-way, multilingual neural machine translation with a shared attention mechanism. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics, pages 866–875. find our card
  6. Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T. Yarman Vural, and Kyunghyun Cho. 2016b. Zero-resource translation with multi-lingual neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 268–277. find
  7. Dan Gillick, Cliff Brunk, Oriol Vinyals, and Amarnag Subramanya. 2015. Multilingual language processing from bytes. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics, pages 1296–1306. find
  8. Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2015. Accurate evaluation of segment-level machine translation metrics. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1183–1191. find
  9. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2017. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, 23(1):3–30. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735– 1780. find
  10. Ray S. Jackendoff. 1992. Semantic Structures, Volume 18. MIT press. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguist… find
  11. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2015. Character-aware neural language models. In Proceedings of the 30th AAAI Conference on Artificial Intelligence, pages 2741–2749. find our card
  12. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference for Learning Representations. find
  13. Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W. Black. 2015. Character-based neural machine translation. arXiv preprint arXiv:1511.04586. find our card
  14. Minh-Thang Luong and Christopher D. Manning. 2016. Achieving open vocabulary neural machine translation with hybrid word-character models. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1054–1063. find our card
  15. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1412–1421. find
  16. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning, pages 1310–1318. find
  17. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715–1725. find
  18. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Edinburgh neural machine translation systems for WMT 16. In Proceedings of the 1st Conference on Machine Translation. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Training very deep networks. In Advances in Neural Inform… find our card
  19. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, Volume 27, pages 3104–3112. find
  20. Yulia Tsvetkov, Sunayana Sitaram, Manaal Faruqui, Guillaume Lample, Patrick Littell, David Mortensen, Alan W. Black, Lori Levin, and Chris Dyer. 2016. Polyglot neural language models: A case study in cross-lingual phonetic representation learning. In Proceedings of the 2016 Conference of the North A… find
  21. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  22. Yijun Xiao and Kyunghyun Cho. 2016. Efficient character-level document classification by combining convolution and recurrent layers. arXiv preprint arXiv:1602.00367. — [arXiv:1602.00367](https://arxiv.org/abs/1602.00367) find
  23. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, Volume 28, pages 649–657. find our card