Elamur Research⌕
☰
home / papers / 2018 / Revisiting Character-Based Neural Machine Translation with Capacity and Compression

Revisiting Character-Based Neural Machine Translation with Capacity and Compression

Paper · 2018 · Colin Cherry, George Foster, Ankur Bapna et al.

Summary

Character-level neural machine translation removes the need for tokenization and BPE segmentation, along with the hyperparameters those steps require. The cost is much longer input and output sequences. Cherry et al. revisit whether special architectures are actually needed for characters. They train a standard deep BiLSTM encoder–decoder with attention, adapted from Chen et al. (2018), on both characters and BPE fragments, keeping everything else identical. With enough depth, the character-level model beats the BPE models for all but the largest training corpora, although it costs far more computation. The authors therefore treat character-specific designs as ways to save compute rather than as better models of long sequences. They compare several methods for compressing the source sequence in the encoder:

  • a reimplementation of Lee et al. (2017)'s convolution-and-pooling encoder,
  • a BiLSTM encoder with fixed-stride mean-pooling layers placed between recurrent layers,
  • the Hierarchical Multiscale LSTM, which learns which timesteps to skip. This is its first use in NMT.

None of these methods matches the uncompressed character baseline. The paper argues that efficient character-level translation is a worthwhile open problem.

Key points

  • On WMT15 DeEn, the deep character model scores 31.6 BLEU. BPE models with vocabularies from 1k to 32k score between 29.6 and 30.5.
  • The reimplementation of Lee et al. (2017) beats the originally reported scores by more than 2 BLEU, likely because of large-batch training. Making this encoder deeper brought no real gains.
  • The pooled BiLSTM encoder places a stride-3 pooling layer after the second BiLSTM layer and a stride-2 layer after the third, shortening its output 6×. It trails BPE-1k by about 0.5 BLEU at similar compression.
  • In small-scale tests, the HM encoder used no-flush and hierarchical modifications, a z-bias initialized to 1, and slope annealing. It reached LSTM parity (28.8 vs. 28.9 BLEU) while saving about 35% of layer-wise computation. Each HM layer kept roughly 60% of the timesteps of the layer below.
  • Compared with fixed-stride pooling, HM gating compresses less but performs better; 3 HM layers beat 2. The speedups are conceptual so far, and realizing them in practice is a significant engineering challenge.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 958 worksEfficient models & compression · 77 works

Cite

Cite as Colin Cherry, George Foster, Ankur Bapna, Orhan Firat and Wolfgang Macherey. 2018. Revisiting Character-Based Neural Machine Translation with Capacity and Compression. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1461

@inproceedings{cherry2018revisiting,
  title = {{Revisiting Character-Based Neural Machine Translation with Capacity and Compression}},
  author = {Colin Cherry, George Foster, Ankur Bapna, Orhan Firat and Wolfgang Macherey},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1461},
  url = {https://doi.org/10.18653/v1/d18-1461}
}

References

23 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
  2. Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432. find
  3. James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. 2016. Quasi-recurrent neural networks. arXiv preprint arXiv:1611.01576. find
  4. William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. 2016. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on, pages 4960–4964. IEEE. find
  5. Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018. The best of both worlds: Combining recent advances in neural machine translation. In Proceedings of the 56th Annual Me…
  6. Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. 2017. Hierarchical multiscale recurrent neural networks. In International Conference on Learning Representations, Toulon, France. find
  7. Marta R. Costa-jussà and José A. R. Fonollosa. 2016. Character-based neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 357–361, Berlin, Germany. Association for Computational Linguistics.
  8. Caglar Gulcehre, Francis Dutil, Adam Trischler, and Yoshua Bengio. 2017. Plan, attend, generate: Character-level neural machine translation with planning. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 228–234, Vancouver, Canada. Association for Computational Linguistic… find
  9. Melvin Johnson, Mike Schuster, Quoc Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernand a Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Asso… find
  10. Łukasz Kaiser, Aurko Roy, Ashish Vaswani, Niki Parmar, Samy Bengio, Jakob Uszkoreit, and Noam Shazeer. 2018. Fast decoding in sequence models using discrete latent variables. arXiv preprint arXiv:1803.03382. — [arXiv:1803.03382](https://arxiv.org/abs/1803.03382) find
  11. Nan Rosemary Ke, Konrad Zolna, Alessandro Sordoni, Zhouhan Lin, Adam Trischler, Yoshua Bengio, Joelle Pineau, Laurent Charlin, and Chris Pal. 2018. Focused hierarchical RNNs for conditional sequence processing. arXiv preprint arXiv:1806.04342. — [arXiv:1806.04342](https://arxiv.org/abs/1806.04342) find
  12. Diederik P Kingma and Jimmy Lei Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  13. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (To Appear).
  14. Ákos Kádár, Marc-Alexandre Côté, Grzegorz Chrupała, and Afra Alishahi. 2018. Revisiting the hierarchical multiscale LSTM. In COLING. find
  15. Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017. Fully character-level neural machine translation without explicit segmentation. Transactions of the Association for Computational Linguistics, 5:365–378.
  16. Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W Black. 2015. Character-based neural machine translation. arXiv preprint arXiv:1511.04586.
  17. Minh-Thang Luong and Christopher D. Manning. 2016. Achieving open vocabulary neural machine translation with hybrid word-character models. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1054–1063, Berlin, Germany. Association…
  18. Matt Post. 2018. A call for clarity in reporting BLEU scores. arXiv preprint arXiv:1804.08771v1.
  19. Colin Raffel, Minh-Thang Luong, Peter J. Liu, Ron J. Weiss, and Douglas Eck. 2017. Online and linear-time attention by enforcing monotonic alignments. In International Conference on Machine Learning, Sydney, Australia. find
  20. Rico Sennrich. 2017. How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs. In EACL, pages 376–382, Valencia, Spain.
  21. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li…
  22. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. arXiv preprint arXiv:1601.04811.
  23. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kaza-waand Keith Stevens, George Kurian,… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cherry-2018-revisiting-character-based-neural-machine/