Elamur Research⌕
☰
home / papers / 2018 / End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification

End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification

Summary

Autoregressive Transformer decoders emit tokens one at a time, which limits parallelism at inference. Earlier non-autoregressive NMT systems by Gu et al. (2017) and Lee et al. (2018) needed separately trained components and multi-step inference, such as fertility prediction or explicit length estimation. Libovický and Helcl recast translation as sequence labeling, making the model fully end-to-end.

Each Transformer encoder state is linearly projected and sliced into k vectors, giving a sequence k times longer than the source. A decoder without the causal mask then labels every position with either a wordpiece or a null symbol. Training uses the Connectionist Temporal Classification (CTC) loss, which sums over all valid null placements via dynamic programming.

The authors tested three variants on WMT English-Romanian and English-German. Single-pass BLEU was roughly on par with comparable non-autoregressive baselines. Decoding was much faster than autoregressive decoding, but quality stayed clearly below it. This shows that one-pass parallel translation can be trained directly without auxiliary models.

Key points

  • Variants compared:
    • a 12-layer deep encoder with no decoder;
    • a 6+6 encoder-decoder;
    • the same encoder-decoder with positional encoding.
  • Shared settings: model dimension 512, feed-forward dimension 4096, 16 attention heads, split factor k = 3.
  • Data:
    • WMT16 en-ro: 613k sentence pairs, 38k shared wordpieces.
    • WMT en-de: 4.6M sentence pairs, 41k wordpieces; validated on WMT13, tested on WMT14 and WMT15.
  • Tools: Neural Monkey for the implementation; SacreBLEU for scoring.
  • Best BLEU (with weight averaging of 5 checkpoints and beam search):
    • en-ro 19.93 and ro-en 25.28;
    • WMT15 en-de 20.67 and de-en 22.55, above the 19.13/21.69 of Lee et al.'s best result.
    • The encoder-decoder variants beat the deep encoder.
  • Manual evaluation (100 sentences per direction, non-autoregressive model):
    • Fully correct: 23% for en→de and 13% for de→en.
    • Comprehensible: 71% and 51% respectively.
    • Common errors: verbs missing at the end of German sentences (35% in en→de) and corrupted named entities and rare multi-wordpiece words.
  • Sentence-length analysis: sentence-level BLEU drops with source length (r = −0.42 versus −0.39 for the autoregressive model) and correlates mildly with the number of null symbols (r = 0.15). The authors suggest a larger k could help, at the cost of GPU memory, and that an external language model with beam search may be worth exploring.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

76 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 73 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1336

@inproceedings{libovicku2018end,
  title = {{End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification}},
  author = {Libovický, Jindřich and Helcl, Jindřich},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1336},
  url = {https://doi.org/10.18653/v1/d18-1336}
}

References

22 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martı́n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenbe… find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. find our card
  3. Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the… find our card
  4. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Kari… find our card
  5. Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013. Findings of the 2013 Workshop on Statistical Machine Translation. In Proceedings of the Eighth Workshop on Statistical Machine Transl… find our card
  6. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 workshop on statistical machine translation. In Proc… find our card
  7. Boxing Chen and Colin Cherry. 2014. A systematic comparison of smoothing techniques for sentence-level bleu. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 362–367, Baltimore, Maryland, USA. Association for Computational Linguistics. find our card
  8. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 1243–1252. find our card
  9. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369–376. ACM. find
  10. Alex Graves, Navdeep Jaitly, and Abdel-rahman Mohamed. 2013. Hybrid speech recognition with deep bidirectional lstm. In Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on, pages 273–278. IEEE. find
  11. Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. 2017. Non-autoregressive neural machine translation. CoRR, abs/1711.02281. find our card
  12. Jindřich Helcl and Jindřich Libovický. 2017. Neural Monkey: An open-source tool for sequence learning. The Prague Bulletin of Mathematical Linguistics, (107):5–17. find
  13. Felix Hieber, Tobias Domhan, Michael Denkowski, David Vilar, Artem Sokolov, Ann Clifton, and Matt Post. 2017. Sockeye: A Toolkit for Neural Machine Translation. arXiv preprint arXiv:1712.05690. — [arXiv:1712.05690](https://arxiv.org/abs/1712.05690) find
  14. Pennsylvania, USA. Association for Computational Linguistics. find
  15. Lawrence R Rabiner. 1989. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2):257– 286. find
  16. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3104–3112. Curran Associates, Inc. find
  17. Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2tensor for neural machine translation. CoRR, abs/1803.07416. find our card
  18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Process… find
  19. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  20. Melvin Johnson, Mike Schuster, Quoc Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viegas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Associ… find
  21. Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural sequence modeling by iterative refinement. CoRR, abs/1802.06901. find our card
  22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/libovicky-2018-end-end-non-autoregressive-neural/