Elamur Research⌕
☰
home / papers / 2019 / Monotonic Infinite Lookback Attention for Simultaneous Machine Translation

Monotonic Infinite Lookback Attention for Simultaneous Machine Translation

Summary

Simultaneous machine translation starts producing output before the speaker finishes a sentence. It must decide when to read more source tokens and when to write target tokens, trading quality against delay. Earlier systems either learned an adaptive schedule on top of an NMT model trained only on full sentences, or trained NMT jointly with a rigid schedule such as wait-k. The authors propose Monotonic Infinite Lookback (MILk) attention for an RNN encoder-decoder. A hard monotonic head decides how much of the source has been read. A soft head can then attend to every token up to that point. Because unlimited lookback removes any built-in incentive to stream, they add a latency term to the training loss. This term is a differentiable, expectation-based version of Average Lagging. The whole model trains end to end as a substitute for standard soft attention. On German-English and English-French (WMT14), MILk beats wait-k, MoChA and Monotonic Attention across a wide range of lags. It keeps full-attention quality at noticeably lower delays, which shows the value of schedules that adapt to the input.

Key points

  • MILk is presented as the first simultaneous NMT system that learns an adaptive read/write schedule jointly with a model attending over all source tokens read so far. Differentiable Average Lagging (DAL) is used as the latency objective.
  • DeEn: MILk holds full-attention quality (28.4 BLEU) down to a lag of about 8.5 tokens, whereas wait-k degrades below 13.3. At a lag of 4 tokens, MILk leads by 1.5 BLEU.
  • EnFr (WMT14): wait-k drops sharply at k = 8 (DAL 8.4, BLEU 39.8), while MILk stays flat until a lag of 7.2 (BLEU 40.5). At a lag of 4.5, MILk leads by 2.3 BLEU.
  • MILk's advantage grows under the AL and AP metrics. However, the tested configuration could not reach lags below 4, while wait-k can still operate at very low latency.
  • MILk's schedule adapts to the input: its initial delays vary widely compared with the nearly constant wait-6, despite similar DAL (5.8 vs 6.5). It tends to wait until noun or verb phrases are complete, which helps it reorder words correctly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics Machine translation

 Institutions

🏢GoogleMountain View · United States272 works · 255,532 citations

Cite

Cite as Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, Chung-Cheng Chiu, Semih Yavuz, Ruoming Pang, Wei Li and Colin Raffel. 2019. Monotonic Infinite Lookback Attention for Simultaneous Machine Translation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1126

@inproceedings{arivazhagan2019monotonic,
  title = {{Monotonic Infinite Lookback Attention for Simultaneous Machine Translation}},
  author = {Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, Chung-Cheng Chiu, Semih Yavuz, Ruoming Pang, Wei Li and Colin Raffel},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1126},
  url = {https://doi.org/10.18653/v1/p19-1126}
}

References

22 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ashkan Alinejad, Maryam Siahbani, and Anoop Sarkar. 2018. Prediction improves simultaneous neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3022–3027. Association for Computational Linguistics. find
  2. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer Normalization. arXiv e-prints, page arXiv:1607.06450. 7 Wait-6 will have delays different from 6 only for source sentences with fewer than 6 tokens. our card
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. our card
  4. Srinivas Bangalore, Vivek Kumar Rangarajan Sridhar, Prakash Kolan, Ladan Golipour, and Aura Jimenez. 2012. Real-time incremental speech-to-speech translation of dialogs. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua… find
  5. Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018. The best of both worlds: Combining recent advances in ne… our card
  6. Colin Cherry and George Foster. 2019. Thinking Slow about Latency Evaluation for Simultaneous Machine Translation. arXiv e-prints, page arXiv:1906.00048. find
  7. Chung-Cheng Chiu and Colin Raffel. 2018. Monotonic chunkwise attention. In International Conference on Learning Representations. find
  8. Kyunghyun Cho and Masha Esipova. 2016. Can neural machine translation do simultaneous translation? CoRR, abs/1606.02012. — [arXiv:1606.02012](https://arxiv.org/abs/1606.02012) find
  9. Fahim Dalvi, Nadir Durrani, Hassan Sajjad, and Stephan Vogel. 2018. Incremental decoding and training methods for simultaneous translation in neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua… find
  10. Christian Fügen, Alex Waibel, and Muntsin Kolss. 2007. Simultaneous translation of lectures and speeches. Machine Translation, 21(4):209–252. find
  11. Alvin Grissom II, He He, Jordan Boyd-Graber, John Morgan, and Hal Daumé III. 2014. Don’t until the final verb wait: Reinforcement learning for simultaneous machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1342–1352. Associa… find
  12. Jiatao Gu, Graham Neubig, Kyunghyun Cho, and Victor O.K. Li. 2017. Learning to translate in real-time with neural machine translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1053–1062. Association f… our card
  13. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations. find
  14. Mingbo Ma, Liang Huang, Hao Xiong, Kaibo Liu, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, and Haifeng Wang. 2018. STACL: Simultaneous Translation with Integrated Anticipation and Controllable Latency. arXiv e-prints, page arXiv:1810.08398. — [arXiv:1810.08398](https://arxiv.org/abs/1810.08398) find
  15. Yusuke Oda, Graham Neubig, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2014. Optimizing segmentation strategies for simultaneous speech translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 551–556. Association… find
  16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. our card
  17. Ofir Press and Noah A. Smith. 2018. You May Not Need Attention. arXiv e-prints, page arXiv:1810.13409. — [arXiv:1810.13409](https://arxiv.org/abs/1810.13409) find
  18. Colin Raffel, Minh-Thang Luong, Peter J. Liu, Ron J. Weiss, and Douglas Eck. 2017. Online and linear-time attention by enforcing monotonic alignments. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2837–2846, Inte… find
  19. Vivek Kumar Rangarajan Sridhar, John Chen, Srinivas Bangalore, Andrej Ljolje, and Rathinavelu Chengalvarayan. 2013. Segmentation strategies for streaming speech translation. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human La… find
  20. Harsh Satija and Joelle Pineau. 2016. Simultaneous machine translation using deep reinforcement learning. In Proceedings of the Abstraction in Reinforcement Learning Workshop. find
  21. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725. Association for Computational Linguistics. our card
  22. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2826. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-monotonic-infinite-lookback-attention-simultaneous/