Elamur Research⌕
☰
home / papers / 2020 / Document-level Neural MT: A Systematic Comparison

Document-level Neural MT: A Systematic Comparison

Summary

Many context-aware neural MT methods exist, but they have been tested on different languages, datasets and model sizes, so their relative merits were unclear. The authors re-implement representative approaches under a shared setup with strong Transformer baselines pre-trained on large sentence-level corpora: Concat2to1/Concat2to2, a multi-source context encoder, and a cache-based model. They also propose the Doc-Star-Transformer, a document-level adaptation of the Star Transformer. It compresses each sentence into a single embedding and lets words attend to these embeddings. Gradient-stopped caching allows it to scale to documents of any length. The systems were tested on two tasks: document translation for EN→FR and EN→DE using public data, and chat translation for EN→FR, EN→DE and EN→PT br using proprietary data. Evaluation combined BLEU, contrastive test sets for pronouns and coherence/cohesion, and human MQM scoring. With strong baselines, context brought only small BLEU gains. The simple Concat2to2 handled discourse phenomena best, and Doc-Star-Transformer received good human ratings. The work also shows that BLEU fails to separate strong systems.

Key points

  • Concat2to2, which passes translated context through the decoder, was the clear winner on contrastive sets. EN→DE pronoun accuracy rose from 45.0% to 70.8%, and one EN→FR anaphora set rose from 50.0% to 82.5%. The multi-source encoder fell below the baseline.
  • In chat translation, Doc-Star-Transformer beat the baseline consistently, with the largest effect on the lowest-resource pair, EN→PT br. Feeding it predicted rather than gold target history lowered its scores.
  • Human MQM scores for EN→PT br were 92.47 for the baseline, 93.20 for Cache, and 95.20 for Star. For EN→FR, Star scored slightly below the baseline (86.80 vs 87.46). BLEU differences were marginal throughout.
  • The authors release a new large-scale EN→FR anaphoric pronoun contrastive test set (GitHub: rbawden/Large-contrastive-pronoun-testset-EN-FR).
  • For chat, speaker tags were added so that models can use context from both speakers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

75 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 71 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{lopes2020document,
  title = {{Document-level Neural MT: A Systematic Comparison}},
  author = {Lopes, António and Farajian, M. Amin and Bawden, Rachel and Zhang, Michael and Martins, André F. T.},
  booktitle = {Proceedings of the 22nd Annual Conference of the European Association for Machine Translation},
  year = {2020},
  url = {https://aclanthology.org/2020.eamt-1.24/}
}

References

41 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bahdanau, D., K. Cho, and Y. Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. find our card
  2. Barrault, Loı̈c, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. *Figure 3: Examples of four challenges for MT of Ope… find our card
  3. Hardmeier, Christian. 2012. Discourse in Statistical Machine Translation. a survey and a case study. Discours, 11. find
  4. Bawden, Rachel, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating Discourse Phenomena in Neural Machine Translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies. find our card
  5. Hassan, Hany, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, et al. 2018. Achieving human parity on automatic Chinese to English news translation. arXiv preprint arXiv:1803.05567. find our card
  6. Bawden, Rachel, Sophie Rosset, Thomas Lavergne, and Eric Bilinski. 2019. DiaBLa: A Corpus of Bilingual Spontaneous Written Dialogues for Machine Translation. find
  7. Junczys-Dowmunt, Marcin. 2019. Microsoft translator at wmt 2019: Towards large-scale document-level neural machine translation. In Proceedings of the Fourth Conference on Machine Translation. find our card
  8. Bawden, Rachel. 2018. Going beyond the sentence: Contextual Machine Translation of Dialogue. Ph.D. thesis, LIMSI, CNRS, Université Paris-Sud, Université Paris-Saclay, Orsay, France. find
  9. Klein, Guillaume, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. Open-NMT: Open-source toolkit for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. find
  10. Cettolo, Mauro, Christian Girardi, and Marcello Federico. 2012. WIT 3 : Web Inventory of Transcribed and Translated Talks. In Proceedings of the 16th Conference of the European Association for Machine Translation. find
  11. Dai, Zihang, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. In Proceedings of the 57th annual meeting on association for computational linguistics. find
  12. Dyer, Chris, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find
  13. Grave, Edouard, Armand Joulin, and Nicolas Usunier. 2016. Improving neural language models with a continuous cache. arXiv preprint arXiv:1612.04426. find
  14. Guo, Qipeng, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019. Star-transformer. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find
  15. Koehn, Philipp, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Pr… find
  16. Läubli, Samuel, Rico Sennrich, and Martin Volk. 2018. Has machine translation achieved human parity? a case for document-level evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. find our card
  17. Laubli, Samuel, Sheila Castilho, Graham Neubig, Rico Sennrich, Qinlan Shen, and Antonio Toral. 2020. A set of recommendations for assessing human–machine parity in language translation. Journal of Artificial Intelligence Research (JAIR), 67. find
  18. Lison, Pierre, Jörg Tiedemann, and Milen Kouylekov. 2018. OpenSubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In Proceedings of the 11th International Conference on Language Resources and Evaluation. find our card
  19. Lommel, Arle Richard. 2013. Multidimensional quality metrics: a flexible system for assessing translation quality Maruf, Sameen, André F. T. Martins, and Gholamreza Haffari. 2018. Contextual neural model for translating bilingual multi-speaker conversations. In Proc. of the 3rd Conference on Machin… find our card
  20. Sennrich, Rico. 2017. How Grammatical is Character-level Neural Machine Translation? In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. find
  21. Maruf, Sameen, André F. T. Martins, and Gholamreza Haffari. 2019. Selective Attention for Context-aware Neural Machine Translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find
  22. Smith, Karin Sim. 2017. On integrating discourse in machine translation. In Proceedings of the Third Workshop on Discourse in Machine Translation. find
  23. Miculicich, Lesly, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018. Document-Level Neural Machine Translation with Hierarchical Attention Networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. find our card
  24. Tiedemann, Jörg and Yves Scherrer. 2017. Neural machine translation with extended context. In Proceedings of the 3rd Workshop on Discourse in Machine Translation. find our card
  25. Toral, Antonio, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation. In Proceedings of the Third Conference on Machine Translation. find our card
  26. Müller, Mathias, Annette Rios, Elena Voita, and Rico Sennrich. 2018. A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation. In Proceedings of the Third Conference on Machine Translation. find our card
  27. Tu, Zhaopeng, Yang Liu, Shuming Shi, and Tong Zhang. 2018. Learning to remember translation history with a continuous cache. Transactions of the Association for Computational Linguistics, 6. find
  28. Ng, Nathan, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook fair’s wmt19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1). find our card
  29. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. find
  30. Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. find our card
  31. Voita, Elena, Rico Sennrich, and Ivan Titov. 2019a. Context-aware monolingual repair for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. find
  32. Pereyra, Gabriel, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton. 2017. Regularizing neural networks by penalizing confident output distributions. In Proceedings of the 5th International Conference on Learning Representations. find our card
  33. Popel, Martin and Ondřej Bojar. 2018. Training tips for the transformer model. The Prague Bulletin of Mathematical Linguistics, 110(1). find
  34. Press, Ofir and Lior Wolf. 2017. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. find our card
  35. Rios Gonzales, Annette, Laura Mascarell, and Rico Sennrich. 2017. Improving word sense disambiguation in neural machine translation with sense embeddings. In Proceedings of the 2nd Conference on Machine Translation. find our card
  36. Sagot, Benoı̂t. 2010. The lefff, a freely available and large-coverage morphological and syntactic lexicon for French. In Proceedings of the 7th International Conference on Language Resources and Evaluation. find our card
  37. Sennrich, Rico, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. of the 54th Annual Meeting of the Association for Computational Linguistics. find
  38. Voita, Elena, Rico Sennrich, and Ivan Titov. 2019b. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. find our card
  39. Wang, Longyue. 2019. Discourse-Aware Neural Machine Translation. Ph.D. thesis, Dublin City University, Dublin, Ireland. find
  40. Yu, Lei, Laurent Sartran, Wojciech Stokowiec, Wang Ling, Lingpeng Kong, Phil Blunsom, and Chris Dyer. 2019. Putting machine translation in context with the noisy channel model. find
  41. Zhang, Jiacheng, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018. Improving the Transformer Translation Model with Document-Level Context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lopes-2020-document-level-neural-mt-systematic/