Elamur Research
☰
home / papers / 2018 / Document-Level Neural Machine Translation with Hierarchical Attention Networks

Document-Level Neural Machine Translation with Hierarchical Attention Networks

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D18-1325.pdf

Summary

Standard NMT systems translate each sentence in isolation. This ignores discourse links and can hurt the cohesion of a translated document. The authors add document context to a Transformer through a hierarchical attention network (HAN) that reuses the model's own hidden states from the k previous sentences instead of training a separate context encoder. One attention level summarizes the words of each earlier sentence. A second level weighs those sentence summaries for the current word. A learned gate then decides how much context to mix into each word's representation.

Separate HANs can sit on the source side (HAN encoder) or the target side (HAN decoder variants), and the two can be combined. Experiments cover Chinese–English and Spanish–English in three domains: TED Talks, subtitles and news. The combined encoder+decoder model gives the best BLEU everywhere. It significantly beats a Transformer baseline and a cache-memory context model, and source and target context prove complementary. The model also improves pronoun and noun translation, lexical cohesion and coherence. This makes it a practical, reusable recipe for context-aware MT.

Key points

  • Data: Zh–En and Es–En TED Talks (IWSLT 2014/2015), Zh–En TV subtitles (Wang et al., 2018, with three references) and an Es–En OpenSubtitles2018 subset. News results are Es–En only, trained on News-Commentaries11 and tested on WMT newstest2009–2013.

  • Baselines: an OpenNMT Transformer "base" model (6 layers, 512 dimensions, 8 heads, 30K vocabulary) and a re-implemented cache model following Tu et al. (2018) with a 25-word memory.

  • Training: two stages, first sentence-level NMT and then the full network with HAN. Using k = 3 previous sentences worked best on the development set.

  • BLEU for HAN encoder + HAN decoder (gain over the Transformer):

    • Zh–En TED: 17.79 (+0.92)
    • Es–En TED: 37.24 (+1.80)
    • Zh–En subtitles: 29.67 (+1.07)
    • Es–En subtitles: 36.23 (+1.03)
    • Es–En news: 22.76 (+1.40)

    Training speed stays close to the cache model.

  • Discourse metrics and release: the model gains on pronoun and noun accuracy, lexical cohesion and LSA-based coherence (Wiki-6 model), though it still falls short of human references. Code is available at github.com/idiap/HAN_NMT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

155 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 151 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1325 · arXiv 1809.01576

@inproceedings{miculicich2018document,
  title = {{Document-Level Neural Machine Translation with Hierarchical Attention Networks}},
  author = {Miculicich, Lesly and Ram, Dhananjay and Pappas, Nikolaos and Henderson, James},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1325},
  eprint = {1809.01576},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d18-1325}
}

References

50 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations, San Diego, USA. find our card
  2. Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating discourse phenomena in neural machine translation. In Proceedings of the 16th Annual Conference of North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, USA.… find our card
  3. Steven Bird. 2006. Nltk: The natural language toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 69–72. Association for Computational Linguistics. find
  4. Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013. Findings of the 2013 Workshop on Statistical Machine Translation. In Proceedings of the Eighth Workshop on Statistical Machine Transl… find our card
  5. Leo Born, Mohsen Mesgar, and Michael Strube. 2017. Using a graph-based coherence model in document-level machine translation. In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 26–35, Copenhagen, Denmark. Association for Computational Linguistics. find
  6. Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. WIT 3 : Web inventory of transcribed and translated talks. In Proceedings of the 16 th Conference of the European Association for Machine Translation (EAMT), pages 261–268, Trento, Italy. find
  7. Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, Roldano Cattoni, and Marcello Federico. 2015. find
  8. The IWSLT 2015 evaluation campaign. In In proceedins of the International Workshop on Spoken Language Translation. find
  9. Christian Hardmeier, Sara Stymne, Jörg Tiedemann, and Joakim Nivre. 2013. cder for phrase-based statistical machine translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 193–198, Sofia, Bulgaria. Association for Computa… find
  10. Sebastien Jean, Stanislas Lauly, Orhan Firat, and Kyunghyun Cho. 2017. Does neural machine translation benefit from larger context? arXiv preprint arXiv:1704.05135. find
  11. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. Open-NMT: Open-source toolkit for neural machine translation. In Proc. ACL. find
  12. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of EMNLP 2004, pages 388–395, Barcelona, Spain. Association for Computational Linguistics. find
  13. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proc… find
  14. Ekaterina Lapshinova-Koltunski and Christian Hardmeier. 2017. Discovery of discourse-related language contrasts through alignment discrepancies in english-german translation. In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 73–81, Copenhagen, Denmark. Association for C… find
  15. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. In NISP 2016 -Deep Learning Symposium paper. find our card
  16. Peter W Foltz, Walter Kintsch, and Thomas K Landauer. 1998. The measurement of textual coherence with latent semantic analysis. Discourse processes, 25(2-3):285–307. find
  17. Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and tv subtitles. In In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016). European Language Resources Association. find our card
  18. Zhengxian Gong, Min Zhang, and Guodong Zhou. 2011. Cache-based document-level statistical machine translation. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 909–919, Edinburgh, Scotland, UK. Association for Computational Linguistics. find
  19. Sharid Loáiciga and Cristina Grisot. 2016. Predicting and using a pragmatic component of lexical aspect. LiLT (Linguistic Issues in Language Technology), 13. find
  20. Christian Hardmeier. 2012. Discourse in statistical machine translation. a survey and a case study. Discours. Revue de linguistique, psycholinguistique et informatique. A journal of linguistics, psycholinguistics and computational linguistics, (11). find
  21. Sameen Maruf and Gholamreza Haffari. 2018. Document context neural machine translation with memory networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1275– 1284. Association for Computational Linguistics. find
  22. Laura Mascarell. 2017. Lexical chains meet word embeddings in document-level statistical machine translation. In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 99– 109, Copenhagen, Denmark. Association for Computational Linguistics. find
  23. Thomas Meyer and Andrei Popescu-Belis. 2012. Using sense-labeled discourse connectives for statistical machine translation. In Proceedings of the Joint Workshop on Exploiting Synergies between Information Retrieval and Machine Translation (ESIRMT) and Hybrid Approaches to Machine Translation (HyTra)… find
  24. Thomas Meyer and Bonnie Webber. 2013. Implicitation of discourse connectives in (machine) translation. In Proceedings of the Workshop on Discourse in Machine Translation, pages 19–26, Sofia, Bulgaria. Association for Computational Linguistics. find
  25. Lesly Miculicich Werlen. 2017. Towards document-level neural machine translation. Technical report, Idiap. find
  26. Lesly Miculicich Werlen, Nikolaos Pappas, Dhananjay Ram, and Andrei Popescu-Belis. 2018. Self-attentive residual decoder for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  27. Lesly Miculicich Werlen and Andrei Popescu-Belis. 2017a. Using coreference links to improve spanish-to-english machine translation. In Proceedings of the 2nd Workshop on Coreference Resolution Beyond OntoNotes (CORBON 2017), pages 30–40, Valencia, Spain. Association for Computational Linguistics. find
  28. Lesly Miculicich Werlen and Andrei Popescu-Belis. 2017b. Validation of an automatic metric for the accuracy of pronoun translation (APT). In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 17–25, Copenhagen, Denmark. Association for Computational Linguistics. find
  29. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Lin… find our card
  30. Nikolaos Pappas and Andrei Popescu-Belis. 2017. Multilingual hierarchical attention networks for document classification. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1015–1025, Taipei, Taiwan. Asian Federation of Natural L… find
  31. Xiao Pu, Laura Mascarell, and Andrei Popescu-Belis. 2017. Consistent translation of repeated nouns using syntactic and semantic cues. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 948–957. Association for… find
  32. Annette Rios Gonzales and Don Tuggener. 2017. Coreference resolution of elided subjects and possessive pronouns in spanish-english statistical machine translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers,… find
  33. Karin Sim Smith. 2017. On integrating discourse in machine translation. In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 110–121, Copenhagen, Denmark. Association for Computational Linguistics. find
  34. Dan Stefanescu, Rajendra Banjade, and Vasile Rus. 2014. Latent semantic analysis models on wikipedia and tasa. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014). European Language Resources Association (ELRA). find
  35. Jörg Tiedemann. 2010. Context adaptation in statistical machine translation using models with exponentially decaying cache. In Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing, pages 8–15, Uppsala, Sweden. Association for Computational Linguistics. find
  36. Jörg Tiedemann and Yves Scherrer. 2017. Neural machine translation with extended context. In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 82–92, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  37. Zhaopeng Tu, Yang Liu, Zhengdong Lu, Xiaohua Liu, and Hang Li. 2017. Context gates for neural machine translation. Transactions of the Association for Computational Linguistics, 5:87–99. find
  38. Zhaopeng Tu, Yang Liu, Shuming Shi, and Tong Zhang. 2018. Learning to remember translation history with a continuous cache. Transactions of the Association for Computational Linguistics, 6. find
  39. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc. find
  40. Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-aware neural machine translation learns anaphora resolution. In Proceedings of the 56th Annual Meeting of the Association for pages 1480–1489, San Diego, California. Association for Computational Linguistics. find our card
  41. Computational Linguistics (Volume 1: Long Papers), pages 1264–1274. Association for Computational Linguistics. find
  42. Longyue Wang, Zhaopeng Tu, Shuming Shi, Tong Zhang, Yvette Graham, and Qun Liu. 2018. Translating pro-drop languages with reconstruction models. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pages 1–9, New Orleans, Louisiana, USA. AAAI Press. find
  43. Biao Zhang, Deyi Xiong, jinsong su, Hong Duan, and Min Zhang. 2016. Variational neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 521–530, Austin, Texas. Association for Computational Linguistics. find
  44. Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. 2017. Exploiting cross-sentence context for neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2826–2831, Copenhagen, Denmark. Association for Computational Linguistics. find
  45. Mingxuan Wang, Zhengdong Lu, Hang Li, and Qun Liu. 2016. Memory-enhanced decoder for neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 278–286, Austin, Texas. Association for Computational Linguistics. find
  46. Billy T. M. Wong and Chunyu Kit. 2012. Extending machine translation evaluation metrics with lexical cohesion to document level. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1060–1068, Jeju Island,… find
  47. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. find
  48. Deyi Xiong, Yang Ding, Min Zhang, and Chew Lim Tan. 2013. Lexical chain based cohesion models for document-level statistical machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1563–1573, Seattle, Washington, USA. Association for Comp… find
  49. Zichao Yang, Zhiting Hu, Yuntian Deng, Chris Dyer, and Alex Smola. 2017. Neural machine translation with recurrent attention modeling. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 383–387, Valencia, Spai… find
  50. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, find our card