Elamur Research⌕
☰
home / papers / 2018 / Coreference and Coherence in Neural Machine Translation: A Study Using Oracle Experiments

Coreference and Coherence in Neural Machine Translation: A Study Using Oracle Experiments

Summary

Standard NMT systems translate each sentence in isolation. They therefore struggle with discourse phenomena such as anaphoric pronouns (coreference) and consistent word choice across a document (coherence). These effects are also hard to measure with BLEU. Stojanovski and Fraser address both problems with automatically built oracle setups for English→German subtitle translation. In these setups the model receives an explicit contextual hint: the gold previous target sentence, the target-side pronouns (marked in the source with XPRONOUN), or the target translation of a word repeated from the prior sentence (marked with XREP). Each oracle also has a fair variant, which uses no target-side knowledge, and a noisy variant, which adds the previous target sentence to test robustness. The authors propose an RNN model whose context attention operates only over word embeddings, plus a context-aware Transformer. They compare both against concatenation baselines. Given clear signals, context-aware models gain strongly in BLEU, and concatenation models score highest but run slower. Because the oracles need no hand-built challenge sets, the method transfers easily to other domains and language pairs.

Key points

  • On English→German subtitles, the coreference oracle yields BLEU improvements of as much as 7.02, while the coherence oracle yields gains of at most 1.89 BLEU.
  • The fair coreference variant obtains antecedents with CorefAnnotator from Stanford CoreNLP. The fair previous-sentence variant uses a baseline system's translation, or the previous source sentence, as context.
  • The models compared are an NMT(RNN) with embedding-level context attention, a context-aware NMT(Transformer), and concatenation models following Tiedemann and Scherrer (2017). The paper also checks its oracles against the challenge sets of Bawden et al. (2018).
  • In decoding the development set, the proposed RNN context model took 1233 s and the concatenation model 2063 s. Concatenation cost roughly 900 s more for each additional context sentence.
  • Attention analysis on noisy oracles shows that the concatenation model locates XREP signals reliably. The embedding-attention model often attends to the correct word but still fails to generate it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution · Machine translation

Cited by

16 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 12 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-6306

@inproceedings{stojanovski2018coreference,
  title = {{Coreference and Coherence in Neural Machine Translation: A Study Using Oracle Experiments}},
  author = {Stojanovski, Dario and Fraser, Alexander},
  booktitle = {Proceedings of the Third Conference on Machine Translation: Research Papers},
  year = {2018},
  doi = {10.18653/v1/w18-6306},
  url = {https://doi.org/10.18653/v1/w18-6306}
}

References

30 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christian Hardmeier, Sara Stymne, Jörg Tiedemann, and Joakim Nivre. 2013. Docent: A document-level decoder for phrase-based statistical machine translation. In ACL 2013 (51st Annual Meeting of the Association for Computational Linguistics); 4-9 August 2013; Sofia, Bulgaria, pages 193–198. find
  2. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450. find our card
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations, ICLR ’15. ArXiv: 1409.0473. find our card
  4. Felix Hieber, Tobias Domhan, Michael Denkowski, David Vilar, Artem Sokolov, Ann Clifton, and Matt Post. 2017. Sockeye: A Toolkit for Neural Machine Translation. ArXiv e-prints. find
  5. Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating Discourse Phenomena in Neural Machine Translation. In NAACL 2018, New Orleans, USA. find our card
  6. Sebastien Jean, Stanislas Lauly, Orhan Firat, and Kyunghyun Cho. 2017. Does neural machine translation benefit from larger context? arXiv preprint arXiv:1704.05135. find
  7. Marine Carpuat. 2009. One translation per discourse. In Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions, pages 19–27. find
  8. Yangfeng Ji, Trevor Cohn, Lingpeng Kong, Chris Dyer, and Jacob Eisenstein. 2015. Document context language models. arXiv preprint arXiv:1511.03962. — [arXiv:1511.03962](https://arxiv.org/abs/1511.03962) find
  9. Marine Carpuat and Michel Simard. 2012. The trouble with smt consistency. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 442–449. find
  10. Ronan Le Nagard and Philipp Koehn. 2010. Aiding pronoun translation with co-reference resolution. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 252–261. find our card
  11. Kevin Clark and Christopher D. Manning. 2016a. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2256–2262. find our card
  12. Sharid Loáiciga, Sara Stymne, Preslav Nakov, Christian Hardmeier, Jörg Tiedemann, Mauro Cettolo, and Yannick Versley. 2017. Findings of the 2017 discomt shared task on cross-lingual pronoun prediction. In The Third Workshop on Discourse in Machine Translation. find
  13. Kevin Clark and Christopher D. Manning. 2016b. Improving coreference resolution by learning entity-level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643–653. find our card
  14. Thomas Mueller, Helmut Schmid, and Hinrich Schütze. 2013. Efficient higher-order CRFs for morphological tagging. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 322–332, Seattle, Washington, USA. find
  15. Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In Advances in neural information processing systems, pages 1019–1027. find our card
  16. Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157–163. find our card
  17. Annette Rios Gonzales, Laura Mascarell, and Rico Sennrich. 2017. Improving word sense disambiguation in neural machine translation with sense embeddings. In Proceedings of the Second Conference on Machine Translation, pages 11–19. find our card
  18. Tim Salimans and Diederik P Kingma. 2016. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pages 901–909. find
  19. Liane Kirsten Guillou. 2016. Incorporating pronoun function into statistical machine translation. Ph.D. thesis, The University of Edinburgh, UK. find
  20. Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. Nematus: a toolkit for neural machine translation. In Proceedings of the Software Demonstrations of… find our card
  21. Christian Hardmeier. 2012. Discourse in statistical machine translation. a survey and a case study. Discours. Revue de linguistique, psycholinguistique et informatique. A journal of linguistics, psycholinguistics and computational linguistics, (11). find
  22. Christian Hardmeier and Marcello Federico. 2010. Modelling pronominal anaphora in statistical machine translation. In IWSLT (International Workshop on Spoken Language Translation); Paris, France; December 2nd and 3rd, 2010., pages 283– 289. find our card
  23. Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C Courville, and Joelle Pineau. 2016. find
  24. Building end-to-end dialogue systems using generative hierarchical neural network models. In AAAI, pages 3776–3784. Jörg Tiedemann and Yves Scherrer. 2017. Neural machine translation with extended context. In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 82–92. find our card
  25. Zhaopeng Tu, Yang Liu, Shuming Shi, and Tong Zhang. 2017. Learning to remember translation history with a continuous cache. arXiv preprint arXiv:1711.09367. find
  26. Ferhan Ture, Douglas W Oard, and Philip Resnik. 2012. Encouraging consistent translation choices. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 417–426. find
  27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000–6010. find
  28. Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-Aware Neural Machine Translation Learns Anaphora Resolution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1264–1274, Melbourne, Australia. find our card
  29. Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. 2017. Exploiting cross-sentence context for neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2826–2831. find
  30. Tian Wang and Kyunghyun Cho. 2016. Larger-context language modelling with recurrent neural network. In 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 - Long Papers, volume 3, pages 1319–1329. find

↑ back to top

This page in Markdown →