Elamur Research
☰
home / papers / 2017 / Evaluating Discourse Phenomena in Neural Machine Translation

Evaluating Discourse Phenomena in Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1711.00513

Summary

Context-aware neural machine translation is usually judged with BLEU, which barely registers whether a system handles phenomena that cross sentence boundaries. Bawden et al. address this for English→French with two hand-crafted contrastive test sets modelled on real dialogue from OpenSubtitles2016. One targets anaphoric pronouns (it/they), and the other targets coherence and cohesion (lexical consistency and disambiguation). Each item pairs a correct and an incorrect French translation, and a model passes if it gives the correct one a higher score. The disambiguating clue always sits in the preceding sentence, so a context-blind system scores exactly 50%. The authors use these sets to compare multi-encoder NMT variants trained on subtitles, a simple concatenation approach, and a new hybrid. The hybrid encodes the previous source sentence separately while decoding the previous and current sentences together. Multi-encoder models barely beat chance despite BLEU gains, whereas the hybrid scores best. This suggests that target-side context fed through the decoder matters most. The work offers a cheap, fully automatic check of real discourse-context use.

Key points

  • Each test set has 200 contrastive pairs. The coreference set has 50 blocks of four pairs, half of them "semi-correct" variants where the antecedent is oddly translated and the pronoun must agree with that choice. The coherence/cohesion set has 100 blocks of two pairs. Both are freely available at diamt.limsi.fr/eval.html.
  • Multi-encoder models using the previous source sentence (S-CONCAT, S-HIER, S-GATE) reach only 50% on coreference and at most 53.5% on coherence/cohesion. Variants using the previous target sentence (T-CONCAT, T-HIER, T-GATE) stay near chance on both sets.
  • Concatenating the previous and current sentences and decoding both (2-TO-2) yields 63.5% on coreference but only 52% on coherence/cohesion.
  • The novel S-HIER-TO-2 model performs best, with 72.5% on coreference and 57% on coherence/cohesion. Adding the previous target sentence as an extra encoder input (S-T-HIER-TO-2) lowers these scores to 56.5% and 51.5%.
  • Attention weights reveal no clear focus on pronoun antecedents. The authors take this as a sign that the useful context is carried in the decoder's recurrent state rather than in the encoder, and they suggest stream decoding as a direction for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Topics: Discourse analysis · Coreference resolution · Machine translation

Cited by

190 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 180 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1711.00513 · arXiv 1711.00513

@misc{bawden2017evaluating,
  title = {{Evaluating Discourse Phenomena in Neural Machine Translation}},
  author = {Bawden, Rachel and Sennrich, Rico and Birch, Alexandra and Haddow, Barry},
  year = {2017},
  doi = {10.48550/arxiv.1711.00513},
  eprint = {1711.00513},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1711.00513}
}

References

26 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of the 3rd International Conference on Learning Representations. ICLR’15. ArXiv: 1409.0473. ↗ our card
  2. Rachel Bawden. 2016. Cross-lingual Pronoun Prediction with Linguistically Informed Features. In Proceedings of the 1st Conference on Machine Translation. Berlin, Germany, WMT’16, pages 564–570. find
  3. Ozan Caglayan, Loı̈c Barrault, and Fethi Bougares. 2016. Multimodal Attention for Neural Machine Translation. In arXiv:1609.03976. ↗
  4. Marine Carpuat. 2009. One translation per discourse. In Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions. Boulder, Colorado, USA, SEW’09, pages 19–27. find
  5. Robert De Beaugrande and Wolfgang Dressler. 1981. Introduction to Text Linguistics. Longman, London. find
  6. Liane Guillou. 2016. Incorporating Pronoun Function into Statistical Machine Translation. Ph.D. thesis, School of Informatics. University of Edinburgh. find
  7. Liane Guillou and Christian Hardmeier. 2016. PROTEST: A Test Suite for Evaluating Pronouns in Machine Translation. In Proceedings of the 10th Language Resources and Evaluation Conference. Portorož, Slovenia, LREC’16, pages 636–643. find
  8. Liane Guillou, Christian Hardmeier, Preslav Nakov, Sara Stymne, Jörg Tiedemann, Yannick Versley, Mauro Cettolo, Bonnie Webber, and Andrei Popescu-Belis. 2016. Findings of the 2016 WMT Shared Task on Cross-lingual Pronoun Prediction. In Proceedings of the 1st Conference on Machine Translation. Berlin, Germany, WMT’16, pages 525– 542. find
  9. Christian Hardmeier. 2014. Discourse in Statistical Machine Translation. Ph.D. thesis, Uppsala University, Department of Linguistics and Philology, Uppsala, Sweden. find
  10. Po-Yao Huang, Frederick Liu, Sz-Rung Shiang, Jean Oh, and Chris Dyer. 2016. Attention-based Multimodal Neural Machine Translation. In Proceedings of the 1st Conference on Machine Translation. Berlin, Germany, volume 2: of WMT’16, pages 639– 645. find
  11. Pierre Isabelle, Colin Cherry, and George Foster. 2017. A Challenge Set Approach to Evaluating Machine Translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen, Denmark, EMNLP’17, pages 2476–2486. find our card
  12. Sebastien Jean, Stanislas Lauly, Orhan Firat, and Kyunghyun Cho. 2017a. Does Neural Machine Translation Benefit from Larger Context? In arXiv:1704.05135. ArXiv: 1704.05135. ↗
  13. Sebastien Jean, Stanislas Lauly, Orhan Firat, and Kyunghyun Cho. 2017b. Neural Machine Translation for Cross-Lingual Pronoun Prediction. In Proceedings of the 3rd Workshop on Discourse in Machine Translation. Copenhagen, Denmark, DIS-COMT’17, pages 54–57. find
  14. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics. Prague, Czech Republic, ACL’07, pages 177–180. find
  15. Jindřich Libovický and Jindřich Helcl. 2017. Attention Strategies for Multi-Source Sequence-to-Sequence Learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Vancouver, Canada, ACL’17, pages 196–202. find
  16. Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proceedings of the 10th Language Resources and Evaluation Conference. Portorož, Slovenia, LREC’16, pages 923–929. find our card
  17. Sharid Loáiciga, Sara Stymne, Preslav Nakov, Christian Hardmeier, Jörg Tiedemann, Mauro Cettolo, and Yannick Versley. 2017. Findings of the 2017 DiscoMT Shared Task on Cross-lingual Pronoun Prediction. In Proceedings of the 3rd Workshop on Discourse in Machine Translation. Copenhagen, Denmark, DISCOMT’17, pages 1–16. find
  18. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Philadelphia, USA, ACL’02, pages 311–318. find our card
  19. Annette Rios Gonzales, Laura Mascarell, and Rico Sennrich. 2017. Improving Word Sense Disambiguation in Neural Machine Translation with Sense Embeddings. In Proceedings of the 2nd Conference on Machine Translation. Copenhagen, Denmark, WMT’17, pages 11–19. find
  20. Carolina Scarton and Lucia Specia. 2015. A Quantitative Analysis of Discourse Phenomena in Machine Translation. Discours [online] (16). https://doi.org/10.4000/discours.9047. ↗
  21. Rico Sennrich. 2017. How Grammatical is Character-level Neural Machine Translation? In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. Valencia, Spain, EACL’17, pages 376–382. find
  22. Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio, Miceli Barone, Jozef Mokry, and Maria Nădejde. 2017. Nematus: a Toolkit for Neural Machine Translation. In Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics. Valencia, Spain, EACL’17, pages 65–68. find our card
  23. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Berlin, Germany, ACL’16, pages 1715– 1725. find
  24. Jörg Tiedemann and Yves Scherrer. 2017. Neural Machine Translation with Extended Context. In Proceedings of the 3rd Workshop on Discourse in Machine Translation. Copenhagen, Denmark, DIS-COMT’17, pages 82–92. find our card
  25. Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. 2017. Exploiting Cross-Sentence Context for Neural Machine Translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Denmark, Copenhagen, EMNLP’17, pages 2816–2821. find
  26. Barret Zoph and Kevin Knight. 2016. Multi-source Neural Translation. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics. San Diego, California, USA, NAACL’16, pages 30–34. find