Elamur Research⌕
☰
home / papers / 2019 / Context-Aware Monolingual Repair for Neural Machine Translation

Context-Aware Monolingual Repair for Neural Machine Translation

Paper · 2019 · Elena Voita, Rico Sennrich and Ivan Titov

Summary

Sentence-level NMT systems translate each sentence on its own, so neighbouring sentences can disagree in speaker gender, formality or word choice even when each one looks fine in isolation. Most context-aware NMT methods need parallel corpora aligned at the document level, and such corpora are rarely available. The authors propose DocRepair, a Transformer that works only on the target side and post-edits a block of sentence-level outputs jointly. It is trained purely on monolingual target-language documents. Fragments of four consecutive sentences are corrupted by sending each sentence separately through a round-trip translation (Russian→English→Russian, with sampling), and the model learns to restore the original coherent text. DocRepair never uses the internals of the first-pass system, so it can sit on top of any black-box translator. On English→Russian OpenSubtitles2018 data, it raises BLEU, gives large gains on contrastive tests for discourse phenomena, and is clearly preferred by human annotators. The work shows that document-level consistency can be improved without scarce document-level parallel data.

Key points

  • Data: The baseline MT system was trained on 6m sentence pairs from OpenSubtitles2018, the same data released by Voita et al. (2019). DocRepair used 30m monolingual fragments of 4 consecutive sentences.

  • Training-data noise: Inputs were built by back-translating each sentence and then sampling forward translations at temperature 0.5. Twenty samples were precomputed per sentence, and 10% of input tokens were randomly replaced.

  • BLEU:

    • baseline: 33.91
    • CADec: 33.86
    • sentence-level repair: 34.12
    • DocRepair: 34.60

  • Contrastive evaluation: The model was tested on four contrastive suites:

    • deixis (3000 instances)
    • lexical cohesion (2000 instances)
    • ellipsis affecting NP inflection (500 instances)
    • VP ellipsis (500 instances)

    VP ellipsis turned out to be the hardest phenomenon to learn from round-trip translations alone.

  • Release: Code and data, including the round-trip translations, are available at github.com/lena-voita/good-translation-wrong-in-context.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom255 works · 68,771 citations🎓University of AmsterdamAmsterdam · Netherlands89 works · 81,337 citations🎓University of ZurichZurich · Switzerland50 works · 11,138 citations🏢YandexMoscow · Russia9 works · 3,341 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Machine translation · 958 works

Cite

Cite as Elena Voita, Rico Sennrich and Ivan Titov. 2019. Context-Aware Monolingual Repair for Neural Machine Translation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1081

@inproceedings{voita2019context,
  title = {{Context-Aware Monolingual Repair for Neural Machine Translation}},
  author = {Elena Voita, Rico Sennrich and Ivan Titov},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1081},
  url = {https://doi.org/10.18653/v1/d19-1081}
}

References

19 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ruchit Agrawal, Turchi Marco, and Negri Matteo. 2018. Contextual Handling in Neural Machine Translation: Look Behind, Ahead and on Both Sides. find
  2. Samuel Läubli, Rico Sennrich, and Martin Volk. 2018. Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4791–4796. Association for Computational Linguistics.
  3. Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating Discourse Phenomena in Neural Machine Translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Paper…
  4. Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018. Opensubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan.
  5. Sameen Maruf and Gholamreza Haffari. 2018. Document context neural machine translation with memory networks. In Proceedings of the 56th Annual Markus Freitag, Isaac Caswell, and Scott Roy. 2019. Ape at scale and its implications on mt evaluation Kaiser, and Illia Polosukhin. 2017. Attention is all y…
  6. Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1275– 1284, Melbourne, Australia. Association for Computational Linguistics. find
  7. Elena Voita, Rico Sennrich, and Ivan Titov. 2019. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1198–1212, Florence,…
  8. Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018. Document-level neural machine translation with hierarchical attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2947–2954, Brussels, Belgium. Association for…
  9. Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-aware neural machine translation learns anaphora resolution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1264–1274, Melbourne, Australia. Associati…
  10. Santanu Pal, Sudip Kumar Naskar, Mihaela Vela, and Josef van Genabith. 2016. A neural network based approach to automatic post-editing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 281–286, Berlin, Germany. Association for… find
  11. Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. 2017. Exploiting Cross-Sentence Context for Neural Machine Translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP’17, pages 2816– 2821, Denmark, Copenhagen. Association for Computational Linguist…
  12. Martin Popel and Ondrej Bojar. 2018. Training Tips for the Transformer Model. pages 43–70.
  13. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational… find
  14. Hao Xiong, Zhongjun He, Hua Wu, and Haifeng Wang. 2019. Modeling coherence for discourse neural machine translation. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):7338–7345. find
  15. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational L…
  16. Michel Simard, Cyril Goutte, and Pierre Isabelle. 2007. Statistical phrase-based post-editing. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 508–515, Rochester, New York. A…
  17. Lucia Specia, Kim Harris, Frédéric Blain, Aljoscha Burchardt, Vivien Macketanz, Inguna Skadin , a, Matteo Negri, , and Marco Turchi. 2017. Translation quality and productivity: A study on rich morphology languages. In Machine Translation Summit XVI, pages 55–71. Asia-Pacific Association for Machine… find
  18. Jörg Tiedemann and Yves Scherrer. 2017. Neural Machine Translation with Extended Context. In Proceedings of the Third Workshop on Discourse in Machine Translation, DISCOMT’17, pages 82–92, Copenhagen, Denmark. Association for Computational Linguistics.
  19. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/voita-2019-context-aware-monolingual-repair-neural/