Elamur Research⌕
☰
home / papers / 2022 / Rethinking Document-level Neural Machine Translation

Rethinking Document-level Neural Machine Translation

Paper · 2022 · Zewei Sun, Mingxuan Wang, Hao Zhou et al.

Summary

Most document-level neural machine translation (DNMT) research adds new modules to inject neighbouring-sentence context, such as hierarchical attention, extra context encoders or cache memories. The authors ask whether a plain Transformer can translate whole documents at once. First they re-examine earlier results. They show that a sentence-level baseline with stronger dropout matches or beats many reported DNMT gains, which suggests those gains came mostly from regularization on small corpora.

They then train an unmodified Transformer document-to-document (Doc2Doc) using Multi-resolutional Training. This mixes full documents with shorter sequences, including single sentences. Earlier work found that naive single-resolution Doc2Doc training fails, and this mixing makes it work. The approach adds no parameters, uses full source and target context, and can absorb extra sentence-level data. It handles documents of up to about 2000 words. Across nine document-level and two sentence-level datasets in six languages, it outperforms sentence-level models and prior DNMT systems on BLEU, lexical indices, new discourse metrics and human judgment. The work matters because it argues for simpler models and stronger baselines in DNMT research.

Key points

  • Baseline check: A 4-layer Transformer-base with dropout=0.3 reaches 19.21 BLEU on TED ZH-EN, above MCN's 19.10. On TED EN-DE it reaches 25.19, equal to the best prior method (QCN). This casts doubt on earlier improvements.
  • Datasets: The nine document-level sets are TED (ZH-EN, EN-DE), News (EN-DE, ES-EN, FR-EN, RU-EN), Europarl (EN-DE), Subtitles (EN-RU) and a newly built News ZH-EN set. Wikipedia (EN-DE) and WMT (ZH-EN) serve as sentence-level corpora.
  • New metrics: Three auxiliary linguistic indicators cover tense consistency, translation of omitted pronouns and conjunction presence. Human ratings on a 1–5 scale correlate strongly with the TCP metric.
  • Context sensitivity: Doc2Doc picks up cross-sentence context. In a probing test, it rendered "chancellor" for German and Austrian contexts but "prime minister" for Italian and French ones, unlike the sentence-level model.
  • Limitation and release: Self-attention cost grows quadratically with document length, so memory use and inference speed are the main bottleneck. Data and evaluation scripts are released at github.com/sunzewei2715/Doc2Doc_NMT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, Santa BarbaraSanta Barbara · United States35 works · 9,287 citations🏢ByteDanceBeijing · China18 works · 3,890 citations🎓Nanjing UniversityNanjing · China28 works · 7,072 citations🔬Peng Cheng LaboratoryShenzhen · China18 works · 2,349 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 958 works

Cite

Cite as Zewei Sun, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Shujian Huang, Jiajun Chen and Lei Li. 2022. Rethinking Document-level Neural Machine Translation. Findings of the Association for Computational Linguistics: ACL 2022.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.findings-acl.279

@inproceedings{sun2022rethinking,
  title = {{Rethinking Document-level Neural Machine Translation}},
  author = {Zewei Sun, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Shujian Huang, Jiajun Chen and Lei Li},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.279},
  url = {https://doi.org/10.18653/v1/2022.findings-acl.279}
}

References

60 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.
  2. Guangsheng Bao, Yue Zhang, Zhiyang Teng, Boxing Chen, and Weihua Luo. 2021. G-transformer for document-level machine translation. In ACL.
  3. Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2017. Evaluating discourse phenomena in neural machine translation. In NAACL-HLT.
  4. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2019. Longformer: The long-document transformer. arXiv, abs/2004.05150.
  5. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv, abs/1904.10509. find
  6. Gonçalo M Correia, Vlad Niculae, and André FT Martins. 2019. Adaptively sparse transformers. In EMNLP-IJCNLP.
  7. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In ACL. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
  9. Eva Martínez Garcia, C. Creus, and C. España-Bonet. 2019. Context-aware neural machine translation decoding. In DiscoMT@EMNLP. find
  10. Sébastien Jean and Kyunghyun Cho. 2019. Context-aware learning for neural machine translation. arXiv, abs/1903.04715. — [arXiv:1903.04715](https://arxiv.org/abs/1903.04715) find
  11. Sébastien Jean, Stanislas Lauly, Orhan Firat, and Kyunghyun Cho. 2017. Neural machine translation for cross-lingual pronoun prediction. In DiscoMT@EMNLP. find
  12. Shu Jiang, Rui Wang, Zuchao Li, Masao Utiyama, Kehai Chen, Eiichiro Sumita, Hai Zhao, and Bao-Liang Lu. 2019. Document-level neural machine translation with inter-sentence attention. arXiv, abs/1910.14528. — [arXiv:1910.14528](https://arxiv.org/abs/1910.14528) find
  13. Marcin Junczys-Dowmunt. 2019. Microsoft translator at wmt 2019: Towards large-scale document-level neural machine translation. In WMT.
  14. Prathyusha Jwalapuram, Barbara Rychalska, Shafiq Joty, and Dominika Basaj. 2020. Can your context-aware mt system pass the dip benchmark tests? : Evaluation benchmarks for discourse phenomena in machine translation. arXiv, abs/2004.14607. find
  15. Samuel Läubli, Rico Sennrich, and Martin Volk. 2018. Has machine translation achieved human parity? a case for document-level evaluation. In EMNLP.
  16. Bei Li, Hui Liu, Ziyang Wang, Yufan Jiang, Tong Xiao, Jingbo Zhu, Tongran Liu, and Changliang Li. 2020. Does multi-encoder help? a case study on context-aware neural machine translation. In ACL. find
  17. Yinhan Liu, Jiatao Gu, Naman Goyal, Xiongmin Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. TACL. find
  18. Shuming Ma, Dongdong Zhang, and Ming Zhou. 2020. A simple and effective unified encoder for document-level machine translation. In ACL.
  19. Sameen Maruf and Gholamreza Haffari. 2018. Document context neural machine translation with memory networks. In ACL.
  20. Sameen Maruf, André F. T. Martins, and Gholamreza" Haffari. 2019. Selective attention for context-aware neural machine translation. In NAACL-HLT.
  21. Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018. Document-level neural machine translation with hierarchical attention networks. In EMNLP.
  22. Mathias Müller, Annette Rios Gonzales, Elena Voita, and Rico Sennrich. 2018. A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation. In WMT.
  23. Xiaomian Kang, Yang Zhao, Jiajun Zhang, and Chengqing Zong. 2020. Dynamic context selection for document-level neural machine translation via reinforcement learning. In EMNLP. find
  24. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL.
  25. Yunsu Kim, Thanh Tran, and Hermann Ney. 2019. When and why is document-level context useful in neural machine translation? In DiscoMT@EMNLP-IJCNLP.
  26. Alec Radford. 2018. Improving language understanding by generative pre-training. find
  27. Diederick P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. find
  28. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  29. Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In ICLR.
  30. Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In ICLR.
  31. Philipp Koehn and Rebecca Knowles. 2017. challenges for neural machine translation. NMT@ACL. find
  32. Danielle Saunders, Felix Stahlberg, and Bill Byrne. 2020. Using context in neural machine translation training objectives. In ACL. find
  33. Six In Shaohui Kuang and Deyi Xiong. 2018. Fusing recency into neural machine translation with an inter-sentence gate model. In COLING. find
  34. Shaohui Kuang, Deyi Xiong, Weihua Luo, and Guodong Zhou. 2018. Modeling coherence for neural machine translation with dynamic and topic caches. In COLING. find
  35. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL.
  36. Zewei Sun, Shujian Huang, Hao-Ran Wei, Xin-yu Dai, and Jiajun Chen. 2020. Generating diverse translation by manipulating multi-head attention. In AAAI. find
  37. Zewei Sun, Mingxuan Wang, and Lei Li. 2021. Multilingual translation via grafting pre-trained language models. In EMNLP-Findings. find
  38. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In CVPR. find
  39. Xin Tan, Longyin Zhang, Deyi Xiong, and Guodong Zhou. 2019. Hierarchical modeling of global context for document-level neural machine translation. In EMNLP-IJCNLP. find
  40. Jörg Tiedemann and Yves Scherrer. 2017. Neural machine translation with extended context. In DiscoMT@EMNLP.
  41. Zhaopeng Tu, Yang P. Liu, Shuming Shi, and Tong Zhang. 2018. Learning to remember translation history with a continuous cache. TACL. find
  42. Inigo Jauregi Unanue, Nazanin Esmaili, Gholamreza Haffari, and Massimo Piccardi. 2020. Leveraging discourse rewards for document-level neural machine translation. In COLING. find
  43. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
  44. Elena Voita, Rico Sennrich, and Ivan Titov. 2019. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In ACL.
  45. Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-aware neural machine translation learns anaphora resolution. In ACL.
  46. Longyue Wang, Zhaopeng Tu, Shuming Shi, Tong Zhang, Yvette Graham, and Qun Liu. 2018a. Translating pro-drop languages with reconstruction models. In AAAI. find
  47. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  48. Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. 2020. Lite transformer with long-short range attention. In ICLR. find
  49. Hao Xiong, Zhongjun He, Hua Wu, and Haifeng Wang. 2018. Modeling coherence for discourse neural machine translation. In AAAI. find
  50. Hongfei Xu, Deyi Xiong, Josef van Genabith, and Qiuhui Liu. 2020. Efficient context-aware neural machine translation with layer-wise weighting and input-aware gating. In DiscoMT@IJCAI. find
  51. Zhengxin Yang, Jinchao Zhang, Fandong Meng, Shuhao Gu, Yang Feng, and Jie Zhou. 2019. Enhancing context modeling with a query-guided capsule network for document-level nmt. In EMNLP-IJCNLP. find
  52. Lei Yu, Laurent Sartran, Wojciech Stokowiec, Wang Ling, Lingpeng Kong, Phil Blunsom, and Chris Dyer. 2020. Better document-level machine translation with bayes’ rule. TACL. find
  53. Hyeongu Yun, Yongkeun Hwang, and Kyomin Jung. 2020. Improving context-aware neural machine translation using self-attentive sentence embedding. In AAAI. find
  54. Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. 2017. Exploiting cross-sentence context for neural machine translation. In EMNLP.
  55. Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang P. Liu. 2018. Improving the transformer translation model with document-level context. In EMNLP.
  56. Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. 2018b. Learning to jointly translate and predict dropped pronouns with a shared reconstruction mechanism. In EMNLP. find
  57. Zaixiang Zheng, Xiang Yue, Shujian Huang, Jiajun Chen, and Alexandra Birch. 2020. Toward making the most of context in neural machine translation. In IJCAI. find
  58. Longyue Wang, Zhaopeng Tu, Xiaojun Zhang, Hang Li, Andy Way, and Qun Liu. 2016a. A novel approach to dropped pronoun translation. In NAACL-HLT. find
  59. Longyue Wang, Xiaojun Zhang, Zhaopeng Tu, Hang Li, and Qun Liu. 2016b. Dropped pronoun generation for dialogue machine translation. In ICASSP. find
  60. Lesly Miculicich Werlen and Andrei Popescu-Belis. 2017. Validation of an automatic metric for the accuracy of pronoun translation (apt). In DiscoMT@EMNLP.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2022-rethinking-document-level-neural-machine/