Elamur Research
☰
home / papers / 2023 / Large Language Models Effectively Leverage Document-level Context for Literary Translation, but Critical Errors Persist

Large Language Models Effectively Leverage Document-level Context for Literary Translation, but Critical Errors Persist

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2023.wmt-1.41.pdf

Summary

LLMs now compete with commercial systems on sentence-level translation benchmarks. It was unclear whether they can use broader context when translating literature, partly because automatic document-level metrics are unreliable and human evaluation is costly. Karpinska and Iyyer compiled 360 source–translation paragraph pairs from 18 novels. The published human translations mostly appeared after 2021, which limits the risk of pretraining contamination. The data span 18 language pairs, with English, Japanese and Polish as target languages. They compared GPT-3.5 (text-davinci-003) translating whole paragraphs via few-shot prompting (PARA) with sentence-by-sentence variants (SENT, PARA_SENT) and Google Translate (GTR). Bilingual translators annotated error spans with an MQM-inspired taxonomy, chose the better of two outputs and justified their choice. In every language pair, PARA produced fewer errors and was preferred. It resolved pronouns, ellipsis, polysemy and cultural references more accurately and kept style more consistent. However, it still made serious mistranslations and omitted content more often than the baselines, so human translators remain necessary. The released annotations support research on evaluating document-level literary MT.

Key points

  • The dataset contains 20 aligned paragraphs from each of 18 recent novel translations (360 in total), with an average of 7.45 sentences per paragraph. Sentence segmentation was done manually, and only about 2% of each novel is released, under fair use.
  • Human annotation covered 720 pairs of translated paragraphs and took about 350 hours of annotation and analysis. Each judgment combined span-level error labels, a preference choice and a free-text justification.
  • Context matters for this kind of text: at least 55% of the reference paragraphs merge or split source sentences.
  • A probe for memorization asked text-davinci-003 to recover masked character names in 171 paragraphs. It almost always failed, and the few correct answers were likely inferable from context.
  • PARA had the fewest errors but omitted more content than SENT and GTR. A preliminary test suggests GPT-4 only partly fixes these problems. Only mid- and high-resource languages were tested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Cited by

91 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 87 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.wmt-1.41

@inproceedings{karpinska2023large,
  title = {{Large Language Models Effectively Leverage Document-level Context for Literary Translation, but Critical Errors Persist}},
  author = {Karpinska, Marzena and Iyyer, Mohit},
  booktitle = {Proceedings of the Eighth Conference on Machine Translation},
  year = {2023},
  doi = {10.18653/v1/2023.wmt-1.41},
  url = {https://doi.org/10.18653/v1/2023.wmt-1.41}
}

References

79 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ruchit Agrawal, Marco Turchi, and Matteo Negri. 2018. Contextual Handling in Neural machine Translation: Look Behind, Ahead and on Both Sides. In 21st Annual Conference of the European Association for Machine Translation, pages 11–20. find
  2. Rachith Aiyappa, Jisun An, Haewoon Kwak, and Yong-Yeol Ahn. 2023. Can we trust the evaluation on ChatGPT? find
  3. R.H. Baayen, D.J. Davidson, and D.M. Bates. 2008. Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4):390–412. find
  4. Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software, 67(1):1–48. find our card
  5. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA. Association for Computing Machinery. ↗
  6. Laurent Besacier. 2014. Machine translation for litterature: a pilot study (traduction automatisée d’une oeuvre littéraire: une étude pilote) [in French]. In Proceedings of TALN 2014 (Volume 2: Short Papers), pages 389–394, Marseille, France. Association pour le Traitement Automatique des Langues. find
  7. Sheila Castilho. 2021. Towards Document-Level human MT Evaluation: On the Issues of Annotator Agreement, Effort and Misevaluation. In Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), pages 34–45, Online. Association for Computational Linguistics. find
  8. Kent K. Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. 2023. Speak, memory: An archaeology of books known to chatgpt/gpt-4. find
  9. Junxuan Chen, Xiang Li, Jiarui Zhang, Chulun Zhou, Jianwei Cui, Bin Wang, and Jinsong Su. 2020. Modeling Discourse Structure for Document-level Neural Machine Translation. In Proceedings of the First Workshop on Automatic Simultaneous Translation, pages 30–36, Seattle, Washington. Association for Computational Linguistics. find
  10. Andrew Chesterman. 1997. Memes of Translation. Benjamins Translation Library. Benjamins (John) North America, Amsterdam, Netherlands. find
  11. Trevor Cohn and Mirella Lapata. 2007. Machine translation by triangulation: Making effective use of multi-parallel corpora. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 728–735, Prague, Czech Republic. Association for Computational Linguistics. find
  12. Marta R. Costa-jussà, Eric Smith, Christophe Ropers, Daniel Licht, Javier Ferrando, and Carlos Escolano. 2022. Toxicity in Multilingual Machine Translation at Scale. find
  13. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (Technology) is Power: A Critical Survey of “Bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454– 5476, Online. Association for Computational Linguistics. ↗ our card
  14. Chenchen Ding, Masao Utiyama, and Eiichiro Sumita. 2014. Document-level re-ranking with soft lexical and semantic features for statistical machine translation. In Proceedings of the 11th Conference of the Association for Machine Translation in the Americas: MT Researchers Track, pages 110–123, Vancouver, Canada. Association for Machine Translation in the Americas. find
  15. Bilingual Book Maker. 2023. Make bilingual epub books Using AI translate (GitHub). https://github.com/yihong0618/bilingual_ book_maker. [Accessed 05-Apr-2023]. ↗
  16. Joanna Drugan. 2013. Quality in professional translation. Bloomsbury Advances in Translation. Continuum Publishing Corporation, New York, NY. find
  17. Eleftheria Briakou, Colin Cherry, and George Foster. 2023. Searching for needles in a haystack: On the role of incidental bilingualism in palm’s translation capability. find
  18. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
  19. Marine Carpuat and Michel Simard. 2012. The Trouble with SMT Consistency. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 442–449, Montréal, Canada. Association for Computational Linguistics. find
  20. Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. 2023. GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models. find
  21. Yukun Feng, Feng Li, Ziang Song, Boyuan Zheng, and Philipp Koehn. 2022. Learn to Remember: Transformer with Recurrent Memory for Document-level Machine Translation. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1409–1420, Seattle, United States. Association for Computational Linguistics. find
  22. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. find our card
  23. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online. Association for Computational Linguistics. ↗ our card
  24. Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. 2022. Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text. find
  25. Marjan Ghazvininejad, Hila Gonen, and Luke Zettlemoyer. 2023. Dictionary-based Phrase-level Prompting of Large Language Models for Machine Translation. find
  26. Nuno M. Guerreiro, Duarte Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. 2023. Hallucinations in Large Multilingual Translation Models. find our card
  27. Yuchen Eleanor Jiang, Tianyu Liu, Shuming Ma, Dongdong Zhang, Mrinmaya Sachan, and Ryan Cotterell. 2023. Discourse-centric evaluation of document-level machine translation with a new densely annotated parallel corpus of novels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7853–7872, Toronto, Canada. Association for Computational Linguistics. find
  28. Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is ChatGPT A Good Translator? Yes with GPT-4 As The Engine. find our card
  29. Marcin Junczys-Dowmunt. 2019. Microsoft translator at WMT 2019: Towards large-scale document-level neural machine translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 225–233, Florence, Italy. Association for Computational Linguistics. find
  30. Christian Hardmeier. 2012. Discourse in Statistical Machine Translation. Discours, (11). find
  31. Xiaomian Kang, Yang Zhao, Jiajun Zhang, and Chengqing Zong. 2020. Dynamic context selection for document-level neural machine translation via reinforcement learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2242–2254, Online. Association for Computational Linguistics. find
  32. Christian Hardmeier, Sara Stymne, Jörg Tiedemann, and Joakim Nivre. 2013. Docent: A Document-level Decoder for Phrase-Based Statistical Machine Translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 193–198, Sofia, Bulgaria. Association for Computational Linguistics. find
  33. Marzena Karpinska, Nader Akoury, and Mohit Iyyer. 2021. The Perils of Using Mechanical Turk to Evaluate Open-ended Text Generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1265–1285, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  34. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation. find
  35. Marzena Karpinska, Nishant Raj, Katherine Thai, Yixiao Song, Ankita Gupta, and Mohit Iyyer. 2022. DEMETR: Diagnosing Evaluation Metrics for Translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9540–9561, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  36. Chao Han. 2020. Translation quality assessment: a critical methodological review. The Translator, 26(3):257–273. find
  37. Klára Jágrová and Tania Avgustinova. 2023. Intelligibility of highly predictable polish target words in sentences presented to czech readers. In Computational Linguistics and Intelligent Text Processing, pages 110–125. Springer Nature Switzerland. find
  38. Sebastien Jean, Stanislas Lauly, Orhan Firat, and Kyunghyun Cho. 2017. Does neural machine translation benefit from larger context? find
  39. Yuchen Jiang, Tianyu Liu, Shuming Ma, Dongdong Zhang, Jian Yang, Haoyang Huang, Rico Sennrich, Ryan Cotterell, Mrinmaya Sachan, and Ming Zhou. 2022. BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1550–1565, Seattle, United States. Association for Computational Linguistics. find our card
  40. Tom Kocmi and Christian Federmann. 2023. Large Language Models Are State-of-the-Art Evaluators of Translation Quality. find
  41. Alexandra Kuznetsova, Per B. Brockhoff, and Rune H. B. Christensen. 2017. lmerTest Package: Tests in Linear Mixed Effects Models. Journal of Statistical Software, 82(13):1–26. find
  42. Russell V. Lenth. 2023. emmeans: Estimated Marginal Means, aka Least-Squares Means. R package version 1.8.5. find
  43. Arle Lommel, Maja Popovic, and Aljoscha Burchardt. 2014a. Assessing inter-annotator agreement for translation error annotation. Reykjavik, Iceland. Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC 14). find
  44. Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014b. Multidimensional Quality Metrics (MQM) : A Framework for Declaring and Describing Translation Quality Metrics. Tradumàtica, pages 0455–463. find
  45. António Lopes, M. Amin Farajian, Rachel Bawden, Michael Zhang, and André F. T. Martins. 2020. Document-level neural MT: A Systematic Comparison. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 225–234, Lisboa, Portugal. European Association for Machine Translation. find
  46. Hongyuan Lu, Haoyang Huang, Dongdong Zhang, Haoran Yang, Wai Lam, and Furu Wei. 2023. Chain-of-dictionary prompting elicits translation in large language models. find
  47. Elman Mansimov, Gábor Melis, and Lei Yu. 2021. Capturing document context inside sentence-level neural machine translation models with self-training. In Proceedings of the 2nd Workshop on Computational Approaches to Discourse, pages 143–153, Punta Cana, Dominican Republic and Online. Association for Computational Linguistics. find
  48. Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018. Document-level neural machine translation with hierarchical attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2947–2954, Brussels, Belgium. Association for Computational Linguistics. find our card
  49. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. ↗ our card
  50. Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find our card
  51. Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André F. T. Martins, and Alon Lavie. 2021. Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared Task. In Proceedings of the Sixth Conference on Machine Translation, pages 1030–1040, Online. Association for Computational Linguistics. find our card
  52. Juan C. Sager. 1998. What Distinguishes Major Types of Translation? The Translator, 4(1):69–89. find
  53. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning Robust Metrics for Text Generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. find
  54. Lucía Molina and Amparo Hurtado Albir. 2004. Translation techniques revisited: A dynamic and functionalist approach. Meta, 47(4):498–512. find
  55. Kristiina Taivalkoski-Shilov. 2019a. Ethical issues regarding machine(-assisted) translation of literary texts. Perspectives, 27(5):689–703. find
  56. Wang Ning and César Domínguez. 2016. Comparative literature and translation: A cross-cultural and interdisciplinary perspective. In Yves Gambier and Luc van Doorslaer, editors, Border crossings. Translation studies and other disciplines, pages 287–308. John Benjamins. find
  57. Kristiina Taivalkoski-Shilov. 2019b. Free indirect discourse: an insurmountable challenge for literary MT systems? In Proceedings of the Qualities of Literary Machine Translation, pages 35–39. find
  58. Sharon O'Brien. 2012. Translation as human–computer interaction. Translation Spaces, 1:101–122. find
  59. OpenAI. 2022. Introducing ChatGPT. find
  60. OpenAI. 2023. GPT-4 technical report. find our card
  61. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. ↗
  62. Xin Tan, Longyin Zhang, Deyi Xiong, and Guodong Zhou. 2019. Hierarchical modeling of global context for document-level neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1576– 1585, Hong Kong, China. Association for Computational Linguistics. find
  63. Dorota Pawlak. 2023. ChatGPT for Translators: How to Use the Tool to Work More Efficiently? find
  64. Katherine Thai, Marzena Karpinska, Kalpesh Krishna, Bill Ray, Moira Inghilleri, John Wieting, and Mohit Iyyer. 2022. Exploring document-level literary machine translation with parallel paragraphs from world literature. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9882–9902, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  65. Matt Post and Marcin Junczys-Dowmunt. 2023. Escaping the sentence-level paradigm in machine translation. find
  66. Craig Thomson, Ehud Reiter, and Barkavi Sundararajan. 2023. Evaluating factual accuracy in complex data-to-text. Computer Speech & Language, 80:101482. find
  67. Jörg Tiedemann and Yves Scherrer. 2017. Neural machine translation with extended context. In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 82–92, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  68. Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. 2020-2022. Label Studio: Data labeling software. Open source software available from https://github.com/heartexlabs/label-studio. ↗
  69. Antonio Toral and Andy Way. 2015. Machine-assisted translation of literary text. Translation Spaces, 4(2):240–267. find
  70. Masao Utiyama and Hitoshi Isahara. 2007. A comparison of pivot methods for phrase-based statistical machine translation. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 484–491, Rochester, New York. Association for Computational Linguistics. find
  71. Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018. Improving the transformer translation model with document-level context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 533–542, Brussels, Belgium. Association for Computational Linguistics. find our card
  72. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. find
  73. Zaixiang Zheng, Xiang Yue, Shujian Huang, Jiajun Chen, and Alexandra Birch. 2020. Towards making the most of context in neural machine translation. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3983–3989. International Joint Conferences on Artificial Intelligence Organization. Main track. find
  74. Giorgos Vernikos, Brian Thompson, Prashant Mathur, and Marcello Federico. 2022. Embarrassingly Easy Document-level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metric. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find our card
  75. David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2022. Prompting palm for translation: Assessing strategies and performance. find
  76. Elena Voita, Rico Sennrich, and Ivan Titov. 2019. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1198–1212, Florence, Italy. Association for Computational Linguistics. find our card
  77. Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. 2023. Document-level machine translation with large language models. find our card
  78. Biao Zhang, Ankur Bapna, Melvin Johnson, Ali Dabirmoghaddam, Naveen Arivazhagan, and Orhan Firat. 2022. Multilingual document-level translation enables zero-shot transfer from sentences to documents. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4176–4192, Dublin, Ireland. Association for Computational Linguistics. find
  79. Biao Zhang, Barry Haddow, and Alexandra Birch. 2023. Prompting large language model for machine translation: A case study. find our card