Elamur Research⌕
☰
home / papers / 2026 / BlAInded by Fluency: How Idiomatic Machine Translation Outputs Affect Student Post-Editors' Edit Types

BlAInded by Fluency: How Idiomatic Machine Translation Outputs Affect Student Post-Editors' Edit Types

Summary

LLM-based translation is increasingly fluent, but it is unclear how that fluency shapes the work of people who post-edit it. This case study tests whether prompting an LLM for richer vocabulary and fewer English-style structures changes the output, and how such output affects the edits made by translation students. The authors compiled 18 recent British editorials and translated each into French with GPT-5.0, using two prompts: a plain translation request (P1) and one asking for lexical and syntactic variety without calques (P2). They compared the outputs with lexical and syntactic metrics. They then picked the most representative editorial and had twelve Master's students post-edit either the P1 or the P2 version. The post-edits were analysed with the same metrics and annotated using the MTPEAS taxonomy. P2 yielded measurably more varied and less source-bound French. After post-editing, the lexical gains persisted but the syntactic differences disappeared. Students working from the more idiomatic version edited less, overlooked more MT errors and corrected fewer of them properly. This suggests that fluent MT can lull post-editors into excessive trust, which matters for PE training and workflow design.

Key points

  • Corpus: 18 editorials (six each from The Guardian, The Telegraph and The Independent), published mid-July to mid-September 2025. The 594-word editorial used for post-editing, on the future of the EU, was chosen by Mahalanobis distance over metrics computed with SpaCy and Coh-Metrix.
  • Metrics and statistics: TTR, MATTR-50, lexical density and syntactic-equivalence measures were tested with paired t-tests, Wilcoxon or Mann-Whitney, and Welch's t-tests. Results were complemented by Hedges' g effect sizes, bootstrap resampling and Bonferroni correction.
  • Post-editing setup: 12 native-French second-year Master's students were split randomly into two groups of six. Each did a full PE in 110 minutes without MT tools and without knowing the study's aims.
  • Main result: P2 significantly raised lexical and syntactic variety in the MTs. In the post-edited texts (PE1 vs PE2), only the lexical differences remained significant.
  • Error handling:
    • Annotators tagged 31 errors in MT1 and 24 in MT2.
    • Students left roughly half of the MT1 errors and over two thirds of the MT2 errors unaddressed.
    • The successful edit rate in PE2 was about half that of PE1, and low in both groups.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{scourneau2026blainded,
  title = {{BlAInded by Fluency: How Idiomatic Machine Translation Outputs Affect Student Post-Editors' Edit Types}},
  author = {Scourneau, Valentin and Pires, Loïc De Faria},
  booktitle = {Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 1)},
  year = {2026},
  url = {https://aclanthology.org/2026.eamt-1.56/}
}

References

44 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alvarez-Vidal, Sergi, Maria Do Campo, Christian Olalla-Soler, and Pilar Sánchez-Gijón. 2025. Using Translation Techniques to Characterize MT Outputs. In Bouillon, Pierrette, Johanna Gerlach, Sabrina Girletti, Lise Volkart, Raphael Rubino, Rico Sennrich, Ana C. Farinha, Marco Gaido, Joke Daems, Dorothy Kenny, Helena Moniz, and Sara Szoc, editors, Proceedings of Machine Translation Summit XX: Volume 1, pages 619–627, Geneva, Switzerland, June. European Association for Machine Translation. find
  2. Baker, M. 1996. Corpus-based Translation Studies: The Challenges that Lie Ahead. In Terminology, LSP and Translation: Studies in Language Engineering in Honour of Juan C. Sager, pages 175–188. John Benjamins Publishing Company. find
  3. Bangalore, Srinivas, Bergljot Behrens, Michael Carl, Maheshwar Gankhot, Arndt Heilmann, Jean Nitzke, Moritz Schaeffer, and Annegret Sturm. 2015. The role of syntactic variation in translation and post-editing. Translation Spaces, 4(1):119–144. John Benjamins. ↗
  4. Bestgen, Yves. 2024. Diversité lexicale et longueur du texte en évaluation du langage. In Actes des 17es Journées internationales d’Analyse statistique des Données Textuelles, pages 89–98, Brussels, Belgium, June. find
  5. Bodart, Romane, Justine Piette, and Marie-Aude Lefer. 2024. The Machine Translation Post-Editing Annotation System (MTPEAS): A standardized and user-friendly taxonomy for student post-editing quality assessment. Translation Spaces, 13(2):265–292. John Benjamins Publishing Company. find
  6. Castilho, Sheila and Natália Resende. 2022. Post-Editese in Literary Translations. Information, 13(2):66. Multidisciplinary Digital Publishing Institute. ↗
  7. Castilho, Sheila, Natália Resende, and Ruslan Mitkov. 2019. What Influences the Features of Post-editese? A Preliminary Study. In Proceedings of the Human-Informed Translation and Interpreting Technology Workshop (HiT-IT 2019), pages 19–27, Varna, Bulgaria, September. find
  8. Chen, Pinzhen, Zhicheng Guo, Barry Haddow, and Kenneth Heafield. 2024. Iterative Translation Refinement with Large Language Models. arXiv:2306.03856 [cs]. ↗ our card
  9. Covington, Michael A. and Joe D. McFall. 2010. Cutting the Gordian Knot: The Moving-Average Type–Token Ratio (MATTR). Journal of Quantitative Linguistics, 17(2):94–100. Routledge. ↗
  10. Daems, Joke, Orphée De Clercq, and Lieve Macken. 2017. Translationese and Post-editese: How comparable is comparable quality? Linguistica Antverpiensia, New Series – Themes in Translation Studies, 16:89–103. ↗
  11. Dou, Zi-Yi and Graham Neubig. 2021. Word Alignment by Fine-tuning Embeddings on Parallel Corpora. In Merlo, Paola, Jorg Tiedemann, and Reut Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2112– 2128, Online, April. Association for Computational Linguistics. ↗
  12. European Language Industry Survey 2026, March. find
  13. Farrell, Michael. 2018. Machine translation markers in post-edited machine translation output. In Chambers, David, Joanna Drugan, Jo ao Esteves-Ferreira, Juliet Margaret Macan, Ruslan Mitkov, and Olaf-Michael Stefanov, editors, Proceedings of the 40th Conference Translating and the Computer, pages 50–59, London, UK, November. AsLing. find
  14. Farrell, Michael. 2023a. Current evidence of post-editese: differences between post-edited neural machine translation output and human translation revealed through human evaluation. In Proceedings of the International Conference on Human-informed Translation and Interpreting Technology 2023, pages 52–63, Naples, Italy, July. find
  15. Farrell, Michael. 2023b. Preliminary evaluation of ChatGPT as a machine translation engine and as an automatic post-editor of raw machine translation output from other machine translation engines. In Proceedings of the International Conference on Human-Informed Translation and Interpreting Technology 2023, pages 108–113, Naples, Italy, July. find
  16. Gellerstam, M. 1986. Translationese in Swedish novels translated from English. Translation studies in Scandinavia, 1:88–95. find
  17. Green, Spence, Jeffrey Heer, and Christopher D. Manning. 2013. The efficacy of human post-editing for language translation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, page 439–448, New York, NY, USA. Association for Computing Machinery. find
  18. Hansen, Damien and Emmanuelle Esperança-Rodier. 2022. Human-Adapted MT for Literary Texts: Reality or Fantasy? In Proceedings of the International Conference New Trends in Translation and Technology 2022, pages 178–190, Rhodes Island, Greece, July. find
  19. Hedges, Larry V. and Ingram Olkin. 2014. Statistical Methods for Meta-Analysis. Academic Press, San Diego. find
  20. Johansson, Victoria. 2008. Lexical diversity and lexical density in speech and writing: a developmental perspective. Working papers / Lund University, Department of Linguistics and Phonetics, 53:61–79. find
  21. Kocmi, Tom, Vilém Zouhar, Eleftherios Avramidis, Roman Grundkiewicz, Marzena Karpinska, Maja Popović, Mrinmaya Sachan, and Mariya Shmatova. 2024. Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation. In Haddow, Barry, Tom Kocmi, Philipp Koehn, and Christof Monz, editors, Proceedings of the Ninth Conference on Machine Translation, pages 1440– 1453, Miami, Florida, USA, November. Association for Computational Linguistics. ↗ our card
  22. Kocmi, Tom, Sweta Agrawal, Ekaterina Artemova, Eleftherios Avramidis, Eleftheria Briakou, Pinzhen Chen, Marzieh Fadaee, Markus Freitag, Roman Grundkiewicz, Yupeng Hou, Philipp Koehn, Julia Kreutzer, Saab Mansour, Stefano Perrella, Lorenzo Proietti, Parker Riley, Eduardo Sánchez, Patricia Schmidtova, Mariya Shmatova, and Vilém Zouhar. 2025. Findings of the WMT25 multilingual instruction shared task: Persistent hurdles in reasoning, generation, and evaluation. In Haddow, Barry, Tom Kocmi, Philipp Koehn, and Christof Monz, editors, Proceedings of the Tenth Conference on Machine Translation, pages 462–483, Suzhou, China, November. Association for Computational Linguistics. find
  23. Kunilovskaya, Maria, Koel Dutta Chowdhury, Heike Przybyl, Cristina España-Bonet, and Josef Genabith. 2024. Mitigating Translationese with GPT-4: Strategies and Performance. In Scarton, Carolina, Charlotte Prescott, Chris Bayliss, Chris Oakley, Joanna Wright, Stuart Wrigley, Xingyi Song, Edward Gow-Smith, Rachel Bawden, Víctor M Sánchez-Cartagena, Patrick Cadwell, Ekaterina Lapshinova-Koltunski, Vera Cabarrão, Konstantinos Chatzitheodorou, Mary Nurminen, Diptesh Kanojia, and Helena Moniz, editors, Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1), pages 411–430, Sheffield, UK, June. European Association for Machine Translation. find
  24. Lefer, Marie-Aude, Justine Piette, and Romane Bodart. 2022. Machine Translation Post-Editing Annotation System. Technical report, UCLouvain University. find
  25. Leufkens, Sterre. 2020. A functionalist typology of redundancy. Revista da ABRALIN, 19(3):79–103. ↗
  26. Li, Yafu, Ronghao Zhang, Zhilin Wang, Huajian Zhang, Leyang Cui, Yongjing Yin, Tong Xiao, and Yue Zhang. 2025. Lost in Literalism: How Supervised Training Shapes Translationese in LLMs, March. arXiv:2503.04369 [cs]. ↗
  27. Loock, Rudy. 2018. Traduction automatique et usage linguistique : une analyse de traductions anglais-français réunies en corpus. Meta: Journal des traducteurs, 63(3):786–806. ↗
  28. Marques, Francisco Paulo Jamil and Camila Mont’Alverne. 2021. What are newspaper editorials interested in? Understanding the idea of criteria of editorial-worthiness. Journalism, 22(7):1812–1830. SAGE Publications. find
  29. McNamara, Danielle S., Arthur C. Graesser, Philip M. McCarthy, and Zhiqiang Cai. 2014. Automated Evaluation of Text and Discourse with Coh-Metrix. Cambridge University Press, Cambridge. ↗
  30. Mukherjee, Ananya and Manish Shrivastava. 2025. Lost in translation? found in evaluation: A comprehensive survey on sentence-level translation evaluation. ACM Computing Surveys, 58(1):1–47. ↗
  31. Poibeau, Thierry. 2022. On “Human Parity” and “Super Human Performance” in Machine Translation Evaluation. In Calzolari, Nicoletta, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6018–6023, Marseille, France, June. European Language Resources Association. find
  32. Qi, Peng, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In Celikyilmaz, Asli and Tsung-Hsien Wen, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101–108, Online, July. Association for Computational Linguistics. find our card
  33. Sawilowsky, Shlomo S. 2009. New Effect Size Rules of Thumb. Journal of Modern Applied Statistical Methods, 8:597–599. ↗
  34. Schumacher, Perrine. 2025. Exploration des répercussions de la TA neuronale sur la langue cible après post-édition en contexte d’apprentissage : qu’en est-il du post-editese ? Langages, 237(1):109–130. Publisher: Armand Collin. ↗
  35. Scourneau, Valentin. 2025. Impact of Automatic Post-Editing and Prompting Strategies on the Linguistic Features of English-to-French Editorial Translations. Tradumàtica tecnologies de la traducció, 23:65–100. ↗
  36. Toral, Antonio. 2019. Post-editese: an Exacerbated Translationese. In Forcada, Mikel, Andy Way, Barry Haddow, and Rico Sennrich, editors, Proceedings of Machine Translation Summit XVII: Research Track, pages 273–281, Dublin, Ireland, August. European Association for Machine Translation. ↗
  37. Vanmassenhove, Eva, Dimitar Shterionov, and Andy Way. 2019. Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translation. In Forcada, Mikel, Andy Way, Barry Haddow, and Rico Sennrich, editors, Proceedings of Machine Translation Summit XVII: Research Track, pages 222–232, Dublin, Ireland, August. European Association for Machine Translation. ↗ our card
  38. Vanmassenhove, Eva, Dimitar Shterionov, and Matthew Gwilliam. 2021. Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation. In Merlo, Paola, Jorg Tiedemann, and Reut Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2203–2213, Online, April. Association for Computational Linguistics. find
  39. Vanroy, Bram, Orphée De Clercq, Arda Tezcan, Joke Daems, and Lieve Macken. 2021. Metrics of syntactic equivalence to assess translation difficulty. In Explorations in empirical translation process research, volume 3, pages 259–294. Springer. ↗
  40. Volkart, Lise and Pierrette Bouillon. 2023. Are post-editese features really universal? In Proceedings of the International Conference on Human-informed Translation and Interpreting Technology 2023, pages 294–304, July. find
  41. Volkart, Lise and Pierrette Bouillon. 2024. Post-editors as Gatekeepers of Lexical and Syntactic Diversity: Comparative Analysis of Human Translation and Post-editing in Professional Settings. In Scarton, Carolina, Charlotte Prescott, Chris Bayliss, Chris Oakley, Joanna Wright, Stuart Wrigley, Xingyi Song, Edward Gow-Smith, Rachel Bawden, Víctor M Sánchez-Cartagena, Patrick Cadwell, Ekaterina Lapshinova-Koltunski, Vera Cabarrão, Konstantinos Chatzitheodorou, Mary Nurminen, Diptesh Kanojia, and Helena Moniz, editors, Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1), pages 387–395, Sheffield, UK, June. European Association for Machine Translation. find
  42. Volkart, Lise, Sabrina Girletti, Johanna Gerlach, Jonathan David Mutal, and Pierrette Bouillon. 2022. Source or target first? Comparison of two post-editing strategies with translation students. Journal of Data Mining and Digital Humanities, Towards robotic translation? INRIA. find
  43. Wang, Longyue, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. 2023. Document-Level Machine Translation with Large Language Models. arXiv:2304.02210 [cs]. ↗ our card
  44. Zhang, Jia and Stephen Doherty. 2025. Investigating novice translation students’ ai literacy in translation education. The Interpreter and Translator Trainer, 19(3-4):234–253. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/scourneau-2026-blainded-fluency-how-idiomatic-machine/