Elamur Research⌕
☰
home / papers / 2026 / Hindsight Quality Prediction Experiments in Multi-Candidate Human-Post-Edited Machine Translation

Hindsight Quality Prediction Experiments in Multi-Candidate Human-Post-Edited Machine Translation

Summary

The authors examine whether established ways of anticipating MT quality still work now that LLMs are part of translation pipelines. They consider two kinds of prediction: estimating source-segment difficulty before translation, and reference-free quality estimation (QE) of candidate outputs. The study reuses data from a real post-editing project that built the OLDI Seed Corpus's French portion. Each English Wikipedia segment has nine French hypotheses, four from sentence-level NMT systems and five from prompted LLMs (Llama-4-Scout variants and DeepSeek-R1). All hypotheses are scored against one final human post-edited reference using TER (a post-editing effort proxy) and COMET (a human judgment proxy). Predictors are rated by their Kendall's τ with these scores.

The usefulness of difficulty features depends strongly on the target metric: length and neural predictors track COMET but not TER. QE scores align better with NMT outputs than with LLM outputs. Document-level LLMs degrade measurably for later segments, but the effect is tiny. The overall message is that quality-prediction tools calibrated on older NMT need rechecking for LLM-based translation.

Key points

  • Dataset: 6,193 English source segments from 203 core Wikipedia articles. The analysis uses 6,028 segments after dropping 165 that DeepSeek-R1 refused to translate. The data and code ("Hindex", GitHub) are released under CC-BY SA 4.0.
  • Systems compared:
    • NMT: OPUS-MT, NLLB-3.3B, NLLB-600M-Distilled, MADLAD-400-3B.
    • LLMs: Llama 4 Scout (109B) in segment-level and document-level settings, the latter with and without guidelines and with Wikipedia context, plus DeepSeek-R1 (671B).
    • DeepSeek-R1 scored clearly best (TER 12.44, COMET 0.9219). OPUS-MT and distilled NLLB scored lowest.
  • Source-side predictors: twelve features in total.
    • Nine readability and linguistic measures: Dale-Chall, SMOG, Flesch, Coleman-Liau, FK grade, length, word rarity, polysemy and tree height.
    • Neural predictors: MT surprisal and Sentinel DA/MQM scores.
  • Candidate-side QE: COMET_QE and MetricX_QE correlate more strongly with gold quality for NMT outputs than for LLM outputs. Against ChrF, for example, average COMET_QE τ is about 0.22 for NMT versus 0.16 for LLMs.
  • Positional bias: the bias in doc2doc LLMs is statistically significant but very weak (e.g., τ ≈ −0.04 for DeepSeek-R1 with ChrF). Supplementary ChrF results mirror the TER patterns.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/24puen8nstzh

@inproceedings{marmonier2026hindsight,
  title = {{Hindsight Quality Prediction Experiments in Multi-Candidate Human-Post-Edited Machine Translation}},
  author = {Marmonier, Malik and Sagot, Benoît and Bawden, Rachel},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/24puen8nstzh},
  url = {https://doi.org/10.63317/24puen8nstzh}
}

References

41 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA. Oral presentation. find our card
  2. Steven Bird and Edward Loper. 2004. NLTK: The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214–217, Barcelona, Spain. Association for Computational Linguistics. find
  3. John Blatz, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueffing. 2004. Confidence estimation for machine translation. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 315–321, Geneva, Sw… find
  4. Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2012. Findings of the 2012 Workshop on Statistical Machine Translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 10–51, Montréal, Canada. Association for Computation… find our card
  5. Aaron Chatterji, Thomas Cunningham, David J. Deming, Zoë Hitzig, Christopher Ong, Carl Yan Shan, and Kevin Wadman. 2025. How People Use ChatGPT. Working Paper 34255, National Bureau of Economic Research. find
  6. Meri Coleman and Ta Lin Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2):283–284. find
  7. David Dale, Laurie V. Burchell, Jean Maillard, Idris Abdulmumin, Antonios Anastasopoulos, Isaac Caswell, and Philipp Koehn. 2025. Findings of the WMT 2025 Shared Task of the Open Language Data Initiative. In Proceedings of the Tenth Conference on Machine Translation (WMT 2025), pages 495–502, Suzhou… find
  8. Edgar Dale and Jeanne S. Chall. 1948. A formula for predicting readability: Instructions. Educational Research Bulletin, pages 37–54. find
  9. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chengg… find our card
  10. Francesco Fernicola, Silvia Bernardini, Federico Garcea, Adriano Ferraresi, and Alberto Barrón-Cedeño. 2023. Return to the source: Assessing machine translation suitability. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 79– 89, Tampere, Finla… find
  11. Rudolph Flesch. 1948. A new readability yardstick. Journal of Applied Psychology, 32(3):221–233. find
  12. Simona Gandrabur and George Foster. 2003. Confidence estimation for translation prediction. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 95–102. find
  13. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous Measurement Scales in Human Evaluation of Machine Translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational… find
  14. Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task. In Proceedings of the Ninth Conference on Machine Translation, pages 492–504, Miami, Florida, USA. Association for Computational Linguistics. find our card
  15. J. Peter Kincaid, Robert P. Fishburne Jr, Richard L. Rogers, and Brad S. Chissom. 1975. Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel. Technical report, Naval Technical Training Command Millington TN Resear… find
  16. Zheng Wei Lim, Ekaterina Vylomova, Charles Kemp, and Trevor Cohn. 2024. Predicting human translation difficulty with neural machine translation. Transactions of the Association for Computational Linguistics, 12:1479–1496. find
  17. Arle Richard Lommel, Aljoscha Burchardt, and Hans Uszkoreit. 2013. Multidimensional quality metrics: a flexible system for assessing translation quality. In Proceedings of Translating and the Computer 35, London, UK. Aslib. find our card
  18. Jean Maillard, Cynthia Gao, Elahe Kalbassi, Kaushik Ram Sadagopan, Vedanuj Goswami, Philipp Koehn, Angela Fan, and Francisco Guzman. 2023. Small Data, Big Impact: Leveraging Minimal Data for Effective Machine Translation. In Proceedings of the 61st Annual Meeting of the Association for Computational… find
  19. Sameen Maruf, Fahimeh Saleh, and Gholamreza Haffari. 2021. A survey on document-level neural machine translation: Methods and evaluation. ACM Computing Surveys, 54(2). find
  20. G. Harry McLaughlin. 1969. Smog grading-a new readability formula. Journal of Reading, 12(8):639–646. find
  21. Meta AI. 2025. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation. find
  22. George A. Miller. 1994. WordNet: A lexical database for English. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994. find our card
  23. Abhijit Mishra, Pushpak Bhattacharyya, and Michael Carl. 2013. Automatically predicting sentence translation difficulty. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 346–351, Sofia, Bulgaria. Association for Computational… find
  24. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hof… find
  25. Joseph Olive. 2005. Global autonomous language exploitation (GALE). DARPA/IPTO Proposer Information Pamphlet. find
  26. Ziqian Peng, Rachel Bawden, and François Yvon. 2024. À propos des difficultés de traduire automatiquement de longs documents. In 35èmes Journées d’Études sur la Parole (JEP 2024), volume 1 : articles longs et prises de position, pages 2–21, Toulouse, France. ATALA & AFPC. find
  27. Ziqian Peng, Rachel Bawden, and François Yvon. 2025. Investigating length issues in document-level machine translation. In Proceedings of Machine Translation Summit XX: Volume 1, pages 4–23, Geneva, Switzerland. European Association for Machine Translation. find
  28. Maja Popović. 2015. chrF: character n-gram Fscore for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. find
  29. Lorenzo Proietti, Stefano Perrella, Vilém Zouhar, Roberto Navigli, and Tom Kocmi. 2025. Estimating machine translation difficulty. find
  30. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  31. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022. find
  32. CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find our card
  33. Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223–231, Cambridge, Massachu… find our card
  34. Robyn Speer. 2022. rspeer/wordfreq: v3.0. Zenodo, Version v3.0.2. find
  35. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NEURIPS’14, page 3104–3112, Cambridge, MA, USA. MIT Press. find
  36. Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT – Building open translation services for the World. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479–480, Lisboa, Portugal. European Association for Machine Translation. find
  37. Nicola Ueffing, Klaus Macherey, and Hermann Ney. 2003. Confidence measures for statistical machine translation. In Proceedings of Machine Translation Summit IX: Papers, New Orleans, USA. find
  38. Zhang Zhuocheng, Shuhao Gu, Min Zhang, and Yang Feng. 2023. Addressing the length bias challenge in document-level neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11545–11556, Singapore. Association for Computational Linguistics. find
  39. Vilém Zouhar, Pinzhen Chen, Tsz Kin Lam, Nikita Moghe, and Barry Haddow. 2024. Pitfalls and outlooks in using COMET. In Proceedings of the Ninth Conference on Machine Translation, pages 1272–1288, Miami, Florida, USA. Association for Computational Linguistics. 10. find
  40. Language Resource References Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A. Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2023. MADLAD-400: A Multilingual And Document-Level Large Audited Dataset. find our card
  41. Malik Marmonier, Benoît Sagot, and Rachel Bawden. 2025. A French Version of the OLDI Seed Corpus. In Proceedings of the Tenth Conference on Machine Translation (WMT 2025), pages 1048–1060, Suzhou, China. Association for Computational Linguistics. 11. Appendix: Readability and Linguistic Complexity M… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/marmonier-2026-hindsight-quality-prediction-experiments-multi/