Elamur Research
☰
home / papers / 2018 / Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation

Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W18-6312.pdf

Summary

This paper re-examines Microsoft's claim (Hassan et al., 2018) that its neural MT system matched human quality on Chinese-to-English news translation. The authors argue that the original evaluation had three blind spots:

  • Half the newstest2017 test set was Chinese translationese produced from English originals.
  • The judges were crowd workers with unclear translation expertise.
  • Sentences were rated in shuffled order, without document context.

The authors ran a new evaluation using Microsoft's released data. Professional translators and non-expert bilingual NLP researchers ranked three outputs for each source sentence: Microsoft's human reference (HT), Microsoft's MT (MS), and Google Translate (GG). Sentences were shown in document order with neighbouring context.

When only Chinese-original input is considered, human translation is significantly better than MT, in both the new evaluation and Microsoft's own data. So parity was not reached. Expert judges agreed with each other more and separated human from machine output more clearly than non-experts. A manual check of the reference translations also revealed errors suggesting non-professional or post-edited work. The study matters because it questions how parity claims are tested and proposes stricter standards for human evaluation.

Key points

  • The WMT 2017 ZH–EN test set (newstest2017) has 2,001 sentence pairs. Half were originally written in Chinese and half were translated into Chinese from English. MT systems score higher when the input is translationese.
  • The new evaluation covered 49 randomly ordered documents (503 sentences): 41 Chinese-original (299 sentences) and 8 English-original (204 sentences). It yielded 6,675 pairwise judgements, collected with the Appraise tool.
  • The authors used relative ranking instead of direct assessment (DA). This let judges see all three translations together with the previous and next sentences. The authors cite earlier work showing the two methods correlate strongly (R ≥ 0.92).
  • Professional translators showed higher inter-annotator agreement than non-experts and were less forgiving of MT errors.
  • Recommendations for future evaluations:
    • Use source texts originally written in the source language.
    • Use professional translators as judges.
    • Evaluate at the document level.
    • Have references translated from scratch by experienced professionals.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

126 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 115 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-6312

@inproceedings{toral2018attaining,
  title = {{Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation}},
  author = {Toral, Antonio and Castilho, Sheila and Hu, Ke and Way, Andy},
  booktitle = {Proceedings of the Third Conference on Machine Translation: Research Papers},
  year = {2018},
  doi = {10.18653/v1/w18-6312},
  url = {https://doi.org/10.18653/v1/w18-6312}
}

References

18 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Should material be made available by Google, SDL or any other MT developers who claim ‘human parity’ or the like, we would be very happy to apply these principles in subsequent rigorous evaluations of actual demonstrable improvements in translation quality. One thing is certain; as Way (2018) observes, “those of us who have seen many paradigms come and go know that overgilding the lily does none of us any good, especially those of us who have been trying to build bridges between MT developers and the translation community for many years.” We trust that our findings in this paper demonstrate that while MT quality does seem to be improving quite dramatically, human translators will continue to find gainful employment for many years to come, despite somewhat grandiose claims to the contrary. find
  2. Luisa Bentivogli, Arianna Bisazza, Mauro Cettolo, and Marcello Federico. 2016. Neural versus phrase-based machine translation quality: a case study. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 257–267, Austin, Texas. find
  3. Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013. Findings of the 2013 Workshop on Statistical Machine Translation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 1–44, Sofia, Bulgaria. find our card
  4. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine translation (wmt17). In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 169–214, Copenhagen, Denmark. find our card
  5. On a final note, we acknowledge that our conclusions and recommendations are somewhat limited in that they are derived from experiments on just one language direction and five evaluators. Therefore we plan as future work to conduct similar experiments on additional language pairs with a higher number of evaluators. In the spirit of Hassan et al. (2018), without which this paper would not have been possible, we too make publicly available our evaluation materials, the anonymised human judgments and the statistical analyses thereof. 13 find
  6. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation, pages 131– 198, Berlin, Germany. find our card
  7. We would like to thank the five expert and non-expert translators that took part in this study. We are also grateful for valuable comments on this paper from Hany Hassan, the lead author on the Microsoft paper. This research was partially supported by the iADAATPA project funded by CEF research action (2016-EU-IA-0132) under grant agreement No.1331703. The ADAPT Centre for Digital Content Technology at Dublin City University is funded under the Science Foundation Ireland Research Centres Programme (Grant 13/RC/ 2106) and is co-funded under the European Regional Development Fund. find
  8. Sheila Castilho, Joss Moorkens, Federico Gaspari, Iacer Calixto, John Tinsley, and Andy Way. 2017a. Is neural machine translation the new state of the art? The Prague Bulletin of Mathematical Linguistics, 108(1):109–120. find
  9. Sheila Castilho, Joss Moorkens, Federico Gaspari, Rico Sennrich, Vilelmini Sosoni, Panayota Georgakopoulou, Pintu Lohar, Andy Way, Antonio Valerio Miceli Barone, and Maria Gialama. 2017b. A Comparative Quality Evaluation of PBSMT and NMT using Professional Translators. In MT Summit 2017, pages 116–131, Nagoya, Japan. available, but the point remains that if these are poor quality human translations, then this is likely to skew results still further. 13 https://github.com/antot/human_ parity_mt Sheila Castilho, Joss Moorkens, Federico Gaspari, Andy Way, Panayota Georgakopoulou, Maria Gialama, Vilelmini Sosoni, and Rico Sennrich. 2017c. Crowdsourcing for NMT evaluation: Professional Workshop on Computational Linguistics for Literature, pages 18–25, Montrèal, Canada. translators versus the crowd. In Translating and the Computer 39, London. https://www.asling. org/tc39/?page_id=3223. ↗
  10. Longyue Wang, Zhaopeng Tu, Andy Way, and Liu Qun. 2017a. Exploiting cross-sentence context for neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2826–2831, Copenhagen, Denmark. find
  11. Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1):37–46. ↗
  12. Christian Federmann. 2012. Appraise: An open-source toolkit for manual evaluation of machine translation output. The Prague Bulletin of Mathematical Linguistics, 98:25–35. find
  13. Yuguang Wang, Shanbo Cheng, Liyang Jiang, Jiajun Yang, Wei Chen, Muze Li, Lin Shi, Yanfeng Wang, and Hongtao Yang. 2017b. Sogou neural machine translation systems for WMT17. In Proceedings of the Second Conference on Machine Translation, WMT 2017, page 410–415, Copenhagen, Denmark. find
  14. Yvette Graham, Timothy Baldwin, and Nitika Mathur. 2015. Accurate evaluation of segment-level machine translation metrics. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1183–1191, Denver, Colorado. find
  15. Andy Way. 2018. Machine translation: Where are we at today? In Angelone E, Massey G, and Ehrensberger-Dow M, editors, The Bloomsbury Companion to Language Industry Studies. Bloomsbury, London. In press. find
  16. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, Will Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. 2018. Achieving Human Parity on Automatic Chinese to English News Translation. https:// arxiv.org/abs/1803.05567. ↗ our card
  17. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. https://arxiv.org/abs/1609.08144. ↗
  18. Samuel Läubli, Rico Sennrich, and Martin Volk. 2018. Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. find our card