Elamur Research
☰
home / papers / 2018 / Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation

Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D18-1512.pdf

Summary

Hassan et al. (2018) reported that a neural system matched professional human translators on WMT Chinese–English news. This paper asks whether that "human parity" result reflects how evaluation is done, specifically the practice of judging sentences in isolation. The authors used the released outputs, REFERENCE-HT as human and COMBO-6 as machine, and ran a 2×2 mixed factorial experiment. The two factors were the condition (adequacy with the source shown, or fluency with the target only) and the unit (single sentence or full document). Instead of crowd workers using direct assessment, professional translators recruited through ProZ made pairwise rankings, with ties allowed. Spam items were included as a quality check.

For adequacy at the sentence level, raters showed no significant preference, which agrees with the original claim. Once raters saw entire documents, however, they significantly favoured the human translation. Inconsistent terminology across an article was one of the errors that only became visible with context. The authors conclude that sentence-level protocols can hide real quality gaps. They argue that evaluation should move to the document level as MT output becomes harder to tell apart from human work.

Key points

  • Data: The authors drew their sample, consisting of 55 documents plus 2×120 individual sentences, from the zh→en test set of WMT 2017. Only the 123 articles originally written in Chinese were eligible, averaging about 8.13 sentences each.
  • Adequacy results:
    • At the sentence level there was no significant difference (x = 86, n = 189, p = .244).
    • At the document level human translation was significantly preferred (x = 104, n = 178, p < .05).
    • Preference for MT fell from 50% to 37%.
  • Fluency results: Human translation won at both levels (sentence p < .01, document p < .001). With document context, MT preference fell from 32% to 22%.
  • Raters: There were 4 adequacy and 4 fluency raters, averaging 13.7 years of experience. Inter-annotator agreement (Cohen's κ) ranged from 0.13 to 0.32. Significance was tested with two-sided Sign Tests.
  • Example and data release: In one article the MT gave three different renderings of the app name "微信挪车", while the human translation used one consistent name. All experimental data is released at github.com/laeubli/parity.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

160 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 152 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1512

@inproceedings{laubli2018has,
  title = {{Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation}},
  author = {Läubli, Samuel and Sennrich, Rico and Volk, Martin},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1512},
  url = {https://doi.org/10.18653/v1/d18-1512}
}

References

26 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of ICLR, San Diego, CA. find our card
  2. Ondrej Bojar, Christian Federmann, Barry Haddow, Philipp Koehn, Matt Post, and Lucia Specia. 2016a. Ten years of WMT evaluation campaigns: Lessons learnt. In Proceedings of the LREC 2016 Workshop “Translation Evaluation – From Fragmented Tools and Data Sets to an Integrated Ecosystem”, pages 27–34, Portorož, Slovenia. find
  3. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 Conference on Machine Translation (WMT17). In Proceedings of WMT, pages 169–214, Copenhagen, Denmark. find our card
  4. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016b. Findings of the 2016 Conference on Machine Translation (WMT16). In Proceedings of WMT, pages 131–198, Berlin, Germany. find our card
  5. Chris Callison-Burch. 2009. Fast, cheap, and creative: Evaluating translation quality using amazon’s mechanical turk. In Proceedings of EMNLP, pages 286–295, Singapore. find
  6. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (Meta-) evaluation of machine translation. In Proceedings of WMT, pages 136–158, Prague, Czech Republic. find
  7. Sheila Castilho, Joss Moorkens, Federico Gaspari, Iacer Calixto, John Tinsley, and Andy Way. 2017a. Is Neural Machine Translation the New State of the Art? The Prague Bulletin of Mathematical Linguistics, 108:109–120. find
  8. Sheila Castilho, Joss Moorkens, Federico Gaspari, Rico Sennrich, Vilelmini Sosoni, Yota Georgakopoulou, Pintu Lohar, Andy Way, Antonio Valerio Miceli Barone, and Maria Gialama. 2017b. A Comparative Quality Evaluation of PBSMT and NMT using Professional Translators. In Proceedings of MT Summit, Nagoya, Japan. find
  9. Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stüker, Katsuitho Sudoh, Koichiro Yoshino, and Christian Federmann. 2017. Overview of the IWSLT 2017 Evaluation Campaign. In Proceedings of IWSLT, pages 2–14, Tokyo, Japan. find our card
  10. Fabien Cromieres, Chenhui Chu, Toshiaki Nakazawa, and Sadao Kurohashi. 2016. Kyoto University participation to WAT 2016. In Proceedings of WAT, pages 166–174, Osaka, Japan. find
  11. John D. Emerson and Gary A. Simon. 1979. Another Look at the Sign Test When Ties Are Present: The Problem of Confidence Intervals. The American Statistician, 33(3):140–142. find
  12. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous Measurement Scales in Human Evaluation of Machine Translation. In Proceedings of the 7th Linguistic Annotation Workshop & Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. find
  13. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2017. Can machine translation systems be evaluated by the crowd alone? Natural Language Engineering, 23(1):3–30. find
  14. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. 2018. Achieving Human Parity on Automatic Chinese to English News Translation. Computing Research Repository, arXiv:1803.05567. ↗ our card
  15. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent Continuous Translation Models. In Proceedings of EMNLP, pages 1700–1709, Seattle, WA. find
  16. Rico Sennrich, Alexandra Birch, Anna Currey, Ulrich Germann, Barry Haddow, Kenneth Heafield, Antonio Valerio Miceli Barone, and Philip Williams. 2017. The University of Edinburgh’s Neural MT Systems for WMT17. In Proceedings of WMT, pages 389–399, Copenhagen, Denmark. find
  17. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Edinburgh Neural Machine Translation Systems for WMT 16. In Proceedings of WMT, pages 368–373, Berlin, Germany. find our card
  18. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In Proceedings of NIPS, pages 3104–3112, Montreal, Canada. find
  19. Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation. In Proceedings of WMT, Brussels, Belgium. find our card
  20. Georges Van Slype. 1979. Critical study of methods for evaluating the quality of machine translation. Research report BR 19142 prepared for the Commission of the European Communities, Bureau Marcel van Dijk, Brussels, Belgium. find
  21. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Proceedings of NIPS, pages 5998– 6008, Long Beach, CA. find
  22. Bonnie Webber, Andrei Popescu-Belis, and Jörg Tiedemann, editors. 2017. Proceedings of the Third Workshop on Discourse in Machine Translation. Copenhagen, Denmark. find
  23. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. Computing Research Repository, arXiv:1609.08144. ↗
  24. Aniket Kittur, Ed H. Chi, and Bongwon Suh. 2008. Crowdsourcing User Studies with Mechanical Turk. In Proceedings of CHI, pages 453–456, Florence, Italy. find
  25. Hans P. Krings. 1986. Was in den Köpfen von Übersetzern vorgeht. Gunter Narr, Tübingen, Germany. find
  26. Minh-Thang Luong and Christopher D. Manning. 2015. Stanford Neural Machine Translation Systems for Spoken Language Domains. In Proceedings of IWSLT, Da Nang, Vietnam. find