Elamur Research
☰
home / papers / 2018 / Achieving Human Parity on Automatic Chinese to English News Translation

Achieving Human Parity on Automatic Chinese to English News Translation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1803.05567

Summary

The paper asks whether machine translation can match professional human translators, and first makes that question testable. A system reaches human parity when bilingual judges score its output and the human translations of the same test set, and the difference between those scores is not statistically significant. Reference-based metrics such as BLEU penalize valid alternative wordings, and references can themselves be poor because they are often crowd-sourced or MT-assisted. For these reasons, the authors use source-based direct assessment with bilingual annotators. They then describe Microsoft's Chinese-to-English NMT system, which adds several techniques to attentional encoder-decoder models:

  • dual learning that exploits monolingual data;
  • joint training of forward and backward models;
  • Deliberation networks for two-pass decoding;
  • agreement regularization between left-to-right and right-to-left decoders;
  • data filtering with a bilingual sentence encoder;
  • combination of heterogeneous systems.

On the WMT 2017 news task (newstest2017, zh→en), the best combined systems were statistically indistinguishable from professional translations and outperformed crowd-sourced references. The work matters as a milestone claim and as a reproducible evaluation protocol, since all judgments and new references were released.

Key points

  • Evaluation: annotators compare the source with a candidate on a 0–100 slider. Scores are converted to z-scores, and systems are clustered by significance using a rank test at p ≤ 0.05. Quality control relies on deliberately degraded "bad references", and the setup is double-blind via the Appraise tool.
  • Compared outputs include the research systems Combo-4, Combo-5 and Combo-6, plus Online-A-1710 and Sogou. These are judged against three references: Reference-WMT (original), Reference-HT (translated from scratch) and Reference-PE (post-edited).
  • BLEU scores varied substantially depending on which reference was used. The authors take this as evidence that automatic metrics become unreliable near human quality.
  • An error analysis of 500 sampled Combo-6 sentences found the following shares of sentences with each error type:
    • Incorrect Words: 7.64%
    • Ungrammatical: 6.33%
    • Missing Words: 5.46%
    • Named Entity: 4.38%
  • The release (aka.ms/Translator-HumanParityData) contains the new references, the Combo outputs, the Online-A-1710 outputs and all anonymized human judgments. The authors do not claim the result generalizes to other language pairs or domains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Topics: Machine translation

Cited by

273 papers of our library cite it (of the 140673 whose bibliographies we read); 23 of them have a page here, 250 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1803.05567 · arXiv 1803.05567

@misc{hassan2018achieving,
  title = {{Achieving Human Parity on Automatic Chinese to English News Translation}},
  author = {Hassan, Hany and Aue, Anthony and Chen, Chang and Chowdhary, Vishal and Clark, Jonathan and Federmann, Christian and Huang, Xuedong and Junczys-Dowmunt, Marcin and Lewis, William and Li, Mu and Liu, Shujie and Liu, Tie-Yan and Luo, Renqian and Menezes, Arul and Qin, Tao and Seide, Frank and Tan, Xu and Tian, Fei and Wu, Lijun and Wu, Shuangzhi and Xia, Yingce and Zhang, Dongdong and Zhang, Zhirui and Zhou, Ming},
  year = {2018},
  doi = {10.48550/arxiv.1803.05567},
  eprint = {1803.05567},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1803.05567}
}

References

52 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Artetxe, M., Labaka, G., Agirre, E., and Cho, K. Unsupervised neural machine translation. In International Conference on Learning Representations (2018). find our card
  2. Axelrod, A., He, X., and Gao, J. Domain adaptation via pseudo in-domain data selection. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (Stroudsburg, PA, USA, 2011), EMNLP ’11, Association for Computational Linguistics, pp. 355–362. find our card
  3. Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014). ↗ our card
  4. Belinkov, Y., and Bisk, Y. Synthetic and natural noise both break neural machine translation. CoRR abs/1711.02173 (2017). ↗
  5. Bengio, S., Vinyals, O., Jaitly, N., and Shazeer, N. Scheduled sampling for sequence prediction with recurrent neural networks. In NIPS (2015), pp. 1171–1179. find
  6. Bojar, O., Chatterjee, R., Federmann, C., Graham, Y., Haddow, B., Huang, S., Huck, M., Koehn, P., Liu, Q., Logacheva, V., Monz, C., Negri, M., Post, M., Rubino, R., Specia, L., and Turchi, M. Findings of the 2017 conference on machine translation (wmt17). In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers (Copenhagen, Denmark, September 2017), Association for Computational Linguistics, pp. 169–214. find our card
  7. Cettolo, M., Federico, M., Bentivogli, L., Niehues, J., Stüker, S., Sudoh, K., Yoshino, K., and Federmann, C. Overview of the iwslt 2017 evaluation campaign. In Proceedings of the 14th International Workshop on Spoken Language Translation (IWSLT) (Tokyo, Japan, December 2017), IWSLT, pp. 2–12. find our card
  8. Cherry, C., and Foster, G. Batch tuning strategies for statistical machine translation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Stroudsburg, PA, USA, 2012), NAACL HLT ’12, Association for Computational Linguistics, pp. 427–436. find
  9. Clark, J. H., Dyer, C., Lavie, A., and Smith, N. A. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2 (2011), Association for Computational Linguistics, pp. 176–181. find
  10. Denkowski, M., and Lavie, A. Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems. In Proceedings of the EMNLP 2011 Workshop on Statistical Machine Translation (2011). find
  11. Devlin, J. Sharp models on dull hardware: Fast and accurate neural machine translation decoding on the cpu. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Copenhagen, Denmark, September 2017), Association for Computational Linguistics, pp. 2810–2815. find
  12. Dreyer, M., and Marcu, D. HyTER: Meaning-Equivalent Semantics for Translation Evaluation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2012), Association for Computational Linguistics, pp. 162–171. find
  13. Edgington, E. S. Validity of Randomization Tests for One-subject Experiments. Journal of Educational Statistics 5, 3 (1980), 235–251. find
  14. Federmann, C. Appraise: An open-source toolkit for manual evaluation of machine translation output. The Prague Bulletin of Mathematical Linguistics 98 (September 2012), 25–35. find
  15. Feng, S., Liu, S., Yang, N., Li, M., Zhou, M., and Zhu, K. Q. Improving attention modeling with implicit distortion and fertility for machine translation. In COLING 2016, 26th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, December 11-16, 2016, Osaka, Japan (2016), pp. 3082–3092. find
  16. Gehring, J., Auli, M., Grangier, D., Yarats, D., and Dauphin, Y. N. Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122 (2017). ↗ our card
  17. Graham, Y., Baldwin, T., Moffat, A., and Zobel, J. Can machine translation systems be evaluated by the crowd alone? Natural Language Engineering, 23(1), 3-30. doi:10.1017/S1351324915000339 (2016). ↗
  18. He, D., Xia, Y., Qin, T., Wang, L., Yu, N., Liu, T., and Ma, W.-Y. Dual learning for machine translation. In Advances in Neural Information Processing Systems (2016), pp. 820– 828. find
  19. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385 (2015). ↗
  20. Johnson, M., Schuster, M., Le, Q. V., Krikun, M., Wu, Y., Chen, Z., Thorat, N., Viégas, F., Wattenberg, M., Corrado, G., et al. Google’s multilingual neural machine translation system: enabling zero-shot translation. arXiv preprint arXiv:1611.04558 (2016). ↗
  21. Kingma, D. P., and Ba, J. abs/1412.6980 (2014). Adam: A method for stochastic optimization. ↗
  22. Lample, G., Conneau, A., Denoyer, L., and Ranzato, M. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representations (2018). find our card
  23. Lin, J., Xia, Y., Qin, T., Chen, Z., and Liu, T.-Y. Conditional image-to-image translation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (July 2018). find
  24. Luo, P., Wang, G., Lin, L., and Wang, X. Deep dual learning for semantic image segmentation. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017 (2017), pp. 2737–2745. find
  25. Mann, H. B., and Whitney, D. R. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics (1947), 50–60. find
  26. Moore, R. C., and Lewis, W. Intelligent selection of language model training data. In Proceedings of the ACL 2010 Conference Short Papers (Stroudsburg, PA, USA, 2010), ACLShort ’10, Association for Computational Linguistics, pp. 220–224. find our card
  27. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics (2002), Association for Computational Linguistics, pp. 311–318. find our card
  28. Sennrich, R., Haddow, B., and Birch, A. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709 (2015). ↗
  29. Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909 (2015). ↗
  30. Shen, W., and Liu, R. Learning residual images for face attribute manipulation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), IEEE, pp. 1225–1233. find
  31. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Driessche, G. V. D., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., and et al. Mastering the game of go with deep neural networks and tree search. Nature, vol. 529, pp. 484489 (2016). CoRR find
  32. Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. A study of translation edit rate with targeted human annotation. In Proceedings of association for machine translation in the Americas (2006), vol. 200. find our card
  33. Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. NIPS (2014). find
  34. Tang, D., Duan, N., Qin, T., and Zhou, M. Question answering and question generation as dual tasks. arXiv preprint arXiv:1706.02027 (2017). ↗
  35. Tu, Z., Lu, Z., Liu, Y., Liu, X., and Li, H. Modeling coverage for neural machine translation. arXiv preprint arXiv:1601.04811 (2016). ↗ our card
  36. van der Wees, M., Bisazza, A., and Monz, C. Dynamic data selection for neural machine translation. CoRR abs/1708.00712 (2017). ↗
  37. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. arXiv preprint arXiv:1706.03762 (2017). ↗
  38. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NIPS (2017). find
  39. Vilar, D., Xu, J., D’Haro, L. F., and Ney, H. Error analysis of statistical machine translation output. In Proceedings of the Fifth International Conference on Language Resources and Evaluation, LREC 2006, Genoa, Italy, May 22-28, 2006. (2006), pp. 697–702. find our card
  40. Wang, Y., Cheng, S., Jiang, L., Yang, J., Chen, W., Li, M., Shi, L., Wang, Y., and Yang, H. Sogou neural machine translation systems for WMT17. In Proceedings of the Second Conference on Machine Translation, WMT 2017, Copenhagen, Denmark, September 7-8, 2017 (2017), pp. 410–415. find
  41. Wang, Y., Li, X., Cheng, S., Jiang, L., Yang, J., Chen, W., Shi, L., Wang, Y., and Yang, H. Sogou neural machine translation systems for wmt17. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers (Copenhagen, Denmark, September 2017), Association for Computational Linguistics, pp. 410–415. find
  42. Wang, Y., Xia, Y., Zhao, L., Bian, J., Qin, T., Liu, G., and Liu, T. Dual transfer learning for neural machine translation with marginal distribution regularization. In AAAI (2018). find
  43. Wilcoxon, F. Individual Comparisons by Ranking Methods. Biometrics Bulletin 1, 6 (1945), 80–83. find
  44. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, Ł., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith, J., Riesa, J., Rudnick, A., Vinyals, O., Corrado, G., Hughes, M., and Dean, J. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. ArXiv e-prints (Sept. 2016). find
  45. Xia, Y., Bian, J., Qin, T., Yu, N., and Liu, T.-Y. Dual inference for machine learning. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17 (2017), pp. 3112–3118. find
  46. Xia, Y., Qin, T., Chen, W., Bian, J., Yu, N., and Liu, T. Dual supervised learning. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 (2017), pp. 3789–3798. find
  47. Xia, Y., Tian, F., Wu, L., Lin, J., Qin, T., Yu, N., and Liu, T.-Y. Deliberation networks: Sequence generation beyond one-pass decoding. In Advances in Neural Information Processing Systems (2017), pp. 1782–1792. find
  48. Xiong, W., Droppo, J., Huang, X., Seide, F., Seltzer, M. L., Stolcke, A., Yu, D., and Zweig, G. Toward human parity in conversational speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing 25, 12 (2017), 2410–2423. find
  49. Yi, Z., Zhang, H., Tan, P., and Gong, M. Dualgan: Unsupervised dual learning for image-to-image translation. ICCV (2017). find
  50. Zhang, Z., Liu, S., Li, M., Zhou, M., and Chen, E. Joint training for neural machine translation models with monolingual data, 2018. find
  51. Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint arXiv:1703.10593 (2017). ↗
  52. Zoph, B., Yuret, D., May, J., and Knight, K. Transfer learning for low-resource neural machine translation. arXiv preprint arXiv:1604.02201 (2016). ↗