Elamur Research⌕
☰
home / papers / 2022 / GuoFeng: A Benchmark for Zero Pronoun Recovery and Translation

GuoFeng: A Benchmark for Zero Pronoun Recovery and Translation

Summary

Pro-drop languages such as Chinese often leave pronouns unstated, and machine translation into English then produces output that is missing a subject or refers to the wrong entity. Earlier work judged such systems mainly by BLEU, which barely reflects these errors. In addition, recovery and translation of dropped pronouns had been studied on separate corpora, one of them only automatically labeled. The authors address this with mZPRT, a human-annotated Chinese-English test set. It marks zero pronouns on the source side and pairs them with reference translations across five genres. They score systems with BLEU and with AZPT, a pronoun-focused accuracy measure adapted from APT, and compare eight models. Larger Transformers raise BLEU but leave pronoun accuracy almost unchanged. Dedicated ZPT and document-level methods help somewhat, but feeding gold pronoun labels yields far larger gains. The benchmark therefore shows that BLEU hides real progress and that much room for improvement remains.

Key points

  • Dataset scale and genres: mZPRT contains 8,093 sentences (about 146K Chinese and 190K English words) from five domains: Movie Subtitle (based on TVsub), Q&A Forum (based on BaiduKnows), Government News, Web Fiction and Personal Profile.
  • Annotation quality and cost: average pairwise kappa is 0.91. Five annotators and two translators worked for two months at a cost of roughly US $10,000.
  • Pronoun types and frequency: subjects make up about 80% of dropped pronouns, and around 44% of sentences contain at least one. These ZP sentences also form a diagnostic subset.
  • Scaling vs. ZPT methods:
    • BIG and DEEP models gain +1.2 and +1.9 BLEU over BASE, yet AZPT barely moves.
    • Existing ZPT methods add +2.2 to +6.8 AZPT over BIG.
    • Oracle ZP input adds about +47.5 AZPT and +1.9 BLEU.
  • Document-level models and release: on subtitles, the CACHE model improves AZPT by 3.8 points (47.4 → 51.2) with no BLEU gain. Data, code, models and guidelines are released at github.com/longyuewangdcu/mZPRT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

13 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 11 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.emnlp-main.774

@inproceedings{xu2022guofeng,
  title = {{GuoFeng: A Benchmark for Zero Pronoun Recovery and Translation}},
  author = {Xu, Mingzhou and Wang, Longyue and Wong, Derek F. and Liu, Hongye and Song, Linfeng and Chao, Lidia S. and Shi, Shuming and Tu, Zhaopeng},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.774},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.774}
}

References

47 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. ↗ our card
  2. Elizabeth Baran, Yaqin Yang, and Nianwen Xue. 2012. Annotating dropped pronouns in chinese newswire text. In LREC 2012. find
  3. Tagyoung Chung and Daniel Gildea. 2010. Effects of empty categories on machine translation. In EMNLP. find
  4. Michael Collins, Philipp Koehn, and Ivona Kučerová. 2005. Clause restructuring for statistical machine translation. In ACL. find our card
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL. find
  6. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In ICML. ↗ our card
  7. Liane Guillou and Christian Hardmeier. 2018. Automatic reference-based evaluation of pronoun translation misses the point. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4797–4802. find
  8. Christian Hardmeier and Marcello Federico. 2010. Modelling pronominal anaphora in statistical machine translation. In IWSLT. find our card
  9. Bei Li, Hui Liu, Ziyang Wang, Yufan Jiang, Tong Xiao, Jingbo Zhu, Tongran Liu, and Changliang Li. 2020. Does multi-encoder help? a case study on context-aware neural machine translation. In ACL. find
  10. Charles N Li and Sandra A Thompson. 1979. Third-person pronouns and zero-anaphora in chinese discourse. In Discourse and syntax. Brill. ↗
  11. Shuming Ma, Dongdong Zhang, and Ming Zhou. 2020. A simple and effective unified encoder for document-level machine translation. In ACL. find
  12. Aya A Mitani, Phoebe E Freer, and Kerrie P Nelson. 2017. Summary measures of agreement and association between many raters’ ordinal classifications. Annals of epidemiology. ↗
  13. Ruslan Mitkov. 2014. ledge. Anaphora resolution. find
  14. Rout-Mathias Müller, Annette Rios, Elena Voita, and Rico Sennrich. 2018. A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation. In WMT. find our card
  15. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. In CL. ↗
  16. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In ACL. find our card
  17. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In ACL. find our card
  18. Jesús Peral and Antonio Ferrández. 2003. Translation of pronominal anaphora between english and spanish: Discrepancies and evaluation. In JAIR. ↗
  19. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In CoNLL-WS. find our card
  20. Sudha Rao, Allyson Ettinger, Hal Daumé III, and Philip Resnik. 2015. Dialogue focus tracking for zero pronoun resolution. In NAACL. find
  21. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In EMNLP. find our card
  22. Ryokan Ri, Toshiaki Nakazawa, and Yoshimasa Tsuruoka. 2021. Zero-pronoun data augmentation for japanese-to-english translation. In WAT. find
  23. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In ACL. find
  24. Linfeng Song, Kun Xu, Yue Zhang, Jianshu Chen, and Dong Yu. 2020. ZPR2: Joint zero pronoun recovery and resolution using multi-task learning and BERT. In ACL. find our card
  25. Hui Su, Xiaoyu Shen, Rongzhi Zhang, Fei Sun, Pengwei Hu, Cheng Niu, and Jie Zhou. 2019. Improving Multi-turn Dialogue Modelling with Utterance ReWriter. In ACL. find
  26. Xin Tan, Longyin Zhang, and Guodong Zhou. 2021. Coupling context modeling with zero pronoun recovering for document-level natural language generation. In EMNLP. find
  27. Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. 2017. Neural machine translation with reconstruction. In AAAI. find
  28. Zhaopeng Tu, Yang Liu, Shuming Shi, and Tong Zhang. 2018. Learning to remember translation history with a continuous cache. In TACL. ↗
  29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017a. Attention is all you need. In NeurIPS. find
  30. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017b. Attention is all you need. In NIPS. find
  31. Elena Voita, Rico Sennrich, and Ivan Titov. 2019. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In ACL. find our card
  32. Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018. Improving the transformer translation model with document-level context. In EMNLP. find our card
  33. Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-aware neural machine translation learns anaphora resolution. In ACL. find our card
  34. Weinan Zhang, Ting Liu, Qingyu Yin, and Yu Zhang. 2019. Neural recovery machine for Chinese dropped pronoun. In Frontiers of Computer Science. ↗
  35. Yu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang, Boxing Chen, Derek Wong, and Lidia Chao. 2022. UniTE: Unified translation evaluation. In ACL. find our card
  36. Longyue Wang, Zhaopeng Tu, Shuming Shi, Tong Zhang, Yvette Graham, and Qun Liu. 2018a. Translating pro-drop languages with reconstruction models. In AAAI. find
  37. Longyue Wang, Zhaopeng Tu, Xing Wang, and Shuming Shi. 2019. One model to learn both: Zero pronoun prediction and translation. In EMNLP-IJCNLP. ↗ our card
  38. Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. 2017a. Exploiting cross-sentence context for neural machine translation. In EMNLP. find
  39. Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. 2018b. Learning to jointly translate and predict dropped pronouns with a shared reconstruction mechanism. In EMNLP. find
  40. Longyue Wang, Zhaopeng Tu, Xiaojun Zhang, Hang Li, Andy Way, and Qun Liu. 2016. A novel approach to dropped pronoun translation. In NAACL. ↗
  41. Longyue Wang, Zhaopeng Tu, Xiaojun Zhang, Siyou Liu, Hang Li, Andy Way, and Qun Liu. 2017b. A novel and robust approach for pro-drop language translation. Machine Translation, 31(1):65–87. ↗
  42. Lesly Miculicich Werlen and Andrei Popescu-Belis. 2017. Validation of an automatic metric for the accuracy of pronoun translation (apt). In DiscoMT. find our card
  43. Mingzhou Xu, Liangyou Li, Derek F. Wong, Qun Liu, and Lidia S. Chao. 2021. Document graph for neural machine translation. In EMNLP. ↗
  44. Jingxuan Yang, Jianzhuo Tong, Si Li, Sheng Gao, Jun Guo, and Nianwen Xue. 2019. Recovering dropped pronouns in Chinese conversations via modeling their referents. In NAACL. ↗
  45. Yaqin Yang, Yalin Liu, and Nianwen Xue. 2015. Recovering dropped pronouns from chinese text messages. In ACL-IJCNLP. find
  46. Yaqin Yang and Nianwen Xue. 2010. Chasing the ghost: recovering empty categories in the chinese treebank. In COLING. find
  47. Lei Yu, Laurent Sartran, Wojciech Stokowiec, Wang Ling, Lingpeng Kong, Phil Blunsom, and Chris Dyer. 2020. Better document-level machine translation with bayes’ rule. In TACL. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/xu-2022-guofeng-benchmark-zero-pronoun-recovery/