Elamur ResearchβŒ•
☰
home / papers / 2020 / ZPR2: Joint Zero Pronoun Recovery and Resolution using Multi-Task Learning and BERT

ZPR2: Joint Zero Pronoun Recovery and Resolution using Multi-Task Learning and BERT

Summary

Chinese often omits pronouns, especially in conversation, which hurts machine translation, QA and dialogue systems. The authors tackle two related problems together: ZP recovery (restoring the omitted pronoun's form) and ZP resolution (finding the mention it refers to). Earlier work handled them separately and usually assumed gold parse trees and gold ZP positions. This paper uses no gold syntax. It builds a heterogeneous multi-task model on a shared Chinese BERT-base encoder, where each training example carries labels for only one task. Recovery is framed as per-word-space classification over pronoun types. Resolution is framed as extractive reading comprehension, with span (0,0), the [CLS] position, marking spaces that have no ZP. A lightly weighted binary ZP detection task is added as shared supervision. This offsets the size gap between the two datasets (OntoNotes 5.0 for resolution, BaiduZhidao for recovery). The joint model outperforms single-task BERT baselines and improves on earlier state-of-the-art results by about 3.4–3.5 F1 on both tasks. This shows that sharing scarce annotations across closely linked tasks pays off.

Key points

  • Data: BaiduZhidao (version cleaned by Yang et al., 2019) has 5504/1175/1178 train/dev/test instances. OntoNotes 5.0 has 36487 training and 6083 test instances, with 20% of training held out for development.
  • Main results (F1):
    • Resolution: 26.07, compared with 24.45 for BERT and 22.7 for ZPMN.
    • Recovery: 46.81, compared with 45.32 for BERT and 43.36 for NDPR-W.
    • Average: 36.44, compared with 34.89 for BERT.
  • Role of detection: Plain multi-task learning (BERT-MTL) helped resolution but reduced recovery F1 to 44.68. Adding the auxiliary detection loss (weight 0.1) resolved this conflict.
  • Gold settings:
    • Gold Tree + Gold ZP: 58.49 F1, compared with 57.3 for AttentionZP.
    • Gold Tree + Auto ZP: 36.55 F1, compared with 34.8 for ZPMN.
  • Automatic trees: Using automatic trees to constrain candidate spans raised resolution F1 from 26.07 to 34.12. The authors caution that this gain relies on accurate parsing of broadcast news. Code is available at github.com/freesunshine0316/lab-zp-joint.

Our summary of the paper, in our own words β€” not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

9 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 5 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.482

@inproceedings{song2020zpr2,
  title = {{ZPR2: Joint Zero Pronoun Recovery and Resolution using Multi-Task Learning and BERT}},
  author = {Song, Linfeng and Xu, Kun and Zhang, Yue and Chen, Jianshu and Yu, Dong},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.482},
  url = {https://doi.org/10.18653/v1/2020.acl-main.482}
}

References

29 references, 16 of them with a card of ours. β†— leads to the work, Β«findΒ» searches for one we have no id for yet.

  1. Chen Chen and Vincent Ng. 2013. Chinese zero pronoun resolution: Some recent advances. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1360–1365. find our card
  2. Chen Chen and Vincent Ng. 2016. Chinese zero pronoun resolution with deep neural networks. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 778–788. find our card
  3. Henry Y Chen, Ethan Zhou, and Jinho D Choi. 2017. Robust coreference resolution and entity linking on dialogues: Character identification on tv show transcripts. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 216–225. find our card
  4. Yu-Hsin Chen and Jinho D Choi. 2016. Character identification on multiparty conversation: Identifying mentions of characters in tv shows. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 90– 100. find
  5. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. Quac: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184. find our card
  6. Tagyoung Chung and Daniel Gildea. 2010. Effects of empty categories on machine translation. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 636– 645. Association for Computational Linguistics. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  8. Ryu Iida and Massimo Poesio. 2011. A cross-lingual ilp solution to zero anaphora resolution. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 804–813. find our card
  9. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  10. Fang Kong and Guodong Zhou. 2010. A tree kernel-based unified framework for chinese zero anaphora resolution. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 882–891. find our card
  11. Ting Liu, Yiming Cui, Qingyu Yin, Wei-Nan Zhang, Shijin Wang, and Guoping Hu. 2017. Generating and exploiting large-scale pseudo training data for zero pronoun resolution. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 102–11… find our card
  12. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 4487–4496. find our card
  13. Hiromi Nakaiwa and Satoshi Shirai. 1996. Anaphora resolution of japanese zero pronouns with deictic reference. In Proceedings of the 16th conference on Computational linguistics-Volume 2, pages 812–817. Association for Computational Linguistics. find our card
  14. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392. find our card
  15. Sudha Rao, Allyson Ettinger, Hal Daumé III, and Philip Resnik. 2015. Dialogue focus tracking for zero pronoun resolution. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 494–503. find
  16. Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266. find our card
  17. Gabi Rolih. 2018. Applying coreference resolution for usage in dialog systems. find
  18. Ryohei Sasano, Daisuke Kawahara, and Sadao Kurohashi. 2008. A fully-lexicalized probabilistic model for japanese zero anaphora resolution. In Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1, pages 769–776. Association for Computational Linguistics. find our card
  19. Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. DREAM: A challenge data set and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics, 7:217–231. find our card
  20. Shunsuke Takeno, Masaaki Nagata, and Kazuhide Yamamoto. 2016. Integrating empty category detection into preordering machine translation. In Proceedings of the 3rd Workshop on Asian Translation (WAT2016), pages 157–165. learning for chinese zero pronoun resolution. In Proceedings of the 56th Annual M… find
  21. Qingyu Yin, Yu Zhang, Weinan Zhang, Ting Liu, and William Yang Wang. 2018b. Zero pronoun resolution with attention-based neural network. In Proceedings of the 27th International Conference on Computational Linguistics, pages 13–23. find our card
  22. Wei-Nan Zhang, Ting Liu, Qingyu Yin, and Yu Zhang. 2016. Neural recovery machine for chinese dropped pronoun. arXiv preprint arXiv:1605.02134. β€” [arXiv:1605.02134](https://arxiv.org/abs/1605.02134) find
  23. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. the Proceedings of ICLR. find our card
  24. Longyue Wang, Zhaopeng Tu, Xiaojun Zhang, Hang Li, Andy Way, and Qun Liu. 2016. A novel approach to dropped pronoun translation. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 983–993. find
  25. Bing Xiang, Xiaoqiang Luo, and Bowen Zhou. 2013. Enlisting the ghost: Modeling empty categories for machine translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 822– 831. find
  26. Jingxuan Yang, Jianzhuo Tong, Si Li, Sheng Gao, Jun Guo, and Nianwen Xue. 2019. Recovering dropped pronouns in chinese conversations via modeling their referents. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech… find
  27. Yaqin Yang, Yalin Liu, and Nianwen Xue. 2015. Recovering dropped pronouns from chinese text messages. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 3… find
  28. Qingyu Yin, Yu Zhang, Weinan Zhang, and Ting Liu. 2017. Chinese zero pronoun resolution with deep memory network. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1309–1318. find our card
  29. Qingyu Yin, Yu Zhang, Weinan Zhang, Ting Liu, and William Yang Wang. 2018a. Deep reinforcement find

↑ back to top