Elamur Research⌕
☰
home / papers / 2018 / Zero Pronoun Resolution with Attention-based Neural Network

Zero Pronoun Resolution with Attention-based Neural Network

Summary

The paper addresses anaphoric zero pronoun resolution in Chinese, a pro-drop language. A zero pronoun is an omitted argument with no surface form, so it must be represented through its context. Earlier neural approaches either used only a few nearby words or treated every context word as equally relevant.

The authors propose a pair-wise neural model with two components:

  • Zero pronoun encoder: a self-attentive mechanism, inspired by Lin et al. (2017), applies several attention hops over the words before and after the gap.
  • Candidate encoder: an attention-based RNN encodes each candidate noun phrase. It uses the zero pronoun's vector to weight the words inside the mention.

The two vectors pass through tanh layers, which output a coreference score for each pronoun–candidate pair.

On the Chinese portion of OntoNotes 5.0, the model outperforms prior systems, including the strongest baseline (Liu et al., 2017), and is reported as state of the art. The work matters because it shows that selectively attending to context and to mention words helps represent elements absent from the text. This idea may carry over to other pro-drop languages.

Key points

  • Candidate selection: noun phrases within two sentences of the gap, limited to max or modifier NPs. This retains about 98% of the true antecedents.
  • Per-genre gains over Liu et al. (2017) in F-score:
    • NW: +5.1
    • BN: +4.6
    • BC: +2.1
    • WB: +1.5
    • MZ: +1.2
    • TC: +0.3
  • Why TC lags: the gain on telephone conversation (TC) is marginal, which the authors attribute to short texts and many filler words.
  • Number of attention hops (r): r = 2 gives the best results among the values tried (2–6).
  • Ablation: removing attention entirely costs 0.9 F-score.
  • Training dynamics: on the development set, F-score rises from about 46% after the first epoch to about 52% after roughly 30 iterations, then levels off.
  • Case study: attention heat maps show the model focusing on informative words such as "数字证书/digital certificate".

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

15 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 9 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{yin2018zero,
  title = {{Zero Pronoun Resolution with Attention-based Neural Network}},
  author = {Yin, Qingyu and Zhang, Yu and Zhang, Weinan and Liu, Ting and Wang, William Yang},
  booktitle = {Proceedings of the 27th International Conference on Computational Linguistics},
  year = {2018},
  url = {https://aclanthology.org/C18-1002/}
}

References

25 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chen Chen and Vincent Ng. 2013. Chinese zero pronoun resolution: Some recent advances. In EMNLP, pages 1360–1365. find our card
  2. Chen Chen and Vincent Ng. 2014. Chinese zero pronoun resolution: An unsupervised approach combining ranking and integer linear programming. In Twenty-Eighth AAAI Conference on Artificial Intelligence. find
  3. Chen Chen and Vincent Ng. 2015. Chinese zero pronoun resolution: A joint unsupervised discourse-aware model rivaling state-of-the-art resolvers. In Proceedings of the 53rd Annual Meeting of the ACL and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pa… find our card
  4. Chen Chen and Vincent Ng. 2016. Chinese zero pronoun resolution with deep neural networks. In Proceedings of the 54rd Annual Meeting of the ACL. find our card
  5. Susan P Converse. 2006. Pronominal anaphora resolution in chinese. find
  6. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121–2159. find
  7. Antonio Ferrández and Jesús Peral. 2000. A computational approach to zero-pronouns in spanish. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, pages 166–172. Association for Computational Linguistics. find
  8. Na-Rae Han. 2006. Korean zero pronouns: analysis and resolution. Ph.D. thesis, Citeseer. find
  9. Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580. find
  10. Ryu Iida and Massimo Poesio. 2011. A cross-lingual ilp solution to zero anaphora resolution. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 804–813. Association for Computational Linguistics. find our card
  11. Ryu Iida, Kentaro Inui, and Yuji Matsumoto. 2006. Exploiting syntactic patterns as clues in zero-anaphora resolution. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 625–632. Associat… find our card
  12. Ryu Iida, Kentaro Inui, and Yuji Matsumoto. 2007. Zero-anaphora resolution by learning rich syntactic pattern features. ACM Transactions on Asian Language Information Processing (TALIP), 6(4):1. find
  13. Ryu Iida, Kentaro Torisawa, Chikara Hashimoto, Jong-Hoon Oh, and Julien Kloetzer. 2015. Intra-sentential zero anaphora resolution using subject sharing recognition. Proceedings of EMNLP’15, pages 2179–2189. find our card
  14. Ryu Iida, Kentaro Torisawa, Jong-Hoon Oh, Canasai Kruengkrai, and Julien Kloetzer. 2016. Intra-sentential subject zero anaphora resolution using multi-column convolutional neural network. In Proceedings of EMNLP. find our card
  15. Hideki Isozaki and Tsutomu Hirao. 2003. Japanese zero pronoun resolution based on ranking rules and machine learning. In Proceedings of the 2003 conference on Empirical methods in natural language processing, pages 184–191. Association for Computational Linguistics. find our card
  16. Fang Kong and Guodong Zhou. 2010. A tree kernel-based unified framework for chinese zero anaphora resolution. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 882–891. Association for Computational Linguistics. find our card
  17. Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130. find our card
  18. Ting Liu, Yiming Cui, Qingyu Yin, Shijin Wang, Weinan Zhang, and Guoping Hu. 2017. Generating and exploiting large-scale pseudo training data for zero pronoun resolution. In ACL. find our card
  19. Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. 2014. Recurrent models of visual attention. In Advances in neural information processing systems, pages 2204–2212. find
  20. Ryohei Sasano and Sadao Kurohashi. 2011. A discriminative approach to japanese zero anaphora resolution with large-scale lexicalized case frames. In IJCNLP, pages 758–766. find our card
  21. Duyu Tang, Bing Qin, and Ting Liu. 2016. Aspect level sentiment classification with deep memory network. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 214–224. find
  22. Naiwen Xue, Fei Xia, Fu-Dong Chiou, and Marta Palmer. 2005. The penn chinese treebank: Phrase structure annotation of a large corpus. Natural language engineering, 11(2):207–238. find
  23. Qingyu Yin, Yu Zhang, Weinan Zhang, and Ting Liu. 2017a. Chinese zero pronoun resolution with deep memory network. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1309–1318. find our card
  24. Qingyu Yin, Yu Zhang, Weinan Zhang, and Ting Liu. 2017b. A deep neural network for chinese zero pronoun resolution. In IJCAI. find
  25. Shanheng Zhao and Hwee Tou Ng. 2007. Identification and resolution of chinese zero pronouns: A machine learning approach. In EMNLP-CoNLL, volume 2007, pages 541–550. find our card

↑ back to top