Elamur Research⌕
☰
home / papers / 2017 / Chinese Zero Pronoun Resolution with Deep Memory Network

Chinese Zero Pronoun Resolution with Deep Memory Network

Summary

Chinese text often omits arguments, leaving "zero pronouns" (ZPs). These gaps carry no gender, number or lexical content, so earlier resolvers relied on syntactic and lexical features and could not measure how semantically close a gap is to a candidate antecedent. Yin et al. address the resolution step for anaphoric ZPs (AZPs) with ZPMN, a zero pronoun-specific memory network.

First, a ZP-centered LSTM builds a vector for the gap from the words around it. Candidate antecedents are encoded as vectors and stored as an external memory. Several attention "hops" then refine the ZP vector by weighting the candidates. The attention distribution from the final hop is read as the probability of each candidate being the antecedent. The whole model is differentiable and trained end-to-end with pre-trained word embeddings.

On the Chinese part of OntoNotes 5.0, ZPMN surpasses the previous state-of-the-art systems across several experimental settings. This shows that semantic representations can be learned even for empty positions in a sentence.

Key points

  • Candidate antecedents follow Chen and Ng (2016): maximal or modifier NPs that come before the ZP and lie no more than two sentences away.
  • The ZP-centered LSTM uses two LSTMs. LSTM p reads the left context forward and LSTM f reads the right context backward, so the words nearest the gap matter most. Their final hidden states are concatenated to form the initial ZP vector.
  • Each hop consists of an attention layer over the candidate memory plus a linear layer, with parameters shared across hops. Adding hops therefore does not increase the parameter count.
  • In ablations, encoding candidates with sentence-level context, inside and outside the NP, beats simpler variants such as AntContent-Head.
  • An attention visualization with six hops shows the model moving its focus across hops from a wrong candidate ("Red Cross Society of China") to the correct one ("the earthquake… in India").

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

16 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 9 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1135

@inproceedings{yin2017chinese,
  title = {{Chinese Zero Pronoun Resolution with Deep Memory Network}},
  author = {Yin, Qingyu and Zhang, Yu and Zhang, Weinan and Liu, Ting},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1135},
  url = {https://doi.org/10.18653/v1/d17-1135}
}

References

41 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR). find our card
  2. Yoshua Bengio, Rejean Ducharme, and Pascal Vincent. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. find
  3. Chen Chen and Vincent Ng. 2013. Chinese zero pronoun resolution: Some recent advances. In EMNLP, pages 1360–1365. find our card
  4. Chen Chen and Vincent Ng. 2014a. Chinese overt pronoun resolution: A bilingual approach. In AAAI, pages 1615–1621. find
  5. Chen Chen and Vincent Ng. 2014b. Chinese zero pronoun resolution: An unsupervised approach combining ranking and integer linear programming. In Twenty-Eighth AAAI Conference on Artificial Intelligence. find
  6. Susan P Converse. 2006. Pronominal anaphora resolution in chinese. find
  7. Jesse Dodge, Andreea Gane, Xiang Zhang, Antoine Bordes, Sumit Chopra, Alexander Miller, Arthur Szlam, and Jason Weston. 2015. Evaluating prerequisite qualities for learning end-to-end dialog systems. arXiv preprint arXiv:1511.06931. find
  8. Jeffrey L Elman. 1991. Distributed representations, simple recurrent networks, and grammatical structure. Machine learning, 7(2-3):195–225. find
  9. Antonio Ferrández and Jesús Peral. 2000. A computational approach to zero-pronouns in spanish. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, pages 166–172. Association for Computational Linguistics. find
  10. Na-Rae Han. 2006. Korean zero pronouns: analysis and resolution. Ph.D. thesis, Citeseer. find
  11. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693– 1701. find our card
  12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  13. Ryu Iida, Kentaro Inui, and Yuji Matsumoto. 2006. Exploiting syntactic patterns as clues in zero-anaphora resolution. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 625–632. Associat… find our card
  14. Ryu Iida, Kentaro Inui, and Yuji Matsumoto. 2007. Zero-anaphora resolution by learning rich syntactic pattern features. ACM Transactions on Asian Language Information Processing (TALIP), 6(4):1. find
  15. Ryu Iida and Massimo Poesio. 2011. A cross-lingual ilp solution to zero anaphora resolution. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 804–813. Association for Computational Linguistics. find our card
  16. Ryu Iida, Kentaro Torisawa, Chikara Hashimoto, Jong-Hoon Oh, and Julien Kloetzer. 2015. Intra-sentential zero anaphora resolution using subject sharing recognition. Proceedings of EMNLP’15, pages 2179–2189. find our card
  17. Ryu Iida, Kentaro Torisawa, Jong-Hoon Oh, Cana-sai Kruengkrai, and Julien Kloetzer. 2016. Intra-sentential subject zero anaphora resolution using multi-column convolutional neural network. In Proceedings of EMNLP. find our card
  18. Kenji Imamura, Kuniko Saito, and Tomoko Izumi. 2009. Discriminative approach to predicate-argument structure analysis with zero-anaphora resolution. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 85–88. Association for Computational Linguistics. find our card
  19. Hideki Isozaki and Tsutomu Hirao. 2003. Japanese zero pronoun resolution based on ranking rules and machine learning. In Proceedings of the 2003 conference on Empirical methods in natural language processing, pages 184–191. Association for Computational Linguistics. find our card
  20. Fang Kong and Guodong Zhou. 2010. A tree kernel-based unified framework for chinese zero anaphora resolution. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 882–891. Association for Computational Linguistics. find our card
  21. Ankit Kumar, Ozan Irsoy, Jonathan Su, James Bradbury, Robert English, Brian Pierce, Peter On-druska, Ishaan Gulrajani, and Richard Socher. 2015. Ask me anything: Dynamic memory networks for natural language processing. arXiv preprint arXiv:1506.07285. find our card
  22. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature, 521(7553):436–444. find
  23. Ting Liu, Yiming Cui, Qingyu Yin, Shijin Wang, Weinan Zhang, and Guoping Hu. 2017. Effective deep memory networks for distant supervised relation extraction. In ACL. find
  24. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. find
  25. Vincent Ng. 2007. Semantic class induction and coreference resolution. In AcL, pages 536–543. find our card
  26. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532– 1543. find
  27. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint Conference on EMNLP and CoNLL-Shared Task, pages 1– 40. Association for Computational Linguistics. find our card
  28. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, and Phil Blunsom. 2015. Reasoning about entailment with neural attention. arXiv preprint arXiv:1509.06664. find our card
  29. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685. find
  30. Ryohei Sasano and Sadao Kurohashi. 2011. A discriminative approach to japanese zero anaphora resolution with large-scale lexicalized case frames. In IJCNLP, pages 758–766. find our card
  31. Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013. Reasoning with neural tensor networks for knowledge base completion. In Advances in neural information processing systems, pages 926–934. find
  32. Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Advances in neural information processing systems, pages 2440–2448. find
  33. Yaming Sun, Lei Lin, Duyu Tang, Nan Yang, Zhenzhou Ji, and Xiaolong Wang. 2015. Modeling mention, context and entity with neural networks for entity disambiguation. In IJCAI, pages 1333–1339. find
  34. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find
  35. Duyu Tang, Bing Qin, and Ting Liu. 2016. Aspect level sentiment classification with deep memory network. In EMNLP. find
  36. Shaolei Wang, Wanxiang Che, and Ting Liu. 2016. A neural attention model for disfluency detection. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 278–287, Osaka, Japan. find
  37. The COLING 2016 Organizing Committee. find
  38. Jason Weston, Sumit Chopra, and Antoine Bordes. 2014. Memory networks. arXiv preprint arXiv:1410.3916. find
  39. Feng Xiaocheng, Guo Jiang, Qin Bing, Liu Ting, and Liu Yongjie. 2017. Effective deep memory networks for distant supervised relation extraction. In IJCAI. find
  40. Ching-Long Yeh and Yi-Chun Chen. 2007. Zero anaphora resolution in chinese with shallow parsing. Journal of Chinese Language and Computing, 17(1):41–56. find
  41. Shanheng Zhao and Hwee Tou Ng. 2007. Identification and resolution of chinese zero pronouns: A machine learning approach. In EMNLP-CoNLL, volume 2007, pages 541–550. find our card

↑ back to top