Elamur Research⌕
☰
home / papers / 2017 / Generating and Exploiting Large-scale Pseudo Training Data for Zero Pronoun Resolution

Generating and Exploiting Large-scale Pseudo Training Data for Zero Pronoun Resolution

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P17-1010.pdf

Summary

Zero pronoun (ZP) resolution in Chinese means finding the antecedent of an omitted pronoun. Progress has been held back by reliance on small, hand-annotated corpora from shared tasks, which are expensive to extend or adapt to new domains. The authors treat the task as a form of cloze-style reading comprehension. From large unlabeled news collections they automatically build pseudo examples. A noun or pronoun that appears at least twice in a document is masked in one sentence, which becomes the query, and the model must recover the hidden word from the document. An attention-based network similar to the Attentive Reader learns to fill these gaps. Training happens in two stages: pre-training on the pseudo data, then adaptation on real OntoNotes 5.0 ZP data to bridge the difference in query style. Each single-word prediction is linked to a noun-phrase antecedent through head-word matching. The system improves on the previous state of the art by 3.1 F-score points (absolute). This suggests cheaply generated data can reduce the need for costly annotation.

Key points

  • Pseudo samples are triples ⟨D, Q, A⟩ (document, query with a blank, answer). Candidate answers are nouns or pronouns, identified with the LTP Toolkit, that occur two or more times in the document. The generation recipe is adapted from Hermann et al. (2015) but needs no document summaries.
  • The model uses a shared embedding matrix, separate bidirectional GRU encoders, an averaged query representation, and soft attention over document tokens. A softmax over the vocabulary predicts the answer word.
  • The two-step "pre-training-then-adaptation" scheme first selects the best model on pseudo data by validation accuracy. It then continues training on task-specific OntoNotes 5.0 data, and this outperforms using either data source alone.
  • Unknown words are mapped to numbered tokens (unk1…unkN) by order of appearance within each sample, so they act as positional cues rather than a single shared UNK.
  • NP candidates are extracted following Chen and Ng (2015) and scanned from nearest to farthest. The first NP whose head word matches the predicted word is chosen as the antecedent. Evaluation on OntoNotes 5.0 shows an absolute F-score gain of 3.1%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

21 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 14 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1010 · arXiv 1606.01603

@inproceedings{liu2017generating,
  title = {{Generating and Exploiting Large-scale Pseudo Training Data for Zero Pronoun Resolution}},
  author = {Liu, Ting and Cui, Yiming and Yin, Qingyu and Zhang, Wei-Nan and Wang, Shijin and Hu, Guoping},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1010},
  eprint = {1606.01603},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p17-1010}
}

References

25 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 . find our card
  2. Jerry R Hobbs. 1978. Resolving pronoun references. Lingua 44(4):311–338. find our card
  3. Ryu Iida, Kentaro Inui, and Yuji Matsumoto. 2006. Exploiting syntactic patterns as clues in zero-anaphora resolution. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics. Association for Computa… find our card
  4. Wanxiang Che, Zhenghua Li, and Ting Liu. 2010. Ltp: A chinese language technology platform. In Proceedings of the 23rd International Conference on Computational Linguistics: Demonstrations. Association for Computational Linguistics, pages 13–16. find
  5. Chen Chen and Vincent Ng. 2013. Chinese zero pronoun resolution: Some recent advances. In EMNLP. pages 1360–1365. find our card
  6. Ryu Iida, Kentaro Inui, and Yuji Matsumoto. 2007. Zero-anaphora resolution by learning rich syntactic pattern features. ACM Transactions on Asian Language Information Processing (TALIP) 6(4):1. find
  7. Chen Chen and Vincent Ng. 2014. Chinese zero pronoun resolution: An unsupervised approach combining ranking and integer linear programming. In Twenty-Eighth AAAI Conference on Artificial Intelligence. find
  8. Ryu Iida and Massimo Poesio. 2011. A cross-lingual ilp solution to zero anaphora resolution. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. Association for Computational Linguistics, pages 804–813. find our card
  9. Chen Chen and Vincent Ng. 2015. Chinese zero pronoun resolution: A joint unsupervised discourse-aware model rivaling state-of-the-art resolvers. In Proceedings of the 53rd Annual Meeting of the ACL and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). pa… find our card
  10. Hideki Isozaki and Tsutomu Hirao. 2003. Japanese zero pronoun resolution based on ranking rules and machine learning. In Proceedings of the 2003 conference on Empirical methods in natural language processing. Association for Computational Linguistics, pages 184–191. find our card
  11. Chen Chen and Vincent Ng. 2016. Chinese zero pronoun resolution with deep neural networks. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pages 778–788. http://aclweb.org/anthology/P16-107… ↗ our card
  12. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 . find
  13. Fang Kong and Guodong Zhou. 2010. A tree kernel-based unified framework for chinese zero anaphora resolution. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 882–891. find our card
  14. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 . find
  15. Lluis Marquez Emili Sapena M Antonia Marti Mariona Taule Veronique Hoste Massimo Poesio Yannick Versley Marta Recasens. 2010. Semeval-2010 task 1: Coreference resolution in multiple languages . find our card
  16. François Chollet. 2015. Keras. https://github. com/fchollet/keras. find
  17. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, pages 1532–1543. http://aclweb.org/anthology… ↗
  18. Susan P Converse. 2006. Pronominal anaphora resolution in chinese . find
  19. Antonio Ferrández and Jesús Peral. 2000. A computational approach to zero-pronouns in spanish. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, pages 166–172. find
  20. Alessandro Moschitti Nianwen Xue Olga Uryupina Yuchen Zhang Sameer Pradhan. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes . find our card
  21. Na-Rae Han. 2006. Korean zero pronouns: analysis and resolution. Ph.D. thesis, Citeseer. find
  22. Ryohei Sasano and Sadao Kurohashi. 2011. A discriminative approach to japanese zero anaphora resolution with large-scale lexicalized case frames. In IJCNLP. pages 758–766. find our card
  23. Andrew M Saxe, James L McClelland, and Surya Ganguli. 2013. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 . find
  24. Theano Development Team. 2016. Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints abs/1605.02688. http://arxiv.org/abs/1605.02688. ↗ our card
  25. Shanheng Zhao and Hwee Tou Ng. 2007. Identification and resolution of chinese zero pronouns: A machine learning approach. In EMNLP-CoNLL. volume 2007, pages 541–550. find our card

↑ back to top