Elamur Research⌕
☰
home / papers / 2021 / Pseudo Zero Pronoun Resolution Improves Zero Anaphora Resolution

Pseudo Zero Pronoun Resolution Improves Zero Anaphora Resolution

Summary

Zero anaphora resolution (ZAR) means finding the antecedent of an omitted argument (a "zero pronoun") in pro-drop languages like Japanese. The authors aim to get more from pretrained masked language models (MLMs) on this task with two ideas. First, they add a pretraining task called pseudo zero pronoun resolution (PZERO), built cheaply from raw text. When a noun phrase recurs in a passage, one occurrence is collapsed into a single [MASK] token. The model must then point to its earlier same-form occurrences, which gives it explicit (if noisy) supervision on anaphoric links. Second, they design a finetuning model, AS-PZERO, that keeps all pretrained parameters. It phrases each query as a masked prompt, which narrows the gap between pretraining and finetuning. On the NAIST Text Corpus (NTC), both ideas improve results and set new state-of-the-art F1 scores. The analysis shows that arguments far from the predicate and arguments of passive predicates remain hard. This makes the work a useful recipe for low-resource anaphora tasks.

Key points

  • PZERO data generation: All NPs sharing a surface form are treated as coreferent, which allows large-scale pseudo data to be mined from raw corpora. The whole masked NP becomes one [MASK] token, so the model cannot use mask count as a clue. The target is the NP's last token, matching NTC's annotation convention. Training minimizes a KL-divergence loss over candidate token positions.
  • Finetuning models: The baseline is an argument-selection model (AS). The proposed AS-PZERO appends a query chunk ([MASK] + case marker + predicate), giving a prompt-like setup with minimal architectural change.
  • Benchmark: NTC with NOM, ACC and DAT roles. Evaluation covers intra-sentential, inter-sentential and exophoric (author/reader/general) arguments in a single formulation.
  • Development-set recall analysis: Between the two compared configurations, recall rises from 68.9 to 71.09 for intra-sentential and from 43.11 to 45.65 for inter-sentential arguments. Passive predicates lag behind active ones (about 52–58 vs 71–73 intra). Arguments outside the input window get 0.0 recall.
  • Code: Available at github.com/Ryuto10/pzero-improves-zar. Analyzing noise in the same-surface-form assumption is left as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

8 papers of our library cite it (of the 140673 whose bibliographies we read).

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.emnlp-main.308 · arXiv 2104.07425

@inproceedings{konno2021pseudo,
  title = {{Pseudo Zero Pronoun Resolution Improves Zero Anaphora Resolution}},
  author = {Konno, Ryuto and Kiyono, Shun and Matsubayashi, Yuichiroh and Ouchi, Hiroki and Inui, Kentaro},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.308},
  eprint = {2104.07425},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.308}
}

References

42 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdulrahman Aloraini and Massimo Poesio. 2020. Cross-lingual Zero Pronoun Resolution. In Proceedings of LREC, pages 90–98. find our card
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  3. Nathanael Chambers and Dan Jurafsky. 2009. Unsupervised Learning of Narrative Schemas and their Participants. In Proceedings of ACL-IJCNLP, pages 602–610. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL, pages 4171– 4186. find
  5. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making Pre-trained Language Models Better Few-shot Learners. In Proceedings of ACL-IJCNLP, pages 3816–3830. find our card
  6. Chaoyu Guan, Yuhao Cheng, and Hai Zhao. 2019. Semantic Role Labeling with Associated Memory Network. In Proceedings of NAACL, pages 3361–3371. find
  7. In this study, we proposed a new pretraining task, PZ ERO , which aims to explicitly teach the model Suchin Gururangan, Ana Marasovic, Swabha anaphoric relational knowledge necessary for ZAR. Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, We also proposed a ZAR model to remedy the and Noah A. Smith.… find
  8. Na-Rae Han. 2006. Korean zero pronouns: Analysis Our analysis suggests that the hard subcategories and resolution. Ph.D. thesis, University of Pennsyl-of ZAR; distant arguments and passive predicates vania. 3798 find
  9. Masatsugu Hangyo, Daisuke Kawahara, and Sadao Kurohashi. 2013. Japanese Zero Reference Resolution Considering Exophora and Author/Reader Mentions. In Proceedings of EMNLP, pages 924–934. find
  10. Luheng He, Kenton Lee, Mike Lewis, and Luke Zettlemoyer. 2017. Deep Semantic Role Labeling: What Works and What’s Next. In Proceedings of ACL, pages 473–483. find our card
  11. Ryu Iida, Mamoru Komachi, Naoya Inoue, Kentaro Inui, and Yuji Matsumoto. 2010. Annotating Predicate-Argument Relations and Anaphoric Relations: Findings from the Building of the NAIST Text Corpus. Natural Language Processing, 17(2):25– 50. find
  12. Ryu Iida, Mamoru Komachi, Naoya Inoue, Kentaro Inui, and Yuji Matsumoto. 2017. NAIST Text Corpus: Annotating Predicate- Argument and Coreference Relations in Japanese, pages 1177–1196. Springer Netherlands. find
  13. Ryu Iida and Massimo Poesio. 2011. A Cross-Lingual ILP Solution to Zero Anaphora Resolution. In Proceedings of ACL, pages 804–813. find our card
  14. Ryu Iida, Kentaro Torisawa, Jong-Hoon Oh, Cana-sai Kruengkrai, and Julien Kloetzer. 2016. Intra-Sentential Subject Zero Anaphora Resolution using Multi-Column Convolutional Neural Network. In Proceedings of EMNLP, pages 1244–1254. find our card
  15. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for Coreference Resolution: Baselines and Analysis. In Proceedings of EMNLP-IJCNLP, pages 5803–5808. find our card
  16. Diederik Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In Proceedings of ICLR. find
  17. Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. 2019. A Surprisingly Robust Trick for the Winograd Schema Challenge. In Proceedings of ACL, pages 4837–4842. find
  18. Ryuto Konno, Yuichiroh Matsubayashi, Shun Kiyono, Hiroki Ouchi, Ryo Takahashi, and Kentaro Inui. 2020. An Empirical Study of Contextual Data Augmentation for Japanese Zero Anaphora Resolution. In Proceedings of COLING, pages 4956–4968. find our card
  19. Taku Kudo and Yuji Matsumoto. 2002. Japanese Dependency Analysis using Cascaded Chunking. In Proceedings of CoNLL. find
  20. Shuhei Kurita, Daisuke Kawahara, and Sadao Kurohashi. 2018. Neural Adversarial Training for Semi-supervised Japanese Predicate-argument Structure Analysis. In Proceedings of ACL, pages 474–484. find
  21. Hector J. Levesque. 2011. The Winograd Schema Challenge. In Proceedings of AAAI. find
  22. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. arXiv preprint arXiv:2107.13586. find our card
  23. Ting Liu, Yiming Cui, Qingyu Yin, Wei-Nan Zhang, Shijin Wang, and Guoping Hu. 2017. Generating and Exploiting Large-scale Pseudo Training Data for Zero Pronoun Resolution. In Proceedings of ACL, pages 102–111. find our card
  24. Yuichiroh Matsubayashi and Kentaro Inui. 2018. Distance-Free Modeling of Multi-Predicate Interactions in End-to-End Japanese Predicate-Argument Structure Analysis. In Proceedings of COLING, pages 94–106. find
  25. Hikaru Omori and Mamoru Komachi. 2019. Multi-Task Learning for Japanese Predicate Argument Structure Analysis. In Proceedings of NAACL, pages 3404–3414. find
  26. Hiroki Ouchi, Hiroyuki Shindo, and Yuji Matsumoto. 2017. Neural Modeling of Multi-Predicate Interactions for Japanese Predicate Argument Structure Analysis. In Proceedings of ACL, pages 1591–1600. find
  27. Manuel Palomar, Antonio Ferrández, Lidia Moreno, Patricio Martínez-Barco, Jesús Peral, Maximiliano Saiz-Noeda, and Rafael Muñoz. 2001. An Algorithm for Anaphora Resolution in Spanish Texts. Computational Linguistics, 27(4):545–567. find our card
  28. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. Technical report, OpenAI. find
  29. Ryohei Sasano, Daisuke Kawahara, and Sadao Kurohashi. 2008. A Fully-Lexicalized Probabilistic Model for Japanese Zero Anaphora Resolution. In Proceedings of COLING, pages 769–776. find our card
  30. Ryohei Sasano and Sadao Kurohashi. 2011. A Discriminative Approach to Japanese Zero Anaphora Resolution with Large-scale Lexicalized Case Frames. In Proceedings of IJCNLP, pages 758–766. find our card
  31. Timo Schick, Helmut Schmid, and Hinrich Schütze. 2020. Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification. In Proceedings of COLING, pages 5569–5578. find
  32. Timo Schick and Hinrich Schütze. 2021a. Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference. In Proceedings of EACL, pages 255–269. find our card
  33. Timo Schick and Hinrich Schütze. 2021b. It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners. In Proceedings of NAACL-HLT, pages 2339–2352. find our card
  34. Tomohide Shibata, Daisuke Kawahara, and Sadao Kurohashi. 2016. Neural Network-Based Model for Japanese Predicate Argument Structure Analysis. In Proceedings of ACL, pages 1235–1244. find
  35. Tomohide Shibata and Sadao Kurohashi. 2018. Entity-Centric Joint Modeling of Japanese Coreference Resolution and Predicate Argument Structure Analysis. In Proceedings of ACL, pages 579–589. find our card
  36. Linfeng Song, Kun Xu, Yue Zhang, Jianshu Chen, and Dong Yu. 2020. ZPR2: Joint Zero Pronoun Recovery and Resolution using Multi-Task Learning and BERT. In Proceedings of ACL, pages 5429–5434. find our card
  37. Hirotoshi Taira, Sanae Fujita, and Masaaki Nagata. 2008. A Japanese Predicate Argument Structure Analysis using Decision Lists. In Proceedings of EMNLP, pages 523–532. find
  38. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lh… find our card
  39. Souta Yamashiro, Hitoshi Nishikawa, and Takenobu Tokunaga. 2018. Neural Japanese Zero Anaphora Resolution using Smoothed Large-scale Case Frames with Word Embedding. In Proceedings of PACLIC. find our card
  40. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Proceedings of NeurIPS, pages 5754–5764. find our card
  41. Qingyu Yin, Yu Zhang, Weinan Zhang, Ting Liu, and William Yang Wang. 2018. Zero Pronoun Resolution with Attention-based Neural Network. In Proceedings of COLING, pages 13–23. find our card
  42. Dataset Training dep NOM 36934 12219 7843 ACC 24654 2136 948 5744 465 294 DAT Development NOM ACC DAT Test ing Japanese dependency parser Cabocha (Kudo and Matsumoto, 2002). The parser divides the sentences into a phrase (Japanese “bunsetsu"). Note that each bunsetsu consists of a sequence of words.… find

↑ back to top