Elamur Research⌕
☰
home / papers / 2020 / An Empirical Study of Contextual Data Augmentation for Japanese Zero Anaphora Resolution

An Empirical Study of Contextual Data Augmentation for Japanese Zero Anaphora Resolution

Summary

Zero anaphora resolution (ZAR) in pro-drop languages like Japanese recovers omitted predicate arguments, but labeled training data is scarce. The authors test whether contextual data augmentation (CDA) can help. CDA uses a pretrained language model to substitute tokens and create new training sentences. Applied naively to ZAR, it has two drawbacks: running a large LM over every sentence is costly, and substitutions such as swapping a verb can silently change which noun a zero pronoun refers to, making the copied gold labels wrong.

They propose two adaptations:

  • [MASK]-based augmentation: the baseline model sees sentences where selected tokens are simply replaced by [MASK], with no symbolic replacement tokens generated. This halves the LM computation.
  • Linguistically-controlled masking: POS tags decide which token types may be masked.

The baseline, MP-MLM, combines a BERT-style encoder with the multi-predicate model of Matsubayashi and Inui (2018). Masking everything except verbs at a fairly high rate yields statistically significant ZAR gains, while standard CDA brings none. This shows augmentation can help ZAR if the predicate structure is preserved.

Key points

  • The task follows the Japanese predicate-argument structure (PAS) setting, where each predicate is assigned NOM, ACC and DAT argument labels, using the same task definition as earlier studies by Matsubayashi and Inui (2017/2018), Ouchi et al. (2017), and Iida et al. (2015).
  • The MP-MLM baseline alone already beat earlier state-of-the-art results by a wide margin.
  • The best setting, called MASKING (S all\verb), masks all token types except verbs with a probability around 0.5. It significantly outperformed the baseline in both single-model and ensemble experiments.
  • Conventional CDA with symbolic token replacement performed roughly the same as the baseline.
  • Analysis of the generated sentences shows why verbs should be kept:
    • Masking verbs can break the relation between predicates and shift or erase a zero pronoun's antecedent.
    • Masking other tokens creates strong contextual variety while keeping the antecedent intact.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

4 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 3 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.coling-main.435 · arXiv 2011.00948

@inproceedings{konno2020empirical,
  title = {{An Empirical Study of Contextual Data Augmentation for Japanese Zero Anaphora Resolution}},
  author = {Konno, Ryuto and Matsubayashi, Yuichiroh and Kiyono, Shun and Ouchi, Hiroki and Takahashi, Ryo and Inui, Kentaro},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.435},
  eprint = {2011.00948},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.coling-main.435}
}

References

49 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chen Chen and Vincent Ng. 2014. Chinese Zero Pronoun Resolution: An Unsupervised Approach Combining Ranking and Integer Linear Programming. In Twenty-Eighth AAAI Conference on Artificial Intelligence. find
  2. Chen Chen and Vincent Ng. 2015. Chinese Zero Pronoun Resolution: A Joint Unsupervised Discourse-Aware Model Rivaling State-of-the-Art Resolvers. In ACL-IJCNLP, pages 320–326. find our card
  3. Chen Chen and Vincent Ng. 2016. Chinese Zero Pronoun Resolution with Deep Neural Networks. In ACL, pages 778–788. find our card
  4. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In EMNLP, pages 1724–1734. find
  5. Susan P. Converse. 2005. Resolving Pronominal References in Chinese with the Hobbs Algorithm. In IJCNLP-SIGHAN. find
  6. Michael Denkowski and Graham Neubig. 2017. Stronger Baselines for Trustable Results in Neural Machine Translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 18–27. find our card
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL, pages 4171–4186. find
  8. Antonio Ferrández and Jesús Peral. 2000. A Computational Approach to Zero-pronouns in Spanish. In ACL, pages 166–172. find
  9. Hagen Fürstenau and Mirella Lapata. 2009. Semi-Supervised Semantic Role Labeling. In EACL, pages 220–228. find
  10. Fei Gao, Jinhua Zhu, Lijun Wu, Yingce Xia, Tao Qin, Xueqi Cheng, Wengang Zhou, and Tie-Yan Liu. 2019. Soft Contextual Data Augmentation for Neural Machine Translation. In ACL, pages 5539–5544. find
  11. Na-Rae Han. 2006. Korean zero pronouns: Analysis and resolution. find
  12. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In CVPR, pages 770–778. find
  13. Ryu Iida and Massimo Poesio. 2011. A Cross-Lingual ILP Solution to Zero Anaphora Resolution. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 804–813. find our card
  14. Ryu Iida, Kentaro Inui, Hiroya Takamura, and Yuji Matsumoto. 2003. Incorporating Contextual Cues in Trainable Models for Coreference Resolution. In Proceedings of the 2003 EACL Workshop on The Computational Treatment of Anaphora. find
  15. Ryu Iida, Kentaro Inui, and Yuji Matsumoto. 2006. Exploiting Syntactic Patterns as Clues in Zero-Anaphora Resolution. In ACL-COLING, pages 625–632. find our card
  16. Ryu Iida, Kentaro Inui, and Yuji Matsumoto. 2007. Zero-anaphora Resolution by Learning Rich Syntactic Pattern Features. ACM Transactions on Asian Language Information Processing, 6(4):1:1–1:22. find
  17. Ryu Iida, Mamoru Komachi, Naoya Inoue, Kentaro Inui, and Yuji Matsumoto. 2010. Annotating Predicate-Argument Relations and Anaphoric Relations: Findings from the Building of the NAIST Text Corpus. Natural Language Processing, 17(2):25–50. find
  18. Ryu Iida, Kentaro Torisawa, Chikara Hashimoto, Jong-Hoon Oh, and Julien Kloetzer. 2015. Intra-sentential Zero Anaphora Resolution using Subject Sharing Recognition. In EMNLP, pages 2179–2189. find our card
  19. Ryu Iida, Kentaro Torisawa, Jong-Hoon Oh, Canasai Kruengkrai, and Julien Kloetzer. 2016. Intra-Sentential Subject Zero Anaphora Resolution using Multi-Column Convolutional Neural Network. In EMNLP, pages 1244–1254. find our card
  20. Ryu Iida, Mamoru Komachi, Naoya Inoue, Kentaro Inui, and Yuji Matsumoto. 2017. NAIST Text Corpus: Annotating Predicate-Argument and Coreference Relations in Japanese. In Handbook of Linguistic Annotation, pages 1177–1196. Springer. find
  21. Hideki Isozaki and Tsutomu Hirao. 2003. Japanese Zero Pronoun Resolution based on Ranking Rules and Machine Learning. In EMNLP-WS, pages 184–191. find our card
  22. Kushal Kafle, Mohammed Yousefhussien, and Christopher Kanan. 2017. Data Augmentation for Visual Question Answering. In INLG, pages 198–202. find
  23. Diederik Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR. find
  24. Sosuke Kobayashi. 2018. Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations. In NAACL, pages 452–457. find our card
  25. Dan Kondratyuk and Milan Straka. 2019. 75 Languages, 1 Model: Parsing Universal Dependencies Universally. In EMNLP-IJCNLP, pages 2779–2795. find our card
  26. Fang Kong and Guodong Zhou. 2010. A Tree Kernel-Based Unified Framework for Chinese Zero Anaphora Resolution. In EMNLP, pages 882–891. find our card
  27. Shuhei Kurita, Daisuke Kawahara, and Sadao Kurohashi. 2018. Neural Adversarial Training for Semi-supervised Japanese Predicate-argument Structure Analysis. In ACL, pages 474–484. find
  28. Ting Liu, Yiming Cui, Qingyu Yin, Wei-Nan Zhang, Shijin Wang, and Guoping Hu. 2017. Generating and Exploiting Large-scale Pseudo Training Data for Zero Pronoun Resolution. In ACL, pages 102–111. find our card
  29. Yuichiroh Matsubayashi and Kentaro Inui. 2017. Revisiting the Design Issues of Local Models for Japanese Predicate-Argument Structure Analysis. In IJCNLP, pages 128–133. find
  30. Yuichiroh Matsubayashi and Kentaro Inui. 2018. Distance-Free Modeling of Multi-Predicate Interactions in End-to-End Japanese Predicate-Argument Structure Analysis. In COLING, pages 94–106. find
  31. Hikaru Omori and Mamoru Komachi. 2019. Multi-Task Learning for Japanese Predicate Argument Structure Analysis. In NAACL, pages 3404–3414. find
  32. Hiroki Ouchi, Hiroyuki Shindo, and Yuji Matsumoto. 2017. Neural Modeling of Multi-Predicate Interactions for Japanese Predicate Argument Structure Analysis. In ACL, pages 1591–1600. find
  33. Manuel Palomar, Antonio Ferrández, Lidia Moreno, Patricio Martı́nez-Barco, Jesús Peral, Maximiliano Saiz-Noeda, and Rafael Muñoz. 2001. An Algorithm for Anaphora Resolution in Spanish Texts. Computational Linguistics, 27(4):545–567. find our card
  34. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In NAACL, pages 2227–2237. find
  35. Kepa Joseba Rodrı́guez, Francesca Delogu, Yannick Versley, Egon W. Stemle, and Massimo Poesio. 2010. Anaphoric Annotation of Wikipedia and Blogs in the Live Memories Corpus. In LREC. find our card
  36. Ryohei Sasano and Sadao Kurohashi. 2011. A Discriminative Approach to Japanese Zero Anaphora Resolution with Large-scale Lexicalized Case Frames. In IJCNLP, pages 758–766. find our card
  37. Ryohei Sasano, Daisuke Kawahara, and Sadao Kurohashi. 2008. A Fully-Lexicalized Probabilistic Model for Japanese Zero Anaphora Resolution. In COLING, pages 769–776. find our card
  38. Kazuhiro Seki, Atsushi Fujii, and Tetsuya Ishikawa. 2002. A Probabilistic Method for Analyzing Japanese Anaphora Integrating Zero Pronoun Detection and Resolution. In COLING. find our card
  39. Tomohide Shibata, Daisuke Kawahara, and Sadao Kurohashi. 2019. Improving Japanese Syntax Parsing with BERT. Natural Language Processing, pages 205–208. find
  40. Jun Suzuki, Sho Takase, Hidetaka Kamigaito, Makoto Morishita, and Masaaki Nagata. 2018. An Empirical Study of Building a Strong Baseline for Constituency Parsing. In ACL, pages 612–618. find
  41. Hirotoshi Taira, Sanae Fujita, and Masaaki Nagata. 2008. A Japanese Predicate Argument Structure Analysis using Decision Lists. In EMNLP, pages 523–532. find
  42. Alex Wang and Kyunghyun Cho. 2019. BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 30–36. find our card
  43. William Yang Wang and Diyi Yang. 2015. That’s So Annoying!!!: A Lexical and Frame-Semantic Embedding Based Data Augmentation Approach to Automatic Categorization of Annoying Behaviors using \#petpeeve Tweets. In EMNLP, pages 2557–2563. find our card
  44. Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019. Conditional BERT Contextual Augmentation. In ICCS, pages 84–95. find
  45. Souta Yamashiro, Hitoshi Nishikawa, and Takenobu Tokunaga. 2018. Neural Japanese Zero Anaphora Resolution using Smoothed Large-scale Case Frames with Word Embedding. In PACLIC. find our card
  46. Qingyu Yin, Yu Zhang, Weinan Zhang, Ting Liu, and William Yang Wang. 2018. Zero Pronoun Resolution with Attention-based Neural Network. In COLING, pages 13–23. find our card
  47. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level Convolutional Networks for Text Classification. In NIPS, pages 649–657. find our card
  48. Shanheng Zhao and Hwee Tou Ng. 2007. Identification and Resolution of Chinese Zero Pronouns: A Machine Learning Approach. In EMNLP-CoNLL, pages 541–550. find our card
  49. Jie Zhou and Wei Xu. 2015. End-to-end Learning of Semantic Role Labeling Using Recurrent Neural Networks. In ACL, pages 1127–1137. *Figure 5: Overview of two replacement approaches.* Method find

↑ back to top