Elamur Research⌕
☰
home / papers / 2020 / Cross-lingual Zero Pronoun Resolution

Cross-lingual Zero Pronoun Resolution

Summary

In pro-drop languages such as Arabic and Chinese, arguments like subjects are often left unexpressed where English would use a pronoun. Working out what these gaps (zero pronouns) refer to matters for machine translation and information extraction. Earlier systems, however, depended on large language-specific feature sets and plenty of training data, which Arabic lacks. The authors tackle one half of the task: choosing the antecedent of an already identified anaphoric zero pronoun (AZP). Multilingual BERT encodes each sentence with frozen weights. The gap is represented by averaging the vectors of its neighbouring words, and each candidate noun phrase by averaging its sub-tokens. Two language-neutral cues are added: whether the AZP and the candidate share a sentence, and their word distance. A small feed-forward network scores every pair, and the highest-scoring candidate is selected. Evaluated on the Arabic and Chinese data in OntoNotes 5.0, the model is described as the first neural resolver for Arabic and as surpassing the prior best Chinese system. This shows that a single lightweight design can handle typologically distant languages.

Key points

  • Data: Arabic and Chinese sections of OntoNotes 5.0. Zero pronouns are marked * in the Arabic text and *pro* in the Chinese text.
  • Encoder: BERT-base Multilingual (12 layers, 768 hidden units, WordPiece tokenizer), used in feature-extraction mode.
  • Architecture: a pairwise <AZP, candidate> classifier. It adds two features, same_sentence and find_distance, and uses three ReLU hidden layers plus a scoring layer.
  • Baselines and analysis: the model is compared with plain BERT fine-tuning and feature-extraction setups. A layer-by-layer study looks for the BERT layer best suited to the task. The previous Chinese state of the art was Yin et al. (2018). The exact scores and the best layer appear in the part of the paper omitted from the provided text.
  • Code: github.com/amaloraini/cross-lingual-ZP

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

10 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 5 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{aloraini2020cross,
  title = {{Cross-lingual Zero Pronoun Resolution}},
  author = {Aloraini, Abdulrahman and Poesio, Massimo},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  url = {https://aclanthology.org/2020.lrec-1.11/}
}

References

48 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aken, B., Winter, B., Löser, A., and Gers, F. A. (2019). How does bert answer questions? a layer-wise analysis of transformer representations. In arXiv preprint arXiv:1908.08593. find our card
  2. Alnajadat, B. M. (2017). Pro-drop in standard arabic. In International Journal of English Linguistics 7.1. find
  3. Aone, C. and Bennett, S. W. (1995). Evaluating automated and manual acquisition of anaphora resolution strategies. In ACL ’95 Proceedings of the 33rd annual meeting on Association for Computational Linguistics, pages 122–129. find our card
  4. Björkelund, A. and Kuhn, J. (2014). Learning structured perceptrons for coreference resolution with latent antecedents and non-local features. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 47–57. find our card
  5. Byron, D. K., Gegg-Harrison, W., and Lee, S.-H. (2006). Resolving zero anaphors and pronouns in korean. In Traitement Automatique des Langues 46.1, pages 91–114. find
  6. Chen, C. and Ng, V. (2012). Combining the best of two worlds: A hybrid approach to multilingual coreference resolution. In Joint Conference on EMNLP and CoNLL-Shared Task, pages 56–63. find our card
  7. Chen, C. and Ng, V. (2013). Chinese zero pronoun resolution: Some recent advances. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1360–1365. find our card
  8. Chen, C. and Ng, V. (2014). Chinese zero pronoun resolution: An unsupervised probabilistic model rivaling supervised resolvers. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. find our card
  9. Chen, C. and Ng, V. (2015). Chinese zero pronoun resolution: A joint unsupervised discourse-aware model rivaling state-of-the-art resolvers. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Proce… find our card
  10. Chen, C. and Ng, V. (2016). Chinese zero pronoun resolution with deep neural network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 778–788. finding syntax in word representations. In Proceedings of the 2019 Conference of th… find
  11. Hobbs, J. (1978). Resolving pronoun references. In Lingua, pages 311–338. find our card
  12. Huang, C.-T. J. (1984). On the distribution and reference of empty pronouns. In Linguistic Inquiry, Vol. 15, No. 4, pages 531–574. find
  13. Iida, R. and Poesio, M. (2011). A cross-lingual ilp solution to zero anaphora resolution. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 804–813. find our card
  14. Iida, R., Inui, K., and Matsumoto, Y. (2006). Exploiting syntactic patterns as clues in zero-anaphora resolution. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistic, pages 625–632. find our card
  15. Iida, R., Inui, K., and Matsumoto., Y. (2007). Zero-anaphora resolution by learning rich syntactic pattern features. In ACM Transactions on Asian Language Information Processing, 6(4). find
  16. Iida, R., Torisawa, K., Hashimoto, C., Oh, J.-H., and Kloet-zer, J. (2015). Intra-sentential zero anaphora resolution using subject sharing recognition. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2179–2189. find our card
  17. Imamura, K., Saito, K., and Izumi, T. (2009). Discriminative approach to predicate argument structure analysis with zero-anaphora resolution. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 85–88. find our card
  18. Isozaki, H. and Hirao, T. (2003). Japanese zero pronoun resolution based on ranking rules and machine learning. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing,, pages 184–191. find our card
  19. Jawahar, G., Sagot, B., and Seddah, D. (2019). What does bert learn about the structure of language? In 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy. find our card
  20. Kameyama, M. (1985). Zero Anaphora: The case of Japanese. Ph.D. thesis, Stanford University, Stanford, CA. find
  21. Kim, Y.-B. and Ehara., T. (1995). Zero-subject resolution method based on probabilistic inference with evaluation function. In Proceedings of the 3rd Natural Language Processing Pacific- Rim Symposium, pages 721––727. find
  22. Kim, Y.-J. (2000). Subject/object drop in the acquisition of korean: A cross-linguistic comparison. In Journal of East Asian Linguistics 9.4, pages 325–351. find
  23. Kong, F. and Zhou, G. (2010). A tree kernel-based unified framework for chinese zero anaphora resolution. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 882–891. find our card
  24. Kovaleva, O., Romanov, A., Rogers, A., and Rumshisky, A. (2019). Revealing the dark secrets of bert. In arXiv preprint arXiv:1908.08593. find our card
  25. Lee, H., Peirsman, Y., Chang, A., Chambers, N., Surdeanu, M., and Jurafsky, D. (2011). Stanford’s multi-pass sieve coreference resolution system at the conll-2011 shared task. In CONLL Shared Task ’11 Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pa… find our card
  26. Li, C. N. and Thompson, S. A. (1979). Third person pronouns and zero anaphora in chinese discourse. In Syntax and Semantics, volume 12: Discourse and Syntax, pages 311–335. find
  27. Academic Press. Liu, T., Cui, Y., Yin, Q., Zhang, W., Wang, S., and Hu, G. (2017). Generating and exploiting large-scale pseudo training data for zero pronoun resolution. In arXiv preprint arXiv:1606.01603. find our card
  28. Mihăilă, C., Ilisei, I., , and Inkpen, D. (2011). Zero pronominal anaphora resolution for the romanian language. In Research Journal on Computer Science and Computer Engineering with Applications, POLIBITS, 42. find
  29. Nair, V. and Hinton, G. (2010). Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807–814. find
  30. Peters, M., Ruder, S., and Smith, N. (2019). To tune or not to tune? adapting pretrained representations to diverse tasks. In arXiv preprint arXiv:1903.05987. find
  31. Poesio, M., Uryupina, O., and Versley, Y. (2010). Creating a coreference resolution system for italian. In LREC 2010 May 19. find our card
  32. Pradhan, S., Moschitti, A., Xue, N., Uryupina, O., and Zhang, Y. (2012). Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint Conference on EMNLP and CoNLL-Shared Task. Association for Computational Linguistics, Association for Computational Linguistics., pag… find our card
  33. Sasano, R. and Kurohashi, S. (2011). discriminative approach to japanese zero anaphora resolution with large-scale lexicalized case frames. In Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 758–766. find our card
  34. Sasano, R., Kawahara, D., and Kurohashi, S. (2008). A fully-lexicalized probabilistic model for japanese zero anaphora resolution. In Proceedings of the 22nd International Conference on Computational Linguistics, pages 769–776. find our card
  35. Sasano, R., Kawahara, D., and Kurohashi, S. (2009). The effect of corpus size on case frame acquisition for discourse analysis. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 521–529. find
  36. Seki, K., Fujii, A., and Ishikawa., T. (2002). A probabilistic method for analyzing japanese anaphora integrating zero pronoun detection and resolution. In Proceedings of the 19th International Conference on Computational Linguistics - Volume 1, pages 1–7. find our card
  37. Taira, H., Fujita, S., and Nagata, M. (2008). A japanese predicate argument structure analysis using decision lists. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 523–532. find
  38. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. find
  39. Watanabe, Y., Asahara, M., and Matsumoto, Y. (2010). A structured model for joint learning of argument roles and predicate senses. In Proceedings of the ACL 2010 Conference Short Papers,, pages 98–101. find
  40. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Łukasz Kaiser, Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith, J., Riesa, J., Rud… find
  41. Yamashiro, S., Nishikawa, H., and Tokunaga, T. (2018). Neural japanese zero anaphora resolution using smoothed large-scale case frames with word embedding. In Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation. find our card
  42. Yeh, C.-L. and Chen, Y.-C. (2006). Zero anaphora resolution in chinese with shallow parsing. In Journal of Chinese Language and Computing 17 (1), pages 41–56. find
  43. Yin, Q., Zhang, Y., Zhang, W., and Liu, T. (2016). A deep neural network for chinese zero pronoun resolution. In arXiv preprint arXiv:1604.05800. — [arXiv:1604.05800](https://arxiv.org/abs/1604.05800) find
  44. Yin, Q., Zhang, Y., Zhang, W., and Liu, T. (2017). Chinese zero pronoun resolution with deep memory network. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1309–1318. find our card
  45. Yin, Q., Zhang, Y., Zhang, W., Liu, T., and Wang, W. Y. (2018). Zero pronoun resolution with attention-based neural network. In Proceedings of the 27th International Conference on Computational Linguistics, pages 13–23. find our card
  46. Yoshikawa, K., Asahara, M., and Matsumoto, Y. (2011). Jointly extracting japanese predicate-argument relation with markov logic. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 1125–1133. find
  47. Yoshino, K., Mori, S., and Kawahara, T. (2013). Predicate argument structure analysis using partially annotated corpora. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 957–961. find
  48. Zhao, S. and Ng, H. T. (2007). Identification and resolution of chinese zero pronouns: A machine learning approach. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 541–550. find our card

↑ back to top