Elamur Research⌕
☰
home / papers / 2021 / Incorporating Syntax and Semantics in Coreference Resolution with Heterogeneous Graph Attention Network

Incorporating Syntax and Semantics in Coreference Resolution with Heterogeneous Graph Attention Network

Summary

Modern neural coreference systems rely on large pretrained encoders and mostly ignore explicit linguistic structure. Jiang and Cohn ask whether dependency syntax and semantic role labelling (SRL) still help such systems. They extend the c2f-coref architecture, using SpanBERT as the encoder, with a heterogeneous graph built for each document:

  • Syntactic sub-graph: token nodes are joined by labelled dependency arcs, and the roots of adjacent sentences are linked.
  • Semantic sub-graph: argument and predicate nodes are joined by SRL labels, and each argument is tied to the tokens it covers.

A Graph Attention Network passes messages through both sub-graphs. The resulting token vectors, enriched with syntax and semantics, are merged with the original embeddings through an attention-based integration layer and a learned gate.

On the English OntoNotes 5.0 benchmark, the model (coref-HGAT) beats a reimplemented SpanBERT baseline by 1.3 average F1 for the base encoder and 1.1 for the large one. It keeps a gain even when features come from automatic parsers. The study suggests that explicit structure still adds value on top of strong pretrained encoders. The code is released on GitHub (Fantabulous-J/coref-HGAT).

Key points

  • Data and metric: OntoNotes 5.0 has 2802/343/348 training/dev/test documents. Evaluation uses average F1 of MUC, B3 and CEAFφ4 with the official CoNLL-2012 scripts.
  • Main results: Average F1 rises from 77.5 to 78.8 for SpanBERT-base and from 80.0 to 81.1 for SpanBERT-large, averaged over 5 seeds. CorefQA reaches 83.1 but was not used as a baseline because it needs 128G GPU memory for training.
  • Ablation: Dependency features alone add +1.0 and SRL alone adds +0.9. Combining them gives +1.3. Two GAT layers worked better than one or three.
  • Predicted features: Results were tested with the Stanford and biaffine dependency parsers (the biaffine parser has about 3% higher LAS on Penn Treebank). SRL came from parsers trained on CoNLL05 or CoNLL12. Gains range from +0.6 to +0.9, and better or closer-domain parsers help more.
  • Document length (dev set): Overall F1 improves from 77.8 to 79.2, with the largest gains on the longest (1153+ tokens, +2.8) and shortest documents. Performance still drops for long texts, and the authors suggest document-level cues such as discourse structure as a remedy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution · Semantic role labeling

Cited by

6 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 4 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.naacl-main.125

@inproceedings{jiang2021incorporating,
  title = {{Incorporating Syntax and Semantics in Coreference Resolution with Heterogeneous Graph Attention Network}},
  author = {Jiang, Fan and Cohn, Trevor},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.125},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.125}
}

References

38 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1–6. find our card
  2. Jasmijn Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Sima’an. 2017. Graph convolutional encoders for syntax-aware neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1957–1967, Copenhagen, Denmark. Associat… find our card
  3. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. find our card
  4. Xavier Carreras and Lluís Màrquez. 2004. Introduction to the CoNLL-2004 shared task: Semantic role labeling. In Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004, pages 89–97, Boston, Massachusetts, USA. Association for Computational Lingu… find
  5. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP… find our card
  6. Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 740–750, Doha, Qatar. Association for Computational Linguistics. find our card
  7. Ben Kantor and Amir Globerson. 2019. Coreference resolution with entity equalization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 673–677. find our card
  8. Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019. Connecting the dots: Document-level neural relation extraction with edge-oriented graphs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lang… find
  9. Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, and Matt Gardner. 2019. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Co… find our card
  10. Marie-Catherine de Marneffe and Christopher D. Manning. 2008. The Stanford typed dependencies representation. In Coling 2008: Proceedings of the workshop on Cross-Framework and Cross-Domain Parser Evaluation, pages 1–8. find our card
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  12. Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In 5th International Conference on Learning Representations, ICLR 2017. find our card
  13. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. find
  14. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015. find
  15. Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR). find our card
  16. Fang Kong and Fu Jian. 2019. Incorporating structural information for better coreference resolution. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5039–5045. International Joint Conferences on Artificial Intelligence Organization. find
  17. Fang Kong, GuoDong Zhou, and Qiaoming Zhu. 2009. Employing the centering theory in pronoun resolution from the semantic perspective. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 987–996. find our card
  18. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197. Association for Computational Linguistics. find our card
  19. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687–… find our card
  20. Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019. A general framework for information extraction using dynamic span graphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tec… find our card
  21. Peng Shi and Jimmy Lin. 2019. Simple BERT models for relation extraction and semantic role labeling. CoRR, abs/1904.05255. find our card
  22. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60. find our card
  23. Dario Stojanovski and Alexander Fraser. 2018. Coreference and coherence in neural machine translation: A study using oracle experiments. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 49–60. find our card
  24. Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1506–1515, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  25. Yuyang Nie, Yuanhe Tian, Yan Song, Xiang Ao, and Xiang Wan. 2020a. Improving named entity recognition with attentive ensemble of syntactic information. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4231–4245. find
  26. Yuyang Nie, Yuanhe Tian, Xiang Wan, Yan Song, and Bo Dai. 2020b. Named entity recognition for social media texts with semantic augmentation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1383–1391. find
  27. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  28. Simone Paolo Ponzetto and Michael Strube. 2006. Exploiting semantic role labeling, WordNet and Wikipedia for coreference resolution. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 192– 199. find our card
  29. Amir Pouran Ben Veyseh, Tuan Ngo Nguyen, and Thien Huu Nguyen. 2020. Graph transformer networks with syntactic and semantic structures for event argument extraction. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3651–3661. find
  30. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40. find our card
  31. Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations. find
  32. David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langua… find our card
  33. Kai Wang, Weizhou Shen, Yunyi Yang, Xiaojun Quan, and Rui Wang. 2020. Relational graph attention network for aspect-based sentiment analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3229– 3238. find
  34. Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020. CorefQA: Coreference resolution as query-based span prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6953–6963. find our card
  35. Patrick Xia, João Sedoc, and Benjamin Van Durme. 2020. Incremental neural coreference resolution in constant memory. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8617–8624, Online. Association for Computational Linguistics. find our card
  36. Liyan Xu and Jinho D. Choi. 2020. Revealing the myth of higher-order inference in coreference resolution. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8527–8533. find our card
  37. Yinchuan Xu and Junlin Yang. 2019. Look again at the syntax: Relational graph convolutional network for gendered ambiguous pronoun resolution. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 96–101, Florence, Italy. Association for Computational Linguistics. find
  38. Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2017. Modeling relational data with graph convolutional networks. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jiang-2021-incorporating-syntax-semantics-coreference-resolution/