Elamur Research
☰
home / papers / 2018 / Graph Convolution over Pruned Dependency Trees Improves Relation Extraction

Graph Convolution over Pruned Dependency Trees Improves Relation Extraction

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D18-1244.pdf

Summary

Relation extraction decides whether two entity mentions in a sentence, a subject and an object, are linked by a relation and which one. Dependency parses help with long-distance links, but earlier dependency-based neural models have two problems. Models restricted to the shortest dependency path can drop decisive words such as negation. Tree-recursive models like Tree-LSTM are slow because varying tree shapes are hard to batch. The authors adapt graph convolutional networks to this task. The parse is treated as an undirected adjacency matrix with self-loops and degree normalization. Max pooling then builds sentence, subject and object vectors, which are concatenated and passed to a feed-forward classifier. A variant called C-GCN first runs a BiLSTM over the words to add word-order context and robustness to parse errors. They also introduce path-centric pruning, which keeps the shortest path plus tokens within K steps of it. On TACRED, the model reaches state-of-the-art results, beating sequence and dependency baselines. It also improves on dependency baselines on SemEval 2010 Task 8 and trains 10–100x faster than tree-based models. Combining it with a sequence model raises scores further.

Key points

  • Datasets: TACRED (the main benchmark, where state of the art is claimed) and SemEval 2010 Task 8. The parse trees follow UD.
  • Speed: on a Titan Xp GPU, one Tree-LSTM minibatch of 50 examples takes about 6.54 s on average. The GCN is roughly 10–100x faster overall because all of its operations are matrix multiplications.
  • Design ablations: separate weights per edge direction did not help, and adding edge-wise gating by dependency type reduced performance.
  • Complementarity with PA-LSTM, measured over 5 seeds on the TACRED dev set:
    • C-GCN is at least 60% more accurate on 847 examples (3.7%).
    • PA-LSTM has the same advantage on 629 examples (2.8%).
    • Dependency models do better when the entities are far apart, while the sequence model makes better use of local word cues.
  • Entity bias in SemEval:
    • A PA-LSTM that sees only "subject and object" still gets 65.1 dev F1 with GloVe and 47.9 without it.
    • An unmasked model falls from 83.6 to 62.4 F1 when dev entities are replaced by <UNK>.
    • A model trained with entity masking holds steady at 74.7.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Dependency parsing

Cited by

199 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 195 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1244

@inproceedings{zhang2018graph,
  title = {{Graph Convolution over Pruned Dependency Trees Improves Relation Extraction}},
  author = {Zhang, Yuhao and Qi, Peng and Manning, Christopher D.},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1244},
  url = {https://doi.org/10.18653/v1/d18-1244}
}

References

27 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Heike Adel, Benjamin Roth, and Hinrich Schütze. 2016. Comparing convolutional neural networks to traditional models for slot filling. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology (NAACL-HLT). find
  2. Razvan C Bunescu and Raymond J Mooney. 2005. A shortest path dependency kernel for relation extraction. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing (EMNLP 2005), pages 724–731. find
  3. Shexia He, Zuchao Li, Hai Zhao, and Hongxiao Bai. 2018. Syntax for semantic role labeling, to be, or not to be. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. find
  4. Nanda Kambhatla. 2004. Combining lexical, syntactic, and semantic features with maximum entropy models for extracting relations. In Proceedings of the ACL 2004 Interactive poster and demonstration sessions. Association for Computational Linguistics. find
  5. Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations (ICLR 2017). find our card
  6. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324. find
  7. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017). find our card
  8. Yang Liu, Furu Wei, Sujian Li, Heng Ji, Ming Zhou, and Houfeng Wang. 2015. A dependency-based neural network for relation classification. Proceedings of the 53th Annual Meeting of the Association for Computational Linguistics (ACL 2015). find
  9. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60. find our card
  10. Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017). find our card
  11. Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 10… find
  12. Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using LSTMs on sequences and tree structures. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016). find our card
  13. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, et al. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of the 10th International Conference on La… find our card
  14. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP 2014), volume 14, pages 1532– 1543. find
  15. Chris Quirk and Hoifung Poon. 2017. Distant supervision for relation extraction beyond the sentence boundary. Proceedings of the 15th Conference of the European Association for Computational Linguistics (EACL 2017). find
  16. Bryan Rink and Sanda Harabagiu. 2010. UTD: Classifying semantic relations by combining lexical and semantic resources. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 256–259. Association for Computational Linguistics. find
  17. Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Tim Lillicrap. 2017. A simple neural network module for relational reasoning. In Advances in neural information processing systems, pages 4974– 4983. find our card
  18. Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. Proceedings of the 53th Annual Meeting of the Association for Computational Linguistics (ACL 2015). find
  19. Ngoc Thang Vu, Heike Adel, Pankaj Gupta, and Hinrich Schütze. 2016. Combining recurrent and convolutional neural networks for relation classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology (N… find
  20. Linlin Wang, Zhu Cao, Gerard de Melo, and Zhiyuan Liu. 2016. Relation classification via multi-level attention CNNs. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016). find
  21. Kun Xu, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2015a. Semantic relation classification via convolutional neural networks with simple negative sampling. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015). find
  22. Yan Xu, Lili Mou, Ge Li, Yunchuan Chen, Hao Peng, and Zhi Jin. 2015b. Classifying relations via long short term memory networks along shortest dependency paths. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015), pages 1785–1794. find
  23. Mo Yu, Wenpeng Yin, Kazi Saidul Hasan, Cicero dos Santos, Bing Xiang, and Bowen Zhou. 2017. Improved neural relation detection for knowledge base question answering. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017). find
  24. Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. 2003. Kernel methods for relation extraction. Journal of machine learning research, 3:1083–1106. find
  25. Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, Jun Zhao, et al. 2014. Relation classification via convolutional deep neural network. In Proceedings of the 24th International Conference on Computational Linguistics (COLING 2014), pages 2335– 2344. find
  26. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017). find
  27. Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016), page 207. find our card