Elamur Research⌕
☰
home / papers / 2019 / Attention Guided Graph Convolutional Networks for Relation Extraction

Attention Guided Graph Convolutional Networks for Relation Extraction

Summary

The paper addresses relation extraction, which means identifying how entities in text are related. Dependency trees help with this task, but earlier systems relied on fixed pruning rules, such as keeping only the shortest dependency path, the LCA subtree, or tokens within distance K of that path. These rules can discard words that matter. The authors propose Attention Guided Graph Convolutional Networks (AGGCNs), which take the complete dependency tree as input. Multi-head self-attention turns the tree into several fully connected graphs with learned edge weights. This acts as a learned "soft pruning" that decides how much each connection matters. Dense connections make deep GCN stacks trainable, so the model can capture both nearby and distant dependencies. On cross-sentence n-ary relation extraction, AGGCN beats the previous best accuracy on multi-class ternary and binary settings by about 8% and 6%. It also outperforms earlier models on the large sentence-level TACRED benchmark. The work suggests that learning to weight syntactic structure is more effective than hand-crafted pruning, and it adds no computational overhead compared with earlier GCNs.

Key points

  • Architecture: AGGCN stacks M identical blocks. Each block has an attention guided layer, which builds N weighted adjacency matrices via multi-head attention. It also has N densely connected GCN layers and a linear combination layer. The original tree adjacency serves as initialization, and attention is applied from the second block onward.
  • Efficiency: The attention-derived matrices are the same n × n size as the original adjacency matrix. As a result, the model costs no more to compute than previous GCNs and, unlike Tree-LSTM, can run in parallel.
  • Pruning experiments: C-AGGCN outperforms the prior state-of-the-art C-GCN (Zhang et al., 2018) even when given pruned trees with various K values. Its advantage over C-GCN grows with sentence length.
  • Data efficiency: Trained on only 80% of the training data, C-AGGCN reaches 66.5 F1, which beats C-GCN trained on the full set.
  • Ablation and code: Removing the attention guided or densely connected layers hurts results, and the attention layer matters more. Dropping the feed-forward layer lowers F1 to 67.8. The code is PyTorch-based and released at statnlp.org.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

75 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 74 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1024

@inproceedings{guo2019attention,
  title = {{Attention Guided Graph Convolutional Networks for Relation Extraction}},
  author = {Guo, Zhijiang and Zhang, Yan and Lu, Wei},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1024},
  url = {https://doi.org/10.18653/v1/p19-1024}
}

References

43 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ryan T. McDonald, Fernando Pereira, Seth Kulick, R. Scott Winters, Yang Jin, and Peter S. White. 2005. Simple algorithms for complex relation extraction with applications to biomedical ie. In Proc. of ACL. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR. find our card
  3. Joost Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Sima’an. 2017. Graph convolutional encoders for syntax-aware neural machine translation. In Proc. of EMNLP. find our card
  4. Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proc. of ACL. find
  5. Joan Bruna. 2014. Spectral networks and deep locally connected networks on graphs. In Proc. of ICLR. find
  6. Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using lstms on sequences and tree structures. In Proc. of ACL. find our card
  7. Razvan C. Bunescu and Raymond J. Mooney. 2005. A shortest path dependency kernel for relation extraction. In Proc. of EMNLP. find our card
  8. Thien Huu Nguyen and Ralph Grishman. 2015. Relation extraction: Perspective from convolutional neural networks. In Proc. of VS@NAACL-HLT. find our card
  9. Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long short-term memory-networks for machine reading. In Proc. of EMNLP. find our card
  10. Adam Paszke, Sam Gross, and Adam Lerer. 2017. Automatic differentiation in pytorch. In Proc. of workshop on NeurIPS. find
  11. Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In Proc. of NeurIPS. find our card
  12. Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen tau Yih. 2017. Cross-sentence n-ary relation extraction with graph lstms. Transactions of the Association for Computational Linguistics, 5:101–115. find our card
  13. Michele Gori, Gabriele Monfardini, and Franco Scarselli. 2005. A new model for learning in graph domains. In Proc. of IJCNN. find
  14. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proc. of EMNLP. find
  15. Zhijiang Guo, Yan Zhang, Zhiyang Teng, and Wei Lu. 2019. Densely connected graph convolutional networks for graph-to-sequence learning. Transactions of the Association of Computational Linguistics. find
  16. Chris Quirk and Hoifung Poon. 2017. Distant supervision for relation extraction beyond the sentence boundary. In Proc. of EACL. find
  17. Mikael Henaff, Joan Bruna, and Yann LeCun. 2015. Deep convolutional networks on graph-structured data. arXiv preprint. find
  18. Bryan Rink and Sanda M. Harabagiu. 2010. Utd: Classifying semantic relations by combining lexical and semantic resources. In SemEval@ACL. find
  19. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In SemEval@ACL. find our card
  20. Adam Santoro, David Raposo, David G. T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter W. Battaglia, and Timothy P. Lillicrap. 2017. A simple neural network module for relational reasoning. In Proc. of NeurIPS. find our card
  21. Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018a. A graph-to-sequence model for amr-to-text generation. In Proc. of ACL. find our card
  22. Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. 2017. Densely connected convolutional networks. In Proc. of CVPR. find
  23. Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018b. N-ary relation extraction using graph state lstm. In Proc. of EMNLP. find
  24. Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proc. of ICLR. find our card
  25. Kenton Lee, Luheng He, Mike Lewis, and Luke S. Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proc. of EMNLP. find our card
  26. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proc. of ACL. find
  27. Yang Liu, Furu Wei, Sujian Li, Heng Ji, Ming Zhou, and Houfeng Wang. 2015. A dependency-based neural network for relation classification. In Proc. of ACL. find
  28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. of NeurIPS. find
  29. Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In Proc. of EMNLP. find our card
  30. Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In Proc. of ICLR. find
  31. Patrick Verga, Emma Strubell, and Andrew McCallum. 2018. Simultaneously self-attending to all mentions for full-abstract biological relation extraction. In Proc. of NAACL-HLT. find
  32. Ngoc Thang Vu, Heike Adel, Pankaj Gupta, and Hinrich Schütze. 2016. Combining recurrent and convolutional neural networks for relation classification. In Proc. of NAACL-HLT. find
  33. Linlin Wang, Zhu Cao, Gerard de Melo, and Zhiyuan Liu. 2016. Relation classification via multi-level attention cnns. In Proc. of ACL. find our card
  34. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015a. Show, attend and tell: Neural image caption generation with visual attention. In Proc. of ICML. find our card
  35. Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken ichi Kawarabayashi, and Stefanie Jegelka. 2018. Representation learning on graphs with jumping knowledge networks. In Proc. of ICML. find
  36. Kun Xu, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2015b. Semantic relation classification via convolutional neural networks with simple negative sampling. In Proc. of EMNLP. find
  37. Yan Xu, Lili Mou, Ge Li, Yunchuan Chen, Hao Peng, and Zhi Jin. 2015c. Classifying relations via long short term memory networks along shortest dependency paths. In Proc. of EMNLP. find our card
  38. Mo Yu, Wenpeng Yin, Kazi Saidul Hasan, Cı́cero Nogueira dos Santos, Bing Xiang, and Bowen Zhou. 2017. Improved neural relation detection for knowledge base question answering. In Proc. of ACL. find
  39. Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. 2002. Kernel methods for relation extraction. In Proc. of EMNLP. find our card
  40. Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jian Zhao. 2014. Relation classification via convolutional deep neural network. In Proc. of COL-ING. find our card
  41. Yuhao Zhang, Peng Qi, and Christopher D. Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. In Proc. of EMNLP. find our card
  42. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proc. of EMNLP. find
  43. Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In Proc. of ACL. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/guo-2019-attention-guided-graph-convolutional-networks/