Elamur Research⌕
☰
home / papers / 2016 / Relation Classification via Multi-Level Attention CNNs

Relation Classification via Multi-Level Attention CNNs

Paper · 2016 · Linlin Wang, Zhu Cao, Gerard de Melo et al.

Summary

The paper addresses relation classification: given a sentence with two marked nominal entities, the system must decide which semantic relation connects them, for example Cause-Effect(e1,e2). Earlier systems typically depended on hand-engineered features, kernels, or dependency parsers. Parsers make errors and transfer poorly to new domains, genres or languages.

The authors propose a CNN trained end-to-end on labeled sentences without external linguistic tools. Each word is represented by its embedding plus two position embeddings encoding its distance to each entity, and these vectors are combined over a sliding window. A first attention layer, built on diagonal matrices, weights words by their relevance to the two entities. After convolution, a second attention-based pooling layer picks out the features most useful for each relation, using learned relation embeddings. Training relies on a new pairwise margin loss. It pulls the normalized network output toward the correct relation embedding and pushes it away from the highest-scoring wrong one.

On SemEval 2010 Task 8 the model reaches 88.0% F1. This beats prior methods that used parsers or lexical resources, which suggests learned attention can stand in for explicit syntactic structure in this task.

Key points

  • Benchmark and result: the model is evaluated on SemEval 2010 Task 8 and reports a state-of-the-art F1 of 88.0%. It outperforms models built on dependency paths, syntactic trees or NLP toolkits.
  • Two model variants:
    • Att-Input-CNN uses only entity-level input attention.
    • Att-Pooling-CNN adds relation-level attention pooling. Its training curve oscillates more, but it ends with slightly higher F1.
  • Input encoding: word vectors are concatenated with word position embeddings (WPE) giving the relative offset to e1 and e2. For example, "and" in the running example has offsets −1 and 6.
  • Objective: the loss uses a margin of 1, a hardest-negative choice of wrong class, and L2 regularization. It is presented as an alternative to the ranking loss of CR-CNN (dos Santos et al., 2015).
  • Analysis:
    • In the attention plots, "inside" gets the highest weight for a Component-Whole sentence.
    • The top trigrams per class are intuitive, e.g. "are caused by" for Cause-Effect(e2,e1).
    • Errors come from metaphorical, unseen phrases such as "revolves around" and from adjacent noun-noun pairs like "sesame oil".

Interesting ideas

  • I1. Entity-conditioned attention instead of a parser. Weighting each word by its affinity to the target entities supplies some of the structural signal a dependency parse would give. This is attractive wherever parsers are weak or unavailable, such as lower-resource languages or noisy domains.
  • I2. Hardest-negative margin loss over learned class embeddings. Each label gets its own vector, and the model is penalized only against the most confusable wrong label. This sharpens decision boundaries and could transfer to any multi-class problem with many similar classes.
  • I3. Two-stage attention, input then pooling. Combining "which words matter for these entities" with "which features matter for this class" is a general pattern. It could suit other pair-centric tasks, such as entity linking or matching trade records.
  • I4. Tracing predictions back to n-grams. Ranking the trigrams that contributed most to correct decisions yields human-readable class lexicons. These are useful for debugging models and for bootstrapping rule-based or annotation resources.
  • I5. Error analysis by construction type. Grouping failures into categories, such as metaphor or unseen predicates versus bare noun-noun compounds, shows where better embeddings rather than better architecture are needed. This kind of analysis is worth repeating when evaluating any relation extractor.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China137 works · 83,262 citations🔬State Key Laboratory of Intelligent Technology and SystemsBeijing · China21 works · 9,641 citations🎓Rutgers, The State University of New JerseyNew Brunswick · United States19 works · 11,379 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works

Cite

Cite as Linlin Wang, Zhu Cao, Gerard de Melo and Zhiyuan Liu. 2016. Relation Classification via Multi-Level Attention CNNs. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1123

@inproceedings{wang2016relation,
  title = {{Relation Classification via Multi-Level Attention CNNs}},
  author = {Linlin Wang, Zhu Cao, Gerard {de Melo} and Zhiyuan Liu},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1123},
  url = {https://doi.org/10.18653/v1/p16-1123}
}

References

38 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nguyen Bach and Sameer Badaskar. 2007. A review of relation extraction. Online at http://www.cs.cmu.edu/%7Enbach/papers/ A-survey-on-Relation-Extraction.pdf . ↗
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations (ICLR). our card
  3. Elizabeth Boschee, Ralph Weischedel, and Alex Zamanian. 2005. Automatic information extraction. In Proceedings of the 2005 International Conference on Intelligence Analysis, McLean, VA, pages 2–4. find
  4. Razvan C Bunescu and Raymond J Mooney. 2005. A shortest path dependency kernel for relation extraction. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 724–731. Association for Computational Linguistics. our card
  5. Kazuma Hashimoto, Makoto Miwa, Yoshimasa Tsuruoka, and Takashi Chikayama. 2013. Simple customization of recursive neural networks for semantic relation classification. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1372–1376, Seattle, Washington, USA… find
  6. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the 5th International… our card
  7. Nanda Kambhatla. 2004. Combining lexical, syntactic, and semantic features with maximum entropy models for extracting relations. In Proceedings of the 42nd Annual Meeting of the Association for Computation Linguistics, page 22. Association for Computational Linguistics. find
  8. Yang Liu, Furu Wei, Sujian Li, Heng Ji, Ming Zhou, and Houfeng Wang. 2015. A dependency-based neural network for relation classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processi… find
  9. Yee Seng Chan and Dan Roth. 2010. Exploiting background knowledge for relation extraction. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 152–160. Association for Computational Linguistics. find
  10. Fandong Meng, Zhengdong Lu, Mingxuan Wang, Hang Li, Wenbin Jiang, and Qun Liu. 2015. Encoding source language with convolutional neural network for machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7the International Joint Confere… find
  11. Jinxiu Chen, Donghong Ji, Chew Lim Tan, and Zhengyu Niu. 2005. Unsupervised feature selection for relation extraction. In Proceedings of IJCNLP. find
  12. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In ICLR Workshop. our card
  13. Jiaqiang Chen, Niket Tandon, and Gerard de Melo. 2015. Neural word representations from large-scale commonsense knowledge. In Proceedings of WI 2015. find
  14. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In HLT-NAACL, pages 746– 751. our card
  15. Cıcero Nogueira dos Santos, Bing Xiang, and Bowen Zhou. 2015. Classifying relations by ranking with convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, vol… our card
  16. Takaaki Hasegawa, Satoshi Sekine, and Ralph Grishman. 2004. Discovering relations among named entities from large corpora. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, page 415. Association for Computational Linguistics. our card
  17. Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using LSTMs on sequences and tree structures. arXiv preprint arXiv:1601.00770. our card
  18. Raymond J Mooney and Razvan C Bunescu. 2005. Subsequence kernels for relation extraction. In Advances in Neural Information Processing Systems, pages 171–178. find
  19. Thien Huu Nguyen and Ralph Grishman. 2014. Employing word representations and regularization for domain adaptation of relation extraction. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Short Papers), pages 68–74. find
  20. Thien Huu Nguyen and Ralph Grishman. 2015. Combining neural networks and log-linear models to improve relation extraction. arXiv preprint arXiv:1511.05926. — [arXiv:1511.05926](https://arxiv.org/abs/1511.05926) find
  21. Longhua Qian, Guodong Zhou, Fang Kong, Qiaoming Zhu, and Peide Qian. 2008. Exploiting constituent dependencies for tree kernel-based semantic relation extraction. In Proceedings of the 22nd International Conference on Computational Linguistics, volume 1, pages 697–704. Association for Computational… find
  22. Longhua Qian, Guodong Zhou, Fang Kong, and Qiaoming Zhu. 2009. Semi-supervised learning for semantic relation classification using stratified sampling strategy. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing. find
  23. Bryan Rink and Sanda Harabagiu. 2010. Utd: Classifying semantic relations by combining lexical and semantic resources. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 256–259. Association for Computational Linguistics. find
  24. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of EMNLP 2015. find
  25. Cicero dos Santos, Ming Tan, Bing Xiang, and Bowen Zhou. 2016. Attentive pooling networks. arXiv preprint arXiv:1602.03609. find
  26. Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1201–1211. our card
  27. Fabian M Suchanek, Georgiana Ifrim, and Gerhard Weikum. 2006. Combining linguistic and statistical analysis to extract relations from web documents. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 712–717. ACM. find
  28. Yan Xu, Lili Mou, Ge Li, Yunchuan Chen, Hao Peng, and Zhi Jin. 2015b. Classifying relations via long short term memory networks along shortest dependency paths. In Proceedings of Conference on Empirical Methods in Natural Language Processing (to appear). our card
  29. Yan Xu, Ran Jia, Lili Mou, Ge Li, Yunchuan Chen, Yangyang Lu, and Zhi Jin. 2016. Improved relation classification by deep recurrent neural networks with data augmentation. arXiv preprint arXiv:1601.03651. find
  30. Wenpeng Yin, Hinrich Schütze, Bing Xiang, and Bowen Zhou. 2015. ABCNN: attention-based convolutional neural network for modeling sentence pairs. arXiv preprint arXiv:1512.05193. our card
  31. Mo Yu, Matthew Gormley, and Mark Dredze. 2014. Factor-based compositional embedding models. In NIPS Workshop on Learning Semantics. find
  32. Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, Jun Zhao, et al. 2014. Relation classification via convolutional deep neural network. In COLING, pages 2335–2344. our card
  33. Shu Zhang, Dequan Zheng, Xinchen Hu, and Ming Yang. 2015. Bidirectional long short-term memory networks for relation classification. In Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation pages, pages 73–78. find
  34. Zhu Zhang. 2004. Weakly-supervised relation classification for information extraction. In In Proceedings of the Thirteenth ACM International Conference on Information and Knowledge Management. find
  35. Guodong Zhou, Su Jian, Zhang Jie, and Zhang Min. 2005. Exploring various knowledge in relation extraction. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pages 427–434. Association for Computational Linguistics. our card
  36. Niket Tandon, Gerard de Melo, and Gerhard Weikum. 2011. Deriving a Web-scale common sense fact database. In Proceedings of the Twenty-fifth AAAI Conference on Artificial Intelligence (AAAI 2011), pages 152–157, Palo Alto, CA, USA. AAAI Press. find
  37. Jason Weston, Samy Bengio, and Nicolas Usunier. 2011. Wsabie: Scaling up to large vocabulary image annotation. In Proceedings of IJCAI, volume 11, pages 2764–2770. find
  38. Kun Xu, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2015a. Semantic relation classification via convolutional neural networks with simple negative sampling. Proceedings of EMNLP 2015. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2016-relation-classification-multi-level-attention/