Elamur Research⌕
☰
home / papers / 2015 / Relation Extraction: Perspective from Convolutional Neural Networks

Relation Extraction: Perspective from Convolutional Neural Networks

Summary

The paper addresses supervised relation extraction (RE): deciding whether two marked entity mentions in a sentence are related and, if so, which relation holds. This is harder than relation classification (RC) because real corpora contain far more non-relation pairs than relation pairs. Traditional feature-based and kernel-based systems depend on parsers, taggers and hand-built resources, and errors from these tools carry into the relation decisions. The authors propose a convolutional neural network that takes only three inputs: the unprocessed sentence, the locations of the head words of both entities, and pre-trained word embeddings. Its design choices are:

  • position embeddings that encode each word's distance to both entities,
  • convolution filters of several window sizes,
  • max pooling,
  • a softmax classifier with dropout.

Word embeddings start from word2vec and are fine-tuned during training. On ACE 2005, the CNN beats lexical baselines in a realistic setting with no gold entity types. On SemEval 2010 RC, it beats prior systems that also avoid external features and matches feature-rich ones. The study offers a resource-light baseline for neural RE.

Key points

  • Architecture: a look-up layer (word embeddings concatenated with two position embeddings), a convolution layer with multiple filter widths, max pooling, and a fully connected softmax layer with dropout. Inputs are trimmed or padded to a fixed length.
  • SemEval 2010 RC (macro-F1): the CNN scores 82.8 with no supervised or manual features.
    • It beats the best comparable feature-free system, FCM (80.6), by 2.2 points.
    • It is on par with feature-enriched systems: FCM 83.0, O-CNN 82.7, MVRNN 82.4, SVM 82.2.
  • Traditional ACE systems: they can exceed 72% only when given hand-labelled entity and mention types. Without those, reported results on ACE 2005 drop below 60%.
  • Class balance on ACE 2005: a 5-fold cross-validation experiment varies the ratio of positive examples to "Other" examples.
    • All models improve as the ratio grows, gaining about 20% at the most balanced setting.
    • The CNN stays ahead of the Words and Words-HM-Wed baselines at every ratio.
  • Gains over O-CNN: the authors credit multiple window sizes, trainable position embeddings and, possibly, the word2vec embeddings.

Interesting ideas

  • I1. Relative-distance embeddings: the model learns one vector for each word's offset to each target entity. This tells the network which tokens are "the arguments" without any parser. The same trick fits any task that asks about specific spans in a sentence, such as event arguments or aspect sentiment.
  • I2. Mixing filter widths: several window sizes in a single convolution layer let the model capture short and long n-gram cues together. It is a cheap gain for any text CNN.
  • I3. Non-static embeddings: initialising from general pre-trained vectors and then fine-tuning them adapts the representation to the task. This is useful for low-resource languages, where unsupervised embeddings may be the only available external resource.
  • I4. Imbalance as a controlled variable: sweeping the positive-to-negative ratio shows how much difficulty comes from class skew alone. It is a good diagnostic for any detection task where a "none" class dominates.
  • I5. Separating RE from RC: the paper frames RC as the easy special case of RE with perfect detection. It also warns that gold entity types inflate scores. Both points argue for honest, realistic evaluation settings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

104 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 96 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w15-1506

@inproceedings{nguyen2015relation,
  title = {{Relation Extraction: Perspective from Convolutional Neural Networks}},
  author = {Nguyen, Thien Huu and Grishman, Ralph},
  booktitle = {Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing},
  year = {2015},
  doi = {10.3115/v1/w15-1506},
  url = {https://doi.org/10.3115/v1/w15-1506}
}

References

52 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. 2001. A Neural Probabilistic Language Model. In Advances in Neural Information Processing Systems 13 (NIPS’00), pages 932938, MIT Press, 2001. find
  2. John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain Adaptation with Structural Correspondence Learning. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, Sydney, Australia. find our card
  3. Elizabeth Boschee, Ralph Weischedel, and Alex Zamanian. 2005. Automatic Information Extraction. In Proceedings of the International Conference on Intelligence Analysis. find
  4. Razvan C. Bunescu and Raymond J. Mooney. 2005a. A Shortest Path Dependency Kenrel for Relation Extraction. In Proceedings of HLT/EMNLP. find
  5. Razvan C. Bunescu and Raymond J. Mooney. 2005b. Subsequence Kernels for Relation Extraction. In Proceedings of NIPS. find
  6. Yee S. Chan and Dan Roth. 2010. Exploiting Background Knowledge for Relation Extraction. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 152-160, Beijing, China, August. find
  7. Ronan Collobert and Jason Weston. 2008. A Unied Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. In International Conference on Machine Learning, ICML, 2008. find
  8. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu and Pavel Kuksa. 2011. Natural Language Processing (Almost) from Scratch. Journal of Machine Learning Research 12:24932537. find our card
  9. Aron Culotta and Jeffrey Sorensen. 2004. Dependency Tree Kernels for Relation Extraction. In Proceedings of ACL 2004. find our card
  10. Hal Daumé III. 2007. Frustratingly Easy Domain Adaptation. In Proceedings of the ACL, pages 256-263, Prague, Czech Republic, June 2007. find our card
  11. Ralph Grishman, David Westbrook and Adam Meyers. 2005. find
  12. NYUs English ACE 2005 System Description. find
  13. ACE 2005 Evaluation Workshop. find
  14. Jing Jiang and ChengXiang Zhai. 2007. A Systematic Exploration of the Feature Space for Relation Extraction. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT’07), pages 113-120, 2007. find
  15. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, Stan Szpakowicz 2010. SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals. find our card
  16. In Proceedings of the 5th International Workshop on Semantic Evaluation, SemEval 2010. find
  17. Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2012. Improving Neural Networks by Preventing Co-Adaptation of Feature Detectors. CoRR, abs/1207.0580. find
  18. Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel Weld. 2011. Knowledge-based Weak Supervision for Information Extraction of Overlapping Relations. In Proceedings of ACL 2011. find
  19. Nal Kalchbrenner, Edward Grefenstette and Phil Blunsom. 2014. A Convolutional Neural Network for Modelling Sentences. In Proceedings of ACL 2014. find our card
  20. Nanda Kambhatla. 2004. Combining Lexical, Syntactic, and Semantic Features with Maximum Entropy Models for Information Extraction. In Proceedings of ACL 2004. find
  21. Yoon Kim. 2014. Convolutional Neural Networks for Sentence Classification. In Proceedings of EMNLP 2014. find our card
  22. Yann LeCun, Léon Bottou, Yoshua Bengio and Patrick Haffner. 1988. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, 86(11):22782324, November, 1988. find
  23. ChunYang Liu, WenBo Sun, WenHan Chao, and WanXiang Che. 2013. Convolution Neural Network for Relation Extraction. In Proceedings of 9th International Conference on Advanced Data Mining and Applications, Part II (ADMA 2013), Hangzhou, China, December, 2013. find
  24. David McClosky, Eugene Charniak, and Mark Johnson. 2010. Automatic Domain Adaptation for Parsing. In Proceedings of Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 28-36, Los Angeles, California, June 2010. find
  25. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of NIPS, 2013. find
  26. Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant Supervision for Relation Extraction without Labeled Data. In Proceedings of ACL, 2009. find
  27. Andriy Mnih and Geoffrey Hinton. 2007. Three new Graphical Models for Statistical Language Modelling. In Proceedings of ICML’07, pages 641-648, Corvallis, OR, 2007. find
  28. Andriy Mnih and Geoffrey Hinton. 2009. A Scalable Hierarchical Distributed Language Model. In NIPS, page 1081-1088. find
  29. Mohit Iyyer, Jordan Boyd-Graber, Leonardo Claudino, Richard Socher and Hal Daumé III. 2014. A Neural Network for Factoid Question Answering over Paragraphs. In Proceedings of EMNLP 2014. find
  30. Thien Huu Nguyen and Ralph Grishman. 2014. Employing Word Representations and Regularization for Domain Adaptation of Relation Extraction. In Proceedings of ACL 2014, pages 68-74, Baltimore, Maryland, USA. find
  31. Truc-Vien T. Nguyen, Alessandro Moschitti, and Giuseppe Riccardi. 2009. Convolution Kernels on Constituent, Dependency and Sequential Structures for Relation Extraction. In Proceedings of EMNLP 09, pages 1378-1387, Stroudsburg, PA, USA. find
  32. Barbara Plank and Alessandro Moschitti. 2013. Embedding Semantic Similarity in Tree Kernels for Domain Adaptation of Relation Extraction. In Proceedings of the ACL 2013, pages 1498-1507, Sofia, Bulgaria. find
  33. Longhua Qian, Guodong Zhou, Qiaoming Zhu and Peide Qian. 2008. Exploiting Constituent Dependencies for Tree Kernel-based Semantic Relation Extraction. In Proceedings of COLING, pages 697-704, Manchester. find
  34. Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling Relations and their Mentions without Labeled Text. In Proceedings of ECML PKDD, 2010. find
  35. Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng and Grégoire Mesnil. 2014. Learning Semantic Representations Using Convolutional Neural Networks for Web Search. In Proceedings of WWW 2014. find
  36. Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. 2011. Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. In Proceedings of NIPS 2011. find
  37. Richard Socher, Brody Huval, Christopher D. Manning and Andrew Y. Ng. 2012. Semantic Compositionality through Recursive Matrix-Vector Spaces. In Proceedings of EMNLP 2012. find our card
  38. Richard Socher, John Bauer, Christopher D. Manning and Andrew Y. Ng. 2013. Parsing with Compositional Vector Grammars. In Proceedings of ACL 2013. find
  39. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Chris Manning, Andrew Ng and Chris Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Proceedings of EMNLP 2013. find
  40. Richard Socher, Danqi Chen, Christopher D. Manning, Andrew Y. Ng. 2013. Reasoning With Neural Tensor Networks for Knowledge Base Completion. In Proceedings of NIPS 2013. find
  41. Ang Sun, Ralph Grishman, and Satoshi Sekine. 2011. Semi-supervised Relation Extraction with Large-scale Word Clustering. In Proceedings of ACL-HLT, pages 521-529, Portland, Oregon, USA. find
  42. Le Sun and Xianpei Han 2014. A Feature-Enriched Tree Kernel for Relation Extraction. In Proceedings of ACL 2014, pages 61-67, Baltimore, Maryland, USA. find
  43. Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati and Christopher D. Manning 2012. Multi-instance Multi-label Learning for Relation Extraction. In Proceedings of EMNLP-CoNLL 2012. find
  44. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL’10), pages 384-394, Uppsala, Sweden, July, 2010. find our card
  45. Wen-tau Yih, Xiaodong He, and Christopher Meek. 2014. Semantic Parsing for Single-Relation Question Answering. In Proceddings of ACL 2014. find our card
  46. Mo Yu, Matthew R. Gormley, and Mark Dredze. 2014. Factor-based Compositional Embedding Models. In the NIPS Learning Semantics Workshop 2014. find
  47. Matthew D. Zeiler. 2012. ADADELTA: An Adaptive Learning Rate Method. CoRR, abs/1212.5701. find our card
  48. Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. 2003. Kernel Methods for Relation Extraction. Journal of Machine Learning Research, 3:10831106. find our card
  49. Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou and Jun Zhao. 2014. Relation Classification via Convolutional Deep Neural Network. In Proceedings of COL-ING 2014. find our card
  50. Min Zhang, Jie Zhang, Jian Su, and GuoDong Zhou. 2006. A Composite Kernel to Extract Relations between Entities with both Flat and Structured Features. In Proceedings of COLING-ACL-06, pages 825-832, Sydney. find
  51. Guodong Zhou, Jian Su, Jie Zhang, and Min Zhang. 2005. Exploring various Knowledge in Relation Extraction. In Proceedings of ACL’05, pages 427-434, Ann Arbor, USA, 2005. find our card
  52. Guodong Zhou, Min Zhang, DongHong Ji, and QiaoMing Zhu. 2007. Tree Kernel-based Relation Extraction with Context-sensitive Structured Parse Tree Information. In Proceedings of EMNLP-CoNLL-07, pages 728-736, Prague. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nguyen-2015-relation-extraction-perspective-convolutional-neural/