Elamur Research⌕
☰
home / papers / 2015 / Classifying Relations by Ranking with Convolutional Neural Networks

Classifying Relations by Ranking with Convolutional Neural Networks

Summary

The paper addresses relation classification: deciding which semantic relation holds between two marked nominals in a sentence. Earlier neural systems still relied on features from WordNet, dependency parsers or NER. The authors propose CR-CNN, a convolutional network that classifies by ranking. A convolutional layer with max pooling turns the sentence into a fixed-size vector. The network also learns one embedding per relation class, and each class is scored by the dot product between the sentence vector and that class embedding. Training uses a new pairwise logistic ranking loss. Each update pushes the correct class score above a positive margin and one wrong class score below a negative margin. The artificial class Other gets no embedding at all; a sentence is labelled Other when every real class scores below zero. On the SemEval-2010 Task 8 dataset, CR-CNN reaches F1 84.1 using only word embeddings and word position embeddings, beating earlier systems that depend on costly external resources. This shows that ranking with class embeddings, combined with special treatment of a catch-all class, can replace hand-engineered features.

Key points

  • Benchmark: SemEval-2010 Task 8, which involves labelling the relation between two tagged nominals (e.g., Component-Whole) with directional classes plus Other.
  • CR-CNN obtains F1 84.1. For comparison, the SVM of Rink and Harabagiu (2010), which uses many external resources, scores 82.2, and FCM (Yu et al., 2014), which uses dependency parse and NER, scores 83.0.
  • Loss settings: scaling factor γ = 2, margins m⁺ = 2.5 and m⁻ = 0.5, and L2 regularization with β = 0.001, optimized by SGD. The negative class used in each update is the highest-scoring incorrect class.
  • The authors report three further findings:
    • Ranking works better than a CNN with a softmax output.
    • Dropping the Other embedding improves both precision and recall.
    • Feeding only the words between the two nominals, with plain word embeddings, is nearly as good as using word position embeddings (WPE).
  • Input uses word embeddings, optionally concatenated with WPE (Zeng et al., 2014). WPE encode each word's relative distance to the two target nouns.

Interesting ideas

  • I1. Classes as learned vectors scored by dot product. Treating labels as embeddings in the same space as the input representation is a general recipe for any classifier. It also scales to very large label sets, because each update touches only two classes.
  • I2. Leaving out the catch-all class. Instead of modelling a heterogeneous "none/other" label, the model trains only the meaningful classes and falls back to Other when no class scores positively. This could help in NER, intent detection or document tagging, where a noisy residual class often dominates.
  • I3. Hardest-negative selection. At each step, the update is made against the wrong class that currently scores highest. This is cheap when there are few classes and focuses learning on the most confusing alternatives.
  • I4. Cropping the input to the span between the targets. Using only the text between the two entities removes noise and makes position features almost unnecessary. This is a simple preprocessing trick worth trying for other pairwise tasks.
  • I5. Tracing scores back to n-grams. Because max pooling links each dimension of the sentence vector to a specific window, one can sum each trigram's contribution to a class score. This yields interpretable, direction-sensitive cue phrases per relation, which is useful for error analysis or for building lexicons.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Named entity recognition · Dependency parsing

Cited by

115 papers of our library cite it (of the 140884 whose bibliographies we read); 9 of them have a page here, 106 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p15-1061

@inproceedings{santos2015classifying,
  title = {{Classifying Relations by Ranking with Convolutional Neural Networks}},
  author = {Santos, Cícero dos and Xiang, Bing and Zhou, Bowen},
  booktitle = {Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2015},
  doi = {10.3115/v1/p15-1061},
  url = {https://doi.org/10.3115/v1/p15-1061}
}

References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio. 2009. Learning deep architectures for ai. Foundations and Trends Machine Learning, 2(1):1–127. find
  2. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. 2010. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference. find
  3. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537. find our card
  4. Cı́cero Nogueira dos Santos and Maı́ra Gatti. 2014. Deep convolutional neural networks for sentiment analysis of short texts. In Proceedings of the 25th International Conference on Computational Linguistics (COLING), Dublin, Ireland. find our card
  5. Cı́cero Nogueira dos Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In Proceedings of the 31st International Conference on Machine Learning (ICML), JMLR: W&CP volume 32, Beijing, China. find
  6. Jianfeng Gao, Patrick Pantel, Michael Gamon, Xiaodong He, and Li Deng. 2014. Modeling interestingness with deep neural networks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  7. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó. Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the 5th Internationa… find our card
  8. Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In Proceedings of the Conference on Neural Information Processing Systems, pages 2042–2050. find
  9. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural netork for modelling sentences. In Proceedings of the 52th Annual Meeting of the Association for Computational Linguistics, pages 655–665, Baltimore, Maryland. find
  10. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods for Natural Language Processing, pages 1746–1751, Doha, Qatar. find our card
  11. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In In Proceedings of Workshop at ICLR. find
  12. Longhua Qian, Guodong Zhou, Fang Kong, and Qiaoming Zhu. 2009. Semi-supervised learning for semantic relation classification using stratified sampling strategy. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1437–1445. find
  13. Bryan Rink and Sanda Harabagiu. 2010. Utd: Classifying semantic relations by combining lexical and semantic resources. In Proceedings of International Workshop on Semantic Evaluation, pages 256–259. find
  14. Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1201–1211. find our card
  15. Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, pages 173–… find our card
  16. Jason Weston, Samy Bengio, and Nicolas Usunier. 2011. Wsabie: Scaling up to large vocabulary image annotation. In Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, pages 2764–2770. find
  17. Jason Weston, Sumit Chopra, and Keith Adams. 2014. \#tagspace: Semantic embeddings from hashtags. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1822–1827. find
  18. Mo Yu, Matthew Gormley, and Mark Dredze. 2014. Factor-based compositional embedding models. In Proceedings of the 2nd Workshop on Learning Semantics, Montreal, Canada. find
  19. Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of the 25th International Conference on Computational Linguistics (COLING), pages 2335–2344, Dublin, Ireland. find our card
  20. Zhu Zhang. 2004. Weakly-supervised relation classification for information extraction. In Proceedings of the ACM International Conference on Information and Knowledge Management, pages 581–588, New York, NY, USA. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/santos-2015-classifying-relations-ranking-convolutional-neural/