Elamur Research⌕
☰
home / papers / 2015 / Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Paths

Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Paths

Summary

The paper addresses relation classification: deciding which predefined semantic relation holds between two marked entities in a sentence. The authors propose SDP-LSTM, a neural model that ignores most of the sentence. It reads only the shortest dependency path (SDP) linking the two entities, obtained with the Stanford parser. The path is split at the entities' common ancestor into two sub-paths, and each is processed bottom-up by an LSTM-based recurrent network, so the model is sensitive to relation direction. Four separate channels run along the path: word embeddings, POS tags, grammatical relations and WordNet hypernyms. Each channel is max-pooled, the results are concatenated, and the combined vector goes through a hidden layer and a softmax. A custom dropout scheme limits overfitting. On SemEval 2010 Task 8 the model reaches an F1 of 83.7%. This beats the competing systems compared under a plain softmax/cross-entropy setup without special handling of the Other class. The work shows that syntactic structure combined with recurrent networks can beat convolutional and recursive neural baselines.

Key points

  • Benchmark result: On SemEval 2010 Task 8, SDP-LSTM scores 83.7% F1, compared with:
    • FCM: 83.0%
    • CR-CNN without its Other-class treatment: 82.7% (84.1% with it)
    • MVRNN: 82.4%
    • Feature-rich SVM: 82.2%
  • Controlled experiments:
    • Replacing LSTM units with a plain RNN gives 82.8%.
    • Running one LSTM over the whole path instead of two sub-paths gives 82.2%.
  • Channel ablation:
    • Words alone: 82.35%
    • Words + POS: 82.98%
    • Words + grammatical relations: 83.21%
    • Words + WordNet: 83.03%
    • All four channels: 83.70%
    • The gains from individual channels do not simply add up when combined.
  • Channel resources:
    • Grammatical relations are grouped into 19 coarse classes, following De Marneffe et al. (2006).
    • Hypernyms are 41 WordNet supersense-style concepts (e.g., noun.food), assigned with the Ciaramita and Altun (2006) tagger.
  • Dropout analysis:
    • Dropout on LSTM units hurts even at a rate of 0.1.
    • Dropout on embeddings adds 2.16%.
    • Dropout on the penultimate layer adds a further 0.16%.

Interesting ideas

  • I1. Prune the input to the syntactic path between the entities. Throwing away words off the dependency path acts as a cheap, linguistically motivated denoising step. It could help for other languages, such as Ukrainian, wherever a UD-style parser is available.
  • I2. Split the path at the common ancestor. Running separate encoders on each side of the ancestor node builds the asymmetry of r(a, b) versus r(b, a) into the architecture. This is useful for any directed pairwise task, such as trade-flow or part-of relations.
  • I3. Keep heterogeneous features in separate channels. Words, tags, dependency labels and coarse semantic classes each get their own recurrent network and are fused only after pooling. This makes ablation simple and lets weak signals such as POS still contribute.
  • I4. Use coarse semantic classes as a bridge. Mapping words to a small set of hypernym categories links rare or unseen words to conceptually similar ones. This is a lightweight trick for low-resource vocabularies.
  • I5. Choose dropout placement deliberately. Applying dropout to embeddings and the penultimate layer, but not to recurrent units, worked best here. Checking each placement separately is a good habit when regularizing LSTMs elsewhere.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Part-of-speech tagging

Cited by

116 papers of our library cite it (of the 140884 whose bibliographies we read); 12 of them have a page here, 104 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1206

@inproceedings{xu2015classifying,
  title = {{Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Paths}},
  author = {Xu, Yan and Mou, Lili and Li, Ge and Chen, Yunchuan and Peng, Hao and Jin, Zhi},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1206},
  url = {https://doi.org/10.18653/v1/d15-1206}
}

References

34 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michele Banko, Michael J. Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. 2007. Open information extraction for the web. In Proceedings of twentieth International Joint Conference on Artificial Intelligence, volume 7, pages 2670– 2676. find
  2. Yoshua Bengio, Aaron Courville, and Pierre Vincent. 2013. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828. find our card
  3. Razvan C. Bunescu and Raymond J. Mooney. 2005. A shortest path dependency kernel for relation extraction. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 724–731. find our card
  4. Razvan Bunescu and Raymond Mooney. 2007. Learning to extract relations from the web using minimal supervision. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, volume 45, pages 576–583. find
  5. Yun-Nung Chen, Dilek Hakkani-Tur, and Gokan Tur. 2014. Deriving local relational surface forms from dependency-based entity embeddings for unsupervised spoken language understanding. In Spoken Language Technology Workshop (SLT), 2014 IEEE, pages 242–247. find
  6. Massimiliano Ciaramita and Yasemin Altun. 2006. Broad-coverage sense disambiguation and information extraction with a supersense sequence tagger. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 594–602. find our card
  7. Marie-Catherine De Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of the International Conference on Language Resources and Evaluation, volume 6, pages 449–454. find our card
  8. Cıcero Nogueira dos Santos, Bing Xiang, and Bowen Zhou. 2015. Classifying relations by ranking with convolutional neural networks. In Proceedings of 53rd Annual Meeting of the Association for Computational Linguistics, pages 626–634. find our card
  9. Javid Ebrahimi and Dejing Dou. 2015. Chain based rnn for relation classification. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1244–1249. find
  10. Katrin Fundel, Robert Küffner, and Ralf Zimmer. 2007. Relexłrelation extraction using dependency parse trees. Bioinformatics, 23(3):365–371. find
  11. Zhou GuoDong, Su Jian, Zhang Jie, and Zhang Min. 2005. Exploring various knowledge in relation extraction. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 427–434. find our card
  12. Kazuma Hashimoto, Makoto Miwa, Yoshimasa Tsuruoka, and Takashi Chikayama. 2013. Simple customization of recursive neural networks for semantic relation classification. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1372–1376. find
  13. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2009. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the Workshop on Seman… find our card
  14. Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580. find
  15. Sepp Hochreiter. 1998. The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 6(02):107–116. find
  16. Nanda Kambhatla. 2004. Combining lexical, syntactic, and semantic features with maximum entropy models for extracting relations. In Proceedings of the ACL 2004 on Interactive Poster and Demonstration Sessions, page 22. Association for Computational Linguistics. find
  17. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013a. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find
  18. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751. find our card
  19. Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2… find
  20. Lili Mou, Hao Peng, Ge Li, Yan Xu, Lu Zhang, and Zhi Jin. 2015. Discriminative neural sentence modeling by tree-based convolution. arXiv preprint arXiv:1504.01106. find
  21. Barbara Plank and Alessandro Moschitti. 2013. Embedding semantic similarity in tree kernels for domain adaptation of relation extraction. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1498–1507. find
  22. Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 151–161. find our card
  23. Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1201–1211. find our card
  24. Richard Socher, Andrej Karpathy, Quoc V. Le, Christopher D. Manning, and Andrew Y. Ng. 2014. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics, 2:207–218. find our card
  25. Chang Wang and James Fan. 2014. Medical relation extraction with manifold models. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 828–838. find
  26. Mengqiu Wang. 2008. A re-examination of dependency path kernels for relation extraction. In Proceedings of the Third International Joint Conference on Natural Language Processing, pages 841–846. find
  27. Fei Wu and Daniel S. Weld. 2010. Open information extraction using wikipedia. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 118–127. find our card
  28. Yan Xu, Ge Li, Lili Mou, and Yangyang Lu. 2014. Learning non-taxonomic relations on demand for ontology extension. International Journal of Software Engineering and Knowledge Engineering, 24(08):1159–1175. find
  29. Xuchen Yao and Benjamin Van Durme. 2014. Information extraction over structured data: Question answering with freebase. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 956–966. find
  30. Mo Yu, Matthew Gormley, and Mark Dredze. 2014. Factor-based compositional embedding models. In NIPS Workshop on Learning Semantics. find
  31. Wojciech Zaremba and Ilya Sutskever. 2014. Learning to execute. arXiv preprint arXiv:1410.4615. find
  32. Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. 2003. Kernel methods for relation extraction. The Journal of Machine Learning Research, 3:1083–1106. find our card
  33. Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2335–2344. find our card
  34. Xiaodan Zhu, Parinaz Sobhani, and Hongyu Guo. 2015. Long short-term memory over tree structures. In Proceedings of The 32nd International Conference on Machine Learning, pages 1604–1612. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2015-classifying-relations-long-short-term/