Elamur Research
☰
home / papers / 2016 / End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures

End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P16-1105.pdf

Summary

The paper addresses end-to-end relation extraction: finding entity mentions and the semantic relations between them in one model, instead of running NER and relation classification as separate pipeline steps. Earlier joint systems depended on hand-crafted features, and LSTM-based relation classifiers had trailed CNNs. The authors build a neural model with two stacked parts. A bidirectional sequential LSTM-RNN over word and POS embeddings tags entities greedily with BILOU labels. On top of it, a bidirectional (bottom-up and top-down) tree-structured LSTM-RNN encodes the dependency path between each pair of detected entities and classifies their relation. The lower layers are shared, so entity and relation labels update the parameters together. Entity pretraining and scheduled sampling compensate for weak entity predictions early in training. On ACE2005 and ACE2004 the model outperforms the strongest feature-based joint system, and on SemEval-2010 Task 8 it matches leading CNN models. The work shows that a neural joint model combining linear and syntactic context can replace feature engineering for this task.

Key points

  • Compared with the previous best feature-based end-to-end system, the model cuts F1 error by 12.1% (relative) on ACE2005 and by 5.7% on ACE2004.
  • On the SemEval-2010 Task 8 test set, the SPTree variant reaches 0.844 Macro-F1 without external resources, against 0.841 for the CNN of dos Santos et al. (2015). With WordNet hypernym embeddings it reaches 0.855, against 0.856 for Xu et al. (2015a).
  • Restricting the tree input to the shortest dependency path (SPTree, 0.851 dev Macro-F1) works better than using the full tree (FullTree, 0.829). The choice of input structure matters more than the type of LSTM cell (Child-Sum, SPSeq, SPXu all perform close to SPTree).
  • In the ablations, removing both the sequence layer and the entity-pair information drops dev Macro-F1 to 0.827. Swapping the Stanford neural dependency parser for the Stanford PCFG parser changes results only slightly.
  • The gains in relation extraction accuracy are attributed to parameter sharing together with label embeddings, scheduled sampling, and pretraining of the entity component.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

274 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 261 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1105 · arXiv 1601.00770

@inproceedings{miwa2016end,
  title = {{End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures}},
  author = {Miwa, Makoto and Bansal, Mohit},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1105},
  eprint = {1601.00770},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p16-1105}
}

References

39 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. arXiv preprint arXiv:1506.03099. find
  2. Razvan C Bunescu and Raymond Mooney. 2005. A shortest path dependency kernel for relation extraction. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 724–731. ACL. find
  3. Yee Seng Chan and Dan Roth. 2011. Exploiting syntactico-semantic structures for relation extraction. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 551–560, Portland, Oregon, USA, June. ACL. find
  4. Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 740–750, Doha, Qatar, October. ACL. find our card
  5. Massimiliano Ciaramita and Yasemin Altun. 2006. Broad-coverage sense disambiguation and information extraction with a supersense sequence tagger. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 594–602, Sydney, Australia, July. ACL. find
  6. George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The automatic content extraction (ace) program – tasks, data, and evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC-2004), Lis… find
  7. Cicero dos Santos, Bing Xiang, and Bowen Zhou. 2015. Classifying relations by ranking with convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Lo… find
  8. James Hammerton. 2001. Clause identification with long short-term memory. In Proceedings of the 2001 workshop on Computational Natural Language Learning-Volume 7, page 22. ACL. find
  9. James Hammerton. 2003. Named entity recognition with long short-term memory. In Walter Daelemans and Miles Osborne, editors, Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 172–175. ACL. find
  10. Kazuma Hashimoto, Pontus Stenetorp, Makoto Miwa, and Yoshimasa Tsuruoka. 2015. Task-oriented learning of word embeddings for semantic relation classification. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning, pages 268–278, Beijing, China, July. ACL. find
  11. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the 5th International… find our card
  12. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991. find our card
  13. Rohit J. Kate and Raymond Mooney. 2010. Joint entity and relation extraction using card-pyramid parsing. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 203–212, Uppsala, Sweden, July. ACL. find
  14. Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, 18(5):602–610. find
  15. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR 2015, San Diego, CA, May. find
  16. Alan Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pages 6645– 6649. IEEE. find
  17. Qi Li and Heng Ji. 2014. Incremental joint extraction of entity mentions and relations. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 402–412, Baltimore, Maryland, June. ACL. find
  18. Jiwei Li, Thang Luong, Dan Jurafsky, and Eduard Hovy. 2015. When are tree structures necessary for deep learning of representations? In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2304–2314, Lisbon, Portugal, September. ACL. find
  19. Wei Lu and Dan Roth. 2015. Joint mention extraction and classification with mention hypergraphs. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 857–867, Lisbon, Portugal, September. ACL. find
  20. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. find
  21. Makoto Miwa and Yutaka Sasaki. 2014. Modeling joint entity and relation extraction with table representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1858– 1869, Doha, Qatar, October. ACL. find
  22. David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Lingvisticae Investigationes, 30(1):3–26. find
  23. Eric W. Noreen. 1989. Computer-Intensive Methods for Testing Hypotheses : An Introduction. Wiley-Interscience, April. find
  24. Romain Paulus, Richard Socher, and Christopher D Manning. 2014. Global belief recursive neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N.D. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 2888–2896. Curran Associates, Inc. find
  25. Anastasia Pentina, Viktoriia Sharmanska, and Christoph H. Lampert. 2015. Curriculum learning of multiple tasks. In IEEE Conference on Computer Vision and Pattern Recognition CVPR, pages 5492–5500, Boston, MA, USA, June. find
  26. Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pages 147– 155, Boulder, Colorado, June. ACL. find our card
  27. Dan Roth and Wen-Tau Yih, 2007. Global Inference for Entity and Relation Identification via a Linear Programming Formulation. MIT Press. find
  28. Sameer Singh, Sebastian Riedel, Brian Martin, Jiaping Zheng, and Andrew McCallum. 2013. Joint inference of entities, relations, and coreference. In Proceedings of the 2013 workshop on Automated knowledge base construction, pages 1–6. ACM. find
  29. Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1201–1211,… find
  30. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958. find
  31. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural La… find
  32. Paul J Werbos. 1990. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78(10):1550–1560. find
  33. Kun Xu, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2015a. Semantic relation classification via convolutional neural networks with simple negative sampling. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 536– 540, Lisbon, Portugal, September. ACL… find
  34. Yan Xu, Lili Mou, Ge Li, Yunchuan Chen, Hao Peng, and Zhi Jin. 2015b. Classifying relations via long short term memory networks along shortest dependency paths. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1785–1794, Lisbon, Portugal, September. AC… find
  35. Bishan Yang and Claire Cardie. 2013. Joint inference for fine-grained opinion extraction. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1640–1649, Sofia, Bulgaria, August. ACL. find
  36. Xiaofeng Yu and Wai Lam. 2010. Jointly identifying entities and extracting relations in encyclopedia text via a graphical model approach. In Coling 2010: Posters, pages 1399–1407, Beijing, China, August. find
  37. Coling 2010 Organizing Committee. find
  38. Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. 2003. Kernel methods for relation extraction. The Journal of Machine Learning Research, 3:1083–1106. find
  39. GuoDong Zhou, Jian Su, Jie Zhang, and Min Zhang. 2005. Exploring various knowledge in relation extraction. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 427–434, Ann Arbor, Michigan, June. ACL. find