Elamur Research
☰
home / papers / 2016 / Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification

Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P16-2034.pdf

Summary

This paper addresses relation classification: deciding which semantic relation holds between two marked nominals in a sentence. Earlier strong systems depended on handcrafted or externally derived features such as WordNet, dependency parses, POS tags and NER output. These features are costly to produce, can introduce errors and may generalize poorly. The authors propose Att-BLSTM. Each word, including the entity position markers, is mapped to an embedding and encoded by a bidirectional LSTM with peephole connections. An attention layer then learns a weight for every time step and pools the word-level states into one sentence vector, which a softmax classifier labels. Dropout and L2 constraints regularize training. On the SemEval-2010 Task 8 benchmark, the model reaches a macro-F1 of 84.0% using only word vectors. That result beats most prior systems and nearly matches a feature-rich BLSTM. The work matters because it shows that attention lets a simple, resource-free model find the decisive words wherever they occur in a sentence. This makes the approach attractive for languages with few NLP tools.

Key points

  • Data: SemEval-2010 Task 8 has 9 directed relation types plus an undirected Other class. It contains 10,717 sentences, split into 8,000 for training and 2,717 for testing. Because no official development set exists, 800 training sentences were held out for validation.
  • Input: The entity markers ⟨e1⟩, ⟨/e1⟩, ⟨e2⟩, ⟨/e2⟩ are treated as ordinary tokens. The model uses no POS, NER, dependency or WordNet features. Forward and backward LSTM outputs are combined by element-wise sum.
  • Results: Att-BLSTM scores 84.0% F1 with 100-dimensional GloVe vectors (Pennington et al., 2014). It scores 82.5% with 50-dimensional Turian et al. (2010) vectors, about 2.5 points above the bidirectional RNN of Zhang and Wang (2015) with the same embeddings.
  • Baselines: The best SemEval-2010 SVM reached 82.2%, the CNN of Zeng et al. (2014) 82.7% and SDP-LSTM 83.7%. The feature-heavy BLSTM of Zhang et al. (2015) performs similarly to Att-BLSTM but is far more complex.
  • Training setup: The model uses AdaDelta with learning rate 1.0, minibatch size 10 and L2 strength 10⁻⁵. Dropout rates are 0.3 on embeddings, 0.3 on the LSTM layer and 0.5 on the penultimate layer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

204 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 203 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-2034

@inproceedings{zhou2016attention,
  title = {{Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification}},
  author = {Zhou, Peng and Shi, Wei and Tian, Jun and Qi, Zhenyu and Li, Bingchen and Hao, Hongwei and Xu, Bo},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-2034},
  url = {https://doi.org/10.18653/v1/p16-2034}
}

References

25 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. find our card
  2. Razvan C Bunescu and Raymond J Mooney. 2005. A shortest path dependency kernel for relation extraction. In Proceedings of the conference on human language technology and empirical methods in natural language processing, pages 724–731. Association for Computational Linguistics. find
  3. Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. 2015. Attention-based models for speech recognition. In Advances in Neural Information Processing Systems, pages 577–585. find our card
  4. Alan Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pages 6645–6649. IEEE. semantic resources. In Proceedings of the 5th International Worksho… find
  5. Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850. find
  6. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2009. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the Workshop on Seman… find our card
  7. Richard Socher, Brody Huval, Christopher D Manning, and Andrew Y Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1201–1211. A… find
  8. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1684– 1692. find our card
  9. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th annual meeting of the association for computational linguistics, pages 384–394. Association for Computational Linguistics. find our card
  10. Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580. find
  11. Fei Wu and Daniel S Weld. 2010. Open information extraction using wikipedia. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 118–127. Association for Computational Linguistics. find
  12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  13. Kelvin Xu, Jimmy Ba, Ryan Kiros, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044. find our card
  14. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In IN-TERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association, Makuhari, Chiba, Japan, September 26-30, 2010, pages 1045–1048… find
  15. Xu Yan, Lili Mou, Ge Li, Yunchuan Chen, Hao Peng, and Zhi Jin. 2015. Classifying relations via long short term memory networks along shortest dependency path. arXiv preprint arXiv:1508.03720. — [arXiv:1508.03720](https://arxiv.org/abs/1508.03720) find
  16. Xuchen Yao and Benjamin Van Durme. 2014. Information extraction over structured data: Question answering with freebase. In ACL (1), pages 956–966. Citeseer. find
  17. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. find
  18. Matthew D Zeiler. 2012. Adadelta: An adaptive learning rate method. arXiv preprint arXiv:1212.5701. find our card
  19. George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41. find our card
  20. Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of COLING, pages 2335–2344. find
  21. Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2… find
  22. Dongxu Zhang and Dong Wang. 2015. Relation classification via recurrent neural network. arXiv preprint arXiv:1508.01006. — [arXiv:1508.01006](https://arxiv.org/abs/1508.01006) find
  23. Shu Zhang, Dequan Zheng, Xinchen Hu, and Ming Yang. 2015. Bidirectional long short-term memory networks for relation classification. find
  24. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532– 1543. find
  25. Bryan Rink and Sanda Harabagiu. 2010. Utd: Classifying semantic relations by combining lexical and find