Elamur Research⌕
☰
home / papers / 2015 / End-to-end learning of semantic role labeling using recurrent neural networks

End-to-end learning of semantic role labeling using recurrent neural networks

Summary

Zhou and Xu ask whether semantic role labeling (SRL) can be done well without syntactic parsing. Earlier strong systems relied on parse trees, hand-crafted feature templates and combinations of several parsers, and the earlier parser-free neural attempt by Collobert et al. (2011) fell short. The authors build an end-to-end tagger called deep bi-directional LSTM (DB-LSTM). It stacks pairs of LSTM layers that alternate between forward and backward directions, eight layers in total, with a CRF layer on top for tag sequence prediction. The input is the raw words plus a few simple predicate-related features; no POS tags or parses are used. On the CoNLL-2005 shared task the model reaches F1 = 81.07, beating the previous best parser-combination system, and on CoNLL-2012 it gets F1 = 81.27. It also runs fast, at about 6.7k tokens per second. Analysis shows its advantage grows on longer sentences, and its internal gates appear to encode syntactic distance. The work matters because it shows SRL does not need to depend on brittle, language-specific parsing pipelines.

Key points

  • The architecture is DB-LSTM: 8 LSTM layers stacked as forward/backward pairs, where each layer feeds the next in the reversed direction, with a CRF output layer. The authors say this topology is critical, and that model depth is the largest source of improvement.
  • Only four features are used:
    • the argument word (argu),
    • the predicate word (pred),
    • the predicate context window (ctx-p),
    • a binary region mark (m_r) that flags words inside that window.
  • Results:
    • CoNLL-2005: F1 81.07, which is +3.15 on test and +2.20 on dev over Koomen et al. (2005). Koomen's system combines 5 parse trees from two parsers. Against Koomen's best single system, the dev gain is +4.79.
    • Gains appear on both the WSJ and Brown test sets.
    • CoNLL-2012: F1 81.27.
  • Collobert et al.'s model without parse features scores 74.15, which is 6.92 below DB-LSTM. Its cost is also O(n_p·L²), while DB-LSTM is linear at O(n_p·L).
  • In the 7th LSTM layer, the average forget-gate value rises monotonically with the dependency-tree distance between argument and predicate.

Interesting ideas

  • I1. Alternating-direction stacking. Instead of running two parallel LSTMs and concatenating them, each layer reads the previous layer's output in the opposite direction. This is a simple way to get deep two-sided context and could carry over to other sequence taggers such as NER or morphology tagging.
  • I2. Cheap structural hints instead of parses. A small window of words around the predicate, plus a flag marking that window, disambiguates which predicate is meant (for example, when the same word occurs twice). This kind of minimal marker could condition any per-target labeling task without needing a parser, which helps for lower-resource languages.
  • I3. One pass per target. The sentence is run once for each predicate, giving cost linear in sentence length rather than quadratic. This is a reusable pattern for tasks keyed on a target, such as relation or event extraction.
  • I4. Probing gates against linguistic structure. Comparing forget-gate activations with gold dependency distance is a lightweight interpretability check. It could test whether a model has implicitly learned syntax for a given language or treebank.
  • I5. Breaking down errors by sentence length. Binning F1 by length (10-word bins) and comparing against a baseline shows where the model's long-range strength actually pays off. This evaluation habit is useful whenever claims about long-distance dependencies are made.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Semantic role labeling · Part-of-speech tagging

Cited by

129 papers of our library cite it (of the 140884 whose bibliographies we read); 20 of them have a page here, 109 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p15-1109

@inproceedings{zhou2015end,
  title = {{End-to-end learning of semantic role labeling using recurrent neural networks}},
  author = {Zhou, Jie and Xu, Wei},
  booktitle = {Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2015},
  doi = {10.3115/v1/p15-1109},
  url = {https://doi.org/10.3115/v1/p15-1109}
}

References

40 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. G. Hinton A. Graves, A. Mohamed. 2013. Speech recognition with deep recurrent neural networks. In IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2013. find
  2. Emanuele Bastianelli, Giuseppe Castellucci, Danilo Croce, and Roberto Basili. 2013. Textual inference and meaning representation in human robot interaction. In Proceedings of the Joint Symposium on Semantic Processing. Textual Inference and Structures in Corpora, pages 65–69. find
  3. Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157–166. find
  4. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155, March. find
  5. Yoshua Bengio, Holger Schwenk, Jean-Sbastien Sencal, Frderic Morin, and Jean-Luc Gauvain. 2006. Neural probabilistic language models. In Innovations in Machine Learning, volume 194 of Studies in Fuzziness and Soft Computing, pages 137–186. Springer Berlin Heidelberg. find
  6. Xavier Carreras and Lluı́s Màrquez. 2005. Introduction to the CoNLL-2005 shared task: Semantic role labeling. In Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005), pages 152–164, Ann Arbor, Michigan, June. Association for Computational Linguistics. find our card
  7. Eugene Charniak and Mark Johnson. 2005. Coarse-to-fine n-best parsing and maxent discriminative reranking. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL ’05, pages 173–180, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
  8. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of the 1st North American Chapter of the Association for Computational Linguistics Conference, NAACL 2000, pages 132–139, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
  9. Michael Collins. 2003. Head-driven statistical models for natural language parsing. Comput. Linguist., 29(4):589–637, December. find our card
  10. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, ICML ’08, pages 160–167, New York, NY, USA. ACM. find
  11. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Marchine Learning Research, 12:2493–2537, November. find our card
  12. Shen Dan and Mirella Lapata. 2007. Using semantic roles to improve question answering. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL). find our card
  13. Alex Graves, Marcus Liwicki, Santiago Fernandez, Roman Bertolami, Horst Bunke, and Jürgen Schmidhuber. 2009. A novel connectionist system for unconstrained handwriting recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(5):855–868. find
  14. Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural turing machines. arXiv:1410.5401. find
  15. S. Hochreiter and J. Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. find
  16. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1746–1751. find our card
  17. Peter Koomen, Vasin Punyakanok, Dan Roth, and Wen-tau Yih. 2005. Generalized inference with multiple semantic role labeling systems. In Proceedings of the 9th Conference on Computational Natural Language Learning, CONLL ’05, pages 181–184, Stroudsburg, PA, USA. Association for Computational Linguist… find
  18. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the 8th International Conference on Machine Learning, ICML ’01, pages 282–289, San Francisco, CA, USA. Morgan Kaufmann Pub… find
  19. Yann Lecun, Lon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pages 2278–2324. find
  20. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of phrases and their compositionality. In Advances on Neural Information Processing Systems. find
  21. Andriy Mnih and Koray Kavukcuoglu. 2013. Learning word embeddings efficiently with noise-contrastive estimation. In Advances in Neural Information Processing Systems, pages 2265–2273. find
  22. Alessandro Moschitti, Paul Morarescu, and Sanda M. Harabagiu. 2003. Open domain information extraction via automatic semantic labeling. In FLAIRS Conference’03, pages 397–401. find
  23. Martha Palmer, Daniel Gildea, and Nianwen Xue. 2010. Semantic Role Labeling. Synthesis Lectures on Human Language Technology Series. Morgan and Claypool. find
  24. Jacob Persson, Richard Johansson, and Pierre Nugues. 2009. Text categorization using predicatecargumen-t structures. In Proceedings of NODALIDA, pages 142–149. find
  25. Sameer Pradhan, Kadri Hacioglu, Wayne Ward, James H. Martin, and Daniel Jurafsky. 2005. Semantic role chunking combining complementary syntactic views. In Proceedings of the 9th Conference on Computational Natural Language Learning, CONLL ’05, pages 217–220, Stroudsburg, PA, US-A. Association for Co… find
  26. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using ontonotes. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 143–152, Sofia, Bul… find
  27. V. Punyakanok, D. Roth, and W. Yih. 2008a. The importance of syntactic parsing and inference in semantic role labeling. Computational Linguistics, 34(2). find our card
  28. Vasin Punyakanok, Dan Roth, and Wen tau Yih. 2008b. The importance of syntactic parsing and inference in semantic role labeling. Computational linguistics, 6(9). find our card
  29. M. Schuster and K. K. Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45:2673–2681. find
  30. Mihai Surdeanu, Sanda Harabagiu, John Williams, and Paul Aarseth. 2003. Using predicate-argument structures for information extraction. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1, ACL ’03, pages 8–15, Stroudsburg, PA, USA. Association for Comput… find our card
  31. Mihai Surdeanu, Lluı́s Màrquez, Xavier Carreras, and Pere R. Comas. 2007. Combination strategies for semantic role labeling. Journal of Artificial Intelligence Research, 29:105–151. find
  32. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances on Neural Information Processing Systems. find
  33. Oscar Täckström, Kuzman Ganchev, and Dipanjan Das. 2015. Efficient inference and structured learning for semantic role labeling. Transactions of the Association for Computational Linguistics, 3:29–41. find
  34. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53st Annual Meeting on Association for Computational Linguistics, ACL ’15, Stroudsburg, PA, USA. Association for Computationa… find
  35. Kristina Toutanova, Aria Haghighi, and Christopher D. Manning. 2008. A global joint model for semantic role labeling. Computational Linguistics, 34:161– 191. find
  36. Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2014. Grammar as a foreign language. arXiv:1412.7449. find
  37. Jason Weston, Sumit Chopra, and Antoine Bordes. 2014. Memory networks. arXiv:1410.3916. find
  38. Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv:1502.05698. find our card
  39. Mo Yu, Matthew Gormley, and Mark Dredze. 2014. Factor-based compositional embedding models. In Advances in Neural Information Processing Systems Workshop on Learning Semantics. find
  40. Xiang Zhang and Yann LeCun. 2015. Text understanding from scratch. arXiv:1502.01710. — [arXiv:1502.01710](https://arxiv.org/abs/1502.01710) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2015-end-end-learning-semantic-role/