Elamur Research⌕
☰
home / papers / 2016 / Neural Semantic Role Labeling with Dependency Path Embeddings

Neural Semantic Role Labeling with Dependency Path Embeddings

Summary

Semantic role labelers that depend on sparse lexical and syntactic indicator features often fail on constructions that are rare in training data. Examples include control and raising verbs, nested coordination or subordination, and uncommon nominal predicates. In the sentence "He had trouble raising funds", four public systems all miss He as the agent of raise. Roth and Lapata address this by representing the dependency path from a predicate to a candidate argument as a sequence. The sequence contains part-of-speech tags, word forms and dependency relations, and an LSTM composes it into one vector. That path embedding is trained together with conventional binary features in a feed-forward network with a ReLU hidden layer and a softmax output. The resulting system, PathLSTM, handles argument identification, argument classification and reranking inside a standard SRL pipeline. On the CoNLL-2009 benchmark it outperforms the previous state of the art for English and improves over its feature-based baseline for Chinese, German and Spanish. The work shows that familiar syntactic path features generalize better when they are built up from their parts rather than treated as atomic symbols.

Key points

  • Path representation: each predicate–argument path is encoded item by item. It starts with the predicate's POS tag and word form, continues through the relations and words along the path, and ends with the argument head word. The final LSTM state serves as the path embedding.
  • LSTM design: the model uses input and output gates, with forget gates and memory-to-gate connections as optional parts. Some model variants leave these optional parts out.
  • Pipeline: the system follows earlier work (Björkelund et al., 2009; Toutanova et al., 2008). Predicate identification and disambiguation use logistic regression. The neural model is applied in the argument identification, argument classification and reranking steps.
  • Multilingual CoNLL-2009 test F1:
    • German: 80.1, the best reported.
    • Chinese: 79.4, the best reported.
    • Spanish: 80.2, slightly below Zhao et al. (2009) at 80.5.
    • PathLSTM beats Björkelund et al. (2009) in every language.
  • Systems compared in the motivating example: mate-tools, mateplus, TensorSRL and easySRL.

Interesting ideas

  • I1. Split sparse structured features into parts. Instead of one symbol per full syntactic path, the path is fed as a sequence of words, tags and relations to a recurrent encoder. Rare paths then share statistics with frequent ones, which could help any task that uses path or template features, such as relation extraction or lemma–syntax patterns in low-resource languages.
  • I2. Train learned and handcrafted features together. The path embedding and the classic binary indicators feed into a shared hidden layer. This lets the model capture interactions between them, and the authors suggest it makes the system less sensitive to parse errors.
  • I3. Tune the order of sequence items. The authors tried several orderings and found that starting from the predicate (POS tag, then word form) and ending at the argument word gave the best validation results. Checking input ordering is a cheap experiment worth repeating whenever a structure is linearized for an RNN.
  • I4. Upgrade one component, keep the pipeline. The proven SRL pipeline stays in place, and only the stages where sparsity hurts most are replaced with neural parts. This incremental approach is practical when improving existing tools for Ukrainian NLP.
  • I5. Use targeted examples to show failures. A single hard sentence (a control construction) shows clearly where all strong systems fail. Pairing aggregate scores with such phenomenon-specific examples makes model analysis more convincing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Semantic role labeling · Part-of-speech tagging

Cited by

81 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 70 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1113

@inproceedings{roth2016neural,
  title = {{Neural Semantic Role Labeling with Dependency Path Embeddings}},
  author = {Roth, Michael and Lapata, Mirella},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1113},
  url = {https://doi.org/10.18653/v1/p16-1113}
}

References

40 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wilker Aziz, Miguel Rios, and Lucia Specia. 2011. Shallow semantic trees for smt. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 316–322, Edinburgh, Scotland. find
  2. Anders Björkelund, Love Hafdell, and Pierre Nugues. 2009. Multilingual semantic role labeling. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning: Shared Task, pages 43–48, Boulder, Colorado. find our card
  3. Anders Björkelund, Bernd Bohnet, Love Hafdell, and Pierre Nugues. 2010. A high-performance syntactic and semantic dependency parser. In Coling 2010: Demonstration Volume, pages 33–36, Beijing, China. find
  4. Bernd Bohnet. 2010. Top accuracy and fast dependency parsing is not a contradiction. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 89–97, Beijing, China. find our card
  5. Wanxiang Che, Zhenghua Li, Yongqiang Li, Yuhang Guo, Bing Qin, and Ting Liu. 2009. Multilingual dependency-based syntactic and semantic parsing. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning: Shared Task, pages 49–54, Boulder, Colorado. find
  6. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537. find our card
  7. Koen Deschacht and Marie-Francine Moens. 2009. Semi-supervised semantic role labeling using the Latent Words Language Model. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 21–29, Singapore. find
  8. Nicholas FitzGerald, Oscar Täckström, Kuzman Ganchev, and Dipanjan Das. 2015. Semantic role labeling with neural network factors. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 960–970, Lisbon, Portugal. find
  9. William Foland and James Martin. 2015. Dependency-based semantic role labeling using convolutional neural networks. In Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics, pages 279–288, Denver, Colorado. find
  10. Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Computational Linguistics, 28(3):245–288. find our card
  11. Jan Hajič, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Antònia Martı́, Lluı́s Màrquez, Adam Meyers, Joakim Nivre, Sebastian Padó, Jan Štěpánek, et al. 2009. The CoNLL-2009 shared task: Syntactic and semantic dependencies in multiple languages. In Proceedings of the Thir… find our card
  12. Tao Lei, Yuan Zhang, Lluı́s Màrquez, Alessandro Moschitti, and Regina Barzilay. 2015. High-order low-rank tensors for semantic role labeling. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1150… find
  13. Mike Lewis, Luheng He, and Luke Zettlemoyer. 2015. Joint A* CCG parsing and semantic role labelling. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1444–1454, Lisbon, Portugal. find
  14. Yang Liu, Furu Wei, Sujian Li, Heng Ji, Ming Zhou, and Houfeng Wang. 2015. A dependency-based neural network for relation classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processi… find
  15. Mingbo Ma, Liang Huang, Bowen Zhou, and Bing Xiang. 2015. Dependency-based convolutional neural networks for sentence embedding. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages… find
  16. Derek Monner and James A Reggia. 2012. A generalized LSTM-like training algorithm for second-order recurrent neural networks. Neural Networks, 25:70– 83. find
  17. James Henderson, Paola Merlo, Ivan Titov, and Gabriele Musillo. 2013. Multilingual joint parsing of syntactic and semantic dependencies with a latent variable model. Computational Linguistics, 39(4):949–998. find
  18. Ahmed Hamza Osman, Naomie Salim, Mohammed Salem Binwahlan, Rihab Alteeb, and Albaraa Abuobieda. 2012. An improved plagiarism detection scheme based on semantic role labeling. Applied Soft Computing, 12(5):1493–1502. find
  19. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. find
  20. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–106. find our card
  21. Fei Huang and Alexander Yates. 2010. Open-domain semantic role labeling by modeling word spans. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 968– 978, Uppsala, Sweden. find
  22. Richard Johansson and Pierre Nugues. 2008. The effect of syntactic representation on semantic role labeling. In Proceedings of the 22nd International Conference on Computational Linguistics, pages 393–400, Manchester, United Kingdom. find
  23. Atif Khan, Naomie Salim, and Yogan Jaya Kumar. 2015. A framework for multi-document abstractive summarization based on semantic role labelling. Applied Soft Computing, 30:737–747. find
  24. Mikhail Kozhevnikov and Ivan Titov. 2014. Cross-lingual model transfer using feature representation projection. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 579–585, Baltimore, Maryland. find
  25. Merin Paul and Sangeetha Jamal. 2015. An improved SRL based plagiarism detection technique using sentence ranking. Procedia Computer Science, 46:223–230. find
  26. Sameer Pradhan, Kadri Hacioglu, Wayne Ward, James H. Martin, and Daniel Jurafsky. 2005. Semantic role chunking combining complementary syntactic views. In Proceedings of the Ninth Conference on Computational Natural Language Learning, pages 217–220, Ann Arbor, Michigan. find
  27. Vasin Punyakanok, Dan Roth, and Wen-tau Yih. 2008. The importance of syntactic parsing and inference in semantic role labeling. Computational Linguistics, 34(2):257–287. find our card
  28. Michael Roth and Mirella Lapata. 2015. Context-aware frame-semantic role labeling. Transactions of the Association for Computational Linguistics, 3:449–460. find our card
  29. Michael Roth and Kristian Woodsend. 2014. Composition of word representations improves semantic role labelling. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 407–413, Doha, Qatar. find
  30. Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. 2012. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, pages 2951–2959, Lake Tahoe, Nevada. find
  31. Jie Zhou and Wei Xu. 2015. End-to-end learning of semantic role labeling using recurrent neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1127–1137, Beijing, Ch… find our card
  32. Vivek Srikumar and Dan Roth. 2013. Modeling semantic relations expressed by prepositions. Transactions of the Association for Computational Linguistics, 1:231–242. find
  33. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural La… find
  34. Kristina Toutanova, Aria Haghighi, and Christopher Manning. 2008. A global joint model for semantic role labeling. Computational Linguistics, 34(2):161–191. find
  35. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579–2605. find
  36. Deyi Xiong, Min Zhang, and Haizhou Li. 2012. Modeling the translation of predicate-argument structure for smt. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, pages 902–911, Jeju Island, Korea. find
  37. Yan Xu, Lili Mou, Ge Li, Yunchuan Chen, Hao Peng, and Zhi Jin. 2015. Classifying relations via long short term memory networks along shortest dependency paths. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1785–1794, Lisbon, Portugal. find our card
  38. Nianwen Xue and Martha Palmer. 2004. Calibrating features for semantic role labeling. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 88–94, Barcelona, Spain. find
  39. Beñat Zapirain, Eneko Agirre, Lluı́s Màrquez, and Mihai Surdeanu. 2013. Selectional preferences for semantic role classification. Computational Linguistics, 39(3):631–663. find
  40. Hai Zhao, Wenliang Chen, Jun’ichi Kazama, Kiyotaka Uchimoto, and Kentaro Torisawa. 2009. Multilingual dependency learning: Exploiting rich features for tagging syntactic and semantic dependencies. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL 2009): Sh… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/roth-2016-neural-semantic-role-labeling-dependency/