Elamur Research
☰
home / papers / 2018 / Linguistically-Informed Self-Attention for Semantic Role Labeling

Linguistically-Informed Self-Attention for Semantic Role Labeling

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1804.08199

Summary

Strong neural semantic role labeling (SRL) systems had dropped explicit syntax. Yet earlier studies showed that gold parse trees can greatly improve SRL accuracy. The authors propose LISA (linguistically-informed self-attention), which brings syntax back without an external preprocessing pipeline. LISA is built on a Transformer-style encoder and learns four tasks jointly: POS tagging, predicate detection, dependency parsing and SRL. POS and predicate tags are predicted together as one combined label. Syntax enters the model directly: in one layer, a single attention head is trained to point each token at its dependency head. Each sentence is encoded only once, and roles for all predicates are scored in parallel with a bilinear function.

Because of this design, a better external parse can be substituted at inference time without retraining. The authors show this with the D&M parser. On CoNLL-2005 and CoNLL-2012, LISA beat the previous best syntax-free systems, both with GloVe and with ELMo embeddings, and its largest gains were on out-of-domain text. The work suggests that targeted supervision of attention is a cheap and effective way to add linguistic structure to neural models.

Key points

  • CoNLL-2005 results (predicted predicates, GloVe): about +2.5 F1 on WSJ and over +3.5 F1 on the Brown out-of-domain set, which cuts errors by nearly 10%. On CoNLL-2012, the gain exceeds 2.5 F1.
  • With ELMo: LISA still leads by roughly 1.0 F1 on news and by more than 2.0 F1 out-of-domain. Supplying gold parses adds about 4 more F1 points.
  • Built-in parser (ELMo): reaches 96.28 UAS / 93.65 LAS on WSJ, close to the standalone D&M parser (96.48 / 94.40).
  • Predicate detection: F1 is 98.4 on WSJ and 97.2 on CoNLL-12. On Brown it reaches 93.7, versus 92.4 for He et al. (2017).
  • Error analysis: most of the gap to gold-parse performance comes from span boundary errors. About half of the split/merge fixes involve PP attachment. The code (TensorFlow) is released at github.com/strubell/LISA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semantic role labeling

Cited by

192 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 183 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1804.08199 · arXiv 1804.08199

@misc{strubell2018linguistically,
  title = {{Linguistically-Informed Self-Attention for Semantic Role Labeling}},
  author = {Strubell, Emma and Verga, Patrick and Andor, Daniel and Weiss, David and McCallum, Andrew},
  year = {2018},
  doi = {10.48550/arxiv.1804.08199},
  eprint = {1804.08199},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1804.08199}
}

References

65 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martın Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. 2015. Tensorflow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org. find
  2. Héctor Martı́nez Alonso and Barbara Plank. 2017. When is multitask learning effective? semantic sequence prediction under varying data conditions. In EACL. find
  3. Miguel Ballesteros, Yoav Goldberg, Chris Dyer, and Noah A. Smith. 2016. Training with exploration improves a greedy stack lstm parser. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2005–2010. find
  4. Marzieh Bazrafshan and Daniel Gildea. 2013. Semantic roles for string to tree machine translation. In ACL. find
  5. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In NIPS. find
  6. Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157–166. find
  7. Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Brad Huang, Christopher D. Manning, Abby Vander Linden, Brittany Harding, and Peter Clark. 2014. Modeling biological processes for reading comprehension. In EMNLP. find
  8. Joachim Bingel and Anders Søgaard. 2017. Identifying beneficial task relations for multi-task learning in deep neural networks. In EACL. find
  9. Xavier Carreras and Lluı́s Màrquez. 2005. Introduction to the conll-2005 shared task: Semantic role labeling. In CoNLL. find our card
  10. Rich Caruana. 1993. Multitask learning: a knowledge-based source of inductive bias. In ICML. find
  11. Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daumé III, and John Langford. 2015. Learning to search better than your teacher. In ICML. find
  12. Yun-Nung Chen, William Yang Wang, and Alexander I Rudnicky. 2013. Unsupervised induction and filling of semantic slots for spoken dialogue systems using frame-semantic parsing. In Proc. of ASRU-IEEE. find
  13. Jinho D. Choi and Martha Palmer. 2011. Getting the most out of transition-based dependency parsing. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: short papers, pages 687–692. find
  14. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug):2493–2537. find our card
  15. Hal Daumé III, John Langford, and Daniel Marcu. 2009. Search-based structured prediction. Machine Learning, 75(3):297–325. find
  16. Timothy Dozat. 2016. Incorporating nesterov momentum into adam. In ICLR Workshop track. find
  17. Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In ICLR. find our card
  18. Nicholas FitzGerald, Oscar Täckström, Kuzman Ganchev, and Dipanjan Das. 2015. Semantic role labeling with neural network factors. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 960–970. find
  19. W. N. Francis and H. Kučera. 1964. Manual of information to accompany a standard corpus of present-day edited american english, for use with digital computers. Technical report, Department of Linguistics, Brown University, Providence, Rhode Island. find
  20. Yoav Goldberg and Joakim Nivre. 2012. A dynamic oracle for arc-eager dependency parsing. In Proceedings of COLING 2012: Technical Papers, pages 959–976. find
  21. Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2017. A joint many-task model: Growing a neural network for multiple nlp tasks. In Conference on Empirical Methods in Natural Language Processing. find our card
  22. Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018. Jointly predicting predicates and arguments in neural semantic role labeling. In ACL. find our card
  23. Luheng He, Kenton Lee, Mike Lewis, and Luke Zettlemoyer. 2017. Deep semantic role labeling: What works and whats next. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. find our card
  24. Richard Johansson and Pierre Nugues. 2008. Dependency-based semantic role labeling of propbank. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 69–78. find our card
  25. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference for Learning Representations (ICLR), San Diego, California, USA. find
  26. Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of the Association for Computational Linguistics, 4:313–327. find our card
  27. Thomas N. Kipf and Max Welling. 2017. Semisupervised classification with graph convolutional networks. In International Conference on Learning Representations. find
  28. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In EMNLP. find our card
  29. Beth Levin. 1993. English verb classes and alternations: A preliminary investigation. University of Chicago press. find
  30. Mike Lewis, Luheng He, and Luke Zettlemoyer. 2015. Joint A* CCG Parsing and Semantic Role Labeling. In EMNLP. find
  31. Ding Liu and Daniel Gildea. 2010. Semantic role features for machine translation. In Proceedings of the 23rd International Conference on Computational Linguistics (COLING). find
  32. Yang Liu and Mirella Lapata. 2018. Learning structured text representations. Transactions of the Association for Computational Linguistics, 6:63–75. find
  33. Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In ICML, volume 30. find
  34. Diego Marcheggiani, Anton Frolov, and Ivan Titov. 2017. A simple and accurate syntax-agnostic neural model for dependency-based semantic role labeling. In CoNLL. find our card
  35. Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  36. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn TreeBank. Computational Linguistics – Special issue on using large corpora: II, 19(2):313–330. find our card
  37. Marie-Catherine de Marneffe and Christopher D. Manning. 2008. The stanford typed dependencies representation. In COLING 2008 Workshop on Cross-framework and Cross-domain Parser Evaluation. find our card
  38. Yurii Nesterov. 1983. A method of solving a convex programming problem with convergence rate o(1/k 2 ). volume 27, pages 372–376. find
  39. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1). find our card
  40. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of the 30 th International Conference on Machine Learning. find
  41. Hao Peng, Sam Thomson, and Noah A. Smith. 2017. Deep multitask learning for semantic dependency parsing. In ACL. find
  42. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP. find
  43. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL. find
  44. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using OntoNotes. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 143– 152. find
  45. Sameer Pradhan, Wayne Ward, Kadri Hacioglu, James Martin, and Dan Jurafsky. 2005. Semantic role labeling using different syntactic views. In Proceedings of the Association for Computational Linguistics 43rd annual meeting (ACL). find
  46. Vasin Punyakanok, Dan Roth, and Wen-Tau Yih. 2008. The importance of syntactic parsing and inference in semantic role labeling. Computational Linguistics, 34(2):257–287. find our card
  47. Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS). find
  48. Michael Roth and Mirella Lapata. 2016. Neural semantic role labeling with dependency path embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1192–1202. find
  49. Anders Søgaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 231–235. find our card
  50. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of machine learning research, 15(1):1929–1958. find
  51. Mihai Surdeanu, Lluı́s Màrquez, Xavier Carreras, and Pere R. Comas. 2007. Combination strategies for semantic role labeling. Journal of Artificial Intelligence Research, 29:105–151. find
  52. Charles Sutton and Andrew McCallum. 2005. Joint parsing and semantic role labeling. In CoNLL. find
  53. Swabha Swayamdipta, Sam Thomson, Chris Dyer, and Noah A. Smith. 2017. Frame-semantic parsing with softmax-margin segmental rnns and a syntactic scaffold. In arXiv:1706.09528. ↗ our card
  54. Oscar Täckström, Kuzman Ganchev, and Dipanjan Das. 2015. Efficient inference and structured learning for semantic role labeling. TACL, 3:29–41. find
  55. Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen, and Xiaodong Shi. 2018. Deep semantic role labeling with self-attention. In AAAI. find
  56. Kristina Toutanova, Aria Haghighi, and Christopher D. Manning. 2008. A global joint model for semantic role labeling. Computational Linguistics, 34(2):161–191. find
  57. Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pages 173–180. Association for Computational Linguistics. find
  58. Gokhan Tur, Dilek Hakkani-Tür, and Ananlada Chotimongkol. 2005. Semi-supervised learning for spoken language understanding using semantic role labeling. In ASRU. find
  59. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In 31st Conference on Neural Information Processing Systems (NIPS). find
  60. Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2015. Machine comprehension with syntax, frames, and semantics. In ACL. find
  61. R. J. Williams and D. Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270–280. find
  62. Daniel Zeman, Martin Popel, Milan Straka, Jan Hajic, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo, Slav Petrov, Martin Potthast, et al. 2017. Conll 2017 shared task: Multilingual parsing from raw text to universal dependencies. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1–19, Vancouver, Canada. Association for Computational Linguistics. find our card
  63. Yuan Zhang and David Weiss. 2016. Stack-propagation: Improved representation learning for syntax. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1557–1566. Association for Computational Linguistics. find
  64. Jie Zhou and Wei Xu. 2015. End-to-end learning of semantic role labeling using recurrent neural networks. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL). Greedy F1 81.99 83.37 86.57 Viterbi F1 82.24 83.58 86.81 ∆ F1 +0.25 +0.21 +0.24 find
  65. CoNLL-2005 LISA +D&M +Gold CoNLL-2012 LISA +D&M +Gold Greedy F1 80.11 81.55 85.94 Viterbi F1 80.70 82.05 86.43 ∆ F1 +0.59 +0.50 +0.49 Table 7 : Comparison of development F1 scores with and without Viterbi decoding at test time. F1 LISA +D&M +Gold 0-10 11-20 21-30 31-40 Sentence length (tokens) Figure 5 : F1 score as a function of sentence length. find