Elamur Research
☰
home / papers / 2017 / Deep Semantic Role Labeling: What Works and What's Next

Deep Semantic Role Labeling: What Works and What's Next

What it is about

A neural SRL model that treats each predicate's arguments as BIO tags over the sentence, with no syntax input. It uses an 8-layer highway BiLSTM with recurrent dropout and orthonormal initialisation, decodes with BIO constraints by A* search, and ensembles 5 models by a product of experts. It reaches 83.2 F1 on CoNLL-2005 and 83.4 F1 on CoNLL-2012, about a 10% relative error reduction. Ablations show that orthonormal initialisation matters a great deal (without it the model reaches only 65 F1 in the first 50 epochs). The error analysis finds that most mistakes are prepositional-phrase attachment and argument-adjunct confusions. Deep models handle long-distance arguments better. 98.7% of gold arguments match a gold constituent, and 94.3% of predicted spans match the parse. Gold syntax gives about 3 F1 more, and predicted predicates cost 1.2 to 3.5 F1 on CoNLL-2005 and up to 6.2 on CoNLL-2012.

Ideas

For Mova

The chapters are languages/en/21-clause-structure.md and languages/en/25-relative-clauses.md. The code is prag/src/roles.rs and en/src/parse.rs. The neural model itself is not for Mova; the error analysis, the constraint study and the sweep are the parts to copy. Run twice to confirm identical output.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P17-1044.pdf

Summary

This work addresses span-based semantic role labeling (SRL), which recovers predicate-argument structure without requiring a syntactic parser as input. The authors frame SRL as BIO sequence tagging and build an 8-layer bidirectional LSTM. Its layers alternate direction and are linked by gated highway connections. Training relies on orthonormal weight initialization, recurrent dropout with a mask shared across timesteps, and GloVe embeddings. At prediction time, an A* search enforces structural constraints without changing training, and several models are combined as a product of experts. On CoNLL 2005 and CoNLL 2012, the ensemble reaches 83.2 and 83.4 test F1, cutting relative error by roughly 10% compared with earlier systems. The paper also includes a broad error analysis. Deep models handle long-distance arguments well but still make obvious structural mistakes. Gold syntax could add notable gains, but current automatic parsers help only slightly. The authors also report a simple predicate-detection step for end-to-end use. Code and models are publicly released.

Key points

  • Architecture and training setup:
    • 8 BiLSTM layers (4 forward, 4 backward, interleaved) with 300-dimensional hidden units.
    • The input is a word embedding plus a binary flag marking the predicate.
    • Embeddings are 100-dimensional GloVe (6B tokens), fine-tuned during training.
    • Optimization uses Adadelta, batch size 80, dropout 0.1, and gradient clipping at norm 1.
  • Results: The ensemble scores 83.2 F1 on the CoNLL 2005 WSJ test set and 83.4 F1 on the CoNLL 2012 test set, surpassing Zhou and Xu (2015), FitzGerald et al., and Täckström et al.
  • Decoding constraints: The authors tested several constraints: BIO validity, a rule that each core role appears at most once, and rules governing continuation and reference roles. Only BIO hard constraints gave a clear gain over the ensemble, so they were the only ones kept for final results.
  • Syntax as a decoding penalty: Each predicted argument that is not a constituent adds a penalty C.
    • Gold parses give up to about 2 F1 in the penalty sweep, and around 3 F1 is cited in the oracle discussion.
    • The Choe and Charniak (2016) parser yields small gains, while the Charniak (2000) parser hurts if trusted too strongly.
    • On CoNLL 2005 dev, automatic syntax raises F1 from 82.7 to 83.2, with the largest benefit on in-domain data such as the NW genre.
  • Resources: Code and models are released at github.com/luheng/deep_srl. Evaluation uses the official CoNLL 2005 script on both datasets.

Interesting ideas

  • Constraints only at inference: Structural rules are applied solely during A* decoding, with a heuristic that sums the best remaining per-token log-probabilities. This keeps training simple and lets rules be added or removed later. It suits any tagger, such as NER or chunking, whose labels must satisfy validity rules.
  • Soft trust knob for outside resources: An external parse is injected as a tunable penalty rather than as an input feature, so how much to trust an imperfect parser is a single parameter tuned on dev data. This is useful when auxiliary tools are noisy or out of domain, as Ukrainian parsers might be.
  • Shared recurrent dropout masks: Reusing one dropout mask across all timesteps of a layer avoids compounding noise along long sequences. The authors show this choice, together with orthonormal initialization and highway gates, matters a lot for training deep recurrent stacks.
  • Oracle-style evaluation: Comparing gold syntax with predicted syntax separates whether a resource could help from whether today's tools are good enough to help. This helps decide where to invest effort, for example in a better treebank-trained parser versus a better downstream model.
  • Error breakdowns by phenomenon and genre: Breaking errors down by argument distance, attachment-type confusions, and genre reveals strengths and weaknesses that a single F1 score hides. The same practice is worth adopting when evaluating models on Ukrainian corpora.

Topics: Semantic role labeling

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1044

@inproceedings{he2017deep,
  title = {{Deep Semantic Role Labeling: What Works and What's Next}},
  author = {He, Luheng and Lee, Kenton and Lewis, Mike and Zettlemoyer, Luke},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1044},
  url = {https://doi.org/10.18653/v1/p17-1044}
}

References

30 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kenton Lee, Mike Lewis, and Luke Zettlemoyer. 2016. Global neural ccg parsing with optimality guarantees. In Proc. of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  2. Claire Bonial, Olga Babko-Malaya, Jinho D Choi, Jena Hwang, and Martha Palmer. 2010. Propbank annotation guidelines. Center for Computational Language and Education Research Institute of Cognitive Science University of Colorado at Boulder . find
  3. Mike Lewis and Mark Steedman. 2014. A* ccg parsing with a supertag-factored model. In Proc. of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 990–1000. find our card
  4. Xavier Carreras and Lluı́s Màrquez. 2005. Introduction to the conll-2005 shared task: Semantic role labeling. In Proceedings of the Ninth Conference on Computational Natural Language Learning. Association for Computational Linguistics, pages 152–164. find our card
  5. Diego Marcheggiani, Anton Frolov, and Ivan Titov. 2017. A simple and accurate syntax-agnostic neural model for dependency-based semantic role labeling. arXiv preprint arXiv:1701.02593 . ↗ our card
  6. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proc. of the First North American chapter of the Association for Computational Linguistics conference (NAACL). Association for Computational Linguistics, pages 132–139. find
  7. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proc. of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 1532–1543. find
  8. Do Kook Choe and Eugene Charniak. 2016. Parsing as language modeling. In Proc. of the 2016 Conference of Empirical Methods in Natural Language Processing (EMNLP). find
  9. Sameer Pradhan, Kadri Hacioglu, Wayne Ward, James H Martin, and Daniel Jurafsky. 2005. Semantic role chunking combining complementary syntactic views. In Proc. of the 2005 Conference on Computational Natural Language Learning (CoNLL). pages 217–220. find
  10. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research 12:2493–2537. find our card
  11. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using ontonotes. In Proc. of the 2013 Conference on Computational Natural Language Learning (CoNLL). pages 143–152. find
  12. Nicholas FitzGerald, Oscar Täckström, Kuzman Ganchev, and Dipanjan Das. 2015. Semantic role labeling with neural network factors. In Proc. of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 960–970. find
  13. Vasin Punyakanok, Peter Koomen, Dan Roth, and Wen-tau Yih. 2005. Generalized inference with multiple semantic role labeling systems. In Proc. of the 2005 Conference on Computational Natural Language Learning (CoNLL). find
  14. Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In Advances in Neural Information Processing Systems. pages 1019–1027. find our card
  15. Vasin Punyakanok, Dan Roth, and Wen-tau Yih. 2008. The importance of syntactic parsing and inference in semantic role labeling. Computational Linguistics 34(2):257–287. find our card
  16. James Henderson, Paola Merlo, Ivan Titov, and Gabriele Musillo. 2013. Multilingual joint parsing of syntactic and semantic dependencies with a latent variable model. Computational Linguistics 39(4):949–998. find
  17. Michael Roth and Mirella Lapata. 2016. Neural semantic role labeling with dependency path embeddings. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL). find
  18. Geoffrey E Hinton. 2002. Training products of experts by minimizing contrastive divergence. Neural computation 14(8):1771–1800. find
  19. Andrew M Saxe, James L McClelland, and Surya Ganguli. 2013. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 . ↗
  20. Paul Kingsbury, Martha Palmer, and Mitch Marcus. 2002. Adding semantic annotation to the penn treebank. In Proceedings of the human language technology conference. pages 252–256. find
  21. Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Training very deep networks. In Advances in neural information processing systems. pages 2377–2385. find
  22. Jonathan K. Kummerfeld, David Hall, James R. Curran, and Dan Klein. 2012. Parser showdown at the wall street corral: An empirical investigation of error types in parser output. In Proc. of the 2012 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 1048–1059. find our card
  23. Swabha Swayamdipta, Miguel Ballesteros, Chris Dyer, and Noah A Smith. 2016. Greedy, joint syntactic-semantic parsing with stack lstms. In Proc. of the 2016 Conference on Computational Natural Language Learning (CoNLL). page 187. find
  24. Oscar Täckström, Kuzman Ganchev, and Dipanjan Das. 2015. Efficient inference and structured learning for semantic role labeling. Transactions of the Association for Computational Linguistics 3:29–41. find
  25. Kristina Toutanova, Aria Haghighi, and Christopher D Manning. 2008. A global joint model for semantic role labeling. Computational Linguistics 34(2):161– 191. find
  26. Zhen Wang, Tingsong Jiang, Baobao Chang, and Zhifang Sui. 2015. Chinese semantic role labeling with bidirectional recurrent neural networks. In Proc. of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 1626– 1631. find
  27. Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA . find
  28. Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 . ↗ our card
  29. Yu Zhang, Guoguo Chen, Dong Yu, Kaisheng Yaco, Sanjeev Khudanpur, and James Glass. 2016. Highway long short-term memory rnns for distant speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 5755–5759. find
  30. Jie Zhou and Wei Xu. 2015. End-to-end learning of semantic role labeling using recurrent neural networks. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL). find