Elamur Research
☰
home / papers / 2017 / Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling

Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1703.04826

Summary

The paper tackles dependency-based semantic role labeling (SRL), which involves finding predicates, disambiguating their senses and assigning semantic roles to their arguments. Earlier neural SRL systems built on LSTMs had largely dropped syntax, partly because there was no easy way to inject it at the word level. The authors adapt graph convolutional networks (GCNs) to labeled, directed dependency trees. Information flows both from heads to dependents and back, and a self-loop is included. Weight matrices depend only on edge direction, while syntactic function labels enter through bias terms. A scalar gate on each edge lets the model reduce the influence of parser edges that are unhelpful or wrong. The GCN layers are stacked on top of a BiLSTM encoder, so the LSTM covers long-distance context and the GCN adds a few hops of syntactic neighbourhood. Evaluated on CoNLL-2009, this combination beats prior systems for both English and Chinese. The work was among the first to show that GCNs are useful in NLP. It also demonstrates that sequential and syntax-based encoders complement each other.

Key points

  • Benchmark: English and Chinese portions of CoNLL-2009. The code is available at github.com/diegma/neural-dep-srl.
  • The model is kept compact for parameter efficiency:
    • There are only three matrix types per layer (along-arc, opposite-arc and self-loop).
    • Fully label-specific weights would have required 97 matrices per layer for English (48 syntactic functions) and 83 for Chinese (41).
  • English in-domain: an ensemble of 3 models (product of experts) reaches 89.1% F1, about 1.2 points above the best previous result.
  • Chinese: the best model beats the earlier state of the art, Roth and Lapata (2016), by 3.1 points.
  • English out-of-domain (parser output is noisier there):
    • The ensemble scores 78.9 F1 and surpasses all earlier syntax-aware systems.
    • The single local model (77.2) roughly matches, but does not beat, the 77.3 achieved by Marcheggiani et al. (2017), whose model uses no syntax.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semantic role labeling

Cited by

227 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 219 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1703.04826 · arXiv 1703.04826

@misc{marcheggiani2017encoding,
  title = {{Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling}},
  author = {Marcheggiani, Diego and Titov, Ivan},
  year = {2017},
  doi = {10.48550/arxiv.1703.04826},
  eprint = {1703.04826},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1703.04826}
}

References

46 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research 3:1137–1155. find
  2. Anders Björkelund, Bernd Bohnet, Love Hafdell, and Pierre Nugues. 2010. A high-performance syntactic and semantic dependency parser. In Proceedings of COLING: Demonstrations. find
  3. Anders Björkelund, Love Hafdell, and Pierre Nugues. 2009. Multilingual semantic role labeling. In Proceedings of CoNLL. find
  4. Jason P. C. Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. TACL 4:357–370. find our card
  5. Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of EMNLP. find
  6. Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2016. Language modeling with gated convolutional networks. arXiv preprint arXiv:1612.08083 . David K. Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams. 2015. Convolutional networks on graphs for learning molecular fingerprints. In Proceedings of NIPS. ↗ our card
  7. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proceedings of ACL. find our card
  8. Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science 14(2):179–211. find
  9. Akiko Eriguchi, Yoshimasa Tsuruoka, and Kyunghyun Cho. 2017. Learning to parse and translate improves neural machine translation. arXiv preprint arXiv:1702.03525 . ↗ our card
  10. Nicholas FitzGerald, Oscar Täckström, Kuzman Ganchev, and Dipanjan Das. 2015. Semantic role labeling with neural network factors. In Proceedings of EMNLP. find
  11. William Foland and James Martin. 2015. Dependency-based semantic role labeling using convolutional neural networks. In Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics (*SEM). find
  12. Andrea Gesmundo, James Henderson, Paola Merlo, and Ivan Titov. 2009. Latent variable model of synchronous syntactic-semantic parsing for multiple languages. In Proceedings of CoNLL. find
  13. Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Computational linguistics 28(3):245–288. find our card
  14. Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. 2017. Neural message passing for quantum chemistry. arXiv preprint arXiv:1704.01212 . ↗ our card
  15. Alex Graves. 2008. Supervised sequence labelling with recurrent neural networks. Ph.D. thesis, München, Techn. find
  16. James Henderson, Paola Merlo, Gabriele Musillo, and Ivan Titov. 2008. A latent variable model of synchronous parsing for syntactic and semantic dependencies. In Proceedings of CoNLL. find
  17. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9(8):1735–1780. ↗
  18. Richard Johansson and Pierre Nugues. 2008. The effect of syntactic representation on semantic role labeling. In Proceedings of COLING. find
  19. Steven Kearnes, Kevin McCloskey, Marc Berndl, Vijay Pande, and Patrick Riley. 2016. Molecular graph convolutions: moving beyond fingerprints. Journal of computer-aided molecular design 30(8):595–608. find
  20. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR. find
  21. Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. TACL 4:313– 327. find our card
  22. Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of ICLR. find our card
  23. Phong Le and Willem Zuidema. 2014. The inside-outside recursive neural network model for dependency parsing. In Proceedings of EMNLP. find
  24. Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. 2001. Gradient-based learning applied to document recognition. In Proceedings of Intelligent Signal Processing. find
  25. Tao Lei, Yuan Zhang, Lluı́s Màrquez, Alessandro Moschitti, and Regina Barzilay. 2015. High-order low-rank tensors for semantic role labeling. In Proceedings of NAACL. find
  26. Beth Levin. 1993. English verb classes and alternations: A preliminary investigation. University of Chicago press. find
  27. Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard S. Zemel. 2016. Gated graph sequence neural networks. In Proceedings of ICLR. find
  28. Wang Ling, Chris Dyer, Alan W Black, and Isabel Trancoso. 2015. Two/too simple adaptations of word2vec for syntax problems. In Proceedings of NAACL. find our card
  29. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. TACL 4:521–535. find our card
  30. Diego Marcheggiani, Anton Frolov, and Ivan Titov. 2017. A simple and accurate syntax-agnostic neural model for dependency-based semantic role labeling. arXiv preprint arXiv:1701.02593 . ↗ our card
  31. Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proceedings of INTERSPEECH. find
  32. Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using lstms on sequences and tree structures. In Proceedings of ACL. find our card
  33. Alessandro Moschitti, Daniele Pighin, and Roberto Basili. 2008. Tree kernels for semantic role labeling. Computational Linguistics 34(2):193–224. find
  34. Lili Mou, Hao Peng, Ge Li, Yan Xu, Lu Zhang, and Zhi Jin. 2015. Discriminative neural sentence modeling by tree-based convolution. In Proceedings of EMNLP. find
  35. Sameer Pradhan, Kadri Hacioglu, Wayne H. Ward, James H. Martin, and Daniel Jurafsky. 2005. Semantic role chunking combining complementary syntactic views. In Proceedings of CoNLL. find
  36. Vasin Punyakanok, Dan Roth, and Wen-tau Yih. 2008. The importance of syntactic parsing and inference in semantic role labeling. Computational Linguistics 34(2):257–287. find our card
  37. Michael Roth and Mirella Lapata. 2016. Neural semantic role labeling with dependency path embeddings. In Proceedings of ACL. find
  38. Rico Sennrich and Barry Haddow. 2016. Linguistic input features improve neural machine translation. In Proceedings of WMT. find our card
  39. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP. find
  40. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of NIPS. find
  41. Swabha Swayamdipta, Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2016. Greedy, joint syntactic-semantic parsing with stack LSTMs. In Proceedings of CoNLL. find
  42. Cynthia A. Thompson, Roger Levy, and Christopher D. Manning. 2003. A generative model for semantic role labeling. In Proceedings of ECML. find
  43. Ivan Titov, James Henderson, Paola Merlo, and Gabriele Musillo. 2009. Online projectivisation for synchronous parsing of semantic and syntactic dependencies. In Proceedings of IJCAI. find
  44. Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves. 2016. Conditional image generation with PixelCNN decoders. In Proceedings of NIPS. find
  45. Hai Zhao, Wenliang Chen, Jun’ichi Kazama, Kiyotaka Uchimoto, and Kentaro Torisawa. 2009. Multilingual dependency learning: Exploiting rich features for tagging syntactic and semantic dependencies. In Proceedings of CoNLL. find
  46. Jie Zhou and Wei Xu. 2015. End-to-end learning of semantic role labeling using recurrent neural networks. In Proceedings of ACL. find