Elamur Research⌕
☰
home / papers / 2017 / Deep Multitask Learning for Semantic Dependency Parsing

Deep Multitask Learning for Semantic Dependency Parsing

Paper · 2017 · Hao Peng, Sam Thomson and Noah A. Smith

Summary

Semantic dependency graphs are costly to annotate, and the available annotation is split across competing formalisms, so each one has relatively little training data. Peng, Thomson and Smith test whether the overlap between formalisms can be turned into extra supervision through multitask learning. Their testbed is the SemEval 2015 Broad-Coverage Semantic Dependency Parsing (SDP) task, which provides English WSJ sentences annotated in parallel with DM, PAS and PSD, plus an out-of-domain Brown test set.

They first build a single-task graph-based parser. A BiLSTM over word and POS embeddings feeds MLPs that score predicates, unlabeled arcs and labeled arcs, and AD³ decoding enforces consistency constraints. This baseline beats prior systems without hand-crafted features or syntactic input. They then add two multitask extensions:

  • Parameter sharing of the encoder across formalisms (SHARED and FREDA variants).
  • Joint decoding with higher-order structures that link arcs across formalisms.

Both extensions help, and combining them gives the best results. The work shows that parallel annotations in different theories can reinforce one another.

Key points

  • The data is SemEval 2015 Task 18 (SDP). DM comes from DeepBank/LinGO ERG, PAS from the Enju Treebank, and PSD from the Prague Czech-English Dependency Treebank. Only English is used.
  • In-domain average labeled F1 (test) is:
    • 87.1 for the strongest prior system, A&M 2015 (open), which uses syntax;
    • 87.4 for BASIC;
    • 88.0 for FREDA3 and SHARED3.
  • Per formalism, FREDA3 reaches 90.4 on DM, 92.7 on PAS and 78.5 on PSD.
  • Out-of-domain (Brown) average labeled F1 rises from 83.3 (A&M open) and 83.6 (BASIC) to 84.4 with FREDA3.
  • Cross-formalism structure overlap is uneven. Ignoring arc direction, all pairs overlap by more than 50% unlabeled F1. With direction, PSD agrees with the other two formalisms only about 26%, while DM and PAS agree about 64–70%. Correspondingly, higher-order joint modeling raised dev unlabeled F1 on DM and PAS but lowered it on PSD (89.0 → 88.6).
  • The code is released as NeurboParser (github.com/Noahs-ARK/NeurboParser).

Interesting ideas

  • I1. Splitting the scorer into two parts. The model separates an input-side representation from an output-side representation and scores parts by their inner product. This makes it simple to choose which half is shared between tasks, which suits any multi-annotation setup, such as several tagsets over the same Ukrainian corpus.
  • I2. Linking tasks at decoding time. Instead of only sharing an encoder, the authors score structures that span several formalisms and decode them jointly. This could work for any parallel-annotated data, for example jointly predicting syntactic and semantic graphs.
  • I3. Lightweight constraints instead of full structural decoding. The decoder enforces three rules: - a word is a predicate exactly when it has outgoing arcs; - each arc has exactly one label; - certain labels may appear at most once per head, with that list taken from the training data. This keeps inference nearly arc-factored and cheap while still keeping outputs consistent.
  • I4. Measuring task overlap before multitasking. Scoring one formalism's gold graphs against another's, both with and without arc direction, shows in advance where sharing should help. It also explained why PSD gained less. This is a cheap check worth running before combining any two annotation schemes.
  • I5. Strong results without syntax. A syntax-free BiLSTM baseline can outperform syntax-based systems. This is useful for languages or domains where a good syntactic parser is unavailable or unreliable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations🎓University of WashingtonSeattle · United States263 works · 180,581 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Dependency parsing · 120 works

Cite

Cite as Hao Peng, Sam Thomson and Noah A. Smith. 2017. Deep Multitask Learning for Semantic Dependency Parsing. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1186

@inproceedings{peng2017deep,
  title = {{Deep Multitask Learning for Semantic Dependency Parsing}},
  author = {Hao Peng, Sam Thomson and Noah A. Smith},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1186},
  url = {https://doi.org/10.18653/v1/p17-1186}
}

References

76 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mariana S. C. Almeida and André F. T. Martins. 2015. Lisbon: Evaluating TurboSemanticParser on multiple languages and out-of-domain data. In Proc. of SemEval. find
  2. Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah Smith. 2016. Many languages, one parser. TACL 4:431–444.
  3. Rie Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. JMLR 6:1817–1853. find
  4. Collin Baker, Michael Ellsworth, and Katrin Erk. 2007. SemEval’07 task 19: Frame semantic structure extraction. In Proc. of SemEval. find
  5. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proc. of LAW VII & ID.
  6. Yoshua Bengio, Rèjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. JMLR 3:1137–1155.
  7. John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspondence learning. In Proc. of EMNLP.
  8. Carlo E. Bonferroni. 1936. Teoria statistica delle classi e calcolo delle probabilità. Pubblicazioni del R. Istituto Superiore di Scienze Economiche e Commerciali di Firenze 8:3–62. find
  9. Xavier Carreras. 2007. Experiments with a higher-order projective dependency parser. In Proc. of CoNLL.
  10. Rich Caruana. 1997. Multitask learning. Machine Learning 28(1):41–75. find
  11. Kyunghyun Cho. 2015. Natural language understanding with distributed representation. ArXiv:1511.07916. — [arXiv:1511.07916](https://arxiv.org/abs/1511.07916) find
  12. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proc. of ICML. find
  13. Ann Copestake and Dan Flickinger. 2000. An open source grammar development environment and broad-coverage English grammar using HPSG. In Proc. of LREC. find
  14. Ann Copestake, Dan Flickinger, Ivan A. Sag, and Carl Pollard. 2005. Minimal recursion semantics: An introduction. Research on Language & Computation 3(4):281–332. find
  15. Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Proc. of ACL.
  16. Marie-Catherine de Marneffe, Timothy Dozat, Natalia Silveira, Katri Haverinen, Filip Ginter, Joakim Nivre, and Christopher D. Manning. 2014. Universal Stanford dependencies: A cross-linguistic typology. In Proc. of LREC.
  17. Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In Proc. of ICLR.
  18. Yantao Du, Fan Zhang, Weiwei Sun, and Xiaojun Wan. 2014. Peking: Profiling syntactic tree parsing techniques for semantic graph parsing. In Proc. of SemEval. find
  19. Yantao Du, Fan Zhang, Xun Zhang, Weiwei Sun, and Xiaojun Wan. 2015. Peking: Building semantic dependency graphs with a hybrid parser. In Proc. of SemEval. find
  20. Nicholas FitzGerald, Oscar Täckström, Kuzman Ganchev, and Dipanjan Das. 2015. Semantic role labeling with neural network factors. In Proc. of EMNLP. find
  21. Jeffrey Flanigan, Sam Thomson, Jaime Carbonell, Chris Dyer, and Noah A. Smith. 2014. A discriminative graph-based parser for the abstract meaning representation. In Proc. of ACL.
  22. Daniel Flickinger, Yi Zhang, and Valia Kordoni. 2012. DeepBank: A dynamically annotated treebank of the Wall Street Journal. In Proc. of TLT. find
  23. Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Computational Linguistics 28(3):245–288.
  24. Alex Graves. 2012. Supervised Sequence Labelling with Recurrent Neural Networks, volume 385 of Studies in Computational Intelligence. Springer. find
  25. Alex Graves. 2013. Generating sequences with recurrent neural networks. ArXiv 1308.0850. find
  26. Jiang Guo, Wanxiang Che, Haifeng Wang, and Ting Liu. 2016. A universal framework for inductive transfer parsing across multi-typed treebanks. In Proc. of COLING. find
  27. Jan Hajič, Eva Hajičová, Jarmila Panevová, Petr Sgall, Ondřej Bojar, Silvie Cinková, Eva Fučı́ková, Marie Mikulová, Petr Pajas, Jan Popelka, Jiřı́ Semecký, Jana Šindlerová, Jan Štěpánek, Josef Toman, Zdeňka Urešová, and Zdeněk Žabokrtský. 2012. Announcing Prague Czech-English d… find
  28. James Henderson, Paola Merlo, Ivan Titov, and Gabriele Musillo. 2013. Multi-lingual joint parsing of syntactic and semantic dependencies with a latent variable model. Computational Linguistics 39(4):949–998. find
  29. Karl Moritz Hermann, Dipanjan Das, Jason Weston, and Kuzman Ganchev. 2014. Semantic frame identification with distributed word representations. In Proc. of ACL.
  30. Frank L. Hitchcock. 1927. The expression of a tensor or a polyadic as a sum of products. Journal of Mathematical Physics 6(1):164–189. find
  31. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9(8):1735–1780. find
  32. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proc. of ACL.
  33. Richard Johansson. 2013. Training parsers on incompatible treebanks. In Proc. of NAACL. find
  34. Jenna Kanerva, Juhani Luotolahti, and Filip Ginter. 2015. Turku: Semantic dependency parsing as a sequence classification. In Proc. of SemEval. find
  35. Tao Lei, Yu Xin, Yuan Zhang, Regina Barzilay, and Tommi Jaakkola. 2014. Low-rank tensors for scoring dependency structures. In Proc. of ACL. find
  36. Xavier Lluı́s, Xavier Carreras, and Lluı́s Màrquez. 2013. Joint arc-factored parsing of syntactic and semantic dependencies. TACL 1:219–230. find
  37. Xuezhe Ma and Eduard Hovy. 2016. Neural probabilistic model for non-projective MST parsing. ArXiv 1701.00874. find
  38. André F. T. Martins and Mariana S. C. Almeida. 2014. Priberam: A turbo semantic parser with second order features. In Proc. of SemEval. find
  39. André F. T. Martins, Miguel B. Almeida, and Noah A. Smith. 2013. Turning on the turbo: Fast third-order non-projective turbo parsers. In Proc. of ACL.
  40. André F. T. Martins, Noah Smith, and Eric Xing. 2009. Concise integer linear programming formulations for dependency parsing. In Proc. of ACL. find
  41. Young-Bum Kim, Karl Stratos, and Ruhi Sarikaya. 2016. Frustratingly easy neural domain adaptation. In Proc. of COLING. find
  42. André F. T. Martins, Noah A. Smith, Pedro M. Q. Aguiar, and Mário A. T. Figueiredo. 2011. Dual decomposition with many overlapping components. In Proc. of EMNLP. find
  43. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proc. of ICLR. find
  44. Ryan McDonald, Koby Crammer, and Fernando Pereira. 2005. Online large-margin training of dependency parsers. In Proc. of ACL.
  45. Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. TACL 4:313– 327.
  46. Yusuke Miyao. 2006. From linguistic theory to syntactic analysis: Corpus-oriented grammar development and feature forest model. find
  47. Tamara G. Kolda and Brett W. Bader. 2009. Tensor decompositions and applications. SIAM Review 51(3):455–500. find
  48. Terry Koo, Alexander M. Rush, Michael Collins, Tommi Jaakkola, and David Sontag. 2010. Dual decomposition for parsing with non-projective head automata. In Proc. of EMNLP.
  49. Meghana Kshirsagar, Sam Thomson, Nathan Schneider, Jaime Carbonell, Noah A. Smith, and Chris Dyer. 2015. Frame-semantic role labeling with heterogeneous annotations. In Proc. of ACL.
  50. Marco Kuhlmann. 2014. Linköping: Cubic-time graph parsing with a simple scoring scheme. In Proc. of SemEval. find
  51. Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, and Noah A. Smith. 2016. Distilling an ensemble of greedy dependency parsers into one MST parser. In Proc. of EMNLP. find
  52. Tom Kwiatkowski, Eunsol Choi, Yoav Artzi, and Luke S. Zettlemoyer. 2013. Scaling semantic parsers with on-the-fly ontology matching. In Proc. of EMNLP.
  53. Andriy Mnih and Geoffrey Hinton. 2007. Three new graphical models for statistical language modelling. In Proc. of ICML. find
  54. Stephan Oepen, Marco Kuhlmann, Yusuke Miyao, Daniel Zeman, Silvie Cinková, Dan Flickinger, Jan Hajič, and Zdeňka Urešová. 2015. find
  55. SemEval 2015 task 18: Broad-coverage semantic dependency parsing. In Proc. of SemEval. find
  56. Stephan Oepen, Marco Kuhlmann, Yusuke Miyao, Daniel Zeman, Dan Flickinger, Jan Hajič, Angelina Ivanova, and Yi Zhang. 2014. find
  57. SemEval 2014 task 8: Broad-coverage semantic dependency parsing. In Proc. of SemEval. find
  58. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics 31(1):71–106.
  59. Wenzhe Pei, Tao Ge, and Baobao Chang. 2015. An effective neural network model for graph-based dependency parsing. In Proc. of ACL. find
  60. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Proc. of EMNLP.
  61. Carl Pollard and Ivan A. Sag. 1994. Head-Driven Phrase Structure Grammar. The University of Chicago Press. find
  62. Yuan Zhang and David Weiss. 2016. Stack-propagation: Improved representation learning for syntax. In Proc. of ACL. find
  63. Mike Schuster and Kuldip K. Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing 45(11):2673–2681. find
  64. David Smith and Jason Eisner. 2008. Dependency parsing by belief propagation. In Proc. of EMNLP.
  65. Anders Søgaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In Proc. of ACL.
  66. Vivek Srikumar and Christopher D Manning. 2014. Learning distributed representations for structured output prediction. In Proc. of NIPS. find
  67. Mihai Surdeanu, Richard Johansson, Adam Meyers, Lluı́s Màrquez, and Joakim Nivre. 2008. The CoNLL-2008 shared task on joint parsing of syntactic and semantic dependencies. In Proc. of CoNLL.
  68. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proc. of NIPS. find
  69. Swabha Swayamdipta, Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2016. Greedy, joint syntactic-semantic parsing with stack LSTMs. In Proc. of CoNLL. find
  70. Ben Taskar, Carlos Guestrin, and Daphne Koller. 2004. Max-margin Markov networks. In Advances in Neural Information Processing Systems 16. find
  71. Hillel Taub-Tabib, Yoav Goldberg, and Amir Globerson. 2015. Template kernels for dependency parsing. In Proc. of NAACL. find
  72. Sam Thomson, Brendan O’Connor, Jeffrey Flanigan, David Bamman, Jesse Dodge, Swabha Swayamdipta, Nathan Schneider, Chris Dyer, and Noah A. Smith. 2014. CMU: Arc-factored, discriminative semantic dependency parsing. In Proc. of SemEval. find
  73. Wenhui Wang and Baobao Chang. 2016. Graph-based dependency parsing with bidirectional LSTM. In Proc. of ACL. find
  74. Naiwen Xue, Fei Xia, Fu-dong Chiou, and Martha Palmer. 2005. The Penn Chinese TreeBank: Phrase structure annotation of a large corpus. Natural Language Engineering 11(2):207–238. find
  75. Xun Zhang, Yantao Du, Weiwei Sun, and Xiaojun Wan. 2016. Transition-based parsing for deep dependency structures. Computational Linguistics 42(3):353–389. find
  76. Yuan Zhang, Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. 2014. Greed is good if randomized: New inference for dependency parsing. In Proc. of EMNLP. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/peng-2017-deep-multitask-learning-semantic-dependency/