Elamur Research⌕
☰
home / papers / 2014 / Improved CCG Parsing with Semi-supervised Supertagging

Improved CCG Parsing with Semi-supervised Supertagging

Paper · 2014 · Mike Lewis and Mark Steedman

Summary

CCG parsers depend heavily on supertagging, which assigns each word a rich lexical category. This step suffers from sparse labelled data, unseen words, and errors passed on from a separate POS-tagger, especially outside newswire. Lewis and Steedman replace the POS-tagging and supertagging stages of the C&C pipeline with a supertagger based on unsupervised word embeddings. It uses only embedding, 2-character suffix and capitalisation features from a context window, with no POS or lexical features. They compare two classifiers: the window-based neural network of Collobert et al. (2011b), which fine-tunes embeddings during training, and the linear-chain CRF of Turian et al. (2010). They test several publicly available embedding sets and train on CCGBank Sections 02-21. Labelled dependency parsing improves over the supervised C&C parser on WSJ, Wikipedia and biomedical text, with the largest gains out of domain. This shows that embeddings help structured prediction when the lexicon encodes hierarchy. It also gives a simpler pipeline that does not need a POS-tagger.

Key points

  • Dependency parsing gains over the C&C baseline are 0.8% on WSJ (CCGBank Section 23), 1.8% on Wikipedia (200 sentences annotated by Honnibal et al., 2009) and 3.4% on biomedical text (Bioinfer, with CCG annotation by Rimell and Clark, 2008).
  • The embeddings compared were Collobert&Weston, Skip-Gram, Turian, HLBL and Mikolov RNNLM, with 25 to 640 dimensions and training corpora of 37M to 100B words.
  • The best embeddings were Turian-50, trained on only 37M words, which suggests that larger unlabelled corpora could help further.
  • The C&C supertagger reaches 91.5% 1-best accuracy on CCGBank Section 00. As in C&C, the models are restricted to the 425 most frequent categories.
  • The largest gains appear on words with incorrect POS-tags and on word–category pairs unseen in training. The code is released in the EasyCCG parser (github.com/mikelewis0/easyccg).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Syntax & grammar formalisms · 214 worksPart-of-speech tagging · 126 works

Cite

Cite as Mike Lewis and Mark Steedman. 2014. Improved CCG Parsing with Semi-supervised Supertagging. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00186

@article{lewis2014improved,
  title = {{Improved CCG Parsing with Semi-supervised Supertagging}},
  author = {Mike Lewis and Mark Steedman},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2014},
  doi = {10.1162/tacl\_a\_00186},
  url = {https://doi.org/10.1162/tacl_a_00186}
}

References

38 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas and Dan Klein. 2014. How much do word embeddings encode about syntax. In Proceedings of ACL. find
  2. Michael Auli and Adam Lopez. 2011a. A comparison of loopy belief propagation and dual decomposition for integrated CCG supertagging and parsing. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 470–480. Associatio… find
  3. Michael Auli and Adam Lopez. 2011b. Training a log-linear parser with loss functions via softmax-margin. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 333–343. Association for Computational Linguistics. find
  4. Srinivas Bangalore and Aravind K Joshi. 1999. Supertagging: An approach to almost parsing. Computational Linguistics, 25(2):237–265.
  5. Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014. Tailoring continuous word representations for dependency parsing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics.
  6. Johan Bos. 2008. Wide-coverage semantic analysis with boxer. In Johan Bos and Rodolfo Delmonte, editors, Semantics in Text Processing. find
  7. STEP 2008 Conference Proceedings, Research in Computational Semantics, pages 277–286. find
  8. College Publications. P.F. Brown, P.V. Desouza, R.L. Mercer, V.J.D. Pietra, and J.C. Lai. 1992. Class-based n-gram models of natural language. Computational Linguistics, 18(4):467–479. find
  9. Stephen Clark and James R Curran. 2007. Wide-coverage efficient statistical parsing with CCG and log-linear models. Computational Linguistics, 33(4):493–552.
  10. Michael Collins. 2002. Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, pages 1–8. Association for Computational Linguistics.
  11. Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. 2011a. Torch7: A Matlab-like environment for machine learning. In BigLearn, NIPS Workshop. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011b. Natural language processing (almost) from scrat…
  12. Ronan Collobert. 2011. Deep learning for efficient discriminative parsing. In Geoffrey J. Gordon, David B. Dunson, and Miroslav Dudk, editors, AIS-TATS, volume 15 of JMLR Proceedings, pages 224– 232. find
  13. JMLR.org. James R Curran, Stephen Clark, and David Vadas. 2006. Multi-tagging for lexicalized-grammar parsing. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 697–704. Association for… find
  14. Tejaswini Deoskar, Markos Mylonakis, and Khalil Sima’an. 2011. Learning structural dependencies of words in the zipfian tail. In Proceedings of the 12th International Conference on Parsing Technologies, pages 80–91. Association for Computational Linguistics. find
  15. Tejaswini Deoskar, Christos Christodoulopoulos, Alexandra Birch, and Mark Steedman. 2014. Generalizing a Strongly Lexicalized Parser using Unlabeled Data. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Lin… find
  16. Mark Dredze, John Blitzer, Partha Pratim Talukdar, Kuzman Ganchev, Joao Graca, and Fernando Pereira. 2007. Frustratingly hard domain adaptation for dependency parsing. In EMNLP-CoNLL, pages 1051–1055.
  17. Julia Hockenmaier and Mark Steedman. 2007. CCGbank: a corpus of CCG derivations and dependency structures extracted from the Penn Treebank. Computational Linguistics, 33(3):355–396.
  18. Matthew Honnibal, Joel Nothman, and James R Curran. 2009. Evaluating a statistical CCG parser on Wikipedia. In Proceedings of the 2009 Workshop on The People’s Web Meets NLP: Collaboratively Constructed Semantic Resources, pages 38–41. Association for Computational Linguistics.
  19. M. Honnibal, J.R. Curran, and J. Bos. 2010. Rebanking CCGbank for improved NP interpretation. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 207–215. Association for Computational Linguistics. find
  20. Terry Koo, Xavier Carreras, and Michael Collins. 2008. Simple semi-supervised dependency parsing.
  21. Jayant Krishnamurthy and Tom M. Mitchell. 2012. Weakly supervised training of semantic parsers. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL ’12, pages 754–765. Association for Computational L…
  22. Tom Kwiatkowski, Luke Zettlemoyer, Sharon Goldwater, and Mark Steedman. 2010. Inducing probabilistic CCG grammars from logical form with higher-order unification. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, EMNLP ’10, pages 1223–1233. Association for Co… find
  23. Tom Kwiatkowski, Eunsol Choi, Yoav Artzi, and Luke Zettlemoyer. 2013. Scaling semantic parsers with on-the-fly ontology matching. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1545–1556, Seattle, Washington, USA, October. Association for Computation…
  24. Mike Lewis and Mark Steedman. 2013a. Combined Distributional and Logical Semantics. Transactions of the Association for Computational Linguistics, 1:179– 192. find
  25. Mike Lewis and Mark Steedman. 2013b. Unsupervised induction of cross-lingual semantic relations. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 681–692, Seattle, Washington, USA, October. Association for Computational Linguistics. find
  26. Mike Lewis and Mark Steedman. 2014. A* CCG Parsing with a Supertag-factored Model. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar, October.
  27. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330.
  28. David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of the main conference on human language technology conference of the North American Chapter of the Association of Computational Linguistics, pages 152–159. Association for Computational Ling…
  29. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
  30. Tomáš Mikolov. 2012. Statistical language models based on neural networks. Ph.D. thesis, Ph. D. thesis, Brno University of Technology. find
  31. Andriy Mnih and Geoffrey E Hinton. 2008. A scalable hierarchical distributed language model. In NIPS, pages 1081–1088. Joakim Nivre, Laura Rimell, Ryan McDonald, and Carlos Gómez Rodrı́guez. 2010. Evaluation of dependency parsers on unbounded dependencies. In Proceedings of the 23rd International C…
  32. Coling 2010 Organizing Committee. Naoaki Okazaki. CRFsuite: a fast implementation of conditional random fields (CRFs). Sampo Pyysalo, Filip Ginter, Juho Heimonen, Jari Björne, Jorma Boberg, Jouni Järvinen, and Tapio Salakoski. 2007. Bioinfer: a corpus for information extraction in the biomedical d… find
  33. Laura Rimell and Stephen Clark. 2008. Adapting a lexicalized-grammar parser to contrasting domains. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 475–484. Association for Computational Linguistics. find
  34. Laura Rimell and Stephen Clark. 2009. Porting a lexicalized-grammar parser to the biomedical domain. Journal of Biomedical Informatics, 42(5):852–865. find
  35. Laura Rimell, Stephen Clark, and Mark Steedman. 2009. Unbounded dependency recovery for parser evaluation. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 2-Volume 2, pages 813–821. Association for Computational Linguistics.
  36. Richard Socher, John Bauer, Christopher D Manning, and Andrew Y Ng. 2013. Parsing with compositional vector grammars. In In Proceedings of the ACL conference. find
  37. Citeseer. Emily Thomforde and Mark Steedman. 2011. Semi-supervised CCG lexicon extension. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1246–1256. Association for Computational Linguistics. find
  38. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 384–394. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lewis-2014-improved-ccg-parsing-semi-supervised/