Elamur Research
☰
home / papers / 2014 / Tailoring Continuous Word Representations for Dependency Parsing

Tailoring Continuous Word Representations for Dependency Parsing

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/P14-2131.pdf

Summary

The paper asks whether continuous word embeddings can give English dependency parsing the same kind of boost that Brown clusters already provide. The authors add embedding-based features to a second-order MSTParser. They benchmark publicly released vectors (SENNA, TURIAN, HUANG) against the BROWN clustering used by Koo et al. (2008) and against word2vec CBOW and SKIP models that they trained on BLLIP after excluding PTB.

Converting single vector dimensions into bucketed indicator features barely helps. Gains appear only once the vectors are hierarchically clustered and bit-string prefixes from that hierarchy are used as features. To adapt embeddings to syntax, the authors shrink the context window and introduce SKIPDEP, a skip-gram model trained on child–parent–grandparent–label tuples taken from an automatically parsed corpus.

On the Penn Treebank and the English Web treebank, every representation improves the baseline. SKIPDEP matches BROWN while training in minutes rather than days, and a simple voting ensemble over all representations performs best. This suggests the representations carry complementary information.

Key points

  • On WSJ, UAS rises from 91.95 (baseline) to 92.69 on test with BROWN features (dev: 92.38 to 93.18). SKIPDEP reaches 93.33 on dev and is statistically indistinguishable from BROWN and CBOW. LAS shows similar improvements.
  • Window size changes the embeddings strongly. Small windows yield POS-like clusters with higher M-1 and lower SIM, while large windows yield topical clusters. SKIP dev UAS drops from 92.94 (w=1) to 92.86 (w=5) to 92.70 (w=10).
  • In the intrinsic evaluation, SKIPDEP has the highest M-1 among continuous models (88.3, versus 89.3 for BROWN). HUANG has the highest SIM score (62.6) but the lowest M-1 apart from SKIP with w=10.
  • Training cost differs widely. word2vec models train in 2–4 minutes on BLLIP, plus about 4 hours of hierarchical clustering. BROWN took 2.5 days and SENNA about 2 months.
  • On the Web treebank, macro-average UAS goes from 83.5 (baseline) to 84.3 (SENNA), 84.2 (BROWN) and 84.1 (SKIPDEP), with the ALL ensemble reaching 84.9. SKIPDEP vectors and bit strings are released at ttic.edu/bansal/data/syntacticEmbeddings.zip.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Dependency parsing · Treebanks

Cited by

128 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 119 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p14-2131

@inproceedings{bansal2014tailoring,
  title = {{Tailoring Continuous Word Representations for Dependency Parsing}},
  author = {Bansal, Mohit and Gimpel, Kevin and Livescu, Karen},
  booktitle = {Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2014},
  doi = {10.3115/v1/p14-2131},
  url = {https://doi.org/10.3115/v1/p14-2131}
}

References

43 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rami Al-Rfou’, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual NLP. In Proceedings of CoNLL. find our card
  2. Jacob Andreas and Dan Klein. 2014. How much do word embeddings encode about syntax? In Proceedings of ACL. find
  3. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155, March. find
  4. Jordan L. Boyd-Graber and David M. Blei. 2008. Syntactic topic models. In Proceedings of NIPS. find
  5. Peter F. Brown, Peter V. Desouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai. 1992. Class-based N-gram models of natural language. Computational Linguistics, 18(4):467–479. find
  6. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537. find our card
  7. Scott C. Deerwester, Susan T. Dumais, Thomas K. Landauer, George W. Furnas, and Richard A. Harshman. 1990. Indexing by latent semantic analysis. JASIS, 41(6):391–407. find
  8. Paramveer Dhillon, Dean P. Foster, and Lyle H. Ungar. 2011. Multi-view learning of word embeddings via CCA. In Proceedings of NIPS. find
  9. Paramveer Dhillon, Jordan Rodu, Dean P. Foster, and Lyle H. Ungar. 2012. Two Step CCA: A new spectral method for estimating vector models of words. In Proceedings of ICML. find
  10. Bradley Efron and Robert J. Tibshirani. 1994. An introduction to the bootstrap, volume 57. CRC press. find
  11. Jenny Rose Finkel, Alex Kleeman, and Christopher D. Manning. 2008. Efficient, feature-based, conditional random field parsing. In Proceedings of ACL. find
  12. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002. Placing search in context: The concept revisited. ACM Transactions on Information Systems, 20(1):116–131. find
  13. Yoav Goldberg and Jon Orwant. 2013. A dataset of syntactic-ngrams over time from a very large corpus of English books. In Second Joint Conference on Lexical and Computational Semantics (* SEM), volume 1, pages 241–247. find
  14. Edouard Grave, Guillaume Obozinski, and Francis Bach. 2013. Hidden markov tree models for semantic class induction. In Proceedings of CoNLL. find
  15. Gholamreza Haffari, Marzieh Razavi, and Anoop Sarkar. 2011. An ensemble model that combines syntactic and semantic clustering for discriminative dependency parsing. In Proceedings of ACL. find
  16. Sorami Hisamoto, Kevin Duh, and Yuji Matsumoto. 2013. An empirical investigation of word representations for parsing the web. In Proceedings of ANLP. find
  17. Eric H. Huang, Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving Word Representations via Global Context and Multiple Word Prototypes. In Proceedings of ACL. find
  18. Fei Huang, Arun Ahuja, Doug Downey, Yi Yang, Yuhong Guo, and Alexander Yates. 2014. Learning representations for weakly supervised natural language processing tasks. Computational Linguistics, 40(1). find
  19. Richard Johansson and Pierre Nugues. 2007. Extended constituent-to-dependency conversion for English. In 16th Nordic Conference of Computational Linguistics. find our card
  20. Terry Koo, Xavier Carreras, and Michael Collins. 2008. Simple semi-supervised dependency parsing. In Proceedings of ACL. find our card
  21. Dekang Lin and Xiaoyun Wu. 2009. Phrase clustering for discriminative learning. In Proceedings of ACL-IJCNLP. find
  22. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19:313–330. find our card
  23. Ryan T. McDonald and Fernando C. Pereira. 2006. Online learning of approximate dependency parsing algorithms. In Proceedings of EACL. find our card
  24. Ryan T. McDonald, Koby Crammer, and Fernando C. Pereira. 2005. Spanning tree methods for discriminative training of dependency parsers. Technical Report MS-CIS-05-11, University of Pennsylvania. find
  25. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In Proceedings of ICLR. find
  26. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS. find
  27. Scott Miller, Jethran Guinness, and Alex Zamanian. 2004. Name tagging with word clusters and discriminative training. In Proceedings of HLT-NAACL. find
  28. Andriy Mnih and Geoffrey Hinton. 2007. Three new graphical models for statistical language modelling. In Proceedings of ICML. find
  29. Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction, AKBC-WEKEX ’12, pages 95–100, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  30. Olutobi Owoputi, Brendan OConnor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A. Smith. 2013. Improved part-of-speech tagging for online conversational text with word clusters. In Proceedings of NAACL. find our card
  31. Sebastian Padó and Mirella Lapata. 2007. Dependency-based construction of semantic space models. Computational Linguistics, 33(2):161–199. find our card
  32. Slav Petrov and Ryan McDonald. 2012. Overview of the 2012 shared task on parsing the web. In Notes of the First Workshop on Syntactic Analysis of Non-Canonical Language (SANCL). find
  33. Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of CoNLL. find our card
  34. Adwait Ratnaparkhi. 1996. A maximum entropy model for part-of-speech tagging. In Proceedings of EMNLP. find
  35. Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. Named entity recognition in tweets: An experimental study. In Proceedings of EMNLP. find
  36. Kenji Sagae and Andrew S. Gordon. 2009. Clustering words by syntactic similarity improves dependency parsing of predicate-argument structures. In Proceedings of the 11th International Conference on Parsing Technologies. find
  37. Valentin I. Spitkovsky, Hiyan Alshawi, Angel X. Chang, and Daniel Jurafsky. 2011. Unsupervised dependency parsing without gold part-of-speech tags. In Proceedings of EMNLP. find
  38. Oscar Täckström, Ryan McDonald, and Jakob Uszkoreit. 2012. Cross-lingual word clusters for direct transfer of linguistic structure. In Proceedings of NAACL. find our card
  39. Stephen Tratz and Eduard Hovy. 2011. A fast, accurate, non-projective, semantically-enriched parser. In Proceedings of EMNLP. find
  40. Joseph Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of ACL. find our card
  41. David Vadas and James R. Curran. 2007. Adding noun phrase structure to the Penn Treebank. In Proceedings of ACL. find our card
  42. Joe H. Ward. 1963. Hierarchical grouping to optimize an objective function. Journal of the American statistical association, 58(301):236–244. find
  43. Hiroyasu Yamada and Yuji Matsumoto. 2003. Statistical dependency analysis with support vector machines. In Proceedings of International Conference on Parsing Technologies. find our card