Elamur Research
☰
home / papers / 2015 / Two/Too Simple Adaptations of Word2Vec for Syntax Problems

Two/Too Simple Adaptations of Word2Vec for Syntax Problems

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/N15-1142.pdf

Summary

Word2Vec's skip-gram and CBOW models ignore the order of context words. That works well for semantic similarity but is a poor fit for syntax, where a word's position relative to its neighbours carries much of the signal. The authors propose two lightweight, order-aware variants. Structured Skip-gram uses a separate output matrix for each relative context position. CWindow replaces CBOW's summed context with an ordered concatenation of context embeddings. They trained 50-dimensional vectors on a large 2014 English Wikipedia dump, WIKI(L), and on a large tweet sample, TWITTER, using a single day on ordinary hardware. They then used these vectors to initialise neural taggers and parsers. On Penn Treebank POS tagging, ARK Twitter POS tagging, and PTB dependency parsing with Stanford Dependencies, both variants beat the original models. They also matched or exceeded SENNA embeddings. Qualitatively, nearest neighbours grouped words more by syntactic class and captured spelling variants in tweets. The work matters because it gives syntax-friendly embeddings with little extra cost or complexity, and the code is released as wang2vec.

Key points

  • Corpora: WIKI(L) has about 1,897M words (60M sentences) and TWITTER has 56M tweets (847M words). WIKI(S), a 16M-word set, was used for contrast. Training used window c = 5, 10 negative samples, a minimum count of 40, and 50 dimensions.
  • POS tagging (test accuracy):
    • On PTB, Structured Skip-gram reached 97.05 and CWindow 97.01, compared with 96.68 for Skip-gram, 96.13 for CBOW and 96.58 for SENNA.
    • On ARK Twitter, Structured Skip-gram reached 89.79, slightly above the 89.50 obtained with Brown clusters alone.
  • Dependency parsing (Chen and Manning parser, PTB test): Structured Skip-gram scored UAS 92.24 / LAS 89.92. The other models scored:
    • Skip-gram: 91.90 / 89.55
    • CBOW: 91.52 / 88.93
    • SENNA: 92.03 / 89.51
  • Speed: Structured Skip-gram processes 34.64k words/s versus 71.35k for Skip-gram. CWindow processes 124.43k words/s versus 342.17k for CBOW. Even so, every embedding set was trained in under a day.
  • Code: released publicly at github.com/wlin12/wang2vec.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Part-of-speech tagging

Cited by

120 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 114 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/n15-1142

@inproceedings{ling2015two,
  title = {{Two/Too Simple Adaptations of Word2Vec for Syntax Problems}},
  author = {Ling, Wang and Dyer, Chris and Black, Alan W. and Trancoso, Isabel},
  booktitle = {Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2015},
  doi = {10.3115/v1/n15-1142},
  url = {https://doi.org/10.3115/v1/n15-1142}
}

References

23 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas and Dan Klein. 2014. How much do word embeddings encode about syntax. In Proceedings of ACL. find
  2. Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014. Tailoring continuous word representations for dependency parsing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find our card
  3. Danqi Chen and Christopher D Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 740–750. find our card
  4. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493– 2537. find our card
  5. Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. 2014. find
  6. Fast and robust neural network joint models for statistical machine translation. In 52nd Annual Meeting of the Association for Computational Linguistics, Baltimore, MD, USA, June. find
  7. Paramveer Dhillon, Dean P Foster, and Lyle H Ungar. 2011. Multi-view learning of word embeddings via cca. In Advances in Neural Information Processing Systems, pages 199–207. find
  8. Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. 2010. Why does unsupervised pre-training help deep learning? The Journal of Machine Learning Research, 11:625–660. find
  9. Manaal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. In Proceedings of EACL. find
  10. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A Smith. 2011. Part-of-speech tagging for twitter: Annotation, features, and experiments. In Proceedings of the 49th Annual Meeting of the Associ… find
  11. Yoav Goldberg and Omer Levy. 2014. word2vec explained: deriving mikolov et al.’s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722. — [arXiv:1402.3722](https://arxiv.org/abs/1402.3722) find
  12. Jiang Guo, Wanxiang Che, Haifeng Wang, and Ting Liu. 2014. Revisiting embedding features for simple semi-supervised learning. In Proceedings of EMNLP. find
  13. Geoffrey E Hinton and Ruslan Salakhutdinov. 2012. A better way to pretrain deep boltzmann machines. In Advances in Neural Information Processing Systems, pages 2447–2455. find
  14. Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 873–882. Association for… find
  15. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP, pages 1700–1709. find
  16. Lingpeng Kong, Nathan Schneider, Swabha Swayamdipta, Archna Bhatia, Chris Dyer, and Noah A. Smith. 2014. A dependency parser for tweets. In Proc. of EMNLP, pages 1001–1012, Doha, Qatar, October. find our card
  17. Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, volume 2. find
  18. Shujie Liu, Nan Yang, Mu Li, and Ming Zhou. 2014. A recursive recurrent neural network for statistical machine translation. In Proceedings of ACL, pages 1491– 1500. find
  19. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find
  20. Olutobi Owoputi, Brendan O’Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A Smith. 2013. Improved part-of-speech tagging for online conversational text with word clusters. In HLT-NAACL, pages 380–390. find our card
  21. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104–3112. find
  22. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, ACL ’10, pages 384–394, Stroudsburg, PA, USA. Association for Computational… find our card
  23. Yi Yang and Jacob Eisenstein. 2015. Unsupervised multi-domain adaptation with feature embeddings. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics. find