Elamur Research
☰
home / papers / 2016 / context2vec: Learning Generic Context Embedding with Bidirectional LSTM

context2vec: Learning Generic Context Embedding with Bidirectional LSTM

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/K16-1006.pdf

Summary

Many NLP tasks need a good representation of the sentence surrounding a target word, yet unsupervised approaches have usually just averaged the word embeddings in a fixed window (AWE), which never optimizes the context as a whole. The authors propose context2vec, an unsupervised model and toolkit that learns a generic embedding function for full, variable-length sentential contexts. Its architecture follows word2vec CBOW, but the averaged window is replaced by two independent LSTMs, one reading left-to-right and one right-to-left, that skip the target word itself. Their outputs are concatenated and passed through an MLP. Context and target embeddings share one space and are trained with word2vec's negative sampling objective on corpora of billions of words. The authors show that this setup implicitly factorizes a PMI matrix between targets and whole contexts. Relying solely on cosine similarity between embeddings, context2vec clearly outperforms AWE and performs on par with or better than the strongest previous systems for completing sentences, substituting words in context, and supervised WSD. The released pre-trained models could therefore serve as a drop-in context representation for unsupervised and semi-supervised systems.

Key points

  • Architecture: a bidirectional LSTM with separate left and right word embeddings, followed by an MLP with ReLU. Training uses negative sampling, and the smoothing factor α controls bias toward rare words, with higher values favoring them.
  • Benchmarks: MSCC (sentence completion), LST-07 and LST-14 (lexical substitution), and Senseval-3 (supervised WSD).
  • Sentence completion: on MSCC, the model trained with more iterations reaches about 64.0, against roughly 48 for AWE and 58.9 for the prior best system.
  • WSD and lexical substitution: on Senseval-3 it scores about 72.8, near the best prior result of 74.1. On the lexical substitution benchmarks it is comparable to Melamud et al. (2015a/b), whose systems rely on exemplars or dependency parsing.
  • Resources: source code and pre-trained models are publicly released by Bar-Ilan University.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Word sense disambiguation

Cited by

152 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 142 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/k16-1006

@inproceedings{melamud2016context2vec,
  title = {{context2vec: Learning Generic Context Embedding with Bidirectional LSTM}},
  author = {Melamud, Oren and Goldberger, Jacob and Dagan, Ido},
  booktitle = {Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning},
  year = {2016},
  doi = {10.18653/v1/k16-1006},
  url = {https://doi.org/10.18653/v1/k16-1006}
}

References

36 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rie Kubota Ando. 2006. Applying alternating structure optimization to word sense disambiguation. In Proceedings of the Tenth Conference on Computational Natural Language Learning, pages 77–84. Association for Computational Linguistics. find
  2. Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014. Tailoring continuous word representations for dependency parsing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find our card
  3. Xinxiong Chen, Zhiyuan Liu, and Maosong Sun. 2014. A unified model for word sense representation and disambiguation. In Proceedings of EMNLP. find
  4. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. find
  5. Quan Liu, Hui Jiang, Si Wei, Zhen-Hua Ling, and Yu Hu. 2015. Learning semantic word embeddings based on ordinal knowledge constraints. Proceedings of ACL. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537. find
  6. Adriano Ferraresi, Eros Zanchetta, Marco Baroni, and Silvia Bernardini. 2008. Introducing and evaluating ukwac, a very large web-derived corpus of English. In Proceedings of the 4th Web as Corpus Workshop (WAC-4). find
  7. Diana McCarthy and Roberto Navigli. 2007. Semeval-2007 task 10: English lexical substitution task. In Proceedings of SemEval. find
  8. Oren Melamud, Ido Dagan, and Jacob Goldberger. 2015a. Modeling word meaning in context with substitute vectors. In Proceedings of ACL. find
  9. Eric H. Huang, Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of ACL. find
  10. Oren Melamud, Omer Levy, and Ido Dagan. 2015b. A simple word embedding model for lexical substitution. In Proceedings of Workshop on Vector Space Modeling for NLP (VSM). find
  11. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410. find
  12. Oren Melamud, David McClosky, Siddharth Patwardhan, and Mohit Bansal. 2016. The role of context types and dimensionality in learning word embeddings. In Proceedings of NAACL. find
  13. Mikael Kågebäck, Fredrik Johansson, Richard Johansson, and Devdatt Dubhashi. 2015. Neural context embeddings for automatic discovery of word senses. In Proceedings of NAACL. find
  14. Rada Mihalcea, Timothy Anatolievich Chklovski, and Adam Kilgarriff. 2004. The senseval-3 english lexical sample task. ACL. find
  15. Kazuya Kawakami and Chris Dyer. 2016. Learning to represent words in context with multilingual supervision. In Workshop in ICLR. find
  16. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  17. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  18. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS. find
  19. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Proceedings of NIPS. find
  20. Piotr Mirowski and Andreas Vlachos. 2015. Dependency recurrent neural language models for sentence completion. arXiv preprint arXiv:1507.01193. find
  21. Gerhard Kremer, Katrin Erk, Sebastian Padó, and Stefan Thater. 2014. What substitutes tell us-analysis of an all-words lexical substitution corpus. In Proceedings of EACL. find
  22. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings EMNLP. find
  23. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360. find
  24. Sascha Rothe and Hinrich Schütze. 2015. Autoextend: Extending word embeddings to embeddings for synsets and lexemes. Proceedings of ACL. find
  25. Omer Levy and Yoav Goldberg. 2014a. Dependency-based word embeddings. In Proceedings of ACL. find
  26. Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, and Christopher D Manning. 2011. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of EMNLP. find our card
  27. Omer Levy and Yoav Goldberg. 2014b. Neural word embeddings as implicit matrix factorization. In Proceedings of NIPS. find
  28. György Szarvas, Róbert Busa-Fekete, and Eyke Hüllermeier. 2013. Learning to rank lexical substitutions. In Proceedings of EMNLP. find
  29. Wang Ling, Lin Chu-Cheng, Yulia Tsvetkov, and Silvio Amir. 2015a. Not all contexts are created equal: Better word representations with variable attention. In Proceedings of EMNLP. find
  30. Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clayton. 2015. Chainer: a next-generation open source framework for deep learning. In Proceedings of Workshop on Machine Learning Systems (LearningSys) in NIPS. find
  31. Wang Ling, Chris Dyer, Alan Black, and Isabel Trancoso. 2015b. Two/too simple adaptations of word2vec for syntax problems. In Proceedings of NAACL. find our card
  32. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semisupervised learning. In Proceedings of ACL. find
  33. Mehmet Ali Yatbaz, Enis Sert, and Deniz Yuret. 2012. Learning syntactic categories using paradigmatic representations of word context. In Proceedings EMNLP. find
  34. Deniz Yuret. 2012. FASTSUBS: An efficient and exact procedure for finding the most likely lexical substitutes based on an n-gram language model. Signal Processing Letters, IEEE, 19(11):725–728. find
  35. Jie Zhou and Wei Xu. 2015. End-to-end learning of semantic role labeling using recurrent neural networks. In Proceedings of ACL. find
  36. Geoffrey Zweig and Christopher JC Burges. 2011. The microsoft research sentence completion challenge. Technical report, Technical Report MSR-TR-2011-129, Microsoft. find