Elamur Research⌕
☰
home / papers / 2016 / Encoding Prior Knowledge with Eigenword Embeddings

Encoding Prior Knowledge with Eigenword Embeddings

Paper · 2016 · Dominique Osborne, Shashi Narayan and Shay B. Cohen

Summary

The paper adds external lexical knowledge to spectral word embeddings. It builds on the eigenword approach of Dhillon et al. (2015), which uses canonical correlation analysis (CCA) to relate two views: a pivot word and its surrounding window of context words. Earlier methods such as retrofitting (Faruqui et al., 2015) apply lexicon information to finished embeddings as a separate post-processing step. The authors instead build the prior knowledge into the CCA derivation itself, so the theoretical guarantees of spectral learning are kept.

They recast CCA as an optimization problem over distances between the two projected views of training examples. These distances are weighted by a Laplacian matrix derived from WordNet, FrameNet and the Paraphrase Database. In practice, this amounts to adding extra counts to the matrix before the SVD step, a small change to standard CCA. The resulting embeddings were tested on many word-evaluation datasets, and the authors report several benefits of encoding the priors. The technique is not specific to word embeddings and could apply to other CCA uses, such as transliteration or visual-attribute representation.

Key points

  • Base method: eigenwords (Dhillon et al., 2015). The context view is a sparse matrix of 2k one-hot vectors, the word view is a one-hot matrix, and SVD is applied to a diagonally normalized co-occurrence matrix. This approximates full CCA for efficiency.
  • Prior knowledge sources: WordNet, FrameNet and the Paraphrase Database (PPDB). They are encoded through a graph Laplacian, similar to Koren and Carmel's (2003) work on PCA and related to Laplacian manifold regularization.
  • Theory: the appendix provides proofs (Lemmas 1–3) linking the Laplacian-weighted objective to an SVD solution.
  • Setup: embedding dimensionality ranged from 50 to 300. The authors used the SWELL package for eigenwords and compared against the retrofitting package of Faruqui et al.
  • Publication: Transactions of the ACL, vol. 4, pp. 417–430, 2016.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations

Also on the byline: University of Strathclyde.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Dominique Osborne, Shashi Narayan and Shay B. Cohen. 2016. Encoding Prior Knowledge with Eigenword Embeddings. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00108

@article{osborne2016encoding,
  title = {{Encoding Prior Knowledge with Eigenword Embeddings}},
  author = {Dominique Osborne, Shashi Narayan and Shay B. Cohen},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.1162/tacl\_a\_00108},
  url = {https://doi.org/10.1162/tacl_a_00108}
}

References

53 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and wordnet-based approaches. In Proceedings of HLT-NAACL.
  2. Francis Bach and Michael Jordan. 2005. A probabilistic interpretation of canonical correlation analysis. Tech Report 688, Department of Statistics, University of California, Berkeley. find
  3. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In Proceedings of ACL.
  4. Simon Baker, Roi Reichart, and Anna Korhonen. 2014. An unsupervised model for instance level subcategorization acquisition. In Proceedings of EMNLP. find
  5. Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014. Tailoring continuous word representations for dependency parsing. In Proceedings of ACL.
  6. Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. 2006. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of Machine Learning Research, 7:2399– 2434. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabi… find
  7. Jiang Bian, Bin Gao, and Tie-Yan Liu. 2014. Knowledge-powered deep learning for word embedding. In Machine Learning and Knowledge Discovery in Databases, volume 8724 of Lecture Notes in Computer Science, pages 132–148. find
  8. Avrim Blum and Tom Mitchell. 1998. Combining labeled and unlabeled data with co-training. In Proceedings of COLT.
  9. Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. Journal of Artificial Intelligence Research, 49:1–47. find
  10. Kai-Wei Chang, Wen-tau Yih, and Christopher Meek. 2013. Multi-relational latent semantic analysis. In Proceedings of EMNLP. find
  11. Jiaqiang Chen and Gerard de Melo. 2015. Semantic information extraction for improved word embeddings. In Proceedings of NAACL Workshop on Vector Space Modeling for NLP. find
  12. Shay B. Cohen, K. Stratos, Michael Collins, Dean P. Foster, and Lyle Ungar. 2014. Spectral learning of latent-variable PCFGs: Algorithms and sample complexity. Journal of Machine Learning Research. find
  13. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of ICML. find
  14. Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391– 407. find
  15. Paramveer S. Dhillon, Dean P. Foster, and Lyle H. Ungar. 2015. Eigenwords: Spectral word embeddings. Journal of Machine Learning Research, 16:3035–3078. find
  16. Manaal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. In Proceedings of EACL.
  17. Manaal Faruqui and Chris Dyer. 2015. Non-distributional word vector representations. In Proceedings of ACL. find
  18. Manaal Faruqui, Jesse Dodge, Sujay K. Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. 2015. Retrofitting word vectors to semantic lexicons. In Proceedings of NAACL. find
  19. Lev Finkelstein, Gabrilovich Evgenly, Matias Yossi, Rivlin Ehud, Solan Zach, Wolfman Gadi, and Ruppin Eytan. 2002. Placing search in context: The concept revisited. ACM Transactions on Information Systems, 20(1):116–131. find
  20. Daniel Fried and Kevin Duh. 2015. Incorporating both distributional and relational semantics in word representations. In Proceedings of ICLR. find
  21. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In Proceedings of NAACL.
  22. Guy Halawi, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. 2012. Large-scale learning of word relatedness with constraints. In Proceedings of ACM SIGKDD. find
  23. Zellig S. Harris. 1957. Co-occurrence and transformation in linguistic structure. Language, 33(3):283–340. find
  24. Xiaofei He and Partha Niyogi. 2004. Locality preserving projections. In Proceedings of NIPS. find
  25. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695.
  26. Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of ACL. Jagadeesh Jagarlamudi and Hal Daumé. 2012. Regularized interlingual projections: Evaluation on multilingual transliterat…
  27. Yehuda Koren and Liran Carmel. 2003. Visualization of labeled data using linear transformations. In Proceedings of IEEE Conference on Information Visualization. find
  28. Thomas K. Landauer, Peter W. Foltz, and Darrell Laham. 1998. An introduction to latent semantic analysis. Discourse Processes, 25:259–284. find
  29. Angeliki Lazaridou, Eva Maria Vecchi, and Marco Baroni. 2013. Fish transporters and miracle homes: How compositional distributional semantics can help NP parsing. In Proceedings of EMNLP. find
  30. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In Proceedings of ICLR Workshop.
  31. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS. Tomas Mikolov, Wen tau Yih, and Geoffrey Zweig. 2013c. Linguistic regularities in continuous space word representations.…
  32. George A. Miller and Walter G. Charles. 1991. Contextual correlates of semantic similarity. Language and Cognitive Processes, 6(1):1–28. find
  33. George A Miller. 1995. WordNet: A lexical database for English. Communications of the ACM, 38(11):39–41.
  34. Andriy Mnih and Geoffrey Hinton. 2007. Three new graphical models for statistical language modelling. In Proceedings of ICML. find
  35. Shashi Narayan and Shay B. Cohen. 2016. Optimizing spectral learning for parsing. In Proceedings of ACL. find
  36. Ankur P. Parikh, Shay B. Cohen, and Eric Xing. 2014. Spectral unsupervised parsing with additive tree metrics. In Proceedings of ACL. find
  37. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP.
  38. Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. 2011. A word at a time: Computing word relatedness using temporal semantic analysis. In Proceedings of ACM WWW. Sascha Rothe and Hinrich Schütze. 2015. AutoExtend: Extending word embeddings to embeddings for synsets and lex… find
  39. Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. find
  40. Carina Silberer, Vittorio Ferrari, and Mirella Lapata. 2013. Models of semantic representation with visual attributes. In Proceedings of ACL. find
  41. Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013. Parsing with compositional vector grammars. In Proceedings of ACL. find
  42. Karl Stratos, Michael Collins, and Daniel Hsu. 2015. Model-based word embeddings from decompositions of count matrices. In Proceedings of ACL. find
  43. Karl Stratos, Michael Collins, and Daniel Hsu. 2016. Unsupervised part-of-speech tagging with anchor hidden markov models. Transactions of the Association for Computational Linguistics, 4:245–257.
  44. Peter D. Turney and Michael L. Littman. 2005. Corpus-based learning of analogies and semantic relations. Machine Learning, 60(1-3):251–278. find
  45. Peter D. Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37(1):141– 188.
  46. Peter D. Turney. 2006. Similarity of semantic relations. Computational Linguistics, 32(3):379–416.
  47. Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph and text jointly embedding. In Proceedings of EMNLP.
  48. Tong Wang, Abdelrahman Mohamed, and Graeme Hirst. 2015. Learning lexical embeddings with syntactic and lexicographic knowledge. In Proceedings of ACL-IJCNLP. find
  49. Chang Xu, Yalong Bai, Jiang Bian, Bin Gao, Gang Wang, Xiaoguang Liu, and Tie-Yan Liu. 2014. RC-NET: A general framework for incorporating knowledge into word representations. In Proceedings of the ACM CIKM. find
  50. Dongqiang Yang and David MW Powers. 2005. Measuring semantic similarity in the taxonomy of Word-Net. In Proceedings of the Australasian Conference on Computer Science. find
  51. David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In Proceedings of ACL.
  52. Wen-tau Yih, Geoffrey Zweig, and John Platt. 2012. Polarity inducing latent semantic analysis. In Proceedings of EMNLP-CoNLL. find
  53. Mo Yu and Mark Dredze. 2014. Improving lexical embeddings with semantic knowledge. In Proceedings of ACL.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/osborne-2016-encoding-prior-knowledge-eigenword-embeddings/