Elamur Research⌕
☰
home / papers / 2018 / Specialising Word Vectors for Lexical Entailment

Specialising Word Vectors for Lexical Entailment

Paper · 2018 · Ivan Vulić and Nikola Mrkšić

Summary

Distributional word embeddings blend many lexical relations together. Plain cosine similarity is symmetric, so it cannot show which word is the more general one in a hyponym–hypernym (IS-A) pair. Vulić and Mrkšić address this with LEAR (Lexical Entailment Attract-Repel), a post-processing method that extends the ATTRACT-REPEL specialisation framework. LEAR takes any pre-trained vector space as a black box and refines it using external constraints such as WordNet links.

The method combines several objectives:

  • Similarity: synonym-like and entailment pairs are pulled together, while antonym-like pairs are pushed apart.
  • Hierarchy: an extra asymmetric term rescales vector norms so that more general concepts end up with larger norms.
  • Preservation: a regulariser keeps the specialised vectors close to the original distributional ones.

At test time, entailment strength is scored by adding cosine distance to a norm-based asymmetric distance.

The authors applied LEAR to four English embedding spaces. It set new state-of-the-art results on hypernymy directionality, hypernymy detection, combined detection/directionality, and graded entailment on HyperLex, while leaving similarity performance intact. This shows that a simple Euclidean model can capture both hierarchy and similarity.

Key points

  • Input spaces: LEAR was tested on SGNS (Polyglot Wikipedia), GloVe Common Crawl, context2vec and fastText, which shows it does not depend on a particular embedding model.
  • Asymmetric distances: three norm-based variants were proposed. These are the raw norm difference, the difference divided by the sum of norms, and the difference divided by the larger norm. The two ratio forms remove the incentive for norms to grow without limit.
  • HyperLex results: Spearman's correlation rose from 0.540 to 0.686 on the full set (2,616 pairs) and from 0.512 to 0.705 on the noun subset (2,163 pairs).
  • Similarity preserved: results on SimLex-999 (≈0.71) and SimVerb-3500 (≈0.70) match the original ATTRACT-REPEL, so the entailment objective leaves cosine similarity unharmed.
  • Code: available at github.com/nmrksic/lear. Planned extensions include meronymy, words unseen in the lexical resources, and other languages or cross-lingual settings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom155 works · 115,562 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Ivan Vulić and Nikola Mrkšić. 2018. Specialising Word Vectors for Lexical Entailment. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-1103

@inproceedings{vulic2018specialising,
  title = {{Specialising Word Vectors for Lexical Entailment}},
  author = {Ivan Vulić and Nikola Mrkšić},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1103},
  url = {https://doi.org/10.18653/v1/n18-1103}
}

References

78 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual NLP. In Proceedings of CoNLL, pages 183–192.
  2. Marco Baroni, Raffaella Bernardi, Ngoc-Quynh Do, and Chung-chieh Shan. 2012. Entailment above the word level in distributional semantics. In Proceedings of EACL, pages 23–32.
  3. Marco Baroni and Alessandro Lenci. 2011. How we BLESSed distributional semantic evaluation. In Proceedings of the GEMS 2011 Workshop, pages 1–10.
  4. Maayan Geffet and Ido Dagan. 2005. The distributional inclusion hypotheses and lexical entailment. In Proceedings of ACL, pages 107–114.
  5. Richard Beckwith, Christiane Fellbaum, Derek Gross, and George A. Miller. 1991. WordNet: A lexical database organized on psycholinguistic principles. Lexical acquisition: Exploiting on-line resources to build a lexicon, pages 211–231. find
  6. Daniela Gerz, Ivan Vulić, Felix Hill, Roi Reichart, and Anna Korhonen. 2016. SimVerb-3500: A large-scale evaluation set of verb similarity. In Proceedings of EMNLP, pages 2173–2182.
  7. Jiang Bian, Bin Gao, and Tie-Yan Liu. 2014. Knowledge-powered deep learning for word embedding. In Proceedings of ECML-PKDD, pages 132– 148. find
  8. Goran Glavaš and Simone Paolo Ponzetto. 2017. Dual tensor model for detecting asymmetric lexico-semantic relations. In Proceedings of EMNLP, pages 1758–1768. find
  9. Or Biran and Kathleen McKeown. 2013. Classifying taxonomic relations between pairs of Wikipedia articles. In Proceedings of IJCNLP, pages 788–794. find
  10. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the ACL, 5:135–146.
  11. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of EMNLP, pages 632–642. find
  12. Benjamin Paul Chamberlain, James Clough, and Marc Peter Deisenroth. 2017. Neural embeddings of graphs in hyperbolic space. CoRR, abs/1705.10359. — [arXiv:1705.10359](https://arxiv.org/abs/1705.10359) find
  13. Danqi Chen and Christopher D. Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of EMNLP, pages 740–750.
  14. Daoud Clarke. 2009. Context-theoretic semantics for natural language: An overview. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics (GEMS), pages 112–119. find
  15. Allan M. Collins and Ross M. Quillian. 1972. Experiments on semantic memory and language comprehension. Cognition in Learning and Memory. find
  16. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537.
  17. Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing textual entailment: Models and applications. Synthesis Lectures on Human Language Technologies, 6(4):1–220. find
  18. John C. Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121–2159. find
  19. Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. 2015. Retrofitting word vectors to semantic lexicons. In Proceedings of NAACL-HLT, pages 1606–1615. find
  20. Christiane Fellbaum. 1998. WordNet. find
  21. James A. Hampton. 2007. Typicality, graded membership, and vagueness. Cognitive Science, 31(3):355– 384. find
  22. Zellig S. Harris. 1954. Distributional structure. Word, 10(23):146–162. find
  23. James Henderson and Diana Popa. 2016. A vector space for distributional semantics for entailment. In Proceedings of ACL, pages 2052–2062. find
  24. Aurélie Herbelot and Mohan Ganesalingam. 2013. Measuring semantic content in distributional vectors. In Proceedings of ACL, pages 440–445. find
  25. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695.
  26. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9(8):1735–1780. find
  27. Sujay Kumar Jauhar, Chris Dyer, and Eduard H. Hovy. 2015. Ontologically grounded multi-sense representation learning for semantic vector space models. In Proceedings of NAACL, pages 683–693. find
  28. Hans Kamp and Barbara Partee. 1995. Prototype theory and compositionality. Cognition, 57(2):129– 191. find
  29. Douwe Kiela, Felix Hill, and Stephen Clark. 2015a. Specializing word embeddings for similarity or relatedness. In Proceedings of EMNLP, pages 2044– 2048. find
  30. Douwe Kiela, Laura Rimell, Ivan Vulić, and Stephen Clark. 2015b. Exploiting image generality for lexical entailment detection. In Proceedings of ACL, pages 119–124. find
  31. Barbara Ann Kipfer. 2009. Roget’s 21st Century Thesaurus (3rd Edition). Philip Lief Group. find
  32. Lili Kotlerman, Ido Dagan, Idan Szpektor, and Maayan Zhitomirsky-Geffet. 2010. Directional distributional similarity for lexical inference. Natural Language Engineering, 16(4):359–389. find
  33. Alessandro Lenci and Giulia Benotto. 2012. Identifying hypernyms in distributional semantic spaces. In Proceedings of *SEM, pages 75–79.
  34. Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of ACL, pages 302–308. find
  35. Quan Liu, Hui Jiang, Si Wei, Zhen-Hua Ling, and Yu Hu. 2015. Learning semantic word embeddings based on ordinal knowledge constraints. In Proceedings of ACL, pages 1501–1511. find
  36. Douglas L. Medin, Mark W. Altom, and Timothy D. Murphy. 1984. Given versus induced category representations: Use of prototype and exemplar information in classification. Journal of Experimental Psychology, 10(3):333–352. find
  37. Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. Context2vec: Learning generic context embedding with bidirectional LSTM. In Proceedings of CoNLL, pages 51–61.
  38. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS, pages 3111– 3119.
  39. Michael Mohler, David Bracewell, Marc Tomlinson, and David Hinote. 2013. Semantic signatures for example-based linguistic metaphor detection. In Proceedings of the First Workshop on Metaphor in NLP, pages 27–35. find
  40. Kim Anh Nguyen, Maximilian Köper, Sabine Schulte im Walde, and Ngoc Thang Vu. 2017. Hierarchical embeddings for hypernymy detection and directionality. In Proceedings of EMNLP, pages 233–243. find
  41. Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. 2016. Integrating distributional lexical contrast into word embeddings for antonym-synonym distinction. In Proceedings of ACL, pages 454–459. find
  42. Maximilian Nickel and Douwe Kiela. 2017. Poincaré embeddings for learning hierarchical representations. In Proceedings of NIPS.
  43. Masataka Ono, Makoto Miwa, and Yutaka Sasaki. 2015. Word embedding-based antonym detection using thesauri and distributional information. In Proceedings of NAACL-HLT, pages 984–989. find
  44. Dominique Osborne, Shashi Narayan, and Shay Cohen. 2016. Encoding prior knowledge with eigenword embeddings. Transactions of the ACL, 4:417–430. find
  45. Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In Proceedings of ACL, pages 425–430.
  46. Ted Pedersen, Siddharth Patwardhan, and Jason Michelizzi. 2004. WordNet::Similarity - Measuring the relatedness of oncepts. In Proceedings of AAAI, pages 1024–1025. find
  47. Nikola Mrkšić, Diarmuid Ó Séaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2017. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of ACL, pages 1777–1788. find
  48. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP, pages 1532– 1543.
  49. Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gašić, Lina Maria Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. In Proceedings of NAACL-HLT. find
  50. Marek Rei and Ted Briscoe. 2014. Looking for hyponyms in vector space. In Proceedings of CoNLL, pages 68–77. find
  51. Nikola Mrkšić, Ivan Vulić, Diarmuid Ó Séaghdha, Ira Leviant, Roi Reichart, Milica Gašić, Anna Korhonen, and Steve Young. 2017. Semantic specialisation of distributional word vector spaces using monolingual and cross-lingual constraints. Transactions of the ACL, 5:309–324. find
  52. Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence, 193:217–250. find
  53. Roberto Navigli, Paola Velardi, and Stefano Faralli. 2011. A graph-based algorithm for inducing lexical taxonomies from scratch. In Proceedings of IJCAI, pages 1872–1877. find
  54. Stephen Roller, Katrin Erk, and Gemma Boleda. 2014. Inclusive yet selective: Supervised distributional hypernymy detection. In Proceedings of COLING, pages 1025–1036.
  55. Eleanor H. Rosch. 1973. Natural categories. Cognitive Psychology, 4(3):328–350. find
  56. Enrico Santus, Alessandro Lenci, Qin Lu, and Sabine Schulte im Walde. 2014. Chasing hypernyms in vector spaces with entropy. In Proceedings of EACL, pages 38–42. find
  57. Enrico Santus, Frances Yung, Alessandro Lenci, and Chu-Ren Huang. 2015. EVALution 1.0: an evolving semantic dataset for training and evaluation of distributional semantic models. In Proceedings of the 4th Workshop on Linked Data in Linguistics: Resources and Applications, pages 64–69. find
  58. Vered Shwartz, Yoav Goldberg, and Ido Dagan. 2016. Improving hypernymy detection with an integrated path-based and distributional method. In Proceedings of ACL, pages 2389–2398.
  59. Julie Weeds, David Weir, and Diana McCarthy. 2004. Characterising measures of lexical distributional similarity. In Proceedings of COLING, pages 1015– 1021. find
  60. Vered Shwartz, Enrico Santus, and Dominik Schlechtweg. 2017. Hypernyms under siege: Linguistically-motivated artillery for hypernymy detection. In Proceedings of EACL, pages 65–75. find
  61. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. From paraphrase database to compositional paraphrase model and back. Transactions of the ACL, 3:345–358.
  62. Rion Snow, Daniel Jurafsky, and Andrew Y. Ng. 2006. Semantic taxonomy induction from heterogenous evidence. In Proceedings of ACL, pages 801–808. find
  63. Luu Anh Tuan, Yi Tay, Siu Cheung Hui, and See Kiong Ng. 2016. Learning term embeddings for taxonomic relation identification using dynamic weighting neural network. In Proceedings of EMNLP, pages 403– 413. find
  64. Joseph P. Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of ACL, pages 384–394.
  65. Shyam Upadhyay, Yogarshi Vyas, Marine Carpuat, and Dan Roth. 2018. Robust cross-lingual hypernymy detection using dependency context. In Proceedings of NAACL-HLT. find
  66. Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. 2016. Order-embeddings of images and language. In Proceedings of ICLR (Conference Track). find
  67. Luke Vilnis and Andrew McCallum. 2015. Word representations via Gaussian embedding. In Proceedings of ICLR (Conference Track). find
  68. Zhibiao Wu and Martha Palmer. 1994. Verb semantics and lexical selection. In Proceedings of ACL, pages 133–138.
  69. Chang Xu, Yalong Bai, Jiang Bian, Bin Gao, Gang Wang, Xiaoguang Liu, and Tie-Yan Liu. 2014. RC-NET: A general framework for incorporating knowledge into word representations. In Proceedings of CIKM, pages 1219–1228. find
  70. Mo Yu and Mark Dredze. 2014. Improving lexical embeddings with semantic knowledge. In Proceedings of ACL, pages 545–550.
  71. Zheng Yu, Haixun Wang, Xuemin Lin, and Min Wang. 2015. Learning term embeddings for hypernymy identification. In Proceedings of IJCAI, pages 1390– 1397. find
  72. Jingwei Zhang, Jeremy Salwen, Michael Glass, and Alfio Gliozzo. 2014. Word semantic representations using bayesian probabilistic tensor factorization. In Proceedings of EMNLP, pages 1522–1531. find
  73. Will Y. Zou, Richard Socher, Daniel Cer, and Christopher D. Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In Proceedings of EMNLP, pages 1393–1398.
  74. Ivan Vulić, Daniela Gerz, Douwe Kiela, Felix Hill, and Anna Korhonen. 2017. Hyperlex: A large-scale evaluation of graded lexical entailment. Computational Linguistics.
  75. Ivan Vulić, Goran Glavaš, Nikola Mrkšić, and Anna Korhonen. 2018. Post-specialisation: Retrofitting vectors of words unseen in lexical resources. In Proceedings of NAACL-HLT. find
  76. Ivan Vulić, Nikola Mrkšić, and Anna Korhonen. 2017a. Cross-lingual induction and transfer of verb classes based on word vector space specialisation. In Proceedings of EMNLP, pages 2536–2548. find
  77. Ivan Vulić, Nikola Mrkšić, Roi Reichart, Diarmuid Ó Séaghdha, Steve Young, and Anna Korhonen. 2017b. Morph-fitting: Fine-tuning word vector spaces with simple language-specific rules. In Proceedings of ACL, pages 56–68. find
  78. Julie Weeds, Daoud Clarke, Jeremy Reffin, David Weir, and Bill Keller. 2014. Learning to distinguish hypernyms and co-hyponyms. In Proceedings of COL-ING, pages 2249–2259.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vulic-2018-specialising-word-vectors-lexical-entailment/