Elamur Research⌕
☰
home / papers / 2015 / SensEmbed: Learning Sense Embeddings for Word and Relational Similarity

SensEmbed: Learning Sense Embeddings for Word and Relational Similarity

Summary

Conventional word embeddings merge every meaning of an ambiguous word into one vector, and they learn only from raw text, ignoring curated lexical knowledge. SensEmbed tackles both problems together. The authors automatically sense-tag English Wikipedia with Babelfy, a WSD and entity-linking system built on BabelNet. They keep only high-confidence decisions and drop cases where Babelfy falls back to the most frequent sense. Word2vec is then trained on this tagged text, so each BabelNet sense gets its own vector. To measure similarity, each word is mapped to its senses plus their BabelNet synonyms, and sense vectors are compared with the Tanimoto measure. A graph vicinity factor can further adjust scores using proximity in the semantic network. On several word similarity datasets and the SemEval-2012 relational similarity task, the method reports state-of-the-art results. It also outperforms a Word2vec baseline trained on the same corpus with identical settings. The work shows that automatically disambiguated text combined with a semantic network gives sharper representations, tied to a multilingual sense inventory.

Key points

  • Sense inventory: BabelNet, which merges WordNet with Wikipedia and other resources. This makes the vectors usable across several lexical resources, and in principle across languages via Babelfy.
  • Corpus: the September-2014 English Wikipedia dump (~3 billion tokens). After Babelfy disambiguation it yields about 1 billion sense-tagged words and 2.5 million distinct word senses.
  • Training setup: Word2vec CBOW with hierarchical softmax, 400-dimensional vectors, a window of about five words per side, and frequent-word subsampling at 10⁻³.
  • Similarity: Tanimoto distance between sense vectors, synonym-based expansion of each word's sense set, and an optional graph vicinity factor derived from BabelNet.
  • Findings: switching from word to sense vectors helps on every dataset and strategy, and the graph vicinity factor adds further gains. Synonym expansion mostly helps sense embeddings but harms plain word embeddings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Word sense disambiguation

Cited by

81 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 80 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p15-1010

@inproceedings{iacobacci2015sensembed,
  title = {{SensEmbed: Learning Sense Embeddings for Word and Relational Similarity}},
  author = {Iacobacci, Ignacio and Pilehvar, Mohammad Taher and Navigli, Roberto},
  booktitle = {Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2015},
  doi = {10.3115/v1/p15-1010},
  url = {https://doi.org/10.3115/v1/p15-1010}
}

References

41 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. 2009. A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the A… find our card
  2. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, volume 1, pages 238–247, Baltimore, Maryl… find our card
  3. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A Neural Probabilistic Language Model. The Journal of Machine Learning Research, 3:1137–1155. find
  4. Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating Embeddings for Modeling Multi-relational Data. In Advances in Neural Information Processing Systems, volume 26, pages 2787–2795. find
  5. Elia Bruni, Nam Khanh Tran, and Marco Baroni. 2014. Multimodal Distributional Semantics. Journal of Artificial Intelligence Research, 49(1):1–47. find
  6. Alexander Budanitsky and Graeme Hirst. Evaluating WordNet-based measures of Lexical Semantic Relatedness. Computational Linguistics, 32(1):13–47. find our card
  7. John A. Bullinaria and Joseph P. Levy. 2012. Extracting Semantic Representations from Word Co-occurrence Statistics: Stop-lists, Stemming and SVD. Behavior Research Methods, 44:890–907. find
  8. Yee Seng Chan and Hwee Tou Ng. 2005. Scaling Up Word Sense Disambiguation via Parallel Texts. In Proceedings of the 20th National Conference on Artificial Intelligence - Volume 3, pages 1037–1042, Pittsburgh, Pennsylvania. find
  9. Xinxiong Chen, Zhiyuan Liu, and Maosong Sun. 2014. A unified model for word sense representation and disambiguation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1025–1035, Doha, Qatar. find our card
  10. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, pages 160–167, Helsinki, Finland. find
  11. Scott C. Deerwester, Susan T. Dumais, Thomas K. Landauer, George W. Furnas, and Richard A. Harshman. 1990. Indexing by latent semantic analysis. Journal of American Society for Information Science, 41(6):391–407. find
  12. Lev Finkelstein, Gabrilovich Evgeniy, Matias Yossi, Rivlin Ehud, Solan Zach, Wolfman Gadi, and Ruppin Eytan. 2002. Placing Search in Context: The Concept Revisited. ACM Transactions on Information Systems, 20(1):116–131. find
  13. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The Paraphrase Database. In Proceedings of Human Language Technologies: The 2013 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 758–764, Atlanta, Georgia. find our card
  14. Jiang Guo, Wanxiang Che, Haifeng Wang, and Ting Liu. 2014. Learning Sense-specific Word Embeddings By Exploiting Bilingual Resources. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 497–507, Dublin, Ireland. find
  15. Felix Hill, Roi Reichart, and Anna Korhonen. 2014. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. arXiv preprint arXiv:1408.3456. find our card
  16. Eric H. Huang, Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving Word Representations Via Global Context And Multiple Word Prototypes. In Proceedings of 50th Annual 2006. Meeting of the Association for Computational Linguistics, volume 1, pages 873–882, Jeju Island, South Kor… find our card
  17. David A. Jurgens, Peter D. Turney, Saif M. Mohammad, and Keith J. Holyoak. 2012. Semeval-2012 task 2: Measuring degrees of relational similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the Main Conference and the Shared Task, an… find
  18. Omer Levy and Yoav Goldberg. 2014. Linguistic regularities in sparse and explicit word representations. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 171–180, Ann Arbor, Michigan. find our card
  19. Jordan Louviere. 1991. Best-Worst Scaling: A Model for the Largest Difference Judgments. Working paper, University of Alberta. find
  20. Arthur B. Markman and Dedre Gentner. 1993. Structural alignment during similarity comparisons. Cognitive Psychology, 25(4):431 – 467. find
  21. Douglas L. Medin, Robert L. Goldstone, and Dedre Gentner. 1990. Similarity involving attributes and relations: Judgments of similarity and difference are not inverses. Psychological Science, 1(1):64–69. find
  22. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient Estimation of Word Representations in Vector Space. CoRR, abs/1301.3781. find
  23. Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013b. Exploiting Similarities among Languages for Machine Translation. arXiv preprint arXiv:1309.4168. find
  24. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013c. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of Human Language Technologies: The 2013 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 746–751, Atlanta, G… find our card
  25. George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. 1993. A Semantic Concordance. In Proceedings of the Workshop on Human Language Technology, pages 303–308, Princeton, New Jersey. find
  26. Andriy Mnih and Geoffrey Hinton. 2007. Three New Graphical Models for Statistical Language Modelling. In Proceedings of the 24th International Conference on Machine Learning, pages 641–648, Corvallis, Oregon. find
  27. Andrea Moro, Alessandro Raganato, and Roberto Navigli. 2014. Entity Linking meets Word Sense Disambiguation: a Unified Approach. Transactions of the Association for Computational Linguistics (TACL), 2:231–244. find
  28. Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The Automatic Construction, Evaluation and Application of a Wide-Coverage Multilingual Semantic Network. Artificial Intelligence, 193:217– 250. find
  29. Roberto Navigli. 2009. Word Sense Disambiguation: A survey. ACM Computing Surveys, 41(2):1–69. find
  30. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of the Empirical Methods in Natural Language Processing (EMNLP), volume 12, pages 1532–1543, Doha, Qatar. find
  31. Mohammad Taher Pilehvar, David A. Jurgens, and Roberto Navigli. 2013. Align, Disambiguate and Walk: a Unified Approach for Measuring Semantic Similarity. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1341–1351, Sofia, Bulgaria. find
  32. Joseph Reisinger and Raymond J. Mooney. 2010. Multi-Prototype Vector-Space Models of Word Meaning. In Proceedings of Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 109–117, Los Angeles, California. find our card
  33. Bryan Rink and Sanda Harabagiu. 2012. UTD: Determining relational similarity using lexical patterns. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth Internationa… find
  34. Bryan Rink and Sanda Harabagiu. 2013. The Impact of Selectional Preference Agreement on Semantic Relational Similarity. In Proceedings of the 10th International Conference on Computational Semantics (IWCS) – Long Papers, pages 204–215, Potsdam, Germany. find
  35. Herbert Rubenstein and John B. Goodenough. 1965. Contextual Correlates of Synonymy. Communications of the ACM, 8(10):627–633. find
  36. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word Representations: A Simple and General Method for Semi-supervised Learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 384–394, Uppsala, Sweden. find our card
  37. Amos Tversky. 1977. Features of similarity. Psychological Review, 84:327–352. find
  38. Dongqiang Yang and David M. W. Powers. 2005. Measuring semantic similarity in the taxonomy of wordnet. In Proceedings of the Twenty-eighth Australasian Conference on Computer Science, volume 38, pages 315–322, Darlinghurst, Australia. find
  39. Mo Yu and Mark Dredze. 2014. Improving Lexical Embeddings with Semantic Knowledge. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, volume 2, pages 545–550, Baltimore, Maryland. find our card
  40. Torsten Zesch, Christof Müller, and Iryna Gurevych. 2008. Using Wiktionary for Computing Semantic Relatedness. In Proceedings of the 23rd National Conference on Artificial Intelligence, volume 2, pages 861–866, Chicago, Illinois. find
  41. Alisa Zhila, Wen-tau Yih, Christopher Meek, Geoffrey Zweig, and Tomas Mikolov. 2013. Combining Heterogeneous Models for Measuring Relational Similarity. In Proceedings of Human Language Technologies: The 2013 Annual Conference of the North American Chapter of the Association for Computational Lingui… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/iacobacci-2015-sensembed-learning-sense-embeddings-word/