Elamur Research⌕
☰
home / papers / 2014 / Learning Semantic Hierarchies via Word Embeddings

Learning Semantic Hierarchies via Word Embeddings

Summary

The paper addresses automatic construction of semantic hierarchies, meaning graphs of concepts connected by "is-a" (hypernym–hyponym) links. A web-mining system can return a flat list of hypernyms for an entity. The goal here is to arrange those hypernyms into a directed acyclic graph by deciding which of them subsume the others.

The authors train Skip-gram word embeddings on a large Chinese corpus. They observe that vector offsets between hyponyms and hypernyms are similar for some word pairs but differ greatly across semantic domains, for example animals versus occupations. Because a single offset cannot capture the relation, they learn linear projection matrices that map a word's vector toward its hypernym's vector. They then refine this into a piecewise model: training pairs are first clustered by their offsets, and a separate projection is learned for each cluster. A candidate pair is accepted when the projected vector falls close enough to the candidate hypernym, as controlled by a threshold.

On a newly annotated dataset of 418 Chinese entities, the method outperforms pattern-based, distributional and resource-based baselines. It also complements manually built resources, which matters because existing thesauri lack many fine-grained relations.

Key points

  • The authors annotated a test set of 418 Chinese entities with their hypernym hierarchies, which they describe as the first dataset for this task.
  • The embedding-based method alone reaches an F-score of 73.74%. Combining it with the Wikipedia/thesaurus extension approach of Suchanek et al. (2008) raises this to 80.29%.
  • Embeddings come from the Skip-gram model. The baselines are Hearst-style patterns, Snow et al. (2005), Fu et al. (2013), and the directional distributional measures proposed by Kotlerman and colleagues in 2010 and by Lenci and Benotto in 2012.
  • At equal recall, the Emb+CilinE combination is more precise than Wiki+CilinE, and at equal precision it has higher recall. CilinE is a Chinese semantic thesaurus that misses roughly half of the relations.
  • A main error source is pairs that are almost synonymous in embedding space. When cosine similarity exceeds 0.8, recall drops to 9.5%, and these cases make up about 10.9% of all errors.

Interesting ideas

  • I1. Learn a projection rather than rely on a constant offset. Fitting a matrix that maps a word to its related word generalizes the familiar "king − queen" vector arithmetic. The same idea could model other asymmetric relations, such as part-of links or administrative region hierarchies in geographic data.
  • I2. Cluster the relation instances first, then learn one model per cluster. Grouping training pairs by offset vectors exposes sub-types of a relation, such as animal taxonomies versus job titles. Fitting each sub-type separately is a cheap way to handle heterogeneous relations without a complex nonlinear model.
  • I3. Visualize offset vectors before modelling. The authors' quick check of offset similarity, followed by a plot of all training offsets, justified their design choice. This kind of sanity check is useful whenever one plans to exploit embedding geometry.
  • I4. Combine a high-coverage statistical method with high-precision curated resources. The embedding method finds many fine-grained links that Wikipedia and CilinE miss, while those resources supply reliable general links. A similar mix could help when extending Ukrainian lexical resources.
  • I5. Analyze errors by similarity bucket. Binning failures by cosine similarity showed that near-synonymous pairs are a systematic blind spot. This evaluation trick reveals where geometric methods break down and suggests targeted additions to the training data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

81 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 79 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p14-1113

@inproceedings{fu2014learning,
  title = {{Learning Semantic Hierarchies via Word Embeddings}},
  author = {Fu, Ruiji and Guo, Jiang and Qin, Bing and Che, Wanxiang and Wang, Haifeng and Liu, Ting},
  booktitle = {Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2014},
  doi = {10.3115/v1/p14-1113},
  url = {https://doi.org/10.3115/v1/p14-1113}
}

References

29 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. The Journal of Machine Learning Research, 3:1137–1155. find
  2. Rahul Bhagat, Patrick Pantel, Eduard H Hovy, and Marina Rey. 2007. Ledir: An unsupervised algorithm for learning directionality of inference rules. In EMNLP-CoNLL, pages 161–170. find
  3. Wanxiang Che, Zhenghua Li, and Ting Liu. 2010. Ltp: A chinese language technology platform. In Coling 2010: Demonstrations, pages 13–16, Beijing, China, August. find
  4. Daoud Clarke. 2009. Context-theoretic semantics for natural language: an overview. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics, pages 112–119. Association for Computational Linguistics. find
  5. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537. find our card
  6. Paramveer Dhillon, Dean P Foster, and Lyle H Ungar. 2011. Multi-view learning of word embeddings via cca. In Advances in Neural Information Processing Systems, pages 199–207. find
  7. Richard Evans. 2004. A framework for named entity recognition in the open domain. Recent Advances in Natural Language Processing III: Selected Papers from RANLP 2003, 260:267–274. find
  8. Ruiji Fu, Bing Qin, and Ting Liu. 2013. Exploiting multiple sources for open-domain hypernym discovery. In EMNLP, pages 1224–1234. find
  9. Maayan Geffet and Ido Dagan. 2005. The distributional inclusion hypotheses and lexical entailment. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 107–114. Association for Computational Linguistics. find our card
  10. Marti A Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In Proceedings of the 14th conference on Computational linguistics-Volume 2, pages 539–545. Association for Computational Linguistics. find
  11. Lili Kotlerman, Ido Dagan, Idan Szpektor, and Maayan Zhitomirsky-Geffet. 2010. Directional distributional similarity for lexical inference. Natural Language Engineering, 16(4):359–389. find
  12. Alessandro Lenci and Giulia Benotto. 2012. Identifying hypernyms in distributional semantic spaces. In Proceedings of the Sixth International Workshop on Semantic Evaluation, pages 75–79. Association for Computational Linguistics. find
  13. Paul McNamee, Rion Snow, Patrick Schone, and James Mayfield. 2008. Learning named entity hyponyms for question answering. In Proceedings of the Third International Joint Conference on Natural Language Processing, pages 799–804. find
  14. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  15. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In Proceedings of NAACL-HLT, pages 746–751. find our card
  16. George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41. find our card
  17. Andriy Mnih and Geoffrey E Hinton. 2008. A scalable hierarchical distributed language model. In Advances in neural information processing systems, pages 1081–1088. find
  18. Rosa M Ortega-Mendoza, Luis Villaseñor-Pineda, and Manuel Montes-y Gómez. 2007. Using lexical patterns for extracting hyponyms from the web. In MICAI 2007: Advances in Artificial Intelligence, pages 904–911. Springer. find
  19. Alan Ritter, Stephen Soderland, and Oren Etzioni. 2009. What is this, anyway: Automatic hypernym discovery. In Proceedings of the 2009 AAAI Spring Symposium on Learning by Reading and Learning to Read, pages 88–93. find
  20. HP Ritzema. applications. 1994. Drainage principles and Erik Tjong Kim Sang. 2007. Extracting hypernym pairs from the web. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, pages 165–168. Association for Computational Linguistics. find
  21. Sidney Siegel and N John Castellan Jr. 1988. Non-parametric statistics for the behavioral sciences. McGraw-Hill, New York. find
  22. Rion Snow, Daniel Jurafsky, and Andrew Y. Ng. 2005. Learning syntactic patterns for automatic hypernym discovery. In Lawrence K. Saul, Yair Weiss, and Léon Bottou, editors, Advances in Neural Information Processing Systems 17, pages 1297–1304. MIT Press, Cambridge, MA. find
  23. Rion Snow, Daniel Jurafsky, and Andrew Y. Ng. 2006. Semantic taxonomy induction from heterogenous evidence. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 801–808, Sydney, Australia, Jul… find
  24. Richard Socher, Eric H Huang, Jeffrey Pennin, Christopher D Manning, and Andrew Ng. 2011a. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Advances in Neural Information Processing Systems, pages 801–809. find
  25. Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, and Christopher D Manning. 2011b. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 151–161. Association for Computa… find our card
  26. Fabian M Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2008. Yago: A large ontology from wikipedia and wordnet. Web Semantics: Science, Services and Agents on the World Wide Web, 6(3):203–217. find
  27. Idan Szpektor, Eyal Shnarch, and Ido Dagan. 2007. Instance-based evaluation of entailment rule acquisition. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 456–463, Prague, Czech Republic, June. Association for Computational Linguistics. find
  28. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 384–394. Association for Computational Linguistics. find our card
  29. Julie Weeds, David Weir, and Diana McCarthy. 2004. Characterising measures of lexical distributional similarity. In Proceedings of the 20th international conference on Computational Linguistics, page 1015. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fu-2014-learning-semantic-hierarchies-word-embeddings/