Elamur Research⌕
☰
home / papers / 2014 / Inclusive yet Selective: Supervised Distributional Hypernymy Detection

Inclusive yet Selective: Supervised Distributional Hypernymy Detection

Summary

The paper asks whether the Distributional Inclusion Hypothesis is a reliable basis for detecting hypernymy. The hypothesis says that a broader word (animal) appears in a superset of the contexts of a narrower one (dog). The authors compare unsupervised inclusion measures (WeedsPrec, balAPinc, ClarkeDE, invCL) against a cosine baseline. They use several count-based spaces (U+W2, U+Sent, TypeDM) and their SVD-reduced versions. The unsupervised measures turn out to be fragile: how well they work depends heavily on which space they run in.

As an alternative, the authors train simple linear classifiers on word pairs. One uses concatenated vectors and one uses dimension-wise differences. These classifiers are stable across spaces and reach .84 accuracy on BLESS and .85 on the ENTAILMENT dataset. Because the difference-based model is linear, its weights can be projected back onto the original dimensions to pick out the dimensions that matter. Applying the unsupervised measures only on those dimensions markedly improves their results.

From this the authors propose the Selective Distributional Inclusion Hypothesis: inclusion does hold, but only over relevant dimensions. This matters for using distributional semantics in RTE and similar inference tasks.

Key points

  • Spaces and data:
    • The U+ corpus combines Gigaword, BNC, English Wackypedia and ukWaC, giving about 133K word types and 2.8B tokens.
    • Vectors use the 20k most frequent content words as dimensions with PPMI weighting.
    • Contexts are either 2-word windows (U+W2) or whole sentences (U+Sent).
    • TypeDM tuples supply the third space, and all spaces are also reduced to 300 SVD dimensions.
  • Evaluation sets:
    • BLESS: 200 concepts in 17 classes, with about 14K noun pairs over four relations (co-hyponymy, hypernymy, meronymy, random).
    • ENTAILMENT (Baroni et al., 2012): 2,770 balanced pairs, with positives drawn from WordNet.
  • Best supervised results: .84 accuracy on BLESS and .85 on ENTAILMENT, using dimension-wise difference features.
  • Selecting 500 dimensions from the supervised model raises invCL's hypernymy MAP on BLESS to .58. When the selector is trained on all of BLESS, giving an upper bound, the figure reaches .67. A control using 500 random dimensions leaves results essentially unchanged.
  • In the difference-squared ("g") features, which mark dimensions irrelevant to hypernymy, 200 of the top 250 dimensions are computer-science terms such as Linux and software. The authors read this as web noise from ukWaC.
    • Removing g features lowers Diff accuracy from .84 to .8.
    • Removing them raises invCL from .58 to .61.

Interesting ideas

  • I1. Backprojecting a linear classifier as a feature selector. A linear hyperplane learned in SVD space is mapped back onto the original interpretable dimensions. The resulting weights then serve as a mask for a simpler unsupervised measure. This is a general way to let a small labelled set guide a heuristic metric.
  • I2. Per-concept z-normalisation before comparing relations. For each concept, similarities to its nearest neighbours in each relation are z-scored. The relations are then compared with Tukey's HSD test. This removes differences in scale between concepts and is reusable for evaluating any relation-sensitive similarity measure.
  • I3. Two-sided difference features. Both the plain difference and the squared difference of the two vectors are used as features. One kind captures dimensions where the broader term dominates; the other flags dimensions that should be ignored. Inspecting the second group reveals corpus noise, which makes this a cheap diagnostic for domain contamination in web corpora.
  • I4. Random-subset control. The authors re-run the measures on randomly chosen dimensions of the same number. This shows the gain comes from which dimensions are selected, not from simply having fewer of them. The same control is worth adding to any feature-selection claim.
  • I5. Testing proper inclusion rather than plain inclusion. invCL rewards a hypernym for having extra contexts that the hyponym lacks. After dimension selection it works best among the measures, which suggests asymmetric relations should be scored in both directions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics Lexical semantics & word embeddings

Institutions

🎓The University of Texas at AustinAustin · United States51 works · 52,106 citations

Cite

Cite as Stephen Roller, Katrin Erk and Gemma Boleda. 2014. Inclusive yet Selective: Supervised Distributional Hypernymy Detection. Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{roller2014inclusive,
  title = {{Inclusive yet Selective: Supervised Distributional Hypernymy Detection}},
  author = {Stephen Roller, Katrin Erk and Gemma Boleda},
  booktitle = {Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers},
  year = {2014},
  url = {https://aclanthology.org/C14-1097/}
}

References

31 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marco Baroni and Alessandro Lenci. 2011. How we BLESSed distributional semantic evaluation. In Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics, pages 1–10, Edinburgh, UK, July. Association for Computational Linguistics. our card
  2. Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The WaCky wide web: A collection of very large linguistically processed web-crawled corpora. Language Resources and Evaluation, 43(3):209–226. find
  3. Marco Baroni, Raffaella Bernardi, Ngoc-Quynh Do, and Chung-chieh Shan. 2012. Entailment above the word level in distributional semantics. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 23–32, Avignon, France, April. Association f… our card
  4. Matthew Berland and Eugene Charniak. 1999. Finding parts in very large corpora. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 57–64, College Park, Maryland, USA, June. Association for Computational Linguistics. our card
  5. Paul Buitelaar, Philipp Cimiano, and Bernardo Magnini. 2005. Ontology Learning from Text: Methods, Evaluation and Applications. Frontiers in Artificial Intelligence and Applications Series. IOS Press, Amsterdam. find
  6. Timothy Chklovski and Patrick Pantel. 2004. Verbocean: Mining the web for fine-grained semantic verb relations. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 33–40. find
  7. Philipp Cimiano, Aleksander Pivk, Lars Schmidt-Thieme, and Steffen Staab. 2005. Learning taxonomic relations from heterogeneous sources of evidence. Ontology Learning from Text: Methods, evaluation and applications. find
  8. Daoud Clarke. 2009. Context-theoretic semantics for natural language: an overview. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics, pages 112–119, Athens, Greece, March. Association for Computational Linguistics. find
  9. Maayan Geffet and Ido Dagan. 2004. Feature vector quality and distributional similarity. In Proceedings of the 20th International Conference on Computational Linguistics, page 247. Association for Computational Linguistics. find
  10. Roxana Girju, Adriana Badulescu, and Dan Moldovan. 2003. Learning semantic constraints for the automatic discovery of part-whole relations. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, page… find
  11. Roxana Girju, Adriana Badulescu, and Dan Moldovan. 2006. Automatic discovery of part-whole relations. Computational Linguistics, 32(1):83–135. find
  12. Marti A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In Proceedings of the 14th Conference on Computational Linguistics, pages 539–545, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  13. Aurélie Herbelot and Mohan Ganesalingam. 2013. Measuring semantic content in distributional vectors. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 440–445, Sofia, Bulgaria, August. Association for Computational Linguistics… find
  14. Lili Kotlerman, Ido Dagan, Idan Szpektor, and Maayan Zhitomirsky-geffet. 2010. Directional distributional similarity for lexical inference. Natural Language Engineering, 16:359–389, 10. find
  15. Alessandro Lenci and Giulia Benotto. 2012. Identifying hypernyms in distributional semantic spaces. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International… our card
  16. Alessandro Lenci. 2008. Distributional approaches in linguistic and cognitive research. Italian Journal of Linguistics, 20(1):1–31. find
  17. Dekang Lin, Shaojun Zhao, Lijuan Qin, and Ming Zhou. 2003. Identifying synonyms among distributionally similar words. In Proceedings of the 18th international Joint Conference on Artificial intelligence, pages 1492– 1493. find
  18. Dekang Lin. 1998. An information-theoretic definition of similarity. In Proceedings of the 15th International Conference on Machine Learning, volume 98, pages 296–304. find
  19. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751, Atlanta, Georgia,… our card
  20. Gregory L. Murphy. 2002. The Big Book of Concepts. MIT Press, Boston, MA. find
  21. Patrick Pantel and Marco Pennacchiotti. 2006. Espresso: Leveraging generic patterns for automatically harvesting semantic relations. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics. our card
  22. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertran Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, MMatthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn… find
  23. Enrico Santus. 2013. SLQS: An entropy measure. Master’s thesis, University of Pisa. find
  24. Rion Snow, Daniel Jurafsky, and Andrew Y. Ng. 2005. Learning syntactic patterns for automatic hypernym discovery. In Lawrence K. Saul, Yair Weiss, and Léon Bottou, editors, Advances in Neural Information Processing Systems 17, pages 1297–1304, Cambridge, MA. MIT Press. find
  25. Rion Snow, Daniel Jurafsky, and Andrew Y. Ng. 2006. Semantic taxonomy induction from heterogenous evidence. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics, ACL-44, pages 801–808, Stroudsbur… find
  26. Peter Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37:141–188. our card
  27. Peter D. Turney. 2006. Similarity of semantic relations. Computational Linguistics, 32(3):379–416. our card
  28. Julie Weeds and David Weir. 2003. A general framework for distributional similarity. In Michael Collins and Mark Steedman, editors, Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, pages 81–88. our card
  29. Julie Weeds, David Weir, and Diana McCarthy. 2004. Characterising measures of lexical distributional similarity. In Proceedings of the 20th International Conference on Computational Linguistics, pages 1015–1021, Geneva, Switzerland, Aug 23–Aug 27. Association for Computational Linguistics, COLING. find
  30. Maayan Zhitomirsky-Geffet and Ido Dagan. 2005. The distributional inclusion hypotheses and lexical entailment. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pages 107–114, Ann Arbor, Michigan, June. Association for Computational Linguistics. our card
  31. Maayan Zhitomirsky-Geffet and Ido Dagan. 2009. Bootstrapping distributional feature vector quality. Computational linguistics, 35(3):435–461. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/roller-2014-inclusive-yet-selective-supervised-distributional/