How we BLESSed distributional semantic evaluation

Summary

Benchmarks commonly used to test distributional semantic models (DSMs), such as TOEFL, WordSim 353 and concept categorization sets like AP, were mostly not built for this purpose. Each yields a single score that blends different semantic relations, can depend on outside components such as clustering algorithms, and gives little insight into how models differ. Baroni and Lenci propose BLESS, a data set for intrinsic evaluation. It consists of tuples linking a concrete, basic-level nominal concept to a relatum through an explicitly typed relation: coordinate, hypernym, meronym, attribute or event. It also includes controlled random relata (nouns, adjectives, verbs) for each concept. Models are compared by looking at the distributions of concept-relatum cosines, z-normalized concept by concept, for each relation type. Applied to several DSMs, the method shows clear profiles. Narrow content-word windows rank coordinates highest. A window over all words drifts toward syntactic similarity. Document-based co-occurrence separates related from random words but distinguishes relation types poorly. The approach exposes model behaviour that a single-number benchmark would hide.

Key points

The critique of existing benchmarks rests on several observations:
- TOEFL has 80 multiple-choice items, so one extra correct answer shifts accuracy by 1.25%.
- In WordSim 353, an annotation by Agirre et al. found that 92 of the 174 above-median pairs are only topically related.
- AP has 402 concepts in 21 classes and includes rare items like casuarina.
BLESS tuples have the form concept–relation–relatum (e.g., coyote-hyper-animal). The relation labels are COORD, HYPER, MERO, ATTRI and EVENT, plus RAN.N, RAN.J and RAN.V for random relata.
Attribute and event relations are intentionally left underspecified, so the authors avoid committing to finer, debatable distinctions.
The compared models are ContentWindow2, ContentWindow20, AllWindow2 and Document, plus two baselines, RelatumFrequency and RelationCardinality. For practical reasons, the Document model keeps 1 million dimensions selected by Local Mutual Information mass.
Neither baseline reproduces the meaningful patterns, so those patterns are not artifacts of relatum frequency or relation cardinality.
In AllWindow2, random nouns are statistically indistinguishable from hypernyms and meronyms. The authors attribute this to function-word contexts.

Interesting ideas

I1. Typed relations plus matched random distractors in one resource. This lets you measure two things separately: whether a model finds related words at all, and which kinds of relations it prefers. The idea transfers to evaluating embeddings for any language, Ukrainian included.
I2. Per-target z-normalization of similarity scores before pooling. This removes differences in cosine scale between target words, so distributions over relation types become comparable across concepts and models.
I3. Reporting full score distributions per relation, with significance tests between relation types. This gives a behavioural profile of a model instead of one quantized number. It suits any similarity-based system.
I4. Trivial baselines built from data-set properties (frequency, relation cardinality). These check that observed patterns are not just artifacts of how the benchmark was built, which is a sanity check worth adding to any new evaluation set.
I5. Underspecifying hard-to-define relation types. Using broad labels like "event" or "attribute" avoids fragile fine-grained labels and keeps gold-standard annotation less controversial.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

154 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 146 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{baroni2011how,
  title = {{How we BLESSed distributional semantic evaluation}},
  author = {Baroni, Marco and Lenci, Alessandro},
  booktitle = {Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics},
  year = {2011},
  url = {https://aclanthology.org/W11-2501/}
}

References

23 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Herv Abdi and Lynne Williams. 2010. Newman-Keuls and Tukey test. In N.J. Salkind, D.M. Dougherty, and B. Frey, editors, Encyclopedia of Research Design. Sage, Thousand Oaks, CA. find
Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasça, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and WordNet-based approaches. In Proceedings of HLT-NAACL, pages 19–27, Boulder, CO. find our card
Abdulrahman Almuhareb. 2006. Attributes in Lexical Acquisition. Phd thesis, University of Essex. find
Marco Baroni and Alessandro Lenci. 2010. Distributional Memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673–721. find our card
Marco Baroni, Stefan Evert, and Alessandro Lenci, editors. 2008. Bridging the Gap between Semantic Theory and Computational Simulations: Proceedings of the ESSLLI Workshop on Distributional Lexical Semantic. FOLLI, Hamburg. find
Marco Baroni, Eduard Barbu, Brian Murphy, and Massimo Poesio. 2010. Strudel: A distributional semantic model based on properties and types. Cognitive Science, 34(2):222–254. find
Alexander Budanitsky and Graeme Hirst. 2006. Evaluating wordnet-based measures of lexical semantic relatedness. Computational Linguistics, 32:13–47. find our card
D. A. Cruse. 1986. Lexical Semantics. Cambridge University Press, Cambridge. find
Stefan Evert. 2005. The Statistics of Word Cooccurrences. Dissertation, Stuttgart University. find
Christiane Fellbaum, editor. 1998. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, MA. find
Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002. Placing search in context: The concept revisited. ACM Transactions on Information Systems, 20(1):116–131. find
Thomas Landauer and Susan Dumais. 1997. A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2):211–240. find
Hugo Liu and Push Singh. 2004. ConceptNet: A practical commonsense reasoning toolkit. BT Technology Journal, pages 211–226. find
Kevin Lund and Curt Burgess. 1996. Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, 28:203–208. find
Ken McRae, George Cree, Mark Seidenberg, and Chris McNorgan. 2005. Semantic feature production norms for a large set of living and nonliving things. Behavior Research Methods, 37(4):547–559. find
George Miller and Walter Charles. 1991. Contextual correlates of semantic similarity. Language and Cognitive Processes, 6(1):1–28. find
Gregory Murphy. 2002. The Big Book of Concepts. MIT Press, Cambridge, MA. find
Timothy Rogers and James McClelland. 2004. Semantic Cognition: A Parallel Distributed Processing Approach. MIT Press, Cambridge, MA. find
Herbert Rubenstein and John Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. find
Karen Sparck Jones and Julia R. Galliers. 1996. Evaluating Natural Language Processing Systems: An Analysis and Review. Springer Verlag, Berlin. find
Peter Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37:141–188. find
Peter Turney. 2006. Similarity of semantic relations. Computational Linguistics, 32(3):379–416. find
Morton E. Winston, Roger Chaffin, and Douglas Herrmann. 1987. A taxonomy of part-whole relations. Cognitive Science, 11:417–444. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/baroni-2011-how-we-blessed-distributional-semantic/

📄 How we BLESSed distributional semantic evaluation