📄 Automatic Retrieval and Clustering of Similar Words

Summary

The paper addresses how to discover semantically similar words automatically from raw text, as a first step toward learning word meaning and building corpus-specific thesauri. Lin parses a 64-million-word newswire corpus with a broad-coverage dependency parser and represents each word by the grammatical contexts it appears in, stored as dependency triples. He then defines an information-theoretic similarity measure: two words are similar to the extent that the information carried by their shared contexts makes up a large part of the total information in both descriptions. Applying this measure to frequent nouns, verbs and adjectives/adverbs yields a thesaurus that lists the 200 nearest neighbours for each word. The paper's main methodological contribution is a direct, objective evaluation. Both WordNet and Roget are converted into the same ranked-neighbour format, and entries for the same word are compared across resources. The automatic thesaurus agrees with WordNet more closely than Roget does. This shows that distributional thesauri can rival hand-built resources and can be measured against them quantitatively.

Key points

• Corpus: 64M words in total, drawn from Wall Street Journal (24M), San Jose Mercury (21M) and AP Newswire (19M). Parsing produced 56.5M dependency triples, of which 8.7M are unique.
• Vocabulary covered: all words occurring at least 100 times, namely 5469 nouns, 2173 verbs and 2632 adjectives/adverbs. Each word's thesaurus entry keeps its top 200 neighbours.
• Mutual reciprocity: the program found 543 noun pairs, 212 verb pairs and 382 adjective/adverb pairs that are respective nearest neighbours, meaning each word is the other's top match.
• Evaluation set: 4294 nouns present in WordNet 1.5 and in Roget.
  • Average similarity to WordNet entries was 0.212 for the proposed measure, compared with 0.178 for Roget.
  • Competing measures scored lower: Hindle_r 0.204, cosine 0.199 and Hindle 0.165.
• Significance: paired per-entry differences have 95% confidence intervals entirely above zero. This supports the ranking sim > Hindle_r > cosine. The comparison also shows that using all dependency types, rather than only subject/object relations, clearly helps.

Interesting ideas

• I1. Common evaluation format. Hand-built resources are converted into the same "word → weighted neighbour list" format as the automatic output. This allows heterogeneous resources to be compared on equal footing, and the trick could be reused to benchmark distributional models for Ukrainian against existing lexical databases.
• I2. Information-weighted overlap. Shared contexts are weighted by their information content (pointwise mutual information), and contexts with non-positive association are discarded. This is a simple way to keep frequent, uninformative contexts from dominating the similarity score.
• I3. Rich syntactic context features. Word context is described through typed dependency relations, including inverse relations such as "obj-of", rather than through bag-of-words windows. The gain over a subject/object-only variant suggests that richer syntactic features pay off, which is relevant whenever a dependency parser (e.g. a UD-trained one) is available.
• I4. Paired significance testing. Systems are compared through per-item differences and the standard deviation of the mean, rather than through raw averages alone. This is a cheap way to make claims about small score gaps statistically defensible.
• I5. Similarity trees for word senses. Each neighbour is attached under whichever already-placed word it most resembles. The resulting branches roughly separate a word's senses, which makes this a lightweight heuristic for exploring polysemy without a sense inventory.