📄 A distributional similarity approach to the detection of semantic change in the Google Books Ngram corpus.

Summary

The paper asks whether distributional semantics can automatically flag English words whose meaning shifted between the 1960s and the 1990s, and whether it does better than a plain frequency-change signal. The authors used the American English 2-grams of the Google Books Ngram corpus and built two time slices, 1960–1964 and 1995–1999. For each word they built two context vectors over one shared set of context dimensions, one per decade. Co-occurrence counts were weighted with local mutual information (LMI). The cosine between a word's 60s and 90s vectors serves as its stability score, and a low cosine suggests change. To test this, five human raters scored 100 sampled words for perceived change. The similarity score tracked the human ranking better than a log-ratio of decade frequencies, though the two signals behaved differently across word groups. Some low-similarity words turned out to reflect a shift in usage context rather than a new sense. The work matters because it offers a simple, quantitatively evaluated baseline for diachronic semantic change detection and novel sense discovery.

Key points

• Data: the Google Books Ngram American English 2-grams. The 60s slice holds about 25M 2-grams and the 90s slice about 28M. Counts were summed over each five-year window.
• Vector space: 250,000 randomly chosen mid-frequency words serve as context dimensions, weighted with LMI and compared by cosine. Of 10,000 mid-frequency target words, 48.4% scored above 0.8, 50% fell between 0.2 and 0.8, and only 1.6% scored below 0.2.
• Gold standard: 100 words, 60% of which grew in frequency and 40% of which declined. Five raters judged them on a 0–3 scale. Average pairwise Pearson agreement among raters was 0.51.
• Correlation with the human ranking:
  • The similarity measure reached 0.386, against 0.301 for the frequency measure.
  • For words that became more frequent, similarity reached 0.445 while frequency reached only 0.184.
  • For words that declined, the pattern reversed: similarity scored 0.163 and frequency 0.310.
• A direction-agnostic version of the frequency measure correlated with human judgments at only 0.04. The two measures correlate with each other at 0.38, so they capture partly different information.

Interesting ideas

• I1. Shared context space across time slices. The method fixes one set of context dimensions and fills it separately from each period's counts. Comparing "word_60s" against "word_90s" then becomes an ordinary within-space cosine, with no alignment step needed. This trick could carry over to comparing domains, registers or regional varieties, not just periods.
• I2. Bigrams as a cheap context window. Treating only immediately adjacent words as context makes it possible to build vectors straight from n-gram releases when full text is unavailable. This suits other copyright-restricted resources distributed only as n-gram counts.
• I3. Benchmarking against inter-rater agreement. The authors judge the model's correlation relative to how well the human raters agree with each other, not in absolute terms. This gives a realistic upper bound for small, subjective gold standards.
• I4. Splitting the evaluation by frequency direction. Evaluating rising and falling words separately showed that the frequency baseline mostly benefits from an asymmetry: words that rise are the ones more likely to change. This kind of breakdown can expose shortcut signals behind seemingly strong baselines.
• I5. Inspecting false positives via top-weighted collocates. Listing the top-weighted collocates of words like 'parent' and 'sleep' showed that some apparent errors are real shifts in usage context. Such context drift is useful in its own right, as possible early evidence of meaning change and as data for social or cultural research.