📄 Temporal Analysis of Language through Neural Language Models

Summary

This paper tackles the automatic detection of lexical change over the twentieth century. Earlier diachronic studies often relied on researchers selecting candidate words by hand. Here, the system both discovers which words shifted and locates when the shift happened. The authors sampled 10 million 5-grams per year from the English fiction portion of the Google Books Ngram corpus, covering 1850–2009. They trained a Skip-gram model one year at a time, seeding each year's vectors with those learned for the preceding year, so all yearly spaces stay comparable without any explicit alignment step. They then compared a word's vector in 2009 with its 1900 vector using cosine similarity. Words with low self-similarity, such as gay, cell, checked and actually, are flagged as changed, while function words barely move. Year-by-year similarity curves place the shift of cell around 1985–2009 and that of gay around the 1970s. The approach is simple and needs no supervision, and the resulting yearly vectors were released publicly.

Key points

• Data: about 10M sampled 5-grams per year from the Google Books Ngram English fiction corpus, 1850–2009. Text was lower-cased, and the vocabulary was limited to words seen at least 10 times overall. The years 1850–1899 served only to warm up the model.
• Model setup:
  • Skip-gram via gensim, with 200-dimensional vectors and a window size of 4.
  • The learning rate decays linearly from 0.01 to 0.0001 within each epoch.
  • Full training took roughly 4 days on a 2.9 GHz machine.
• Most changed words: checked had the lowest 1900-vs-2009 cosine similarity (0.3831), followed by check, gay (0.4079), actually, supposed, guess and cell (0.4413). Words occurring fewer than 500 times were excluded from this ranking.
• Least changed words: these are almost entirely function words, such as by (0.9331), than and for.
• Semantic shifts shown by neighbours:
  • gay moved from cheerful/pleasant towards lesbian/bisexual.
  • cell moved from closet/dungeon towards phone/cordless.
  • checked shifted from "restrain" towards "verify".
• Main limitation: a word whose usage declined or disappeared keeps an essentially frozen vector, so the method misses it.

Interesting ideas

• I1. Chronological warm-start initialisation. Each period's embeddings start from the previous period's vectors. This keeps all time slices in one shared coordinate space, so cross-period cosine comparisons are meaningful without a post-hoc alignment step. The trick could transfer to any sequence of corpora, for example by region or domain.
• I2. Convergence measured by angular change. Training on a slice stops once the mean angle between a word's vectors in consecutive epochs falls below a threshold. This criterion is defined directly on the embeddings and could replace a fixed epoch count when retraining models incrementally.
• I3. A chance baseline from global drift. Plotting the average self-similarity of all words gives a reference line for movement caused by sampling and random drift. A word only counts as changed if it falls clearly below this band, which is a cheap way to calibrate drift-based alerts in other settings.
• I4. Neighbour trajectories as an explanation tool. Tracking a word's similarity to its old and new nearest neighbours over time shows when a change happened and in what direction. Pairing this with real example sentences from each period makes the result easy for humans to verify.
• I5. Combining distance with frequency (proposed fix). Since fading words keep stable vectors, the authors suggest mixing cosine distance with frequency change into one metric. This is a useful reminder that embedding stability can reflect missing evidence rather than a lack of change.