Elamur Research
☰
home / papers / 1993 / Accurate Methods for the Statistics of Surprise and Coincidence

Accurate Methods for the Statistics of Surprise and Coincidence

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/J93-1003.pdf

Summary

Statistical text analysis often relies on tests such as Pearson's χ² and z-scores, which assume that counts are roughly normally distributed. Dunning shows that this assumption fails for rare events, and rare events make up much of real text. In Reuters newswire, words rarer than one per 50,000 tokens account for roughly a fifth to a third of running text. When the expected count is small, the normal approximation inflates significance by amounts that vary widely, so the errors cannot simply be corrected afterwards.

As an alternative, the paper proposes the generalized log-likelihood ratio statistic (−2 log λ), built on binomial and multinomial models of word counts. It is easy to compute, behaves well on small samples, and lets rare and common phenomena be compared directly. Where χ² is valid, the two measures give nearly the same values.

In a bigram-association experiment, χ² ranks one-off co-occurrences at the top. The log-likelihood ranking instead surfaces meaningful collocations such as "the swiss," "can be" and "previous year." The measure became a standard tool for collocation and term extraction.

Key points

  • The usual rule of thumb is that the normal approximation to the binomial holds when np(1−p) > 5. Below 1 it breaks down badly: for n = 100, normal estimates overstate significance about 40× at np = 0.1 and about 4×10²⁰ at np = 0.01.
  • In a half-million-word Reuters sample, everyday words such as "landscape" or "winter" appeared fewer than five times. In a 50,000-word sample, their expected count would fall below one.
  • Ranking bigrams by χ² gives the maximum score (31777) to pairs seen only once. Under log-likelihood, the top pair is "the swiss" with −2 log λ = 270.72.
  • The paper contrasts its approach with the IBM machine-translation group, which relied on corpora of about 1 billion words plus the Canadian Hansards, as well as with the count-based mutual information and z-scores used by Gale and Church.
  • Software implementing the method was to be distributed through the Consortium for Lexical Research. Suggested future work includes Fisher's exact test and Poisson-based models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

399 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 390 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@misc{dunning1993accurate,
  title = {{Accurate Methods for the Statistics of Surprise and Coincidence}},
  author = {Dunning, Ted},
  year = {1993},
  url = {https://aclanthology.org/J93-1003/}
}

References

9 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bradley, James V. (1968). Distribution-Free Statistical Tests. Prentice Hall. find
  2. Brown, Peter E; Cocke, John; Della Pietra, Stephen A.; Della Pietra, Vincent J.; Jelinek, Frederick; Lafferty, John D.; Mercer, Robert L.; and Roossin, Paul S. (1989). "A statistical approach to machine translation." Technical Report RC 14773 (\#66226), IBM Research Division. find
  3. Church, Ken W.; Gale, William A.; Hanks, Patrick; and Hindle, Donald (1989). "Parsing, word associations and typical predicate-argument relations." In Proceedings, International Workshop on Parsing Technologies, CMU. find
  4. Dumais, S.; Furnas, G.; Landauer, T.; Deerwester, S.; and Harshman, R. (1988). "Using latent semantic analysis to improve access to textual information." In Proceedings, CHI '88. 281-285. find
  5. Gale, William A., and Church, Ken W. k(~ A .., B) 29114 29121 29159 29077 29152 29154 29155 29163 29165 29158 29027 A the at the be months increased 1986 level again the as B swiss the volume the the the the the the increased the kji ~j kji Pji - qj = ~kjlogpj J (1993). "A program for aligning sente… find our card
  6. Gale, William A., and Church, Ken W. (in press). "Identifying word correspondences in parallel texts." McDonald, James E.; Plate, Tony; and Schvaneveldt, Roger (1990). "Using Pathfinder to extract semantic information from text." In Pathfinder Associative Networks: Studies in Knowledge Organization,… find
  7. Mood, A. M.; Graybill, E A.; and Boes, D. C. (1974). Introduction to the Theory of Statistics. McGraw Hill. find
  8. Schvaneveldt, Roger, ed. (1990). Pathfinder Associative Networks: Studies in Knowledge Organization. Ablex. find
  9. Salton, Gerald, and McGill, M. J. (1983). Introduction to Modern Information Retrieval. McGraw Hill. find