Word Association Norms, Mutual Information, and Lexicography

Summary

Psycholinguists have traditionally gathered word association norms by asking thousands of people to respond to a few hundred stimulus words, as in Palermo and Jenkins (1964). Church and Hanks argue that this approach is expensive, small in scale and unreliable. They propose the association ratio, a corpus statistic derived from Fano's mutual information. It compares how often word x precedes word y within a short window against what chance would predict, so association norms can be estimated automatically for tens of thousands of words. Using Associated Press newswire, they show that high-scoring pairs capture several kinds of relation: semantic links (doctor/nurses), fixed phrases (bread and butter), compounds (United States) and verb–preposition constraints (save…from). The paper argues the measure is useful for:

language models in OCR and speech recognition,
parsing disambiguation,
information retrieval,
above all, lexicography, where the tables act as an index for organizing concordance lines into usage-based senses.

The authors also note limitations. The measure relies only on distributional evidence, so it cannot tell compositional pairs from idiomatic ones, and it does not group words into syntactic classes without a separate preprocessing step.

Key points

Formula and settings: I(x,y) = log2 P(x,y) / (P(x)P(y)). The window is w = 5 words. Pairs with f(x,y) ≤ 5 are discarded, and scores above roughly 3 tend to be interesting.
Asymmetry: unlike true mutual information, the ratio is deliberately left asymmetric so that word order is kept. For example, save…from occurs 176 times versus 18 for the reverse order in 1988 AP (44M words).
Corpora: 1987 AP (15M words), 1988 AP (36M words) and a tagged corpus of 8.6M tokens.
Doctor example: the top associates of doctor are dentists, nurses, treating, treat and hospitals. These broadly resemble the Palermo and Jenkins norms, which covered 200 stimulus words.
Window choice: mean and variance of word separation help pick a window size. Fixed expressions show variance 0, compounds have a separation near 1, and semantic pairs such as man/woman show high variance.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

773 papers of our library cite it (of the 140673 whose bibliographies we read); 23 of them have a page here, 750 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@misc{church1990word,
  title = {{Word Association Norms, Mutual Information, and Lexicography}},
  author = {Church, Kenneth Ward and Hanks, Patrick},
  year = {1990},
  doi = {10.5555/89086.89095},
  url = {https://doi.org/10.5555/89086.89095}
}

References

17 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Church, K. 1988 "A Stochastic Parts Program and Noun Phrase Parser for Unrestricted Text," Second Conference on Applied Natural Language Processing, Austin, TX. find
Church, K.; Gale, W.; Hanks, P.; and Hindle, D. 1989 "Parsing, Word Associations and Typical Predicate-Argument Relations," International Workshop on Parsing Technologies, CMU. find
Fano, R. 1961 Transmission of Information: A Statistical Theory of Communications. MIT Press, Cambridge, MA. find
Firth, J. 1957 "A Synopsis of Linguistic Theory 1930-1955," in Studies in Linguistic Analysis, Philological Society, Oxford; reprinted in Palmer, F. (ed.) 1968 Selected Papers of J. R. Firth, Longman, Harlow. find
Francis, W. and Ku~era, H. 1982 Frequency Analysis of English Usage. Houghton Mifflin Company, Boston, MA. find
Good, I. J. 1953 The Population Frequencies of Species and the Estimation of Population Parameters. Biometrika, Vol. 40, 237-264. find
Hanks, P. 1987 "Definitions and Explanations," in J. Sinclair (ed.), Looking Up: An Account of the COBUILD Project in Lexical Computing. Collins, London and Glasgow. find
Hindle, D. 1983a "Deterministic Parsing of Syntactic Non-fluencies." In Proceedings of the 23rd Annual Meeting of the Association for Computational Linguistics. find
Hindle, D. 1983b "User Manual for Fidditch, a Deterministic Parser." Naval Research Laboratory Technical Memorandum \#7590-142. find
Hornby, A. 1948 The Advanced Learner's Dictionary, Oxford University Press, Oxford, U.K. find
Jelinek, F. 1982. (personal communication) find
Kahan, S.; Pavlidis, T.; and Baird, H. 1987 "On the Recognition of Printed Characters of any Font or Size," IEEE Transactions PAMI, 274-287. find
Meyer, D.; Schvaneveldt, R.; and Ruddy, M. 1975 "Loci of Contextual Effects on Visual Word-Recognition," in P. Rabbitt and S. Dornic (eds.), Attention and Performance V, Academic Press, New York. find
Palermo, D. and Jenkins, J. 1964 "Word AssociationNorms." University of Minnesota Press, Minneapolis, MN. find
Sinclair, J.; Hanks, P.; Fox, G.; Moon, R.; and Stock, P. (eds.) 1987a Collins Cobuild English Language Dictionary. Collins, London and Glasgow. find
Sinclair, J. 1987b "The Nature of the Evidence," in J. Sinclair (ed.), Looking Up: An Account of the COBUILD Project in Lexical Computing. Collins, London and Glasgow. find
Smadja, F. In press. "Microcoding the Lexicon with Co-Occurrence Knowledge," in Zernik (ed.), Lexical Acquisition: Using On-Line Resources to Build a Lexicon, MIT Press, Cambridge, MA. NOTES find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/church-1990-word-association-norms-mutual-information/

📄 Word Association Norms, Mutual Information, and Lexicography