Elamur Research
☰
home / papers / 1990 / Word Association Norms, Mutual Information, and Lexicography

Word Association Norms, Mutual Information, and Lexicography

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/J90-1003.pdf

Summary

Psycholinguists have traditionally gathered word association norms by asking thousands of people to respond to a few hundred stimulus words, as in Palermo and Jenkins (1964). Church and Hanks argue that this approach is expensive, small in scale and unreliable. They propose the association ratio, a corpus statistic derived from Fano's mutual information. It compares how often word x precedes word y within a short window against what chance would predict, so association norms can be estimated automatically for tens of thousands of words. Using Associated Press newswire, they show that high-scoring pairs capture several kinds of relation: semantic links (doctor/nurses), fixed phrases (bread and butter), compounds (United States) and verb–preposition constraints (save…from). The paper argues the measure is useful for:

  • language models in OCR and speech recognition,
  • parsing disambiguation,
  • information retrieval,
  • above all, lexicography, where the tables act as an index for organizing concordance lines into usage-based senses.

The authors also note limitations. The measure relies only on distributional evidence, so it cannot tell compositional pairs from idiomatic ones, and it does not group words into syntactic classes without a separate preprocessing step.

Key points

  • Formula and settings: I(x,y) = log2 P(x,y) / (P(x)P(y)). The window is w = 5 words. Pairs with f(x,y) ≤ 5 are discarded, and scores above roughly 3 tend to be interesting.
  • Asymmetry: unlike true mutual information, the ratio is deliberately left asymmetric so that word order is kept. For example, save…from occurs 176 times versus 18 for the reverse order in 1988 AP (44M words).
  • Corpora: 1987 AP (15M words), 1988 AP (36M words) and a tagged corpus of 8.6M tokens.
  • Doctor example: the top associates of doctor are dentists, nurses, treating, treat and hospitals. These broadly resemble the Palermo and Jenkins norms, which covered 200 stimulus words.
  • Window choice: mean and variance of word separation help pick a window size. Fixed expressions show variance 0, compounds have a separation near 1, and semantic pairs such as man/woman show high variance.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

773 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 760 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@misc{church1990word,
  title = {{Word Association Norms, Mutual Information, and Lexicography}},
  author = {Church, Kenneth Ward and Hanks, Patrick},
  year = {1990},
  url = {https://aclanthology.org/J90-1003/}
}

References

17 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Church, K. 1988 "A Stochastic Parts Program and Noun Phrase Parser for Unrestricted Text," Second Conference on Applied Natural Language Processing, Austin, TX. find
  2. Church, K.; Gale, W.; Hanks, P.; and Hindle, D. 1989 "Parsing, Word Associations and Typical Predicate-Argument Relations," International Workshop on Parsing Technologies, CMU. find
  3. Fano, R. 1961 Transmission of Information: A Statistical Theory of Communications. MIT Press, Cambridge, MA. find
  4. Firth, J. 1957 "A Synopsis of Linguistic Theory 1930-1955," in Studies in Linguistic Analysis, Philological Society, Oxford; reprinted in Palmer, F. (ed.) 1968 Selected Papers of J. R. Firth, Longman, Harlow. find
  5. Francis, W. and Ku~era, H. 1982 Frequency Analysis of English Usage. Houghton Mifflin Company, Boston, MA. find
  6. Good, I. J. 1953 The Population Frequencies of Species and the Estimation of Population Parameters. Biometrika, Vol. 40, 237-264. find
  7. Hanks, P. 1987 "Definitions and Explanations," in J. Sinclair (ed.), Looking Up: An Account of the COBUILD Project in Lexical Computing. Collins, London and Glasgow. find
  8. Hindle, D. 1983a "Deterministic Parsing of Syntactic Non-fluencies." In Proceedings of the 23rd Annual Meeting of the Association for Computational Linguistics. find
  9. Hindle, D. 1983b "User Manual for Fidditch, a Deterministic Parser." Naval Research Laboratory Technical Memorandum \#7590-142. find
  10. Hornby, A. 1948 The Advanced Learner's Dictionary, Oxford University Press, Oxford, U.K. find
  11. Jelinek, F. 1982. (personal communication) find
  12. Kahan, S.; Pavlidis, T.; and Baird, H. 1987 "On the Recognition of Printed Characters of any Font or Size," IEEE Transactions PAMI, 274-287. find
  13. Meyer, D.; Schvaneveldt, R.; and Ruddy, M. 1975 "Loci of Contextual Effects on Visual Word-Recognition," in P. Rabbitt and S. Dornic (eds.), Attention and Performance V, Academic Press, New York. find
  14. Palermo, D. and Jenkins, J. 1964 "Word AssociationNorms." University of Minnesota Press, Minneapolis, MN. find
  15. Sinclair, J.; Hanks, P.; Fox, G.; Moon, R.; and Stock, P. (eds.) 1987a Collins Cobuild English Language Dictionary. Collins, London and Glasgow. find
  16. Sinclair, J. 1987b "The Nature of the Evidence," in J. Sinclair (ed.), Looking Up: An Account of the COBUILD Project in Lexical Computing. Collins, London and Glasgow. find
  17. Smadja, F. In press. "Microcoding the Lexicon with Co-Occurrence Knowledge," in Zernik (ed.), Lexical Acquisition: Using On-Line Resources to Build a Lexicon, MIT Press, Cambridge, MA. NOTES find