Elamur Research⌕
☰
home / papers / 1992 / Word-Sense Disambiguation Using Statistical Models of Roget's Categories Trained on Large Corpora

Word-Sense Disambiguation Using Statistical Models of Roget's Categories Trained on Large Corpora

Summary

This paper tackles word-sense disambiguation for English in unrestricted text without hand-labeled training data. Each sense of a word is equated with one of the categories under which it appears in Roget's International Thesaurus (Fourth Edition). The system then picks the most probable category given the surrounding context. Instead of training a separate model per ambiguous word, it trains one model per Roget category. For each category, it gathers 100-word contexts around every occurrence of every member word in the 10 million word Grolier's Encyclopedia. It then finds words that are unusually frequent in those contexts and weights them by a smoothed log ratio of category-conditional to overall probability. At test time, weights of words within ±50 words are summed in a Bayesian fashion, and the best-scoring listed category wins. On 12 polysemous words studied in earlier work, the method reached 92% accuracy. This matters because class-level training sidesteps the knowledge acquisition bottleneck and scales to the whole vocabulary from raw monolingual text.

Key points

  • The models cover 1042 Roget categories. Training data are concordances from the June 1991 Grolier's Encyclopedia (about 10M words); TOOLS/MACHINERY alone yields 30,924 context lines.
  • Word weight is the log of Pr(w|RCat)/Pr(w). Local estimates are interpolated with global Pr(w) following Gale, Church and Yarowsky (1992). A word occurring k times contributes 1/k per context, so frequent members do not dominate.
  • Classification uses a wide ±50-word window and a uniform category prior. It considers only the categories listed for the target word, typically around six. Statistics are computed on lemmas, and part-of-speech distinctions are handled beforehand by Church's stochastic tagger.
  • The overall result is 92% accuracy on 12 previously studied polysemous words. For crane (74 occurrences: 36 animal, 38 machinery), only one case was wrong, giving 99%.
  • Roget tags can be linked to translations using the Canadian Hansards. For duty, PRICE/FEE-tagged instances mostly align with droit(s)/douane, while DUTY (obligation) instances align with devoir.

Interesting ideas

  • I1. Train on classes, apply to members. The method learns context profiles for semantic classes rather than for individual ambiguous words, which removes the need for sense-labeled examples. This could work with any category inventory, such as WordNet-style hierarchies or a Ukrainian thesaurus, to bootstrap WSD in low-resource settings.
  • I2. Noise dilutes across classes. Spurious contexts from polysemous members spread thinly over the other 1041 categories, while the true signal concentrates in one. This argument for why noisy, automatically gathered training data still works applies broadly to weak supervision.
  • I3. Downweight frequent members. Giving each occurrence a 1/k weight keeps a few common words from dominating a class profile. This simple trick is useful whenever pooled data is skewed toward a few high-frequency items.
  • I4. Smooth local with global. Interpolating sparse local estimates with global corpus statistics balances measurement error against irrelevance. This is a reusable recipe for any small-sample association score.
  • I5. Score margin as confidence. The gap between the top two category totals signals reliability; the one misclassified crane had low scores for all models. Margins like this could support abstention or active-learning selection.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Word sense disambiguation

Cited by

119 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 109 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{yarowsky1992word,
  title = {{Word-Sense Disambiguation Using Statistical Models of Roget's Categories Trained on Large Corpora}},
  author = {Yarowsky, David},
  booktitle = {COLING 1992 Volume 2: The 14th International Conference on Computational Linguistics},
  year = {1992},
  doi = {10.3115/992133.992140},
  url = {https://doi.org/10.3115/992133.992140}
}

References

24 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bar-Hillel (1960). "Automatic Translation of Languages," in Advances in Coenpmera, Donald Booth and R. E. Meagher, eds., Academic, New York. Black, Ez~ (1988), "'An Experiment in Computational Discrimination of English Word Sea~s." IBM Journal of Research and Dev¢lopmenl, v 32. pp 185-194. find
  2. Brown, Pr.mr, Stephen Della Pietra, Vincent Della Pinata, and Robert Mercer (1991), "Word Sense Disambiguition using Statistical Methods," Prooteding$ of the 2¢th Annual Meeting of the Association [or Computational Linguistics, pp 264-270. find
  3. Brown, Peter, Vil,,c~at Delh Pintra, Peter deSouza, and Rck~rt Mercer (1990), "clasa-based n-gram Modeht of Natural Language," Proceedings of the IBM Natural Language ITL, Paris, Fnmce, pp 283-298. find
  4. C~apman, Robert (1977). Roget's International Thesaur~ (Fourth Edition), Haq~r and Row, New York, Choueka, Ymmov, and Serge Lusignam (1985). "'Disambiguation by Short Contexts," Computera and the ltwnanities, v 19. pp. 14%158. find
  5. Omtch, Kmneth (1989), "A Stochastic Parts Program an Noun Phnse Parser for Un~strict~d Text, '~ Proceeding, IEEE International Conference on Acovatics, Speech and Signal Processing, Glasgow. find
  6. Clear, Jeremy (1989). "'An Experiment in Automatic Word Sense ld~RificJtlon." Internal Doctwnent, Oxford Univerlity Press, Oxford. find
  7. Courdl, Garriton (1989). A Connectionist Appre.ach to Word Sense Disamblguatioa, Pitman, London. find
  8. Dagan, 13o, Alon leaS, and Olrike Schwall (1991), "Two Languages am Infmmative than One," Proceedings of the 29th Annual Meeting of the Aesoclation for Computatiosal Linguistics. pp 130-137. find
  9. Gale, William, Kenneth (:hutch, and David Yarowsky (1992), "Ditcriminatlon Decisions for 100,000-Dimensional Spaces" AT&T Statistical Retear¢.h Report No. 103. find
  10. Gale, William, Kenneth Church, and David Yarowsky (1992), "A AcrEs DE COLING-92, NANTES. 23-28 AO~r 1992 find
  11. Method for Disarnbiguating Word S~ses in • Large Ca~.ts," to appear in Computers and llumdnitits, Gnmger, Richard (1977), "FOUL-UP A program that figures out meanings of w o from ¢~ntext," HCAII-77, pp. 172-178. find
  12. Guthile, J., L Guthrle, Y. Walks, and H. Aidinejad (1991), "Subject-Dependent Co-oc.cunea~ and Word Sense Disambiguation," Proceedings of the 29th Annual M, teeing of the Association for Compulmlanal Linguistics, pp 146-152. find
  13. Hanks, Patrick (ed.) (1979), Collins English Dictionary, Collins, London and Glasgow, He.am, Matti (1991), '*Noun Hctnograph Disambiguation Using Local Context in Large Text Corpora," Using Corpora, Univenrity of Waterloo, Waterloo, Ontario, ]tirst, Graerne. (1987), Stmamic luterpretation and the Re… find
  14. Lask, Michael (1986), "Automatic Sense Disambiguadoa: How to tell • Pine Cone from an Ice Cream Cone," Proceeding of the 1986 SIGDOC Conference, Association for Ct~nputing Machinery, New York. find
  15. Miller, George (1990), "Woednea: An On-line Leaical Database," InterncUionalJournal of Lexicography, 4(3), 1990. (Special Issue). Moate].ler, Fredrick, and David Wallace (1964). Inference and Disputed Authorxhip: The Federalist, Addison-Wesley, Reading, Mastacinamtts. find
  16. Salton, G. (1989), Automatic Text Processing, Addis0n-Wesley Publishing Co. Sinclair, J., Ilanks, P., Fox, G., Moon, R., Stock, P. et el. (edl.) (1987) Collin~ Cobuild English Language Dictionary, Collins, London and Glasgow. find
  17. Slator, Brian (1991), "Using Context for Sense Prefenmce," in Zcmik (ed.) Lexical AcqioMtion: E:9~ioitiog On-Line ResoW:ces to B~Id a L~icon, Lawrence Edbamn, Hillsdale, N'J. find
  18. Smadja, F. and K. McKeown (1990), "'Automatically Ext~cting and Representing Collocations for Language Generation," Proceedings of the 21tth Annual Meeting of the Association for Computational Linguistics. find
  19. Small, S. and C. Rieger (1982), "'Parring and Contprehending with Word Experts (A Theory and its Realization),'* in StrategiesfgrNalural Language Processing, W, Lehnert and M, Ringle, eds., Lawrence Erlbaum Associates, Hillsdale, NJ. find
  20. Walker, Donald (1987), "Ka]owledge Resource Tools for Aeo~'ssing Large Text Files," in Machine Translation: Theoretical and Methodological Issues, Serges Nirenberg, ed., Cambridge University Pm~s, Cambridge, England. find
  21. Weiss, Stephen (1973), "Learning to Disamb/guate," Information Storage and Rari~val. v. 9. pp 33-4 I, Veronis, Jean and Nancy lde (1990), "Word Sense Disamliiguation wilh Very Large Neural Networks Extracted from Machine Readable Dictionaries," in Proceedings COLING-90 , pp 389-394. find
  22. Yngve, Victor (1955), "Syntax and the Problem of Multiple Meaning," in Machine Translation of Languages. William Locke and Donald Booth, eds., Wiley, New York. find
  23. Zemik, Un (1990) "Tagging Word Senses in a Corpus: The Nee.die in the Haystack Revisited," in Text-Bated Intelligenl Systems: Currem Research in Text Analysis, Information Extraction, and Retrisval, P.S. Jacobs, ed., GE Research & Devdopmemt Center, Schetw..oady, New York. find
  24. PROC. OF COL1NG-92, NANrFES, AUG. 23-28, 1992 find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yarowsky-1992-word-sense-disambiguation-statistical-models/