Elamur Research⌕
☰
home / papers / 2003 / Using the Web to Obtain Frequencies for Unseen Bigrams

Using the Web to Obtain Frequencies for Unseen Bigrams

Paper · 2003 · Frank Keller and Mirella Lapata

Summary

Statistical NLP struggles with data sparseness: many plausible word combinations never occur even in a large corpus, so their counts are zero. Keller and Lapata test whether hit counts from a Web search engine can replace these missing corpus counts. Their work extends earlier studies by Grefenstette (1998) and Zhu and Rosenfeld (2001). Instead of targeting a single application, they randomly sampled predicate-argument pairs from the British National Corpus (BNC). These were adjective-noun, noun-noun and verb-object combinations, each split into pairs attested in the BNC and pairs absent from it. The authors validated Web counts for these pairs in four ways:

  • comparison with frequencies from two corpora
  • comparison with human plausibility ratings
  • comparison with estimates from class-based smoothing
  • performance on a pseudodisambiguation task using data sets from earlier work

Web figures matched corpus counts closely and correlated reliably with both human judgments and smoothed estimates. They also performed well in pseudodisambiguation. Web data is noisier than corpus data, but its scale makes it a practical, annotation-free source of counts for rare or unseen word pairs.

Key points

  • Source corpus: The bigram samples come from the BNC, which has about 90 million words of written text and 10 million words of speech.
  • Sample size: There are three relation types, each with 90 seen and 90 unseen bigrams. Each type is built around 30 predicate words (adjectives, nouns or verbs) chosen for low sense ambiguity in WordNet and unambiguous part of speech.
  • Seen bigrams: For each predicate word, seen bigrams were picked from three frequency bands (high, medium, low) based on log-transformed co-occurrence counts. This covers the full frequency range.
  • Extraction tools: Gsearch was used for adjective-noun and verb-object pairs, over CLAWS4-tagged text. Noun-noun compounds were identified with Lauer's (1995) heuristic, which excludes pairs embedded in longer noun sequences.
  • Motivation: Banko and Brill (2001a, 2001b) found that learners keep improving with more training data. At the time of writing, Google indexed roughly 3,033 million pages.
  • Missing figures: The exact correlation values are not given in the excerpt provided for this summary.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom255 works · 68,771 citations🎓University of SheffieldSheffield · United Kingdom76 works · 20,808 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksInformation Systems · 523 works

Cite

Cite as Frank Keller and Mirella Lapata. 2003. Using the Web to Obtain Frequencies for Unseen Bigrams. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/089120103322711604

@article{keller2003using,
  title = {{Using the Web to Obtain Frequencies for Unseen Bigrams}},
  author = {Frank Keller and Mirella Lapata},
  journal = {Computational Linguistics},
  year = {2003},
  doi = {10.1162/089120103322711604},
  url = {https://doi.org/10.1162/089120103322711604}
}

References

59 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abney, Steve. 1996. Partial parsing via finite-state cascades. In John Carroll, editor, Workshop on Robust Parsing Eighth European Summer School in Logic, Language and Information, pages 8–15, Prague, Czech Republic. find
  2. Abney, Steve and Marc Light. 1999. Hiding a semantic class hierarchy in a Markov model. In Andrew Kehler and Andreas Stolcke, editors, Proceedings of the ACL Workshop on Unsupervised Learning in Natural Language Processing, pages 1–8, College Park, MD. find
  3. Agirre, Eneko and David Martinez. 2000. Exploring automatic word sense disambiguation with decision lists and the Web. In Proceedings of the 18th International Conference on Computational Linguistics, pages 11–19, Saarbrücken, Germany. find
  4. Banko, Michele and Eric Brill. 2001a. Mitigating the paucity-of-data problem: Exploring the effect of training corpus size on classifier performance for natural language processing. In James Allan, editor, Proceedings of the First International Conference on Human Language Technology Research. Morga… find
  5. Banko, Michele and Eric Brill. 2001b. Scaling to very very large corpora for natural language disambiguation. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics and the 10th Conference of the European Chapter of the Association for Computational Linguistics, p…
  6. Bard, Ellen Gurman, Dan Robertson, and Antonella Sorace. 1996. Magnitude estimation of linguistic acceptability. Language, 72(1):32–68. find
  7. Briscoe, Ted and John Carroll. 1997. Automatic extraction of subcategorization from corpora. In Proceedings of the Fifth Conference on Applied Natural Language Processing, pages 356–363, Washington, DC.
  8. Burnard, Lou, editor, 1995. Users Reference Guide, British National Corpus. British National Corpus Consortium, Oxford University Computing Services, Oxford, England. find
  9. Carroll, Glenn and Mats Rooth. 1998. Valence induction with a head-lexicalized PCFG. In Nancy Ide and Atro Voutilainen, editors, Proceedings of the Third Conference on Empirical Methods in Natural Language Processing, pages 36–45, Granada, Spain. find
  10. Clark, Stephen and David Weir. 2001. Class-based probability estimation using a semantic hierarchy. In Proceedings of the Second Conference of the North American Chapter of the Association for Computational Linguistics, pages 95–102, Pittsburgh, PA. find
  11. Clark, Stephen and David Weir. 2002. Class-based probability estimation using a semantic hierarchy. Computational Linguistics, 28(2):187–206. find
  12. Corley, Martin and Christoph Scheepers. 2002. Syntactic priming in English sentence production: Categorical and latency evidence from an Internet-based study. Psychonomic Bulletin and Review, 9(1):126–131. find
  13. Corley, Steffan, Martin Corley, Frank Keller, Matthew W. Crocker, and Shari Trewin. 2001. Finding syntactic structure in unparsed corpora: The Gsearch corpus query system. Computers and the Humanities, 35(2):81–94. find
  14. Cowart, Wayne. 1997. Experimental Syntax: Applying Objective Methods to Sentence Judgments. Sage, Thousand Oaks, CA. find
  15. Curran, James. 2002. Scaling context space. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 231–238, Philadelphia. find
  16. Dagan, Ido, Lillian Lee, and Fernando Pereira. 1999. Similarity-based models of word cooccurrence probabilities. Machine Learning, 34(1):43–69. find
  17. Grefenstette, Gregory. 1994. Explorations in Automatic Thesaurus Discovery. Kluwer Academic, Boston. find
  18. Grefenstette, Gregory. 1998. The World Wide Web as a resource for example-based machine translation tasks. In Proceedings of the ASLIB Conference on Translating and the Computer, London. find
  19. Grefenstette, Gregory and Jean Nioche. 2000. Estimation of English and non-English language use on the WWW. In Proceedings of the RIAO Conference on Content-Based Multimedia Information Access, pages 237–246, Paris. find
  20. Grishman, Ralph and John Sterling. 1994. Generalizing automatically generated selectional patterns. In Proceedings of the 15th International Conference on Computational Linguistics, pages 742–747, Kyoto, Japan. find
  21. Hindle, Donald and Mats Rooth. 1993. Structural ambiguity and lexical relations. Computational Linguistics, 19(1):103–120.
  22. Jones, Rosie and Rayid Ghani. 2000. Automatically building a corpus for a minority language from the Web. In Proceedings of the Student Research Workshop at the 38th Annual Meeting of the Association for Computational Linguistics, pages 29–36, Hong Kong. find
  23. Karp, Daniel, Yves Schabes, Martin Zaidel, and Dania Egedi. 1992. A freely available wide coverage morphological analyzer for English. In Proceedings of the 14th International Conference on Computational Linguistics, pages 950–954, Nantes, France. find
  24. Katz, Slava M. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics Speech and Signal Processing, 33(3):400–401. find
  25. Keller, Frank and Theodora Alexopoulou. 2001. Phonology competes with syntax: Experimental evidence for the interaction of word order and accent placement in the realization of information structure. Cognition, 79(3):301–372. find
  26. Keller, Frank and Ash Asudeh. 2001. Constraints on linguistic coreference: Structural vs. pragmatic factors. In Johanna D. Moore and Keith Stenning, editors, Proceedings of the 23rd Annual Conference of the Cognitive Science Society, pages 483–488. Erlbaum, Mahwah, NJ. find
  27. Keller, Frank, Martin Corley, Steffan Corley, Lars Konieczny, and Amalia Todirascu. 1998. WebExp: A Java toolbox for Web-based psychological experiments. Technical Report HCRC/TR-99, Human Communication Research Centre, University of Edinburgh. find
  28. Keller, Frank, Maria Lapata, and Olga Ourioupina. 2002. Using the Web to overcome data sparseness. In Jan Hajič and Yuji Matsumoto, editors, Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 230–237, Philadelphia. find
  29. Lapata, Maria. 2001. A corpus-based account of regular polysemy: The case of context-sensitive adjectives. In Proceedings of the Second Conference of the North American Chapter of the Association for Computational Linguistics, pages 63–70, Pittsburgh, PA. find
  30. Lapata, Maria. 2002. The disambiguation of nominalizations. Computational Linguistics, 28(3):357–388. find
  31. Lapata, Maria and Frank Keller. 2003. Evaluating the performance of unsupervised Web-based models for a range of NLP tasks. Unpublished manuscript, University of Sheffield, Sheffield, England, and University of Edinburgh, Edinburgh, Scotland. find
  32. Lapata, Maria, Frank Keller, and Scott McDonald. 2001. Evaluating smoothing algorithms against plausibility judgments. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics and the 10th Conference of the European Chapter of the Association for Computational Lingu… find
  33. Lapata, Maria, Scott McDonald, and Frank Keller. 1999. Determinants of adjective-noun plausibility. In Proceedings of the Ninth Conference of the European Chapter of the Association for Computational Linguistics, pages 30–36, Bergen, Norway. find
  34. Lauer, Mark. 1995. Designing Statistical Language Learners: Experiments on Compound Nouns. Ph.D. thesis, Macquarie University, Sydney, Australia. find
  35. Lee, Lilian. 1999. Measures of distributional similarity. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 25–32, College Park, MD.
  36. Leech, Geoffrey, Roger Garside, and Michael Bryant. 1994. The tagging of the British national corpus. In Proceedings of the 15th International Conference on Computational Linguistics, pages 622–628, Kyoto, Japan. find
  37. Levi, Judith N. 1978. The Syntax and Semantics of Complex Nominals. Academic Press, New York. find
  38. Li, Hang and Naoki Abe. 1998. Generalizing case frames using a thesaurus and the MDL principle. Computational Linguistics, 24(2):217–244.
  39. Lin, Dekang. 1994. PRINCIPAR—An efficient broad-coverage, principle-based parser. In Proceedings of the 15th International Conference on Computational Linguistics, pages 482–488, Kyoto, Japan. find
  40. Lin, Dekang. 1998. Dependency-based evaluation of MINIPAR. In Proceedings of the LREC Workshop on the Evaluation of Parsing Systems, pages 48–56, Granada, Spain. find
  41. Lin, Dekang. 2001. LaTaT: Language and text analysis tools. In Proceedings of the First International Conference on Human Language Technology Research. Morgan Kaufmann, San Francisco. find
  42. Markert, Katja, Malvina Nissim, and Natalia N. Modjeska. 2003. Using the Web for nominal anaphora resolution. In Proceedings of the EACL Workshop on the Computational Treatment of Anaphora, Budapest, pages 39–46. find
  43. McCarthy, Diana. 2000. Using semantic preferences to identify verbal participation in role switching alternations. In Proceedings of the First Conference of the North American Chapter of the Association for Computational Linguistics, pages 256–263, Seattle, WA. find
  44. Mihalcea, Rada and Dan Moldovan. 1999. A method for word sense disambiguation of unrestricted text. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 152–158, College Park, MD. find
  45. Miller, George A., Richard Beckwith, Christiane Fellbaum, Derek Gross, and Katherine J. Miller. 1990. Introduction to WordNet: An on-line lexical database. International Journal of Lexicography, 3(4):235–244. find
  46. Pereira, Fernando, Naftali Tishby, and Lillian Lee. 1993. Distributional clustering of English words. In Proceedings of the 31st Annual Meeting of the Association for Computational Linguistics, pages 183–190, Columbus, OH.
  47. Pollard, Carl and Ivan A. Sag. 1994. Head-Driven Phrase Structure Grammar. University of Chicago Press, Chicago. find
  48. Prescher, Detlef, Stefan Riezler, and Mats Rooth. 2000. Using a probabilistic class-based lexicon for lexical ambiguity resolution. In Proceedings of the 18th International Conference on Computational Linguistics, pages 649–655, Saarbrücken, Germany. find
  49. Ratnaparkhi, Adwait. 1998. Unsupervised statistical models for prepositional phrase attachment. In Proceedings of the 17th International Conference on Computational Linguistics and 36th Annual Meeting of the Association for Computational Linguistics, pages 1079–1085, Montréal. find
  50. Resnik, Philip. 1993. Selection and Information: A Class-Based Approach to Lexical Relationships. Ph.D. thesis, University of Pennsylvania, Philadelphia. find
  51. Resnik, Philip. 1999. Mining the Web for bilingual text. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 527–534, College Park, MD.
  52. Resnik, Philip. 2000. Measuring verb similarity. In Lila R. Gleitman and Aravid K. Joshi, editors, Proceedings of the 22nd Annual Conference of the Cognitive Science Society, pages 399–404. Erlbaum, Mahwah, NJ. find
  53. Rooth, Mats, Stefan Riezler, Detlef Prescher, Glenn Carroll, and Franz Beil. 1999. Inducing a semantically annotated lexicon via EM-based clustering. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 104–111, College Park, MD.
  54. Sampson, Geoffrey. 1995. English for the Computer: The SUSANNE Corpus and Analytic Scheme. Oxford University Press, Oxford. find
  55. Schütze, Carson T. 1996. The Empirical Base of Linguistics: Grammaticality Judgments and Linguistic Methodology. University of Chicago Press, Chicago. find
  56. Stevens, S. S. 1975. Psychophysics: Introduction to its Perceptual, Neural, and Social Prospects. John Wiley, New York. find
  57. Volk, Martin. 2001. Exploiting the WWW as a corpus to resolve PP attachment ambiguities. In Paul Rayson, Andrew Wilson, Tony McEnery, Andrew Hardie, and Shereen Khoja, editors, Proceedings of the Corpus Linguistics Conference, pages 601–606, Lancaster, England. find
  58. Weiss, Sholom M. and Casimir A. Kulikowski. 1991. Computer Systems That Learn: Classification and Prediction Methods from Statistics, Neural Nets, Machine Learning, and Expert Systems. Morgan Kaufmann, San Mateo, CA. find
  59. Zhu, Xiaojin and Ronald Rosenfeld. 2001. Improving trigram language modeling with the World Wide Web. In Proceedings of the International Conference on Acoustics Speech and Signal Processing, Salt Lake City, UT. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/keller-2003-web-obtain-frequencies-unseen-bigrams/