Elamur Research⌕
☰
home / papers / 1993 / Introduction to the Special Issue on Computational Linguistics Using Large Corpora

Introduction to the Special Issue on Computational Linguistics Using Large Corpora

Paper · 1993 · Kenneth W. Church and Robert L. Mercer

Summary

This editorial opens a Computational Linguistics special issue. It argues that statistical, corpus-driven analysis of language is returning after a long period dominated by rule-based and knowledge-based work. Church and Mercer link the earlier decline of empiricism to critiques such as Chomsky's objections to n-grams and Minsky and Papert's attack on perceptrons. They trace the revival mainly to speech recognition. In the 1970s and 1980s that field dropped hand-crafted expert systems in favour of stochastic techniques: Shannon's noisy channel model, n-grams and HMMs. Shared datasets such as TIMIT and the DARPA Resource Management database helped drive the change. The authors then show how these tools carry over to text: probabilistic tagging, preference-based parsing using collocations, entropy-based evaluation, noisy-channel translation and bilingual lexicography, and association statistics for lexicographers. The paper proposes no new system. Its contribution is a historical and methodological frame for the issue's articles. It names three drivers of the empirical turn: cheaper computing, abundant machine-readable text, and stronger demand for measurable results.

Key points

  • Corpus scale changed dramatically. A decade earlier, the one-million-word Brown Corpus counted as large. By 1993, many sites held text collections of hundreds of millions to billions of words, distributed through ACL/DCI, ECI, ICAME, BNC, LDC, CLR, EDR, and standards like TEI.
  • Probabilistic part-of-speech taggers based on the noisy channel model label about 95% of words in unseen text correctly. They clearly beat earlier methods that ignored lexical probabilities.
  • Every speech recognition paper at IEEE ICASSP 1988 used the DARPA RM database, according to LDC. Abstracts were due only months after the database was released.
  • IBM's Raleigh system later became a foundation for data-driven speech work. Its front end processed 20,000 samples/s into 80 filter-bank outputs, producing 100 phoneme-like labels per second.
  • A t-score analysis of Associated Press news contrasts words collocating with strong versus powerful. It led lexicographer Hanks to suggest that strong describes an intrinsic quality and powerful an extrinsic one. The issue includes four papers on preference-based parsing and five on computational lexicography.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬IBM Research - Thomas J. Watson Research CenterYorktown Heights · United States41 works · 46,197 citations🏢AT&TBedminster · United States29 works · 15,393 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Literature and Literary Theory · 340 works

Cite

Cite as Kenneth W. Church and Robert L. Mercer. 1993. Introduction to the Special Issue on Computational Linguistics Using Large Corpora. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@article{church1993introduction,
  title = {{Introduction to the Special Issue on Computational Linguistics Using Large Corpora}},
  author = {Kenneth W. Church and Robert L. Mercer},
  journal = {Computational Linguistics},
  year = {1993},
  doi = {10.5555/972450.972452},
  url = {https://doi.org/10.5555/972450.972452}
}

References

80 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ayuso, D.; Bobrow, R.; MacLaughlin, D.; Meteer, M.; Ramshaw, L.; Schwartz, R.; and Weischedel, R. (1990). "Toward understanding text with a very large vocabulary." DARPA Speech and Natural Language (,Vorkshop. San Mateo, CA, 354-358. Morgan Kaufmann. find
  2. Bahl, L.; Baker, J.; Cohen, P.; Dixon, N.; Jelinek, E; Mercer, R.; and Silverman, H. (1975). "Preliminary results in the performance of a system for the automatic recognition of continuous speech." IBM Technical Report \#RC 5654. find
  3. Bahl, L.; Baker, J.; Cohen, P.; Jelinek, E; Lewis, B.; and Mercer, R. (1978). "Recognition of a continuously read natural corpus." In Proceedings, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). find
  4. Bahl, L., and Mercer, R. (1976). "Part of speech assignment by a statistical decision algorithm." In Abstracts of Papers from the International Symposium on Information Theory. find
  5. Baker, J. (1979). "Trainable grammars for speech recognition." In Speech Communication Papers for the 97th Meeting of the Acoustical Society of America, edited by Klatt and Wolf, 547-550. find
  6. Bakis, R. (1976). "Continuous speech recognition via centisecond acoustic states." In Proceedings of 91st Meeting of the Acoustic Society of America. find
  7. Baum, L. (1972). "An inequality and associated maximization technique in statistical estimation of probabilistic functions of a Markov process." Inequalities, 3, 1-8. find
  8. Bell, T.; Cleary, J.; and Witten, I. (1990). Text Compression. Prentice Hall. find
  9. Biber, D. (1993). "Representativeness in corpus design." Computational Linguistics, 19(2). In press. find
  10. Boggess, L.; Agarwal, R.; and Davis, R. (1991). "Disambiguation of prepositional phrases in automatically labelled technical text." AAAI, 155-159. find
  11. Boguraev, B. (1990). "Looking Up: an account of the COBUILD project in lexical computing." Computational Linguistics, 16(3), 184-185. find
  12. Brent, M. (1993). "Robust acquisition of subcategorization features from arbitrary text: syntactic knowledge meets unsupervised learning." Computational Linguistics, 19(2). In press. find
  13. Brown, P.; Cocke, J.; Della Pietra, S.; Della Pietra, V.; Jelinek, F.; Lafferty, J.; Mercer, R.; and Rossin, P. (1990). "A statistical approach to machine translation." Computational Linguistics, 16(2), 79-85. find
  14. Brown, P.; Della Pietra, S.; Della Pietra, V.; Lai, J.; and Mercer, R. (1992). "An estimate of an upper bound for the entropy of English." Computational Linguistics, 18(1), 31–40. our card
  15. Brown, P.; Della Pietra, S.; Della Pietra, V.; Mercer, R. (1993). "The mathematics of machine translation." Computational Linguistics, 19(2). In press. find
  16. Chomsky, N. (1957). Syntactic Structures. Mouton. find
  17. Church, K. (1988). "A stochastic parts program and noun phrase parser for unrestricted text." In Proceedings, Second Conference on Applied Natural Language Processing. (ACL), Austin, TX, 136-143. our card
  18. Church, K.; Hanks, P.; Hindle, D.; and Gale, W. (1991). "Using statistics in lexical analysis." In Lexical Acquisition: Using On-Line Resources to Build a Lexicon, edited by Zernik. Lawrence Erlbaum, 115-164. find
  19. Church, K., and Gale, W. (1991). "Concordances for parallel text." In Proceedings, Seventh Annual Conference of the UW Centre for the New Oxford English Dictionary and Text Research. find
  20. Cohen, P., and Mercer, R. (1974). "The phonological component of an automatic speech-recognition system." In Speech Recognition, Invited Papers Presented at the 1974 IEEE Symposium, edited by R. Reddy, 275-320. find
  21. Cover, T., and Thomas, J. (1991). Elements of Information Theory. John Wiley & Sons. find
  22. Croft, W.; Turtle, H.; and Lewis, D. (1991). "The use of phrases and structured queries in information retrieval." In SIGIR Forum, edited by A. Bookstein, Y. Chiaramella, G. Salton, and V. Raghavan, 32-45. find
  23. Damerau, F. (1964). "A technique for computer detection and correction of spelling errors." Communications of the ACM, 7(3), 171-176. deMarcken, C. (1990). "Parsing the LOB corpus." Association for Computational Linguistics, 243-251. find
  24. DeRose, S. (1988). "Grammatical category disambiguation by statistical optimization." Computational Linguistics, 14(1), 31-39. our card
  25. Deroualt, A., and Merialdo, B. (1986). "Natural language modeling for phoneme-to-text transcription." IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-8(6), 742-749. find
  26. Fano, R. (1961). Transmission of Information. MIT. Press. find
  27. Firth, J. (1957). "A synopsis of linguistic theory 1930-1955." In Studies in Linguistic Analysis, Philological Society, Oxford; reprinted in Selected Papers of J. R. Firth, edited by E Palmer. find
  28. Ford, M.; Bresnan, J.; and Kaplan, R. (1982). "A competence based theory of syntactic closure." In The Mental Representation of Grammatical Relations, edited by J. Bresnan. MIT Press, 727-796. find
  29. Frakes, W., and Baeza-Yates, R., eds. (1992). Information Retrieval: Data Structures and Algorithms. Prentice Hall. find
  30. Francis, W., and Ku~era, H. (1982). Frequency Analysis of English Usage, Houghton Mifflin. find
  31. Furui, S. (1989). Digital Speech Processing, Synthesis, and Recognition. Marcel Dekker. find
  32. Garside, R.; Leech, G.; and Sampson, G. (1987). The Computational Analysis of English. Longman. find
  33. Good, I. (1953). "The population frequencies of species and the estimation of population parameters." Biometrika, 40, 237-264. find
  34. Halliday, M. (1966). "Lexis as a linguistic level," In In Memory of J. R. Firth, edited by C. Bazell, J. Catford, M. Halliday, and R. Robins. Longman. find
  35. Hanks, P. (1990). "Evidence a'nd intuition in lexicography." In Meaning and Lexicography, edited by J. Tomaszczyk and B. Lewandowska-Tomaszczyk, 31-41. John Benjamins Publishing Company. find
  36. Hearst, M. (1991). "Toward noun homonym disambiguation using local context in large text corpora." In Proceedings, Seventh Annual Conference of the UW Centre for the New OED and Text Research. University of Waterloo, Waterloo, Ontario, 1-22. find
  37. Henisz-Dostert, B.; Ross Macdonald, R.; and Zarechnak, M., eds. (1979). Machine Translation. Mouton. find
  38. Hindle, D. (1989). "Acquiring disambiguation rules from text." ACL, 118-125. find
  39. Isabelle, P. (1992). "Bi-textual aids for translators." In Proceedings, Eighth Annual Conference of the UW Centre for the New OED and Text Research. University of Waterloo, Waterloo, Ontario, 76-89. find
  40. Jelinek, E (1985). "Self-organized language modeling for speech recognition." IBM Report. Reprinted in W & L, 450-506. find
  41. Jelinek, E, and Mercer, R. (1980). "Interpolated estimation of Markov source parameters from sparse data." In Proceedings, Workshop on Pattern Recognition in Practice. North-Holland. find
  42. Jelinek, F., and Mercer, R. (1985). "Probability distribution estimation from sparse data." IBM Technical Disclosure Bulletin, 28, 2591-2594. find
  43. Jelinek, E; Mercer, R.; and Roukos, S. (1991). "Principles of lexical language modeling for speech recognition." In Advances in Speech Signal Processing, edited by S. Furui and M. Mohan. Marcel Dekker, 651-700. find
  44. Karlsson, E (1990). "Constraint grammar as a framework for parsing running text." In Proceedings, 15th International Conference on Computational Linguistics (COLING-90), 168-173. our card
  45. Klavans, J., and Tzoukermann, E. (1990a). "The BICORD system." In Proceedings, 15th International Conference on Computational Linguistics (COLING-90), Helsinki, Finland, 174-179. find
  46. Klavans, J., and Tzoukermann, E. (1990b). "Linking bilingual corpora and machine readable dictionaries with the BICORD system." In Proceedings, Sixth Annual Conference of the UW Centre for the New Oxford English Dictionary and Text Research, 19-30. find
  47. Klatt, D. (1977). "Review of the ARPA speech understanding project." Journal of the Acoustical Society of America. Reprinted in W & L, 554-575. find
  48. Klatt, D. (1980). "Scriber and lafs: Two new approaches to speech analysis." In Trends in Speech Recognition, edited by W. Lea. Prentice-Hall. find
  49. Kukich, K. (1992). "Techniques for automatically correcting words in text." ACM Computing Surveys, 24(4), 377-439. find
  50. Kupiec, J. (1989). "Augmenting a hidden Markov model for phrase-dependent word tagging." DARPA Speech and Natural Language Workshop, San Mateo, CA, 92-98. Morgan Kaufmann. find
  51. Kupiec, J. (1992). "Robust part-of-speech tagging using a hidden Markov model." Computer Speech and Language, 6, 225-242. find
  52. Lari, K., and Young, S. (1991). "Applications of stochastic context-free grammars using the inside-outside algorithm." Computer Speech and Language, 237-258. find
  53. Leech, G.; Garside, R.; and Atwell, E. (1983). "The automatic grammatical tagging of the LOB corpus." ICAME News 7, 13-33. find
  54. Leonard, R. (1984). "A database for speaker-independent digit recognition." Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 3, 1-4. find
  55. Liberman, M., and Church, K. (1991). "Text analysis and word pronunciation in text-to-speech synthesis." In Advances in Speech Signal Processing, edited by S. Furui and M. Mohan. Marcel Dekker, 791-832. find
  56. Merialdo, B. (1991). "Tagging text with a probabilistic model." IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 809-812. find
  57. Meyer, D.; Schvaneveldt, R.; and Ruddy, M. (1975). "Loci of contextual effects on visual word-recognition." In Attention and Performance V, edited by P. Rabbitt and S. Dornie. Academic Press, 98-116. find
  58. Minsky, M., and Papert, S. (1969). Perceptrons; An Introduction to Computational Geometry. MIT Press. find
  59. Mish, F., ed. (1983). Webster's Ninth New Collegiate Dictionary. Merriam, Webster. find
  60. Mosteller, E, and Tukey, J. (1977). Data Analysis and Regression. Addison-Wesley. find
  61. Mosteller, Fredrick, and Wallace, David (1964). Inference and Disputed Authorship: The Federalist. Addison-Wesley. find
  62. Murray, K. (1977). Caught in the Web of Words: James Murray and the Oxford English Dictionary. Yale University Press. find
  63. Palermo, D., and Jenkins, J. (1964). Word Association Norms. University of Minnesota Press. find
  64. Price, P.; Fisher, W.; Bernstein, J.; and Pallett, D. (1988). "The DARPA 1000-word resource management database for continuous speech recognition." In Proceedings, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1, 651-654. find
  65. Pustejovsky, J.; Berger, S.; and Anick, P. (1993). "Lexical semantic techniques for corpus analysis." Computational Linguistics, 19(2). In press. find
  66. Rabiner, L. (1989). "A tutorial on hidden Markov models and selected applications in speech recognition." In Proceedings, IEEE, 77(2), 257-286. Reprinted in W & L, 267-296. find
  67. Salton, G. (1989). Automatic Text Processing. Addison-Wesley. find
  68. Salton, G.; Zhao, Z.; and Buckley, C. (1990). "A simple syntactic approach for the generation of indexing phrases." Technical Report 90-1137, Department of Computer Science, Cornell University. find
  69. Sato, S., and Nagao, M. (1990). "Towards memory based translation." In Proceedings, 15th International Conference on Computational Linguistics (COLING-90), 247-252. find
  70. Shannon, C. (1948). "The mathematical theory of communication." Bell System Technical Journal, 27, 398-403. find
  71. Shannon, C. (1951). "Prediction and entropy of printed English." Bell Systems Technical Journal, 30, 50-64. find
  72. Sinclair, J.; Hanks, P.; Fox, G.; Moon, R.; and Stock, P., eds. (1987). Collins COBUILD English Language Dictionary. Collins. find
  73. Sinclair, J., ed. (1987). Looking Up: An Account of the COBUILD Project in Lexical Computing. Collins. find
  74. Tukey, J. (1977). Exploratory Data Analysis. Addison-Wesley. find
  75. Voutilainen, A.; Heikkila, J.; and Anttila, A. (1992). "Constraint grammar of English: A performance-oriented introduction." Publication No. 21, University of Helsinki, Department of Linguistics, Helsinki, Finland. find
  76. Waibel, A., and Lee, K., eds. (1990). Readings in Speech Recognition. Morgan Kaufmann. find
  77. Warwick-Armstrong, S., and Russell, G. (1990). "Bilingual concordancing and bilingual lexicography." Euralex 1990. find
  78. Weaver, W. (1949). "Translation." Reproduced in Machine Translation of Languages, edited in 1955 by W. Locke and A. Booth. MIT Press, 15-23. find
  79. Welch, T. (1984). "A technique for high performance data compression." Computer, 17(6), 8-19. find
  80. Woods, W. (1970). "Transition networks for natural language analysis." CACM, 13(10), 591-606. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/church-1993-introduction-special-issue-computational-linguistics/