Elamur Research⌕
☰
home / papers / 1987 / Pattern Recognition Applied to the Acquisition of a Grammatical Classification System From Unrestricted English Text

Pattern Recognition Applied to the Acquisition of a Grammatical Classification System From Unrestricted English Text

Paper · 1987 · Eric Steven Atwell and Nicos Frixou Drakos

Summary

Atwell and Drakos ask whether a part-of-speech classification can be discovered automatically from raw, untagged English text using statistical pattern recognition. At the time, such methods were common in speech processing but rarely used in text NLP. The starting point is CLAWS, a first-order Markov tagger. CLAWS showed that a simple Markov model is good enough for practical tagging of the LOB Corpus. Learning its parameters from text that is already tagged is just learning from examples, so the authors try to get word classes from unannotated text instead.

They tried several approaches:

  • clustering words by their immediate neighbouring words;
  • Baker's Markov re-estimation scheme, with and without simplifications such as grouping words by suffix and seeding the model with real tag-pair statistics;
  • Wolff's context-based class induction, with a cap on how many classes a word may belong to.

Every variant produced some plausible groupings, but all were held back by enormous computing cost. The authors admit the case is not yet proven. The paper matters as an early argument for corpus-driven, unsupervised grammar induction and for using induced classes to test existing tagsets.

Key points

  • CLAWS model components:
    • 133 tags;
    • a 133×133 tag-pair frequency matrix;
    • a wordlist plus suffix and prefix fallbacks;
    • likelihood-in-context formulae.
  • CLAWS accuracy: about 96% of words in the LOB Corpus tagged correctly. The LOB Corpus has 500 British English samples of roughly 2,000 words each, over a million words in total.
  • Context clustering: run on a 200,000-word sample, it took weeks. Only 175 words were frequent enough to cluster, yet it still found groups such as modal verbs (will, should, could, must, may, might) and prepositions (in, for, on, by, at, during).
  • Baker's scheme: on raw LOB text (about 50,000 word-forms), it would need roughly 18,000 transition values and 900,000,000 emission values. Grouping by the last five letters still meant about 66 hours for the first iteration on a single LOB text.
  • Wolff's second algorithm: it needed 5 CPU hours on an 11,000-word sample to produce classes such as <had was> and <a an it one the>. By comparison, Berwick's example-driven learner recovered about 70% of Marcus's hand-written parsing rules.

Interesting ideas

  • I1. Context clustering with a falling threshold: words are grouped by their left and right neighbours, merging only very similar context sets first and loosening the threshold step by step, so the most confident classes form first.
  • I2. Generic ambiguity cap: a general limit of about five classes per word is used instead of stating each word's ambiguity by hand, which keeps induction tractable without encoding English-specific knowledge.
  • I3. Seeding from real statistics: the model is started from tag-pair probabilities taken from a tagged corpus rather than random values, to speed up convergence of unsupervised re-estimation.
  • I4. Induced classes as a tagset test: if an induced class matches no existing tag's distribution, that is objective evidence the tagset could be improved, and rival schemes such as LDOCE and OALD codes can be compared the same way.
  • I5. Suffix as a class signal: the last few letters of a word are assumed to predict its class, which shrinks the parameter space; CLAWS needed only about 7,000 listed words plus a suffix list.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): A staged clustering pass could suggest candidate tags for unknown or rare words, with the merge stage recorded as an explanation.
  • I2 (tokens): The lexicon could enforce a small global cap on candidate tags per form, so the tagger's search stays bounded and principled.
  • I3 (learn): Self-training and repair-rule induction could start from treebank-derived statistics, making each iteration cheaper and easier to audit.
  • I4 (tree): When converting between UD dialects or treebanks, distributional mismatches between labels could flag categories whose definitions need revision.
  • I5 (tokens): A suffix-keyed fallback would give transparent guesses for out-of-vocabulary words in the en crate's tagger and lemmatizer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of LeedsLeeds · United Kingdom40 works · 1,822 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Eric Steven Atwell and Nicos Frixou Drakos. 1987. Pattern Recognition Applied to the Acquisition of a Grammatical Classification System From Unrestricted English Text. Third Conference of the European Chapter of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{atwell1987pattern,
  title = {{Pattern Recognition Applied to the Acquisition of a Grammatical Classification System From Unrestricted English Text}},
  author = {Eric Steven Atwell and Nicos Frixou Drakos},
  booktitle = {Third Conference of the European Chapter of the Association for Computational Linguistics},
  year = {1987},
  doi = {10.3115/976858.976868},
  url = {https://doi.org/10.3115/976858.976868}
}

References

7 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akkennan, Erik, Pieter Masereeuw, and Willem Meijs 1985 Designing a computerized lexicon for linguistic purposes Rudopi, Amsterdam Alshawi, Hiyan, Branimir Boguraev, and Ted Briscoe 1985, "Towards a lexicon support environment for real time parsing" in Proceedings of the Second Conference of the European Chapter of the Association for Computational Linguistics, Geneva Atwell, Eric Steven 1981 LOB Corpus Tagging Project: Manual Pre-edit Handbook. Departments of Computer Studies and Linguistics, University of Lancaster Atwell, Eric Steven 1982 LOB Corpus Tagging Project: Manual Postedit Handbook (A mini-grammar of LOB Corpus English, examining the types of error commonly made during automatic (computational) analysis of ordinary written English.) Departments of Computer Studies and Linguistics, University of Lancaster Atwell, Eric Steven 1983 "Constituent-Likelihood Grammar' in Newsletter of the International Computer Archive of Modern English (ICAME NEWS) 7: 34-67, Norwegian Computing Centre for the Humanities, Bergen University Atwell, Eric Steven 1986a Extracting a Natural Language grammar from raw text Department of Computer Studies Research Report no.208, University of Leeds Atwell, Eric Steven 1986b, "A parsing expert system which learns from corpus analysis" in Willem Meijs (ed) Corpus Linguistics and Beyond." Proceedings of the Seventh International Conference on English Language Research on Computerised Corpora, Amsterdam, Netherlands Rodopi, Amsterdam Atwell, Eric Steven 1986c, "How to detect grammatical errors in a text without parsing it" Department of Computer Studies Research Report no.212, University of Leeds; to appear in Proceedings of the Association for Computational Linguistics Third European Chapter Conference, Copenhagen, Denmark (elsewhere in this book). find
  2. Atwell, Eric Steven 1986d "Beyond the micro: advanced software for research and teaching from computer science and artificial intelligence" in Leech, Geoffrey and Candlin, Christopher (eds.) Computers in English language teaching and research: selected papers from the British Council Symposium on computers in English language education and research, Lancaster, England 167-183, Longman Atwell, Eric Steven (forthcoming a) "A lexical database for English leamera and users: the Oxford Advanced Learner's Dictionary" to appear in Proceedings of ICDBHSS87, the 1987 International Conference on DataBases in the Humanities and Social Sciences, Montgomery, Alabama, USA Atwell, Eric Steven (forthcoming b) "Transforming a Parsed Corpus into a Corpus Parser", to appear in Proceedings of the 1987 ICAME 8th International Conference on English Language Research on Computerised Corpora, Heisinki, Finland Atwell, Eric Steven (forthcoming c) "An Expert System for the Automatic Discovery of Particles" to appear in Proceedings of the 1987 International Conference on the Study of Particles, Berlin, East Germany Atwell, Eric Steven, Geoffrey Leech and Roger Garside 1984, "Analysis of the LOB Corpus: progress and prospects", in Jan Aarts and Willem Meijs (ed), Corpus Linguistics; Proceedings of the [CAME Conference on the use of computer corpora in English Language Research, Nijmegen, Netherlands Rodopi. find
  3. Baker, J K 1975 "Stochastic modeling for automatic speech understanding" in D R Reddy (ed) Speech recognition Academic Press Baker, J K 1979 '"I'rainable grammars for speech recognition" in Klatt, D H and Wolf J J (eds.) Speech communication papers for the 97th meeting of the acoustical society of America: 547-550 find
  4. Berwick, R 1985 The acquisition of syntactic knowledge MIT Press, Cambridge (MA) and London Chomsky, Noam 1957 Syntactic Structures Mouton, The Hague Drakos, Nicos Frixou 1986 Electrical circuit analysis using algebraic m nip,,' 'ion and belief revision Department of Computer Studies, Leeds University Leech, Geoffrey, Roger Garside, and Eric Steven Atwell 1983a, "Recent developments in the use of computer corpora in English language research" in Transactiona of the Philological Society 1983: 23-40. find
  5. Leech, Geoffrey, Garside, Roger and Atwell, Eric Steven 1983b "The Automatic Grammatical Tagging of the LOB Corpus" in Newsletter of the International Computer Archive of Modern English ([CAME NEWS) 7: 13-33, Norwegian Computing Centre for the Humanities, Bergen University Gazdar, Gerald, Ewan Klein, Geoffrey Pullum, and Ivan Sag 1985 Generalized Phrase Structure Grammar Black'well, Oxford Homby, A S, with Cowie, A P (eds.) 1974 Oxford Advanced Learner's Dictionary of Current English (third edition) Oxford University Press Johausson, Stig, Geoffrey Leech and Helen Goodluck 1978 Manual of information to accompany the Lancaster-OslolBergen Corpus of British English, for use with digital computers Department of English, Oslo University Johansson, Stig, Eric Atwell, Roger Garside, and Geoffrey Leech 1986 The Tagged LOB Corpus Norwcgian Computing Centre for the Humanities, University of Bergen, Norway. find
  6. Marcus, M P 1980 A Theory of Syntactic Reco n.ion Natural Language MIT Press, Cambridge, MA Procter, Paul (editor-in-chief) 1978 Longman Dictionary of Contemporary English Longman Sinclair, J, Jones, S, and Daley, R 1970 English lexieal studies, Report to OSTI on project C/LP/08; Dept of English, Birmingham University Wolff, J G 1976 "Frequency, Conceptual Structure and Pattern Recognition" in British Journal of Psychology 67:377-390 find
  7. Wolff, J G 1978 "The Discovery of Syntagmatic and Paradigmatic Classes" in Bulletin 6(1):141 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/atwell-1987-pattern-recognition-applied-acquisition-grammatical/