Elamur Research⌕
☰
home / papers / 2002 / Robust Accurate Statistical Annotation of General Text

Robust Accurate Statistical Annotation of General Text

Summary

The paper presents RADISP, a statistical parser for English released in the ANLT toolkit. It aims to be accurate without being tied to one domain, unlike lexicalised parsers optimised on the Wall Street Journal treebank. It also aims to be more accurate and complete than cascaded finite-state partial parsers. The pipeline runs in stages:

  • a Flex-based tokeniser,
  • an HMM tagger that passes on several plausible tags rather than just one,
  • a finite-state lemmatiser,
  • a generalised LR parser driven by a hand-written grammar over PoS and punctuation tags,
  • a probabilistic disambiguator that picks the n-best parses from the forest.

Results can be output as syntactic trees, weighted grammatical relations (GRs) or a robust, underspecified MRS semantics. On a 500-sentence Susanne test suite scored by GRs, the parser reaches an F-score of 76.5%, which the authors consider competitive given the harder, genre-mixed test data. The system has annotated large corpora, including the BNC. These parses have supported lexical acquisition, word sense disambiguation, anaphora resolution and question answering. This makes it a practical, freely available tool for new domains.

Key points

  • The tagger is a bigram HMM using the 155 CLAWS-2 tags, trained on Susanne, LOB and part of the BNC. It reaches about 97% per-word accuracy on similar text, but its unknown-word model is only about 80% accurate.
  • The morphological analyser has about 1400 Flex rules, an error rate below 0.07%, and processes over 200K words per second.
  • The grammar has about 400 unification-based rules over tag sequences, with lexical probabilities for 23 subcategorisation frames. It finds a sentence-rooted parse for about 80% of Susanne but only 67% of the BNC; otherwise it returns a sequence of partial parses.
  • On GR evaluation it scores an F-score of 76.5% on 500 Susanne sentences, and 45% recall at 90% precision.
  • It has parsed over 98% of the 90-million-word written BNC, plus LOB, the Spoken English Corpus, Susanne and the TREC8 QA documents. It is free for non-commercial use.

Interesting ideas

  • I1. Delay hard decisions in pipelines. Each stage hands on a thresholded set of alternatives, for example about 1.3 tags per word chosen by Forward-Backward posteriors, instead of a single best guess. This prevents early errors from becoming permanent and could be applied to any staged NLP system.
  • I2. Parse over tags rather than words. A grammar written over PoS and punctuation sequences, with lexical information added only where it is available, keeps the parser largely independent of domain. This suits languages or domains that lack large treebanks.
  • I3. Evaluate with grammatical relations. Scoring the head-to-head relations a parser recovers, rather than tree bracket similarity, compares systems more meaningfully. Outputting weighted relations also lets downstream users choose their own balance of precision and recall.
  • I4. Leave semantics underspecified. Splitting predicates into separate argument relations, including an unspecified "argN", lets semantics be built incrementally even when a predicate's number of arguments is unknown. A shallow parser can therefore still produce output that deeper analyses can match against.
  • I5. Bootstrap lexical knowledge from the parser's own output. Information such as subcategorisation frames is learned from parses made without it and then fed back into the system. This adds lexical knowledge without hand-annotating each new domain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Word sense disambiguation · Treebanks

Cited by

104 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 95 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{briscoe2002robust,
  title = {{Robust Accurate Statistical Annotation of General Text}},
  author = {Briscoe, Ted and Carroll, John},
  booktitle = {Proceedings of the Third International Conference on Language Resources and Evaluation (LREC'02)},
  year = {2002},
  doi = {10.63317/2aswu2tg29np},
  url = {https://doi.org/10.63317/2aswu2tg29np}
}

References

37 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abney, S. (1996) ‘Part-of-speech tagging and partial parsing’ in Church, K. et al. (ed.), Corpus-based Methods in Language and Speech, Kluwer, Dordrecht. find
  2. Appelt, D., J. Hobbs, J. Bear, D. Israel, M. Kameyama, A. Kehler, D. Martin, K. Myers and M. Tyson (1995) ‘The SRI FASTUS system, MUC-6 test results and analysis’, Proceedings of the 6th Message Understanding Conference, Morgan Kaufmann, San Mateo, CA, pp. 237–248. find
  3. Briscoe E. and J. Carroll (1993) ‘Generalised probabilistic LR parsing of natural language (corpora) with unification-based grammars’, Computational Linguistics, vol.19.1, 25–60. find
  4. Briscoe, E. and J. Carroll (1995) ‘Developing and Evaluating a Probabilistic LR Parser of Part-of-Speech and Punctuation Labels’, Proceedings of the 4th International Workshop on Parsing Technologies (IWPT’95), Prague / Karlovy Vary, Czech Republic, pp. 48–58. find
  5. Briscoe, E., J. Carroll, J. Graham and A. Copestake (2002a) ‘Relational evaluation schemes’, Proceedings of the Workshop at LREC’02 on Beyond PARSEVAL: Towards Improved Evaluation Measures for Parsing Systems, Gran Canaria. find
  6. Briscoe, E., A. Copestake and S. Teufel (2002b) CSTIT MPhil, Language Practical 2, Computer Laboratory, University of Cambridge. find
  7. Carroll, J. (1994) ‘Relating complexity to practical performance in parsing with wide-coverage unification grammars’, Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics, Las Cruces, NM, pp. 287-294. find
  8. Carroll, J. and E. Briscoe (2001) ‘High precision extraction of grammatical relations’, Proceedings of the 7th ACL/SIGPARSE International Workshop on Parsing Technologies (IWPT’01), Beijing, China. find
  9. Carroll, J., E. Briscoe and A. Sanfilippo (1998) ‘Parser evaluation: a survey and a new proposal’, Proceedings of the International Conference on Language Resources and Evaluation, Granada, pp. 447–454. find
  10. Carroll, J. and D. McCarthy (2000) ‘Word sense disambiguation using automatically acquired verbal preferences’, Computers and the Humanities, vol.31.1–2, 109– 111. find
  11. Carroll, J., G. Minnen, and E. Briscoe (1998) ‘Can subcategorisation probabilities help a statistical parser?’, Proceedings of the ACL SIGDAT 6th Workshop on Very Large Corpora (WVLC’98), Montreal, pp. 118–126. find
  12. Charniak, E. (2000) ‘A maximum entropy inspired parser’, Proceedings of the 1st Conference of the North American Chapter of the Association for Computational Linguistics, Seattle, WA, pp. 132–139. find our card
  13. Clark, S. and D. Weir (2001) ‘Class-based probability estimation using a semantic hierarchy’, Proceedings of the 2nd Conference of the North American Chapter of the Association of Computational Linguistics, Carnegie Mellon University, pp. 95–102. find
  14. Collins, M. (1999) Head-driven statistical models for natural language parsing, PhD Dissertation, Computer and Information Science, University of Pennsylvania. find
  15. Collins, M. (2000) ‘Discriminative reranking for natural language parsing’, Proceedings of the 17th International Conference on Machine Learning (ICML2000), Morgan Kaufmann, San Mateo, CA. find our card
  16. Copestake, A., D. Flickinger, I. Sag and C. Pollard (1999) Minimal Recursion Semantics: An introduction, http://www-csli.stanford.edu/˜aac/papers/newmrs.pdf . ↗
  17. Copestake, A., A. Lascarides and D. Flickinger (2001) ‘An algebra for semantic construction in constraint-based grammars’, Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics, Toulouse, France, pp. 132–139. de Marcken, C. (1990) ‘Parsing the LOB corpus’, Proceedin… find our card
  18. Elworthy, D. (1994) ‘Does Baum-Welch re-estimation help taggers?’, Proceedings of the 4th ACL Conference on Applied NLP, Stuttgart, Germany, pp. 53–58. find
  19. Grover, C., J. Carroll and E. Briscoe (1993) The Alvey natural language tools grammar (4th release), Computer Laboratory, Cambridge University, UK, Technical Report 284. find
  20. Grover, C. and A. Lascarides (2001) ‘XML-based data preparation for robust deep parsing’, Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics, Toulouse, France, pp. 252–259. find
  21. Grover, C., C. Matheson, A. Mikheev and M. Moens (2000) ‘LT TTT – A flexible tokenisation tool’, Proceedings of the 2nd International Conference on Language Resources and Evaluation, Athens, Greece. find
  22. Hektoen, E. (1997) ‘Probabilistic parse selection based on semantic cooccurrences’, Proceedings of the 5th International Workshop on Parsing Technologies (IWPT’97), MIT, pp. 113–122. find
  23. Inui, K., V. Sornlertlamvanich, H. Tanaka and T. Tokunaga (1997) ‘A new formalization of probabilistic GLR parsing’, Proceedings of the 5th International Workshop on Parsing Technologies (IWPT’97), MIT, pp. 123–134. find
  24. Kiefer, B., H-U. Krieger, J. Carroll and R. Malouf (1999) ‘A bag of useful techniques for efficient and robust parsing’, Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, University of Maryland, pp. 473–480. find
  25. Korhonen, A. (2002) Subcategorisation acquisition, PhD Dissertation, Computer Laboratory, University of Cambridge. find
  26. Lambeau, F. (2001) Verb sense disambiguation from argument relations, MPhil dissertation, Dept. of Engineering, University of Cambridge.. find
  27. Levine, J., T. Mason and D. Brown (1992) Lex & Yacc, 2nd Edition, O’Reilly, Sebastopol, CA. find
  28. Lin, D. (1998) ‘Dependency-based evaluation of MINI-PAR’, Proceedings of the Workshop at LREC’98 on The Evaluation of Parsing Systems, Granada, Spain. find
  29. McCarthy, D. and A. Korhonen (1998) ‘Detecting verbal participation in diathesis alternations’, Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics, vol. 2, Montreal, Canada, pp. 1493– 1495. find
  30.   Minnen, G., J. Carroll and D. Pearce (2001) ‘Applied morphological processing of English’, Natural Language Engineering, vol.7.3, 225–250. find
  31. Oepen, S. and J. Carroll (2000) ‘Ambiguity packing in constraint-based parsing — practical results’, Proceedings of the 1st Conference of the North American Chapter of the Association for Computational Linguistics, Seattle, WA, pp. 162–169. find
  32. Parsons, T. (1990) Events in the semantics of English, MIT Press: Cambridge, MA. find
  33. Piano, L. (1996) Adaptation of Acquilex tagger to unknown words — release 2, University of Cambridge Computer Laboratory, unpublished memo. find
  34. Preiss, J. (2002) ‘Anaphora resolution with memory-based learning’, Proceedings of the 5th Annual CLUK Research Colloquium, Sheffield. find
  35. Srinivas, B. (2000) ‘A lightweight dependency analyzer’, Natural Language Engineering, vol.6.2, 113–138. find
  36. Weischedel, R., M. Meteer, R. Schwartz, L. Ramshaw and J. Palmucci (1993) ‘Coping with ambiguity and unknown words through probabilistic models’, Computational Linguistics, vol.19.2, 359–382. find
  37. Yeh, A. (2000) ‘Using existing systems to supplement small amounts of annotated grammatical relations training data’, Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, Hong Kong, pp. 126–132. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/briscoe-2002-robust-accurate-statistical-annotation-general/