📄 The Second Release of the RASP System

Summary

RASP is a robust English parsing toolkit that annotates raw, unannotated text with syntactic structure. Its first release had been downloaded by more than 120 sites, and this paper describes the second release, which fixes several of the first version's weaknesses. The authors improved every module in the Unix-style pipeline: tokeniser, PoS tagger, lemmatiser, unification-based grammar, and generalised LR parser. They also enriched the tagger lexicon for rare words, enlarged the grammar, and redesigned the grammatical relation (GR) output to be more semantically motivated. The parse ranking model is now trained semi-supervisedly from only an unlabelled bracketing of the Susanne Treebank, which makes retraining after grammar changes much easier. Users can customise the system by pre-annotating input in XML, disabling grammar rules, or supplying domain-specific lexical data. On a GR-compatible reannotation of DepBank (from WSJ section 23), the system reaches a micro-averaged F1 of 76.3%. According to the authors, this is comparable to the PARC XLE parser. The release matters because it is a free, retrainable, wide-coverage parser for non-commercial use.

Key points

• Tagger: a bigram HMM tagger with 150 CLAWS-derived PoS/punctuation labels, trained on Susanne, LOB and a BNC subset. It scores just over 97% accuracy on the DepBank portion of WSJ section 23, and its thresholded (multi-tag) mode gives over 99% tag recall.
• Morphological analyser: about 1400 Flex finite-state rules, with an error rate below 0.07%.
• Grammar: grew from 400 to 689 unification-based rules, and roughly 20% of them are flagged as marked. It finds a full S-rooted parse for about 85% of Susanne and 84% of the WSJ test data, and falls back to connected partial parses otherwise.
• Evaluation data: 560 randomly chosen DepBank sentences, scored over a GR hierarchy.
• Results: micro-averaged F1 is 76.3% and macro-averaged F1 is about 62.9%. Rare relations such as csubj, obj2 and pmod score much lower than frequent ones like aux and det.

Interesting ideas

• I1. Hierarchical evaluation with partial credit. A mislabelled relation still counts as correct at any more general level of the relation hierarchy that covers both the predicted and gold labels. One evaluation therefore yields everything from fine-grained labelled scores down to unlabelled dependency accuracy. This could suit any task with a taxonomy of labels, such as UD relation subtypes.
• I2. Tag thresholding instead of a single forced tag. The forward-backward algorithm can pass several plausible tags to the parser. This trades some speed for higher recall and reportedly gives slightly better coverage and accuracy, so it is worth trying wherever tagging errors cascade into later stages.
• I3. Genre tuning without retraining. Marking some grammar rules as special lets users switch them off for a given text type. This gives a cheap adaptation mechanism when no in-domain training data exists.
• I4. Training from unlabelled bracketing. The ranking model learns only from bracket structure, not full labelled trees. This lowers annotation cost and makes it practical to retrain after the grammar changes, which is attractive for low-resource languages.
• I5. Weighted dependencies from the whole parse forest. An inside-outside variant gives each GR a weight reflecting how much support it has across all derivations, rather than trusting only the single best parse. Such confidence weights are useful for downstream extraction tasks.