📄 XTAG System - A Wide Coverage Grammar for English

Summary

The paper describes XTAG, a grammar development environment for building a broad-coverage English grammar in Lexicalized Tree Adjoining Grammar (LTAG). In LTAG, each word anchors elementary trees that encode its local syntactic and predicate-argument constraints. The system chains several components:

• a large morphological database,
• a trigram part-of-speech tagger whose N-best output prunes morphological ambiguity,
• a two-pass Earley-style parser that first selects trees for each word and then combines them by substitution and adjunction,
• a statistics database of tree frequencies,
• an X-window tool for editing and debugging grammars.

The parser outputs a derived phrase-structure tree and a derivation tree, which can be read as an unlabeled dependency graph. Parses are ranked by hand-written structural preferences. The authors give preliminary coverage figures on WSJ, IBM manual and ATIS sentences, none of which involve corpus training. They also compare XTAG with the IBM statistical parser on crossing brackets and with the Alvey ANLT parser on LDOCE noun phrases, where the two performed comparably. The work matters because it shows a hand-built, linguistically rich lexicalized grammar running at scale with practical speedups.

Key points

• The morphology database has about 317,000 inflected forms across 13 POS categories, including about 213,000 noun forms and 46,500 verb forms. It was derived from the Collins and Oxford Advanced Learner's dictionaries.

• The grammar resources are large:

  • The syntactic database has more than 37,000 entries, which point either to individual trees or to tree families.
  • The tree database holds 385 trees grouped into 40 tree families (subcategorization frames), plus 62 standalone trees.

• Filtering by the tagger cuts parse time by about 93% on average. Tree-pruning heuristics plus the Lex Prob DB statistics give roughly 87% further runtime savings.

• In preliminary coverage tests, the share of sentences parsed was:

  Corpus	Sentences parsed	Average parses per sentence
  WSJ (sentences of at most 15 words)	39.09%	7.53
  IBM manual	75.42%	6.14
  ATIS	74.42%	6.0

• The external comparisons were small:

  • Against the IBM statistical parser, only 67 IBM-manual sentences overlapped with the Lancaster treebank and could be scored with parseval.
  • Against ANLT, the test set was 143 LDOCE noun phrases, with results reported both with and without the tagger.

Interesting ideas

• I1. A "POS Blender" step uses the N-best sequences from a statistical tagger to filter the candidate analyses proposed by a dictionary-based morphological analyzer, so ambiguity is reduced before parsing begins rather than resolved afterwards.
• I2. The derivation tree, which records which word's tree attached to which other word's tree, doubles as a dependency structure between words. The authors argue that parsers should be judged on these dependencies, not on bracketings, since identical brackets can hide different derivations.
• I3. Parse ranking relies on explicit, ordered structural preferences. Examples include favoring arguments over adjuncts, attaching most PPs to the nearest non-proper-noun site, treating noun sequences as left-branching, and giving coordinated modifiees high attachment.
• I4. Lexical entries select whole tree families that correspond to subcategorization frames, with feature templates layered on top to capture a word's quirks (for example, several entries for "have").
• I5. For speed, the parser first tries only the three most frequent trees for each word given its tag, and falls back to the full lexical set only on failure. It succeeds about half the time without the fallback.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): crate en could intersect lexicon-licensed tags with the tagger's N-best list, keeping a traceable record of why each reading was dropped.
• I2 (tree): this supports evaluating gparse and the perceptron parser on head-dependent attachments rather than constituent overlap, and storing derivation-style provenance for each arc.
• I3 (tree): such named preferences could serve as readable tie-breakers or repair rules when parser witnesses disagree, each one citable in an explanation.
• I4 (world): verb-class knowledge could map each verb sense to a family of frame realizations (active, passive, wh-extracted), so frames link to VerbNet-like classes systematically rather than word by word.
• I5 (learn): a frequency-pruned first pass with a guaranteed full retry is a cheap way to speed up parsing silver data without losing coverage, and the fallback rate itself becomes a useful diagnostic.