📄 Very Large Annotated Database of American English

Summary

This 1991 workshop report describes early work on the Penn Treebank at the University of Pennsylvania. The project aims to assemble a large body of American English, drawn from both written sources and transcribed speech, marked up with part-of-speech tags and syntactic structure. The resource has three intended uses: training data for automatic language learning, a gold standard for evaluating parts of NLP systems, and material for studying the grammar of real spoken English. The report covers three strands of work. The first is data released so far, both POS-tagged and bracketed. The second is tooling, including a retrainable version of Church's PARTS tagger and Pearl, a probabilistic chart parser built with UNISYS. The third is an empirical study of how fast human annotators can correct automatic parser output. The central practical finding is that a stripped-down, skeletal bracketing scheme lets correctors work several times faster than full parses do. That speed makes millions of treebanked words per year achievable, and it shaped how large treebanks were later built.

Key points

• Up to 4.5 million words of preliminary POS-tagged material went to 10 US sites. In addition, 351 MUC-3 messages (about 100K words) were POS-tagged, and 88 of them were syntactically annotated and released by anonymous FTP.
• Pearl is a bottom-up chart parser with Earley-style top-down prediction that always expands the best-scoring hypothesis. It was trained on 1100 Voyager sentences with the PUNDIT string grammar and got 35 of 40 unseen test sentences right (about 88%).
• Annotators correcting full Fidditch output reached 100–200 words/hour after three weeks and 400–500 words/hour after two months. Learning to treebank took much longer than learning POS tagging.
• A skeletal representation raised correction speed to 700–750 words/hour, which supports roughly 2.5 million treebanked words per year with a single correction pass per sentence.
• About 250K words were annotated since December 1, with one third bracketed by more than one annotator. Planned work includes:
  • automatic consistency checks between annotators;
  • retagging the corpus with the retrained PARTS and adjudicating disagreements, aiming for well under 1% tag error at about 5 extra minutes per 1000 words;
  • grammar induction that combines information-theoretic bracketing with X-bar theory.

Interesting ideas

• I1. Annotators should correct a machine draft rather than annotate from scratch, and the draft should be simplified to a skeletal form. Speed then rises severalfold while the essential structure is kept.
• I2. Annotation throughput should be measured over time for each scheme, and the scheme chosen partly on that cost, not only on linguistic richness.
• I3. Retagging an existing corpus with a retrained tagger and adjudicating only where the old and new tags disagree drives the error rate down cheaply.
• I4. Having part of the data bracketed independently by several annotators provides a basis for automatic inter-annotator consistency checking.
• I5. A tagger rebuilt to be independent of any particular tag set can be retrained on new inventories without rewriting its code.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Silver trees agreed by the witness parsers could be shown to the LLM teacher or human checkers in a reduced skeletal view, so that verification is cheaper and covers more sentences.
• I2 (learn): Track how many sentences per hour each teacher tier verifies under different tree presentations, and use those numbers to plan the size of silver data.
• I3 (tokens): Run the current tagger in crate en against older tags and route only the disagreements to a teacher, which gives a low-cost way to clean up lemma and POS errors.
• I4 (tree): Keep a slice of treebank sentences parsed by several witnesses and flag systematic disagreements as candidates for UD dialect conversion fixes or repair rules.
• I5 (tokens): Keep the tagger in crate en parameterized by tag inventory, so that switching between PTB-style and UD tags, or adding finer tags, needs only retraining.