📄 The American National Corpus: A Standardized Resource for American English

Summary

The paper reports on the first two years of the American National Corpus (ANC), proposed at LREC 1998. No large, current, freely obtainable corpus of American English existed. The Brown Corpus has only a million words and dates from the 1960s, and the BNC reflects British vocabulary and grammar. The ANC aims to match the BNC in size (100 million words) and genre balance, using texts from 1990 onward so that the two varieties can be compared directly. A consortium of dictionary publishers and software firms from the US, Japan and Europe funds the work and contributes texts. The Linguistic Data Consortium handles processing and distribution. The corpus will be encoded in XCES (XML) and released in two phases. First comes a cheap, fully automatic base level with Penn TreeBank POS tags. A later phase adds richer, hand-checked annotation. The resource matters for lexicography and NLP because it offers uniform annotation, broad genre coverage and global access for research.

Key points

• Target size is about 100 million words, balanced by BNC domain and medium criteria. Because the texts are contemporary, the ANC overlaps only with the BNC's second time period.

• The corpus has two parts:

  • a static core of about 100M words that never changes;
  • a dynamic part that grows by roughly 10% every five years, in chronological layers.

  A further varied (not balanced) section will hold email, rap lyrics, historically important novels and similar material.

• The base level follows CES Level 0. It is produced by automatically converting printer codes into XCES structure, plus minimal BNC-style headers added by hand. POS tags use the Penn TreeBank tagset and are checked only by sampling.

• In the second phase, a "gold standard" subset (perhaps 10%) gets hand-validated sentences, tokens, names, dates and POS tags. Tags may be mapped to the EAGLES morpho-syntactic categories, and XSLT scripts will support tasks such as concordancing.

• Access rules:

  • Consortium members pay fees over three years and get three years of exclusive commercial rights.
  • Non-profit users get the data from the start for a nominal fee, with no geographic limits (unlike the EU-only BNC).
  • An open, redistributable sub-corpus is also planned.

Interesting ideas

• I1. Keeping a frozen core separate from periodically added, dated layers lets results stay comparable over years while the resource still tracks change.
• I2. The project first runs a fully automatic, spot-checked pass over everything, then spends human effort only on a small gold slice (about a tenth).
• I3. Annotations are stored as separable layers in a standard format and converted with transformation scripts, so new annotation can be added later without touching the base text.
• I4. Genres such as email, manuals and help files differ in their named-entity structures (formulas, addresses, currencies) as well as in syntax. Small samples across many genres support research on genre adaptation.
• I5. The project gathered user requirements through workshops before designing the full annotation scheme, instead of fixing the format up front.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Freeze a fixed evaluation set and put newly collected silver data into versioned, dated layers, so score changes can be traced to model changes rather than data drift.
• I2 (learn): Mirror this by letting witness parsers produce silver trees at scale, and reserve LLM-teacher or human checking for a fixed fraction that serves as the gold reference.
• I3 (tree): Store each UD dialect or conversion as its own layer over the same tokens, with explicit, explainable conversion rules instead of rewriting treebanks in place.
• I4 (tokens): Build a small multi-genre test suite stressing addresses, currencies and formulas, to catch tokenizer and tagger failures that newswire-style text never exposes.
• I5 (answer): Before extending world or answer representations, collect concrete failing questions from the school tests and fable QA, so that the needs of the readers drive the design.