📄 The New Oxford English Dictionary Project

Summary

Benbow describes how Oxford University Press began turning the Oxford English Dictionary into a computerised resource. The 12-volume OED (completed 1928) and its 4-volume Supplement (1972–1986) needed constant revision, and printing more supplements or revising on paper was judged impractical. The project ran in phases. Phase 1 aimed to print one integrated edition with Supplement entries merged into the main text. Later phases would build, extend and publish an electronic lexical database. Phase 1 involved an analysis of the text's logical structure, manual keying of the whole text with structure codes, and proofreading backed by software checks. Merging was automatic where possible, with lexicographers editing the harder cases online. Later steps were cross-reference repair, conversion of the old pronunciation notation to IPA, and phototypesetting. The key finding was that layout does not map simply onto structure, so a parser was needed to complete SGML tagging. The work matters because it shows how a huge, irregular reference text can become a queryable database without losing information.

Key points

• Scale: 16 volumes, about 21,000 pages, 60 million words, 350 million characters, 320,000 entries and 2.35 million quotations.
• International Computaprint Corporation (ICC) keyed the text in 18 months, finishing in June 1986, with an error rate of at most 7 keystrokes per 10,000. OCR was rejected because the typography is intricate and the print is degraded.
• Partners and funding:
  • IBM UK donated equipment and expertise worth about £1 million and seconded three experts.
  • The University of Waterloo provided a computer scientist to build the structural parser and was also designing the future database.
  • The Department of Trade and Industry gave a grant of £285,000.
• The working system ran on IBM VM with IBM's SQL database. Custom programs handled parsing, integration, cross-referencing, conversion of James Murray's notation to IPA, and an editor with structure-integrity checks.
• A user survey run with Waterloo got over 50% response, and two-thirds of respondents suggested further uses. A new-words unit, NEWS (New English Words Series), was set up in 1983 with four staff.

Interesting ideas

• I1. A single visual device, such as a font or punctuation mark, can play different structural roles depending on where it sits in an entry, so structure must be inferred from context rather than surface form alone.
• I2. Keyers inserted only part of the markup, and a parser then added the missing tags and converted everything into one standard scheme.
• I3. Each Supplement entry carried its own typed merge instruction (add, replace the definition, insert earlier examples), which let most merging run automatically.
• I4. Merging and parsing could be done in small chunks, but cross-references could only be fixed after the whole text was assembled.
• I5. The editor refused to save an amended entry until its structural integrity was verified.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): Tokenizer and tagger rules for punctuation, capitals and abbreviations should be keyed on position and neighbours, not on the character itself.
• I2 (tree): UD dialect conversion could follow this pattern by first accepting partial or foreign annotations and then running a deterministic completer that turns them into canonical UD.
• I3 (world): Updates to entities and states in the world model could be expressed as explicit typed patch operations, so every change is traceable.
• I4 (frames): Coreference could be split the same way, with local frames built sentence by sentence and global links resolved once the full document is available.
• I5 (learn): Self-induced repair rules should be admitted only if every tree they change still passes UD well-formedness checks, as a built-in negative control.