📄 How to age BERT Well: Continuous Training for Historical Language Adaptation

Summary

Taggers and parsers for historical languages struggle because these varieties have little annotated data, unstable spelling and rich inflection. This paper asks whether continuing to pre-train an existing BERT on raw text from an older stage of a language helps. The authors take Old English (OE) and continue masked language modeling for two starting models, English bert-base-uncased and bert-base-multilingual-uncased. They use roughly two million words of unlabeled OE from the Complete Corpus of Anglo-Saxon Poetry and a 2018 Plaintext Wikipedia dump. They then fine-tune both models for POS tagging on the manually annotated ISWOC corpus, which is written in West Saxon. Continued pre-training improves tagging for both models, both in-domain and on held-out texts. The two models rank differently depending on the measure. The English model does better at masked-token prediction, but the multilingual model tags better and transfers better to unseen texts. The result shows that a cheap, label-free adaptation step can substantially help low-resource historical varieties. It also warns that language-modeling scores are a poor guide for choosing the best base model.

Key points

• Data and splits:
  • The unlabeled OE pre-training text totals about 2.04M words; fully capitalized texts were removed.
  • ISWOC provides 2,541 annotated sentences (29,406 tokens).
  • Of these, 2,230 sentences are used for train/dev, and 311 sentences from Ælfric's Lives of Saints and Orosius form an out-of-domain test set.
• In-domain POS accuracy, before → after continued pre-training:
  • EN BERT: 86.37 → 92.16.
  • mBERT: 88.79 → 93.70.
• Out-of-domain POS accuracy, before → after:
  • EN BERT: 71.96 → 76.71.
  • mBERT: 77.87 → 84.13.
  • The adaptation gain is larger on unseen texts.
• Best hyperparameters:
  • Masked language modeling: peak learning rate 1e-3, batch size 16.
  • POS fine-tuning: learning rate 2e-5, batch size 8.
  • EN BERT reached about 83% masked-token accuracy.
• Per-tag results across 18 OE tags:
  • Personal pronouns (Pp) score best, with F1 0.98 for mBERT and 0.97 for EN BERT.
  • Both models never correctly tag foreign words (F-) or interjections (I-).
  • EN BERT never predicts interrogative adverbs (Du), even though they occur 53 times in the Gospels.

Interesting ideas

• I1. Running extra masked-LM training on unlabeled target-variety text before supervised fine-tuning is a cheap adaptation step, and its benefit is largest out of domain.
• I2. The model that wins on masked-token prediction lost on the downstream task, so intrinsic LM scores should not be used to pick a base model or checkpoint.
• I3. Before training, the authors checked how each tokenizer handles OE-specific letters and found that one splits them into isolated characters while the other only knows them inside other languages' subwords.
• I4. The models were sometimes highly confident in wrong tags and unsure about correct ones, so their softmax scores are a poor filter for correctness.
• I5. A per-tag breakdown exposed rare categories that a model never outputs at all, which overall accuracy completely hides.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): gparse's encoder could get a round of continued pre-training on raw fables, tales and school texts before parser fine-tuning, to make it more robust on those genres.
• I2 (learn): select encoder checkpoints by downstream parse/tag accuracy on held-out treebank data, never by LM loss alone.
• I3 (tokens): routinely audit how the en crate and gparse's subword vocabulary segment rare characters, archaic spellings or symbols, since bad segmentation silently caps tagger quality.
• I4 (learn): when building silver data, rely on agreement between several witness parsers rather than one model's confidence to accept or reject a tree.
• I5 (tokens): add an evaluation report that flags any UPOS/XPOS tag with zero predictions or near-zero recall, so rare-class gaps get targeted training data.