📄 Playing with Words at the National Library of Sweden -- Making a Swedish BERT

Summary

The authors, working at KBLab of the National Library of Sweden (KB), set out to build a Swedish-specific BERT, because multilingual models tend to underperform on smaller languages and suitable training data is scarce. They assembled a corpus of modern Swedish (1940s to late 2019) from KB's holdings: OCR'd newspapers, government reports (SOU), legal e-deposits, forum comments and Swedish Wikipedia. Before pre-training they cleaned the text heavily. They repaired systematic OCR errors using a lexicon, rebuilt sentences and paragraphs with spaCy plus custom scripts, and kept emojis as meaningful tokens. They then built a SentencePiece vocabulary and pre-trained the model with Google's original BERT code. On NER and POS tagging with the Stockholm-Umeå Corpus, the resulting KB-BERT beat both Google's M-BERT and an earlier Swedish model from Arbetsförmedlingen. The NER gains were clear, while the POS gains were marginal. The model is publicly released. The work shows that a national library's archives can supply a lower-resourced language with a strong base model, and that diverse registers in the corpus matter.

Key points

• The cleaned corpus has about 260M sentences, roughly 3,497M words and about 18.3 GB of text. Newspapers (allnews.txt, about 3.0B words) dominate, followed by SOU, e-deposit, social and sw-wiki. Fiction is underrepresented.
• The OCR repair targeted m ↔ rn confusions, such as "om" being turned into "örn" and "-arna" into "-ama". Candidates were checked against Saldo, the morphological resource from Språkbanken. Over 30,000 mappings were found, and about 13 million occurrences were corrected.
• The tokenizer is SentencePiece with a vocabulary of about 50k, chosen because Swedish has many compounds. Emojis and skin-tone modifiers were added to it by hand.
• Pre-training ran in three stages:
  • 1M steps with sequence length 128 and batch size 512;
  • 100k steps with sequence length 512 and batch size 128;
  • roughly 1M more steps at 128/512.
• Compute came partly from in-house hardware and partly from Google TFRC Cloud TPUs, which were used only for non-copyrighted material.
• On NER (SUC 3.0, split 70/20/10), average FB1 was 0.927 for KB-BERT, 0.906 for M-BERT and 0.876 for AF-AI. HFST-SweNER is shown in the results but is not directly comparable. For POS, KB-BERT improved on M-BERT by less than 1%. NER showed little gain between 350k and 1M pre-training steps.

Interesting ideas

• I1. Lexicon-checked OCR repair. For a known confusion pair, generate the swapped spelling of each word. Add the swap to a correction table only when the original is missing from a morphological lexicon and the variant exists. This is cheap and targeted, and it could fit OCR'd Ukrainian or other Cyrillic archives with their own typical character confusions.
• I2. A minimum paragraph length as a noise filter. Fragments shorter than about ten sentences were discarded, along with broken sentences at paragraph edges. This removes context-poor OCR debris before pre-training.
• I3. Emojis as content and punctuation. Emojis were kept in the vocabulary and allowed to end sentences where they act as end punctuation. This is useful for any model expected to handle social-media text.
• I4. Diversity over sheer volume. The authors argue that adding small corpora of new registers (forums, official prose) helps robustness more than adding more of an existing genre. They report that messy social-media data seemed to help beyond that domain. This is a sensible principle when assembling corpora for any lower-resourced language.
• I5. Splitting compute by copyright status. Copyrighted material was trained in-house, while freely usable text went to external cloud compute. This is a practical pattern for institutions holding rights-restricted collections.