📄 Language Adaptation on a Tight Academic Compute Budget: Tokenizer Swapping Works and Pure bfloat16 Is Enough

Summary

The paper asks whether an academic lab with only a handful of server-grade GPUs and a few days of access can usefully adapt a 7B LLM to a new language through continued pretraining. Starting from the LeoLM recipe for German, the authors shrink it to fit such a budget and adapt Mistral-7B to German, using filtered OSCAR23.01 data. They later run a follow-up "hindsight" study on Arabic. Two efficiency levers are tested. The first is training entirely in bfloat16 instead of mixed precision. The second is replacing the original tokenizer with a language-specific one. Pure bfloat16 proves fast and essentially harmless for continued pretraining. Tokenizer swapping gives more compact tokenization and matches the original vocabulary in quality, but it does not raise German scores. The German models actually fall below the untouched base model, while the Arabic models beat several baselines. The practical lesson is that adaptation pays off mainly for languages the base model covers poorly. It also shows that cheap training tricks can make such adaptation feasible on small budgets.

Key points

• The budget is 7,680 steps with batch size 256 and 4,096-token context, about 8 billion tokens in total. That is 12.5% of LeoLM's 64 billion tokens. The learning rate is raised to 4 × 10⁻⁵, and the data is German only, with no English or code mixed in.
• On two 80GB GPUs, pure bfloat16 ran 39% faster than mixed precision. On a single 80GB GPU, mixed precision could not run at all because of out-of-memory errors.
• In bfloat16, an update u is lost when u < W/128, since there are only 7 mantissa bits. Mistral-7B's RMSNorm weights are much larger than its other weights, so they effectively stayed frozen. This had no visible harm on loss or downstream tasks.
• The swapped tokenizer is a sentencepiece BPE model with byte-fallback and a vocabulary of 32,768. That size was chosen to suit Nvidia Tensor Cores. The tokenizer was trained on a subset of the German training data.
• German adaptation hurt performance. For comparison, LeoLM spent 8× the compute and gained only 0.6 points. Arabic, whose script the original tokenizer did not cover, improved significantly with tokenizer swapping.

Interesting ideas

• I1. Before choosing a low-precision regime, plot weight magnitudes per parameter type and compare them with typical update sizes; this reveals which parameters will silently stop learning.
• I2. Replacing a vocabulary, rather than extending it, keeps the embedding and softmax matrices small, and sizing the vocabulary to a hardware-friendly number speeds up training.
• I3. Choose a target that the base model already handles well as a deliberate lower bound, then confirm with a second, poorly covered target; this separates the effect of the method from the effect of how much room there is to improve.
• I4. Measure tokenizer quality by fertility, meaning how many words a fixed compute budget covers, not by token counts alone.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): When training gparse in bf16, check norm and bias parameters against learning-rate-scaled updates so frozen layers are found by analysis, not by accident.
• I2 (tree): The subword vocabulary of gparse's transformer encoder could be retrained on English treebank-style text at a Tensor-Core-friendly size, giving cheaper training and more sensible word pieces.
• I3 (learn): Silver-data retraining experiments should report gains both on domains where the parser is already strong and on weak ones, with the untouched model as a negative control.
• I4 (tokens): Comparing the crate en tokenizer with gparse's subword splitter in words per token would show where segmentation wastes compute or breaks morphology.