📄 Quantitative Measurement of Language Distance among Closely Related Indo-European Languages Using Pretrained Language Models: A Case Study on the North Germanic Branch

Summary

Measuring how far apart closely related languages are has traditionally been qualitative. Each study used its own word lists and criteria, and most quantitative work on mainland Scandinavian concerned speech rather than writing. Bai proposes a written-modality framework built on pretrained models and tests it on Danish, Norwegian Bokmål and Swedish. A sample of 150 sentence triplets aligned across the three languages, taken from Tatoeba, keeps meaning constant so that differences reflect linguistic form. Three signals are computed:

• distance between sentence embeddings from LaBSE, with mBERT as an independent check;
• how badly text fragments when one language's monolingual BERT vocabulary is applied to another language's version of the same sentence;
• confidence and entropy of mBERT masked-token prediction.

All three signals agree that Danish–Norwegian is the closest pair. This matches the history of Danish serving as Norway's written language from 1380 to 1814. The author argues that the main contribution is not the ordering itself. It is a reproducible pipeline, calibrated on a well-understood case, that can then be applied to less-studied language groups.

Key points

• Data: Danish Tatoeba sentences linked to both Norwegian Bokmål (nob) and Swedish (swe) yielded 456 triplets. From these, 150 were sampled with random seed 42. Tatoeba text is licensed CC BY 2.0 FR.
• Distance definition: distance is 1 minus the median cosine similarity over the 150 pairs.
  • LaBSE: da–no 0.012, no–sv 0.016, da–sv 0.020.
  • mBERT (google-bert/bert-base-multilingual-cased, [CLS] vector of the last hidden state, L2-normalised): da–no 0.016, no–sv 0.045, da–sv 0.046.
• Model resolution: LaBSE similarities sit in a narrow 0.980–0.988 band (std ≈ 0.004) yet still separate three levels. mBERT separates only two tiers. The author attributes this to LaBSE's training on translation pairs versus mBERT's MLM objective. mBERT remains necessary because only it supports the predictability metric.
• Comparison with speech: Gooskens (2007) found that how well listeners understood the spoken language tracked phonetic distance closely (r = −0.80), whereas the link with lexical distance did not reach significance. Spoken Danish drifted away from its neighbours through stød and other sound changes, and Norwegian acts as a bridge in speech. The written-text results are presented as a complement to that spoken-modality work.
• Limitations and plans: stated limitations are the small sample, a single branch, and MLM confidence being confounded by training-corpus distribution. Planned extensions cover Iberian Romance and West Slavic, XLM-R and multilingual-E5, n-gram cross-entropy and perplexity, and speech via Whisper / Wav2Vec2.

Interesting ideas

• I1. Validate a measuring method first on a case whose answer is already known (here, a branch with a settled history), and only then apply it to open questions; this separates trust in the instrument from trust in its findings.
• I2. Hold content fixed with parallel sentences so that any measured difference can only come from form, not meaning.
• I3. Apply one language's subword vocabulary to text from a related variety and treat the increase in fragmentation as a cheap, model-free proxy for orthographic distance.
• I4. Accept a ranking only when two independently trained models with different objectives agree, and use the more discriminating one for fine distinctions.
• I5. Summarise many per-item similarities with the median and a fixed sampling seed, so a few odd items do not drive the result and anyone can rerun it exactly.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Before trusting a new evaluation metric on unknown inputs, run it on pairs whose relative difficulty or similarity is already established and require it to recover that order, much like a negative or positive control.
• I2 (back): Paraphrase sets of one proposition could test the text → graph → text compressor, since all versions should map to the same meaning graph and any divergence points at a form-sensitive bug.
• I3 (tokens): Running the en tokenizer's vocabulary over British, American, archaic or informal text and tracking fragmentation would flag where the lexicon needs extra entries or spelling-variant rules.
• I4 (learn): This mirrors witness-parser agreement for silver data; it also suggests recording each witness's resolution, so near-ties from a coarse witness are not read as real disagreements.
• I5 (tree): Comparing perceptron parser and gparse output, or a treebank before and after UD dialect conversion, could use per-sentence agreement aggregated by median over a seeded sample, giving a stable, reproducible distance between annotation styles.