📄 Evaluating the Performance of Transformers in Translating Low-Resource Languages through Akkadian

Summary

Akkadian is an extinct East Semitic language. Huge numbers of its cuneiform tablets have been transliterated, but few have been translated. Unlike most low-resource languages, it has a large, carefully annotated parallel corpus. That makes it a convenient laboratory for comparing translation architectures without heavy data augmentation. The authors collected transliterated Akkadian–English pairs from Oracc and CDLI and normalised mixed ASCII/Unicode transliteration conventions. They then fine-tuned several pretrained models on consumer hardware using quantisation and LoRA:

• seq2seq models: MarianMT variants initialised from Arabic→English and Spanish→English, and T5
• decoder-only LLMs: Mistral-7B and Qwen-0.5B

Krueger's published T5 served as an external reference. Evaluation used BLEU, ROUGE-2, BERTScore, pairwise significance tests, length-bucketed analysis and inference timing.

The Arabic-initialised MarianMT gave the best overall balance and was by far the fastest. Mistral-7B edged it on BLEU but was much slower. The Spanish-initialised Marian performed far worse, which points to real transfer value from a related language. The authors caution that repetition in the corpus inflates absolute scores.

Key points

• Translation gap: in the Ur III corpus, roughly 72,000 texts are transcribed, yet only about 2.2% have modern translations.

• BERTScore results:

  • MarianAr scored 0.931, and Mistral, Qwen and Krueger's T5 scored about 0.930.
  • The authors' own T5 (15 epochs) reached 0.914, and MarianEs only 0.842.

• Short references (<4 words): the LLMs did best on BLEU. Mistral reached 0.602, Qwen 0.593 and Krueger 0.586, against 0.541 for MarianAr.

• Speed (500 sentences):

  Model	Total time	Per sentence
  MarianAr	108 s	0.22 s
  T5	242 s	0.48 s
  Qwen	348 s	0.70 s
  Mistral	1786 s	3.57 s

• Significance and training notes: all pairwise BLEU differences had p < 0.05. Mistral-7B was fine-tuned for just one epoch and Qwen-0.5B for three. Krueger's T5 outperformed the authors' 15-epoch T5, which the paper credits to his 30 epochs and bidirectional training.

Interesting ideas

• I1. Picking the pretrained starting point by linguistic kinship (Arabic rather than Spanish) changed results dramatically, so the choice of initialisation deserves deliberate testing as a variable. — For Mova (tree): When adapting gparse to a new UD dialect or genre, compare initialising from different source treebanks and keep the one closest in structure, not just the largest.
• I2. Splitting evaluation by reference length showed that big LLMs shine mainly on very short segments, a pattern that a single averaged score hides. — For Mova (learn): Report parser and reader accuracy per sentence-length bucket, so gains limited to short sentences are not mistaken for general improvement.
• I3. The authors openly flag that near-duplicate phrases occur in both train and test splits and inflate scores, while noting that model comparisons remain valid. — For Mova (learn): Add a near-duplicate check between silver training data and test sets as a standard negative control before trusting absolute numbers.
• I4. Measuring wall-clock inference cost next to quality made the compact model the practical winner despite a slightly lower BLEU. — For Mova (tree): Publish sentences-per-second alongside LAS when comparing the perceptron parser and gparse, so pipeline choices reflect the real speed–accuracy trade-off.
• I5. Training a model to translate in both directions (source→English and back) was reported to stabilise convergence for a small seq2seq model. — For Mova (back): Train the text→graph and graph→text halves of the compressor jointly, so each direction regularises the other.