📄 Are Small Language Models the Silver Bullet to Low-Resource Languages Machine Translation?

Summary

The paper examines whether compact language models can translate low-resource languages (LRLs) reliably. Such models are attractive where cost, latency, privacy or offline use matter, but they often hallucinate on underrepresented languages. The authors first benchmark several small decoder-only models (Llama-3.x, Gemma2, Qwen2.5, Phi-3/3.5, Ministral-8B) against GPT-4o-mini on all of FLORES-200. They translate into English and score outputs mainly with an LLM-as-a-Judge (LLMaaJ) metric, plus BLEU. Small models lose far more quality on LRLs than larger ones do, and African, Austronesian and Indigenous American languages score lowest. To close this gap, the authors build synthetic training data by having strong teacher models translate mostly monolingual LRL text, then fine-tune the small models on it. Luxembourgish is the main case study. With this distillation, 2B–3B models can reach or beat systems with up to 70B parameters. General abilities are preserved. This offers a cheap and practical route to deployable LRL translation.

Key points

• Benchmark: All 200 FLORES-200 languages were evaluated in the source→English direction. LLMaaJ scores range from 0 to 1, and a score of at least 0.8 is treated as a good translation.
• Training method: Full-parameter fine-tuning beat LoRA, and decoder-only teachers (Llama 3.3-70B, GPT-4o-mini) beat the encoder–decoder NLLB-200-3.3B teacher.
• Effect of data size: Increasing training data from 25% to 100% roughly doubled spBLEU for EN→LB with Llama-3.2-3B-Instruct and Gemma-2-2b-it. Gemma learned faster with little data but gained less beyond 50%.
• Generalisation to other languages: The pipeline was repeated with Llama-3.2-3B-Instruct on Khasi, Assamese and Ukrainian, using 10,000 monolingual sentences and 1,000 validation pairs per language.
  • Gains were largest where the base model started weak, such as En–Kh and En–As.
  • NLLB could not serve as a teacher for Khasi because it does not support that language.
• No catastrophic forgetting: SuperGLUE scores changed only slightly after fine-tuning. Training ran on 4×H100 GPUs with TRL, and the code and models are released openly.