📄 A Comparative Evaluation of Open-Source Models for Russian-Kazakh Translation

Summary

This paper, submitted to the Russian–Kazakh track of the Turkic LoRes MT shared task at LoResMT 2026, asks which freely available translation system works best for Kazakh without any extra training. The organizers supplied only a test set. The author found that it contained noise: some target fields copied the Russian source, and others held unrelated Russian or Kazakh text. A rule-based filter removed most of the Russian-only near-duplicates, and two bad pairs were dropped by hand. All results are reported on both the original and cleaned versions.

Several model families were run under the same zero-shot inference conditions and scored with ChrF1:

• dedicated Russian–Kazakh models,
• NLLB-200 variants,
• TranslateGemma,
• Kazakh-tuned Llama models,
• Tencent HY-MT1.5.

The specialized deepvk/kazRush-ru-kk model clearly outperformed all multilingual and LLM-based systems. The study also showed that output in the wrong script can be partly rescued by transliteration followed by a small spell-correction model. The practical lesson is that, for low-resource pairs, a purpose-built translator still beats large general models.

Key points

• Test data and cleaning: the test set had 4,626 sentence pairs, split into public and private halves. Cleaning removed 278 pairs. The filter used lowercasing, punctuation stripping, a Russian-alphabet check, and a longest-common-substring threshold of more than 30 characters. Semantically unrelated Kazakh targets were left in.

• Best system and baseline: on the private set, deepvk/kazRush-ru-kk scored 76.24 ChrF1 on original data and 80.28 on cleaned data. A copy-the-source baseline scored only 22.32 and 17.23.

• Multilingual and LLM systems: these clustered in the low-to-mid 50s on the private set (original / cleaned ChrF1):

  Model	Original	Cleaned
  facebook/nllb-200-3.3B	54.08	56.57
  issai/LLama-3.1-KazLLM-1.0-8B	51.12	53.39
  google/translategemma-27b-it	51.08	53.36
  PolynomeAI/Llama-3.1-8B-kkru	49.64	51.80

• TranslateGemma scaling: quality grew strongly with model size; the 4B variant reached only about 39.

• Tencent HY-MT1.5:

  • The 1.8B model often answered in the wrong language, so it was excluded.
  • The 7B model wrote Kazakh in Arabic script, which required transcription to Cyrillic.
  • A google/byt5-small spell corrector was then applied. It was trained on 50,000 noised Kazakh Wikipedia sentences, split into 7-word segments with a corruption probability of 0.35.
  • This post-processing lifted scores to 33.00 on original data and 34.17 on cleaned data.