Elamur Research⌕
☰
home / papers / 2026 / Evaluating Frontier LLM Translation Capability for Lakota

Evaluating Frontier LLM Translation Capability for Lakota

Summary

LLM providers advertise translation for many languages without disclosing how much training data each one had. For critically endangered Lakota, with fewer than 2,000 first-language speakers and no large parallel corpus, it was unclear how well these models actually perform. This study gives the first zero-shot test of frontier LLMs on Lakota. Seven models, four proprietary and three open-weight, translated 200 conversational sentence pairs from the New Lakota Dictionary in both directions. Each model ran once with reasoning disabled or minimal and once with maximum extended thinking. Outputs were scored with chrF++ and BLEU, a diacritic-stripped chrF++ variant, and two LLM judges assessing meaning equivalence. The paper also examines self-reported confidence and refusals.

No model translated reliably. Gemini 3.1 Pro performed best, yet meaning was preserved in only about 60% of Lakota→English cases and as few as 6% for the weakest model. Extended reasoning helped only slightly. For open-weight models, it mainly increased refusals rather than improving quality. These results warn against trusting LLM output for endangered languages and provide a reproducible benchmark.

Key points

  • Models evaluated:
    • Proprietary: Gemini 3.1 Pro Preview, Claude Opus 4.6, Claude Sonnet 4.6 and GPT-5.2.
    • Open-weight (via Together AI): DeepSeek-V4-Pro, Qwen3.6-Plus and GLM-5.1.
    • All were accessed through litellm with schema-enforced JSON output.
  • Top scores: Gemini achieved a mean chrF++ of 59.4 (Lakota→English) and 42.6 (English→Lakota). DeepSeek-V4-Pro was the best open-weight model but still lagged well behind the leaders.
  • Semantic judging: Gemini 3 Flash Preview and Llama-3.3-70B-Instruct-Turbo served as judges and agreed with Cohen's κ = 0.75. Their equivalence rates ranged from 6% (GPT-5.2) to 60% (Gemini) and often diverged from chrF++. BERTScore proved uninformative, with F1 between 0.906 and 0.959 regardless of translation quality.
  • Effect of reasoning and diacritics:
    • Extended thinking added 1–7 chrF++ points for proprietary models.
    • Diacritic normalization showed that base characters were mostly correct, but marks such as carons and ogoneks were placed inconsistently.
  • Phrase-level patterns and availability:
    • Frequent phrasebook expressions scored near 100, while kinship, ceremonial and morphologically complex sentences scored near zero.
    • Code and aggregate results are available at github.com/robotson/lakota-translation-benchmark; the NLD data itself is not redistributed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.americasnlp-6.2

@inproceedings{robertson2026evaluating,
  title = {{Evaluating Frontier LLM Translation Capability for Lakota}},
  author = {Robertson, Lance},
  booktitle = {Proceedings of the Sixth Workshop on NLP for Indigenous Languages of the Americas (AmericasNLP)},
  year = {2026},
  doi = {10.18653/v1/2026.americasnlp-6.2},
  url = {https://doi.org/10.18653/v1/2026.americasnlp-6.2}
}

References

15 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Seth Aycock, David Stap, Di Wu, Christof Monz, and Khalil Sima’an. 2025. Can LLMs really learn to translate a low-resource language from one grammar book? In Proceedings of the Thirteenth International Conference on Learning Representations (ICLR). ArXiv:2409.19151. ↗
  2. Abteen Ebrahimi, Manuel Mager, Shruti Rijhwani, and 1 others. 2023. Findings of the Americas-NLP 2023 shared task on machine translation into indigenous languages. In Proceedings of the Third Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP), pages 206–219. find
  3. Jan F. Ullrich. 2008. New Lakota Dictionary. Lakota Language Consortium, Bloomington, IN. find
  4. Albert Sr. White Hat. 1999. Reading and Writing the Lakota Language. University of Utah Press, Salt Lake City. find
  5. Tasha R. Hauff. 2020. Beyond numbers, colors, and animals: Strengthening Lakota/Dakota teaching on the Standing Rock Indian Reservation. Journal of American Indian Education, 59(1):5–25. ↗
  6. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In Proceedings of the Eighth International Conference on Learning Representations (ICLR). ↗
  7. Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is Chat-GPT a good translator? Yes with GPT-4 as the engine. arXiv preprint arXiv:2301.08745. ↗ our card
  8. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation (EAMT), pages 193–203. ↗
  9. Muhammed Yusuf Kocyigit, Eleftheria Briakou, Daniel Deutsch, Jiaming Luo, Colin Cherry, and Markus Freitag. 2025. Overestimation in LLM evaluation: A controlled large-scale study on data contamination’s impact on machine translation. In Proceedings of the 42nd International Conference on Machine Learning (ICML). ArXiv:2501.18771. ↗
  10. Manuel Mager, Arturo Oncevay, Abteen Ebrahimi, and 1 others. 2021. Findings of the Americas-NLP 2021 shared task on open machine translation for indigenous languages of the americas. In Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP), pages 202–217. find
  11. Christopher Moseley, editor. 2010. Atlas of the World’s Languages in Danger, 3rd edition. UN-ESCO Publishing, Paris. find
  12. Kaushik Parankusham, Rodrigue Rizk, and K.C. Santosh. 2025. LakotaBERT: A transformer-based model for low resource Lakota language. arXiv preprint arXiv:2503.18212. ↗
  13. Maja Popović. 2015. chrF: character n-gram Fscore for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395. find
  14. Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation (WMT), pages 612–618. find our card
  15. Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi. 2024. A benchmark for learning to translate a new language from one grammar book. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR). ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/robertson-2026-evaluating-frontier-llm-translation-capability/