Elamur Research⌕
☰
home / papers / 2026 / IndigiEval: Evaluating LLMs in North American Indigenous Languages

IndigiEval: Evaluating LLMs in North American Indigenous Languages

Summary

LLMs are increasingly considered as tools for revitalizing endangered languages. Their errors are especially risky for second-language learners, who often cannot detect mistakes. Standard benchmarks require thousands of test items or heavy annotation by fluent speakers, which small communities cannot afford and may see as extractive. The authors, citizens of the Muscogee and Choctaw Nations, propose IndigiEval, a lightweight qualitative framework built to be small-scale, non-extractive and accessible. It draws on existing dictionaries, text collections and learning materials, and a second-language speaker can run it, with elders checking only part of the output. The framework has four categories: cultural knowledge, translation (as a proficiency check), text generation and speech. Several commercial LLMs were tested on Mvskoke and Choctaw in every category, and on Cheyenne, Cherokee and Hawaiian in the proficiency part. No model did well across the board. All of them invented orthography, grammar, vocabulary and cultural facts. The main contribution is a reusable template that helps communities decide whether and how to adopt LLM tools, not a single ranking score.

Key points

  • Languages: five endangered US Indigenous languages were covered: Mvskoke (mus), Choctaw (cho), Cheyenne (chy), Cherokee (chr) and Hawaiian (haw).
    • Estimated speakers range from about 300 (Mvskoke) to about 7,000 (Hawaiian).
    • Mvskoke and Choctaw have zero Wikipedia articles and zero Common Crawl presence.
    • Hawaiian has 2,968 Wikipedia articles and 6,343 Common Crawl items.
  • Translation: 10 sentences per language, chosen to vary in syntax and vocabulary, were scored with chrF++ against references from language documentation. The authors explicitly do not recommend LLMs over conventional MT for these languages.
  • Speech recognition: GPT 4o Transcribe and Gemini 3.1 Pro were compared using error rates, including CER.
  • Speech synthesis: one model read the Mvskoke letter ⟨v⟩ as /v/ instead of /ə/. Gemini produced that vowel correctly, but other phonetic errors made its audio unintelligible.
  • Conclusions and caveats:
    • LLMs appear unsuitable where learners cannot spot errors.
    • The datasets are small, so the results are indicative rather than statistically precise.
    • English is involved in every task.
    • The authors raise ethical concerns about tribal data sovereignty and the absence of any way for communities to correct model errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.americasnlp-6.8

@inproceedings{mainzinger2026indigieval,
  title = {{IndigiEval: Evaluating LLMs in North American Indigenous Languages}},
  author = {Mainzinger, Julia and Brixey, Jacqueline},
  booktitle = {Proceedings of the Sixth Workshop on NLP for Indigenous Languages of the Americas (AmericasNLP)},
  year = {2026},
  doi = {10.18653/v1/2026.americasnlp-6.8},
  url = {https://doi.org/10.18653/v1/2026.americasnlp-6.8}
}

References

62 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2026. Claude opus 4.6. find
  2. Roberts Darg̀is, Guntis Barzdins, Inguna Skadiņa, Normunds Gruzitis, and Baiba Saulīte. 2024. Evaluating open-source LLMs in low-resource languages: Insights from latvian high school exams. In Proceedings of the 4th International Conference on Natural Language Processing for Digital Humanities , pag… find
  3. Israel Abebe Azime, Atnafu Lambebo Tonja, Tadesse Destaw Belay, Yonas Chanie, Bontu Fufa Balcha, Negasi Haile Abadi, Henok Biadglign Ademtew, Mulubrhan Abebe Nerea, Debela Desalegn Yadeta, Derartu Dagne Geremew, Assefa Atsbiha Tesfu, Philipp Slusallek, Thamar Solorio, and Dietrich Klakow. 2025. Prov… find
  4. Ona De Gibert, Robert Pugh, Ali Marashian, Raul Vazquez, Abteen Ebrahimi, Pavel Denisov, Enora Rice, Edward Gow-Smith, Juan Prieto, Melissa Robles, Rubén Manrique, Oscar Moreno, Angel Lino, Rolando Coto-Solano, Aldo Alvarez, Marvin Agüero-Torales, John E Ortega, Luis Chiruzzo, Arturo Oncevay, and 3… find
  5. Rosalia Badhorse. 2023. Tsetsėhestȧhese and So’Taeo’o (Cheyenne) Language: Grammar Sketch for Learners . Ph.D. thesis, University of Arizona. find
  6. Rachel Baker-Ramos, Will Gelder, Leah Cho, Jahnavi Kolakaluri, and Josiah Hester. 2025. Kumu connect: Design thinking for place-based generative educational technology in hawaiian immersion schools. In Proceedings of the 2025 Conference on Research on Equitable and Sustained Participation in Enginee… find
  7. Abteen Ebrahimi, Manuel Mager, Adam Wiemerslage, Pavel Denisov, Arturo Oncevay, Danni Liu, Sai Koneru, Enes Yavuz Ugan, Zhaolin Li, Jan Niehues, Monica Romero, Ivan G Torre, Tanel Alumäe, Jiaming Kong, Sergey Polezhaev, Yury Belousov, Wei-Rui Chen, Peter Sullivan, Ife Adebara, and 15 others. 2022. F… find
  8. Anita Bardwell, Joe Bardwell, Lopaka Weltman, and Hoaloha Westcott. 2020. He Papa Kuhikuhi Pilina‘ōlelo: Reference Grammar of the Hawaiian Language . University of Hawai’i. find
  9. Matthias Brenzinger and Patrick Heinrich. 2013. The return of hawaiian: language networks of the revival movement. Current Issues in Language Planning , 14(2):300–316. find
  10. Marc T J Elliott, Deepak P, and Muiris Maccarthaigh. 2025. Evolving generative ai: entangling the accountability relationship. Digital Government: Research and Practice , 6(1):1–13. find
  11. Jacqueline Brixey and David Traum. 2022. Towards an automatic speech recognizer for the choctaw language. In Proc. S4SG 2022 , pages 6–9. find
  12. Cyrus Byington. 1870. Grammar of the Choctaw language. Proceedings of the American Philosophical Society , 11:317–367. Edited by Daniel Garrison Brinton. Also published as a monograph by McCalla and Stavely, Philadelphia, 1870. find
  13. Maxim Enis and Mark Hopkins. 2024. From llm to nmt: Advancing low-resource machine translation with claude. find
  14. Louise Fisher, Lenora Holliman, Wayne Leman, Leroy Pine Sr., and Marie Sanchez. 2023. Cheyenne Dictionary . Chief Dull Knife College. find
  15. Chaka Chaka. 2024. Currently available GenAI-powered large language models and low-resource languages: Any offerings? wait until you see. International Journal of Learning, Teaching and Educational Research , 23:148–173. find
  16. Earnest Gouge, Edited, Translated by Jack B. Martin, and Juanita McGirt. 2004. Totkv Mocvse / New Fire: Creek Folktales . Norman: University of Oklahoma Press. find
  17. Inyoung Cheong, King Xia, K. J. Kevin Feng, Quan Ze Chen, and Amy X. Zhang. 2024. (a)i am not a lawyer, but...: Engaging legal experts towards responsible llm policies for legal advice. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency , FAccT Daya Guo, Dejian Y… find
  18. Zoey Liu, Crystal Richardson, Richard Hatcher, and Emily Prud’hommeaux. 2022. Not always about you: Prioritizing community needs when developing endangered language technology. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages… find
  19. Mary R. Haas. 1979. Southeastern languages. In Lyle Campbell and Marianne Mithun, editors, The Languages of Native America: Historical and Comparative Assessment , pages 299–326. University of Texas Press. find
  20. Julia Mainzinger and Gina-Anne Levow. 2024. Fine-tuning ASR models for very low-resource languages: A study on mvskoke. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop) , pages 76–82, Bangkok, Thailand. Association for C… find
  21. Mary R. Haas, James H. Hill, Jack B. Martin, Margaret McKane Mauldin, and Juanita McGirt. 2015. Creek (Muskogee) Texts . University of California Publications. find
  22. Zachary Arao Hanson. 2025. Indigenous (mis)representation in emerging LLM research methodologies. UC Riverside Undergraduate Research Journal , 19. find
  23. Jack B Martin. 2011. A grammar of creek (Muskogee) . UNP - Nebraska. find
  24. Michael Townsen Hicks, James Humphries, and Joe Slater. 2024. Chatgpt is bullshit. Ethics and Information Technology , 26(2):1–10. find
  25. Jack B Martin and Margaret Mckane Mauldin. 2000. A Dictionary of Creek/Muskogee . University of Nebraska Press. find
  26. Leanne Hinton. 2013. Bringing our languages home: Language revitalization for families . Heyday Books. find
  27. Paul J Meighan. 2024. Indigenous language revitalization using TEK-nology : how can traditional ecological knowledge (TEK) and technology support intergenerational language transmission? Journal of Multilingual and Multicultural Development , 45:3059–3077. find
  28. Gregg Howard and Rick Eby. 1990. Introduction to Cherokee . Various Indian Peoples Publishing Co. find
  29. Lester James Validad Miranda, Elyanah Aco, Conner G. Manuel, Jan Christian Blaise Cruz, and Joseph Marvin Imperial. 2025. FilBench: Can LLMs understand and generate Filipino? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages 2496–2529, Suzhou, China. A… find
  30. Charles Julian. 2010. A History of the Iroquoian Languages . Ph.D. thesis, University of Manitoba. find
  31. Daniel Heath Justice. 2025. Our Fire Survives the Storm: A Cherokee Literary History . University of Minnesota Press. find
  32. Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. 2023. Large language models only pass primary school exams in indonesia: A comprehensive test on IndoMMLU. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 12359–12374. Association for Computat… find
  33. Sjur Nørstebø Moshagen, Lene Antonsen, Linda Wiechetek, and Trond Trosterud. 2024. Indigenous language technology in the age of machine learning. Acta Borealia , 41(2):102–116. find
  34. Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, Victor Gutierrez Basulto, Yazmin Ibanez-Garcia, Hwaran Lee, Shamsuddeen Hassan Muhammad, Kiwoong Park, Anar Sabuhi Rzayev, Nina White, Seid Muhie… find our card
  35. James N Leiker and Ramon Powers. 2011. The northern Cheyenne exodus in history and memory . University of Oklahoma Press. find
  36. Wayne Leman. 1980. A reference grammar of the Cheyenne language . Chief Dull Knife College. find
  37. �Aha Pūnana Leo and Hale Kuamo�o. 2003. Māmaka Kaiao . University of Hawai�i Press. find
  38. Jason Edward Lewis, Hundefinedmi Whaanga, and Ceyda Yolgörmez. 2024. Abundant intelligences: placing ai within indigenous knowledge frameworks. AI & Society , 40(4):2141–2157. find
  39. Graham Neubig, Shruti Rijhwani, Alexis Palmer, Jordan MacKenzie, Hilaria Cruz, Xinjian Li, Matthew Lee, Aditi Chaudhary, Luke Gessler, Steven Abney, Shirley Anugrah Hayati, Antonios Anastasopoulos, Olga Zamaraeva, Emily Prud’hommeaux, Jennette Child, Sara Child, Rebecca Knowles, Sarah Richard Little… find
  40. Moeller, Jeffrey Micher, and 5 others. 2020. A summary of the first workshop on language technology for language documentation and revitalization. In Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-R… find
  41. Clark, Stephen R Pfohl, Heather Cole-Lewis, Darlene Neal, Qazi Mamunur Rashid, Mike Schaekermann, Amy Wang, Dev Dash, Jonathan H Chen, Nigam H Shah, Sami Lachgar, Philip Andrew Mansfield, and 16 others. 2025. Toward expert-level medical question answering with large language models. Nature Medicine… find
  42. Yewei Song, Lujun Li, Cedric Lothritz, Saad Ezzini, Lama Sleem, Niccolo’ Gentile, Radu State, Tegawendé F Bissyandé, and Jacques Klein. 2026a. Are small language models the silver bullet to low-resource languages machine translation? In Proceedings for the Ninth Workshop on Technologies for Machine… find our card
  43. Mississippi Band of Choctaw Indians. 2026. Choctaw language dictionary. find
  44. �Ōiwi Parker Jones. 2018. Hawaiian. Journal of the International Phonetic Association , 48(1):103– 115. find
  45. Brock Pitawanakwat. 2018. Strategies and methods for anishinaabemowin revitalization. The Canadian Modern Language Review , 74(3):460–482. find
  46. Yewei Song, Lujun Li, Cedric Lothritz, Saad Ezzini, Lama Sleem, Niccolo’ Gentile, Radu State, Tegawendé F Bissyandé, and Jacques Klein. 2026b. Are small language models the silver bullet to low-resource languages machine translation? In Proceedings for the Ninth Workshop on Technologies for Machine… find our card
  47. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation , pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. find our card
  48. Pukui and Elbert. 2003. Hawaiian Dictionary . University of Hawai�i Press. find
  49. David Stap and Ali Araabi. 2023. ChatGPT is not a good indigenous translator. In Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (Americas-NLP) . Association for Computational Linguistics. find
  50. Majdi Quttainah, Vinaytosh Mishra, Somayya Madakam, Yotam Lurie, and Shlomo Mark. 2024. Cost, usability, credibility, fairness, accountability, transparency, and explainability framework for safe and effective large language models in medical education: Narrative review and qualitative study. JMIR A… find
  51. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazari… find our card
  52. Nathaniel Robinson, Perez Ogayo, David R. Mortensen, and Graham Neubig. 2023. Chat-GPT MT: Competitive for high- (but not low-) resource languages. In Proceedings of the Eighth Conference on Machine Translation , pages 392– 418, Singapore. Association for Computational Linguistics. find
  53. The Choctaw Nation of Oklahoma Dictionary Committee. 2016. Chahta Anumpa Tosholi Himona: New Choctaw Dictionary , 1st edition. Choctaw Print Services. find
  54. Mike Rogers. 2021. Choctaw Nation members talk about impact of losing native speakers to COVID-19. News 12 . find
  55. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, Akshay Nathan, Alan Luo, Alec Helyar, Aleksander Madry, Aleksandr Efremov, Aleksandra Spyra, Alex Baker-Whitcomb, Alex Beutel, Alex Karpenko, and 465 others. 2025… find our card
  56. Ashmal Vayani, Dinura Dissanayake, Hasindri Watawana, Noor Ahsan, Nevasini Sasikumar, Omkar Thawakar, Henok Biadglign Ademtew, Yahya Hmaiti, Amandeep Kumar, Kartik Kuckreja, Mykola Maslych, Wafa Al Ghallabi, Mihail Mihaylov, Chao Qin, Abdelrahman M Shaker, Mike Zhang, Mahardika Krisna Ihsani, Amiel… find
  57. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Linda Wiechetek, Flammie Pirinen, Maja Lisa Kappfjell, Trond Trosterud, Børre Gaup, and Gary F. Simons and Charles D. Fennig, editors. 2018. Ethnologue: Languages of the World , twenty-first edition. SIL… find
  58. Sjur Nørstebø Moshagen. 2024. The ethical question – use of indigenous corpora for large language models. In Proceedings of the Language Resources and Evaluation Conference , pages 15922– 15931. European Language Resources Association (ELRA) and ICCL. sample. Each sample is from high-quality recordi… find
  59. Shiyue Zhang, Benjamin Frey, and Mohit Bansal. 2022. How can NLP help revitalize endangered languages? A case study and roadmap for the Cherokee language. In ACL 2022 . find
  60. GPT 4o transcribe WER CER 0.83 0.41 1.00 0.52 0.70 0.15 0.94 0.37 lang mus mus cho cho Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023a. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. In Thirty-seventh Conference on Neural… find
  61. Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2024. Multilingual machine translation with large language models: Empirical results and analysis. In Findings of the Association for Computational Linguistics: NAACL 2024 , pages 2765– 2781. As… find our card
  62. Malgorzata Ćavar, Damir Ćavar, and Hilaria Cruz. 2016. Endangered language documentation: Bootstrapping a chatino speech corpus, forced aligner, ASR. In Proceedings of the Language Resources and Evaluation Conference , pages 4004–4011. European Language Resources Association (ELRA). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mainzinger-2026-indigieval-evaluating-llms-north-american/