Elamur Research⌕
☰
home / papers / 2026 / Measuring Linguistic Competence of LLMs on Indigenous Languages of the Americas

Measuring Linguistic Competence of LLMs on Indigenous Languages of the Americas

Paper · 2026 · Justin Vasselli, Arturo Martínez Peguero, Frederikus Hudi et al.

Summary

The paper asks how much LLMs actually know about Indigenous languages of the Americas. Existing benchmarks mostly cover high-resource languages or focus narrowly on machine translation. The authors build a prompting-only probing benchmark, with no fine-tuning, from AmericasNLP 2025 Shared Task development data covering 13 languages, among them Bribri, Guarani, Nahuatl and Quechua. It has three tasks of increasing depth:

  • Language identification in four variants with different distractors.
  • Spanish cloze completion, where accuracy with and without the aligned Indigenous sentence is compared.
  • Grammatical feature classification, covering tense, person, mood and similar features.

They test GPT-4.1, Gemini 2.0 Flash, DeepSeek-V3 and smaller open LLaMA and Qwen models. Competence turns out to be concentrated in a few languages, mainly those with their own Wikipedia editions, and in the strongest API models. Most other model–language pairs stay close to chance. Recognizing a language seems to be a prerequisite for deeper understanding but does not guarantee it. The benchmark gives a diagnostic tool for low-resource, typologically diverse settings.

Key points

  • Language identification setup: 459 sentences of at least five tokens per language, covering 13 Indigenous languages plus Spanish.
  • Language identification results: With high-resource distractors, accuracy was 94–100%. In the open-answer setting, even GPT-4.1 and Gemini reached only about 45–48%. Quechua, Guarani, Aymara, Nahuatl and Wayuu were identified best; all five have Wikipedias, ranging from roughly 24,000 articles for Quechua down to 695 for Wayuu.
  • Cloze task: The dataset has 5,529 four-option items, with gpt-4o-mini used to filter items and generate distractors. Adding the Indigenous sentence produced significant gains (McNemar's test) mainly for Guarani, Quechua, Nahuatl and Aymara, up to +45 points for Quechua with Gemini. No model showed gains for Asháninka, Chatino or Rarámuri.
  • Grammatical feature task: It covers only Bribri and Nahuatl. Large models reached 55–66% on Nahuatl, while all models stayed near chance on Bribri (19–29%).
  • Limitations: Results may depend on prompt wording, the grammar task includes just two languages, and no fine-tuning or adaptation was tested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nara Institute of Science and TechnologyIkoma · Japan46 works · 4,565 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksLow-resource languages · 267 works

Cite

Cite as Justin Vasselli, Arturo Martínez Peguero, Frederikus Hudi, Haruki Sakajo and Taro Watanabe. 2026. Measuring Linguistic Competence of LLMs on Indigenous Languages of the Americas. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-short.21

@inproceedings{vasselli2026measuring,
  title = {{Measuring Linguistic Competence of LLMs on Indigenous Languages of the Americas}},
  author = {Justin Vasselli, Arturo Martínez Peguero, Frederikus Hudi, Haruki Sakajo and Taro Watanabe},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-short.21},
  url = {https://doi.org/10.18653/v1/2026.eacl-short.21}
}

References

31 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Are… — MasakhaNER: Named Entity Recognition for African Languages
  2. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, and 29 others. 2023. Qwen technical report. Preprint, arXiv:2309.16609.
  3. Adolfo Constenla Umaña, Feliciano Elizondo Figueroa, and Fransisco Pereira Mora. 1998. Curso Básico de Bribri. Universidad de Costa Rica, San José. find
  4. Ona De Gibert, Robert Pugh, Ali Marashian, Raul Vazquez, Abteen Ebrahimi, Pavel Denisov, Enora Rice, Edward Gow-Smith, Juan Prieto, Melissa Robles, Rubén Manrique, Oscar Moreno, Angel Lino, Rolando Coto-Solano, Aldo Alvarez, Marvin Agüero-Torales, John E. Ortega, Luis Chiruzzo, Arturo Oncevay, and 3… find
  5. DeepSeek-AI, :, Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, and 69 others. 2024. Deepseek llm: Scaling open-source language models with longterm… find
  6. Cheikh M. Bamba Dione, David Ifeoluwa Adelani, Peter Nabende, Jesujoba Alabi, Thapelo Sindane, Happy Buzaaba, Shamsuddeen Hassan Muhammad, Chris Chinenye Emezue, Perez Ogayo, Anuoluwapo Aremu, Catherine Gitau, Derguene Mbaye, Jonathan Mukiibi, Blessing Sibanda, Bonaventure F. P. Dossou, Andiswa Buku… find
  7. Gemini, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou,… — Gemini: A Family of Highly Capable Multimodal Models
  8. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. The FLORES evaluation datasets for low-resource machine translation: Nepali–English and Sinhala– English. In Proceedings of the 2019 Conference on Empirical Metho…
  9. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization. CoRR, abs/2003.11080.
  10. Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020. X-FACTR: Multilingual factual knowledge retrieval from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5943–5959, Online. Ass… find
  11. Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, and Arianna Bisazza. 2025. Multiblimp 1.0: A massively multilingual benchmark of linguistic minimal pairs. Preprint, arXiv:2504.02768. find
  12. Katharina Kann, Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, John E Ortega, Annette Rios, Angela Fan, Ximena Gutierrez-Vasques, Luis Chiruzzo, Gustavo A Giménez-Lugo, Ricardo Ramos, Ivan Vladimir Meza Ruiz, Elisabeth Mager, Vishrav Chaudhary, Graham Neubig, Alexis Palmer, Rolando Coto-Solano, and… find
  13. Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021. Multilingual LAMA: Investigating knowledge in multilingual pretrained language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3250–3258, Online. Associ…
  14. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, and 5 others. 2020. XGLUE: A New Benchmark Dataset for Cross-l…
  15. Manuel Mager, Arturo Oncevay, Abteen Ebrahimi, John Ortega, Annette Rios, Angela Fan, Ximena Gutierrez-Vasques, Luis Chiruzzo, Gustavo Giménez-Lugo, Ricardo Ramos, Ivan Vladimir Meza Ruiz, Rolando Coto-Solano, Alexis Palmer, Elisabeth Mager-Hois, Vishrav Chaudhary, Graham Neubig, Ngoc Thang Vu, and… — Findings of the AmericasNLP 2021 Shared Task on Open Machine Translation for Indigenous Languages of the Americas
  16. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others.… — GPT-4 Technical Report
  17. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural…
  18. Max Ploner, Jacek Wiland, Sebastian Pohl, and Alan Akbik. 2025. LM-pub-quiz: A comprehensive framework for zero-shot evaluation of relational knowledge in language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: H… find
  19. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376, Online. Association…
  20. Sebastian Ruder, Jonathan H. Clark, Alexander Gutkin, Mihir Kale, Min Ma, Massimo Nicosia, Shruti Rijhwani, Parker Riley, Jean-Michel A Sarr, Xinyi Wang, John Wieting, Nitish Gupta, Anna Katanova, Christo Kirov, Dana L. Dickinson, Brian Roark, Bidisha Samanta, Connie Tao, David I. Adelani, and 8 oth… find
  21. Lutfi Kerem Senel and Hinrich Schütze. 2021. Does she wink or does she nod? a challenging benchmark for evaluating word understanding of language models. find
  22. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 532–538, Online. Association for Computational Linguistics. find
  23. Taiga Someya and Yohei Oseki. 2023. JBLiMP: Japanese benchmark of linguistic minimal pairs. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1581–1594, Dubrovnik, Croatia. Association for Computational Linguistics. find
  24. Ekaterina Taktasheva, Maxim Bazhukov, Kirill Koncha, Alena Fenogenova, Ekaterina Artemova, and Vladislav Mikhailov. 2024. RuBLiMP: Russian benchmark of linguistic minimal pairs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9268–9299, Miami, Florida… find
  25. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. Preprint…
  26. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377– 392.
  27. Wikipedia contributors. 2026. List of wikipedias. https://en.wikipedia.org/wiki/List_of_ ↗
  28. Wikipedias. Accessed January 2026. find
  29. Beilei Xiang, Changbing Yang, Yu Li, Alex Warstadt, and Katharina Kann. 2021. CLiMP: A benchmark for Chinese language model evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2784–2790, Online. Association fo…
  30. A Bribri is a tonal Chibchan language spoken in southern Costa Rica by 7,000 people. It features morphological ergativity, SOV word order, and gendered speech registers (Constenla Umaña et al., 1998). find
  31. Chatino is a Zapotecan language group in Oaxaca, Mexico. The San Juan Quiahije variant has around 5,000 speakers. It is tonal, with complex inflection and variable word order. Guarani is a Tupi-Guarani language spoken by 4–6.5 million people, mainly in Paraguay. It is agglutinative, with nasal harmo… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vasselli-2026-measuring-linguistic-competence-llms-indigenous/