Elamur Research⌕
☰
home / papers / 2026 / Nearest-Neighbor Retrieval for Indigenous Image Captioning

Nearest-Neighbor Retrieval for Indigenous Image Captioning

Summary

This NAIST paper presents the team's entry for the AmericasNLP 2026 shared task, which focuses on generating image captions in Indigenous languages. The task covers Bribri, Guaraní, Nahuatl, Wixárika, and Yucatec Maya, all of which have very little training data. The authors compare two systems.

The first is a simple nearest-neighbor method. It embeds the query image with OpenCLIP (ViT-B/32, LAION-2B), finds the closest development-set image by cosine similarity, and reuses that image's caption unchanged. The second is a five-stage pipeline built on GPT models:

  • scene analysis,
  • dictionary-based lexical pruning,
  • retrieval of structurally similar examples,
  • interlinear gloss generation,
  • surface realization, either by the LLM or by a rule-based conversion script.

The pipeline relies on hand-made glossed captions, morpheme alignment tables, and bilingual dictionaries. Retrieval clearly won on chrF++ and placed at or near the top of the leaderboard for several languages. The pipeline produced short, over-constrained captions, hampered by sparse dictionaries and spelling mismatches between resources. Human ratings told a more mixed story: retrieved captions are fluent but can describe the wrong scene. The work shows that retrieval is a strong baseline for low-resource captioning and that surface-overlap metrics have clear limits.

Key points

  • Four retrieval similarity strategies were tested on the dev set: CLIP, DINOv2 (ViT-L/14), and LaBSE embeddings of English or Spanish captions generated by gpt-4o-mini. CLIP performed best for four languages, and DINOv2 edged ahead only for Wixárika (19.83 vs. 19.71 chrF++).

  • The test-set chrF++ scores for the CLIP retrieval system were:

    LanguagechrF++Rank
    Bribri19.371st of 14
    Wixárika19.841st of 14
    Nahuatl20.933rd of 12
    Maya15.805th of 9
    Guaraní19.4118th of 24

  • The gloss pipeline (GPT-5.4) scored far lower, for example 4.66–14.23 chrF++ on the dev set. Its outputs were roughly half the length of the references. It was not finalized for Wixárika.

  • Rule-based gloss conversion helped clearly only for Nahuatl, with an average gain of +4.88 chrF++ and 43 of 50 examples improved. It gave a small gain for Bribri and slight average losses for Maya and Guaraní.

  • After human evaluation, the retrieval system ranked 1st for Wixárika (average rating 3.79), 3rd for Bribri, and 4th for Maya, Guaraní, and Nahuatl. Nahuatl's average rating was only 1.22, which shows the gap between chrF++ scores and human judgments of caption accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.americasnlp-6.26

@inproceedings{vasselli2026nearest,
  title = {{Nearest-Neighbor Retrieval for Indigenous Image Captioning}},
  author = {Vasselli, Justin and Peguero, Arturo Martínez and Ozaki, Shintaro and Hudi, Frederikus and Sakajo, Haruki and Watanabe, Taro},
  booktitle = {Proceedings of the Sixth Workshop on NLP for Indigenous Languages of the Americas (AmericasNLP)},
  year = {2026},
  doi = {10.18653/v1/2026.americasnlp-6.26},
  url = {https://doi.org/10.18653/v1/2026.americasnlp-6.26}
}

References

21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Bolles and Alejandra Bolles. 2001. A Grammar of the Yucatecan Mayan Language. Foundation for the Advancement of Mesoamerican Studies, Crystal River, FL. find
  2. Minh Duc Bui, David Guzmán, Abteen Ebrahimi, Franklin Morales, Marvin Agüero-Torales, Raquel Insfrán, Cecilia González, Ramón Araujo, Luca Cernuzzi, Carlos Raul Noh Chi, Carlos Eduardo Tec Cahun, Sindi Estrella Poot Cohuo, Daniel Ricardo Benítez Chi, Santos Natanael Palomo Arévalo, Jessica Elizabeth… find
  3. Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. 2023. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco… find
  4. Bruno Estigarribia. 2020. A Grammar of Paraguayan Guarani. Grammars of World and Minority Languages. UCL Press. find
  5. Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. 2010. Every picture tells a story: Generating sentences from images. In Computer Vision – ECCV 2010, pages 15–29, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  6. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891, Dublin, Ireland. Association for Computation… find our card
  7. Michael Ginn, Mans Hulden, and Alexis Palmer. 2024. Can we teach language models to gloss endangered languages? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 5861– 5876, Miami, Florida, USA. Association for Computational Linguistics. find
  8. Carla Victoria Jara. 2018. Gramática de la lengua bribri. éditeur non identifié. find
  9. Ministerio de Educación de la Provincia de Corrientes. 2022. Avañe’ẽ del Taragui: Diccionario guaraní-español, español-guaraní. Ministerio de Educación de la Provincia de Corrientes, Corrientes, Argentina. Coordinación de Educación Intercultural Bilingüe. find
  10. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, and 7 o… find our card
  11. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. find our card
  12. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th Internatio… find our card
  13. SIL International. 2002. Diccionario náhuatl de la sierra norte de puebla. find
  14. SIL International. 2012. Diccionario huichol–español. find
  15. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, and 1 others. 2026. Openai gpt-5 system card. Preprint, arXiv:2601.03267. find our card
  16. Chihiro Taguchi and Richard Sproat. 2026. Creating conlangs to probe the metalinguistic grammatical knowledge of llms. Preprint, arXiv:2510.07591. find
  17. David H. Tuggy. 2004. Náhuatl: Lecciones para prin-cipiantes. Originally published in 1991. Electronic edition copyright 2004. find
  18. Justin Vasselli, Arturo Martínez Peguero, Junehwan Sung, and Taro Watanabe. 2024. Applying linguistic expertise to LLMs for educational material development in indigenous languages. In Proceedings of the 4th Workshop on Natural Language Processing for Indigenous Languages of the Americas (Americas-N… find
  19. Justin Vasselli, Arturo Mp, Frederikus Hudi, Haruki Sakajo, and Taro Watanabe. 2026. Measuring linguistic competence of LLMs on indigenous languages of the Americas. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers… find
  20. Shigeto Yoshida and Angel Abraham Ucan Dzul. 2025. Diccionario de Uso del Maya Yucateco, primera edición corregida edition. Publicación independiente, México. Open Educational Resource (REA). find
  21. Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. 2011. Im2text: Describing images using 1 million captioned photographs. In Neural Information Processing Systems (NIPS). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vasselli-2026-nearest-neighbor-retrieval-indigenous-image/