Elamur Research⌕
☰
home / papers / 2026 / LLM Probe: Evaluating LLMs for Low-Resource Languages

LLM Probe: Evaluating LLMs for Low-Resource Languages

Paper · 2026 · Hailay Kidu Teklehaymanot, Gebrearegawi Gebremariam and Wolfgang Nejdl

Summary

How well large language models handle low-resource, morphologically rich languages is poorly understood, because such languages have little annotated data and no standard test suites. The authors propose LLM Probe, an evaluation framework that uses an annotated bilingual lexicon as its test material instead of running text. They apply it to Tigrinya, a Semitic language written in Ge'ez script with root-and-pattern morphology. They digitized a printed Tigrinya–English dictionary from Swansea using OCR plus manual correction and added entries contributed by native-speaker experts. Linguists then annotated the entries for POS, gender, number and other features, reaching Cohen's κ of 0.84–0.91. Models are tested with example-based prompts on four tasks: lexical alignment, POS tagging, morphosyntactic feature identification, and English→Tigrinya translation. The study compares five causal LLMs against three sequence-to-sequence models. Encoder–decoder models do better on morphosyntax and translation. Causal models align words well but translate poorly. The work shows that a small, hand-verified, structured probe can expose specific weaknesses where large benchmarks are missing. It also lowers the risk of test-set contamination. The framework and data are released openly.

Key points

  • Four probing tasks: lexical alignment, POS tagging, morphosyntactic probing (gender, number, agreement, noun class), and translation fidelity against curated references.
  • Models tested:
    • causal: Falcon-10B, Gemma-2B, Gemma-7B, Mistral-7B, Qwen-7B
    • sequence-to-sequence: mT5-base, mT5-large, ByT5
  • Construction of the bidirectional English–Tigrinya lexicon:
    • starting point: the Swansea Tigrinya–English dictionary, digitized via OCR, error fixing and orthographic normalization
    • extra entries from printed dictionaries, teaching materials and community glossaries
  • Inter-annotator agreement is high (Cohen's κ 0.84–0.91). Entries keep polysemy (several English words → one Tigrinya form), multiple target variants, and masculine/feminine noun pairs.
  • Stated limitations: no proprietary models, general vocabulary only, sensitivity to prompt wording, and a lack of Tigrinya tokenizers, parsers and morphological analyzers.

Interesting ideas

  • I1. An annotated lexicon can serve as a cheap, controlled test bed that is less likely to overlap with pretraining data than corpus benchmarks, so scores better reflect real competence.
  • I2. Gold entries are stored as many-to-many sets (one form for "me, I"; several renderings for "but"), so scoring must accept any valid variant rather than a single string.
  • I3. Paired masculine/feminine entries for the same concept (two forms of "applauder") act as natural minimal pairs for checking feature sensitivity.
  • I4. Prompts force a rigid output shape, such as arrow-joined word pairs or a lowercase comma-separated feature list, so answers can be checked automatically.
  • I5. Breaking results down per task and per architecture showed complementary strengths (alignment vs. translation and morphosyntax) instead of one overall ranking.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Build a small hand-checked word list with POS and features to audit LLM teacher tiers and silver data. This gives an evaluation set the teachers are unlikely to have memorized.
  • I2 (back): Generation and translation through the meta-world should be scored against a set of acceptable realizations. This avoids penalizing legitimate paraphrases or synonyms.
  • I3 (tokens): Minimal-pair lists that differ in one feature (number, person, tense) would make morphology and tagger tests reveal exactly which feature fails.
  • I4 (learn): LLM teacher queries could require a fixed, parseable format for tags, lemmas or roles. Malformed answers can then be rejected before they enter silver data.
  • I5 (learn): Choose different teacher models or witness parsers per subtask based on per-task scores rather than trusting one model for everything.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Leibniz University HannoverHanover · Germany5 works · 599 citations🎓Aksum UniversityAksum · Ethiopia2 works · 2 citations🔬L3S Research CenterHanover · Germany3 works · 4 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksLow-resource languages · 267 works

Cite

Cite as Hailay Kidu Teklehaymanot, Gebrearegawi Gebremariam and Wolfgang Nejdl. 2026. LLM Probe: Evaluating LLMs for Low-Resource Languages. arXiv:2603.29517.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2603.29517 · arXiv 2603.29517

@misc{teklehaymanot2026llm,
  title = {{LLM Probe: Evaluating LLMs for Low-Resource Languages}},
  author = {Hailay Kidu Teklehaymanot, Gebrearegawi Gebremariam and Wolfgang Nejdl},
  year = {2026},
  doi = {10.48550/arxiv.2603.29517},
  eprint = {2603.29517},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2603.29517}
}

References

19 references, 8 of them with a card of ours. ↗ leads to the work.

  1. Hafte Abera and Sebsibe Hailemariam. Tigrinya automatic speech recognition with morpheme based recognition units. In Rossana Cunha, Samira Shaikh, Erika Varis, Ryan Georgi, Alicia Tsai, Antonios Anastasopoulos, and Khyathi Raghavi Chandu, editors, Proceedings of the Fourth Widening Natural Language Processing Workshop, pages 46–50, Seattle, USA, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.winlp-1.12. URL https://aclanthology.org/2020. winlp-1.12/. ↗
  2. Firoj Alam, Shammur Absar Chowdhury, Sabri Boughorbel, and Maram Hasanain. LLMs for low resource languages in multilingual, multimodal and dialectal settings. In Mohsen Mesgar and Sharid Loáiciga, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Abstracts, pages 27–33, St. Julian’s, Malta, March 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.eacl-tutorials. 5. URL https://aclanthology.org/ 2024.eacl-tutorials.5/. ↗
  3. Ekaterina Artemova and Barbara Plank. Low-resource bilingual dialect lexicon induction with large language models. In Tanel Alumäe and Mark Fishel, editors, Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 371–385, Tórshavn, Faroe Islands, May 2023. University of Tartu Library. URL https://aclanthology.org/2023.nodalida-1.39/. ↗
  4. Jiayi Chang, Mingqi Gao, Xinyu Hu, and Xiaojun Wan. Exploring the multilingual nlg evaluation abilities of llm-based evaluators. arXiv preprint arXiv:2503.04360, 2025. ↗
  5. Krishno Dey, Prerona Tarannum, Md Arid Hasan, Imran Razzak, and Usman Naseem. Better to ask in english: Evaluation of large language models on english, low-resource and cross-lingual settings. arXiv preprint arXiv:2410.13153, 2024. ↗
  6. Fitsum Gaim and Jong C. Park. Natural language processing for Tigrinya: Current state and future directions. arXiv preprint arXiv:2507.17974, 2025. ↗
  7. Fitsum Gaim and Jong C. Park. Natural Language Processing for Tigrinya: Current State and Future Directions, July 2025. URL https://arxiv.org/abs/2507.17974. ↗
  8. Fitsum Gaim, Wonsuk Yang, and Jong C. Park. Monolingual pre-trained language models for Tigrinya. In 5th Widening NLP (WiNLP) Workshop, co-located with EMNLP 2021, 2021. URL https://aclanthology.org/2021.winlp-1.3/. ↗
  9. Fitsum Gaim, Wonsuk Yang, and Jong C. Park. GeezSwitch: Language identification in typologically related low-resourced East African languages. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6578–6584, Marseille, France, June 2022. European Language Resources Association. URL https://aclanthology.org/2022. lrec-1.707/. ↗
  10. Fitsum Gaim, Wonsuk Yang, Hancheol Park, and Jong Park. Question-answering in a low-resourced language: Benchmark dataset and models for Tigrinya. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11857–11870, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.661. URL https://aclanthology.org/2023. acl-long.661/. ↗
  11. Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, and Sunayana Sitaram. Are large language model-based evaluators the solution to scaling up multilingual evaluation? In Yvette Graham and Matthew Purver, editors, Findings of the Association for Computational Linguistics: EACL 2024, pages 1051–1070, St. Julian’s, Malta, March 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024. findings-eacl.71/. ↗
  12. Daoyang Li, Haiyan Zhao, Qingcheng Zeng, and Mengnan Du. Exploring multilingual probing in large language models: A cross-language analysis. In Hao Fei, Kewei Tu, Yuhui Zhang, Xiang Hu, Wenjuan Han, Zixia Jia, Zilong Zheng, Yixin Cao, Meishan Zhang, Wei Lu, N. Siddharth, Lilja Øvrelid, Nianwen Xue, and Yue Zhang, editors, Proceedings of the 1st Joint Workshop on Large Language Models and Structure Modeling (XLLM 2025), pages 61–70, Vienna, Austria, August 2025. Association for Computational Linguistics. ISBN 979-8-89176-286-2. doi: 10.18653/v1/2025.xllm-1.7. URL https://aclanthology.org/2025.xllm-1.7/. ↗
  13. Fenghua Liu, Yulong Chen, Yixuan Liu, Zhujun Jin, Solomon Tsai, and Ming Zhong. The gold medals in an empty room: Diagnosing metalinguistic reasoning in llms with camlang. arXiv preprint arXiv:2509.00425, 2025. ↗
  14. Hengyu Luo, Zihao Li, Joseph Attieh, Sawal Devkota, Ona de Gibert, Shaoxiong Ji, Peiqin Lin, Bhavani Sai Praneeth Varma Mantina, Ananda Sreenidhi, Raúl Vázquez, et al. Gloteval: A test suite for massively multilingual evaluation of large language models. arXiv preprint arXiv:2504.04155, 2025. ↗
  15. Xuan-Phi Nguyen, Mahani Aljunied, Shafiq Joty, and Lidong Bing. Democratizing LLMs for low-resource languages by leveraging their English dominant abilities with linguistically-diverse prompts. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3501–3516, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long. 192. URL https://aclanthology.org/ 2024.acl-long.192/. ↗
  16. Samridhi Raj Sinha, Rajvee Sheth, Abhishek Upperwal, and Mayank Singh. Eka-eval: A comprehensive evaluation framework for large language models in indian languages. arXiv preprint arXiv:2507.01853, 2025. ↗
  17. Hailay Kidu Teklehaimanot. A mobile based tigrigna language learning tool, 2015. ↗
  18. Hailay Kidu Teklehaymanot, Dren Fazlija, Niloy Ganguly, Gourab Kumar Patro, and Wolfgang Nejdl. TIGQA: An expert-annotated question-answering dataset in Tigrinya. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16142–16161, Torino, Italia, May 2024. ELRA and ICCL. URL https://aclanthology.org/2024. lrec-main.1404/. ↗
  19. Tianyang Zhong, Zhenyuan Yang, Zhengliang Liu, Ruidong Zhang, Yiheng Liu, Haiyang Sun, Yi Pan, Yiwei Li, Yifan Zhou, Hanqi Jiang, et al. Opportunities and challenges of large language models for low-resource languages in humanities research. arXiv preprint arXiv:2412.04497, 2024. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/teklehaymanot-2026-llm-probe-evaluating-llms-low/