Elamur Research⌕
☰
home / papers / 2026 / When LLMs Annotate: Reliability Challenges in Low-Resource NLI

When LLMs Annotate: Reliability Challenges in Low-Resource NLI

Paper · 2026 · Solmaz Panahi, John Kelleher and Vasudevan Nedumpozhimana

Summary

The paper asks whether large language models can stand in for human annotators on a demanding semantic task in a low-resource language: Natural Language Inference (NLI) in Farsi. The authors ran six instruction-tuned models, three proprietary and three open-weight, zero-shot on 750 items from the ParsiNLU training split. Each model saw eight prompt configurations, built by varying instruction language, instruction length and the order of premise and hypothesis. Reliability was judged on accuracy and also on prompt sensitivity, class-wise consistency and agreement between models.

Claude-Opus-4 and GPT-4o were both more accurate and more stable than the open-weight systems. Placing the hypothesis before the premise disrupted predictions more than any other prompt change. Neutral was the least consistent label for every model. When the authors manually inspected the most unstable items, roughly half traced back to ambiguous or doubtful gold labels. They therefore argue that model disagreement can be used to audit benchmark quality, which matters for under-resourced languages, where annotation noise strongly affects evaluation.

Key points

  • Models and data: The models were Claude-Opus-4, GPT-4o, GPT-4o-mini, Qwen3-30B-Instruct, Llama3.3-70B-Instruct and Aya-23-35B. The data was the ParsiNLU training split (275 entailment, 241 neutral, 234 contradiction). Runs used temperature 0 and a 10-token output cap. Smaller models of about 7B parameters reached only 53–56% accuracy and were left out of the main analysis.
  • Prompt grid: The prompts formed a 2×2×2 design: English or Farsi system prompt, elaborate or concise wording, and original or reversed premise/hypothesis order. The Farsi inputs themselves were never translated.
  • Accuracy range: The best score was 0.83, achieved by Claude-Opus-4 with the original_concise_fa prompt. The worst was 0.61, from Aya-23-35B with reverse_elaborate_fa. Every model peaked under the original ordering.
  • Metrics: Sensitivity was measured as the normalized entropy of predictions across prompt variants. Consistency was computed per class, and Neutral scored lowest.
  • Gold-label problems: About 49% of high-sensitivity samples involved questionable labels or genuine ambiguity. Re-evaluating on a cleaned subset raised scores for all models, with the proprietary ones gaining most. The authors treat these gains as an upper bound rather than a measure of true model competence.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Trinity College DublinDublin · Ireland24 works · 1,231 citations🎓National University of Ireland, MaynoothMaynooth · Ireland5 works · 10 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksLow-resource languages · 267 works

Cite

Cite as Solmaz Panahi, John Kelleher and Vasudevan Nedumpozhimana. 2026. When LLMs Annotate: Reliability Challenges in Low-Resource NLI. Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.loreslm-1.17

@inproceedings{panahi2026when,
  title = {{When LLMs Annotate: Reliability Challenges in Low-Resource NLI}},
  author = {Solmaz Panahi, John Kelleher and Vasudevan Nedumpozhimana},
  booktitle = {Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loreslm-1.17},
  url = {https://doi.org/10.18653/v1/2026.loreslm-1.17}
}

References

42 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Erik Arakelyan, Zhaoqi Liu, and Isabelle Augenstein. 2024. Semantic sensitivities and inconsistent predictions: Measuring the fragility of nli models. Preprint, arXiv:2401.14440. find
  2. Nitay Calderon, Roi Reichart, and Rotem Dror. 2025. The alternative annotator test for llm-as-a-judge: How to statistically justify replacing human annotators with llms. Preprint, arXiv:2501.10970.
  3. Xinyun Chen, Ryan A. Chi, Xuezhi Wang, and Denny Zhou. 2024. Premise order matters in reasoning with large language models. ArXiv, abs/2402.08939. find
  4. Lu Cheng, Mansooreh Karami, Huan Liu, Jundong Li, Song Wang, Amrita Bhattacharjee, Dawei Li, Zhen Tan, Alimohammad Beigi, and Bohan Jiang. 2024. Large language models for data annotation: A survey. Preprint, arXiv:2402.13446. — [arXiv:2402.13446](https://arxiv.org/abs/2402.13446) find
  5. Lu Cheng, Kai Shu, Tianhao Wu, Huan Liu, Canyu Chen, Amrita Bhattacharjee, Yuxuan Jiang, Dawei Li, Zhen Tan, Alimohammad Beigi, Bohan Jiang, Chengshuai Zhao, and Liangjie Huang. 2025. From generation to judgment: Opportunities and challenges of llm-as-a-judge. Preprint, arXiv:2411.16594. find
  6. Cheng-Han Chiang and Hung yi Lee. 2023. Can large language models be an alternative to human evaluations? Preprint, arXiv:2305.01937. find
  7. Ona de Gibert, Joseph Attieh, Teemu Vahtola, Mikko Aulamo, Zihao Li, Raúl Vázquez, Tiancheng Hu, and Jörg Tiedemann. 2025. Scaling low-resource mt via synthetic data generation with llms. Preprint, arXiv:2505.14423.
  8. Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023. Is GPT-3 a good data annotator? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11173–11195, Toronto, Canada. Association for…
  9. Federico Errica, Giuseppe Siracusano, Davide Sanvito, and Roberto Bifulco. 2025. What did i do wrong? quantifying llms’ sensitivity and consistency to prompt engineering. Preprint, arXiv:2406.12334.
  10. Xiyan Fu and Wei Liu. 2025. How reliable is multilingual llm-as-a-judge? Preprint, arXiv:2505.12201. find
  11. Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024. Bias and fairness in large language models: A survey. Preprint, arXiv:2309.00770.
  12. Chengguang Gan and Tatsunori Mori. 2023. Sensitivity and robustness of large language models to prompt template in japanese text classification tasks. Preprint, arXiv:2305.08714. find
  13. Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020. Pushing the limits of rule-based reasoning in transformers through natural language satisfiability. arXiv preprint arXiv:2012.10052. — [arXiv:2012.10052](https://arxiv.org/abs/2012.10052) find
  14. Raviraj Joshi, Abhay Shanbhag, Suramya Jadhav, Amogh Thakurdesai, and Ridhima Sinare. 2025. On limitations of llm as annotator for low resource languages. Preprint, arXiv:2411.17637. find
  15. Jean Kaddour and Qi Liu. 2024. Synthetic data generation in low-resource settings via fine-tuning of large language models. Preprint, arXiv:2310.01119. — [arXiv:2310.01119](https://arxiv.org/abs/2310.01119) find
  16. Daniel Khashabi, Arman Cohan, Siamak Shakeri, Pedram Hosseini, Pouya Pezeshkpour, Malihe Alikhani, Moin Aminnaseri, Marzieh Bitaab, Faeze Brahman, Sarik Ghazarian, Mozhdeh Gheini, Arman Kabiri, Rabeeh Karimi Mahabadi, Omid Memarrast, Ahmadreza Mosallanezhad, Erfan Noury, Shahab Raji, Mohammad Sadegh… — ParsiNLU: A Suite of Language Understanding Challenges for Persian
  17. Yinheng Li, Rogerio Bonatti, Sara Abdali, Justin Wagle, and Kazuhito Koishida. 2024a. Data generation using large language models for text classification: An empirical case study. Preprint, arXiv:2407.12813. — [arXiv:2407.12813](https://arxiv.org/abs/2407.12813) find
  18. Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, Yuxuan Lai, Chongyang Tao, and Shuai Ma. 2024b. Leveraging large language models for nlg evaluation: Advances and challenges. Preprint, arXiv:2401.07103.
  19. Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. 2024. On llms-driven synthetic data generation, curation, and evaluation: A survey. arXiv preprint arXiv:2406.15126.
  20. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30).
  21. Nailia Mirzakhmedova, Marcel Gohsen, Chia Hao Chang, and Benno Stein. 2024. Are large language models reliable argument quality annotators? CoRR. find
  22. Muhammad Uzair Ul Haq, Davide Rigoni, and Alessandro Sperduti. 2025. Llms as data annotators: How close are we to human performance. Preprint, arXiv:2504.15022. — [arXiv:2504.15022](https://arxiv.org/abs/2504.15022) find
  23. Barbara Plank. 2022. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  24. Jia He, Arshdeep Sekhon, David Koleczek, Mukund Rungta, Franklin X Wang, and Sadid Hasan. 2024a. Does prompt formatting have any impact on llm performance? Preprint, arXiv:2411.10541. — [arXiv:2411.10541](https://arxiv.org/abs/2411.10541) find
  25. Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024b. Annollm: Making large language models to be better crowdsourced annotators. Preprint, arXiv:2303.16854.
  26. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. Unnatural instructions: Tuning language models with (almost) no human labor. Preprint, arXiv:2212.09689.
  27. Nan-Jiang Jiang, Chenhao Tan, and Marie-Catherine de Marneffe. 2023. Understanding and predicting human label variation in natural language inference through explanation. Preprint, arXiv:2304.12443.
  28. Mehdi Rezagholizadeh, Tanya Roosta, Peyman Passban, and Bryan Guan. 2025. The order effect: Investigating prompt sensitivity in closed-source llms. Preprint, arXiv:2502.04134. — [arXiv:2502.04134](https://arxiv.org/abs/2502.04134) find
  29. Vinay Samuel, Houda Aynaou, Arijit Ghosh Chowdhury, Karthik Venkat Ramanan, and Aman Chadha. 2024. Can llms augment low-resource reading comprehension datasets? opportunities and challenges. Preprint, arXiv:2309.12426.
  30. Mikayla Sclar, Yejin Choi, Yulia Tsvetkov, and Maarten Sap. 2024. Quantifying language models’ sensitivity to spurious features in prompt design. arXiv preprint arXiv:2402.16892. — [arXiv:2402.16892](https://arxiv.org/abs/2402.16892) find
  31. Yoshi Suhara, Rickard Stureborg, and Dimitris Alikaniotis. 2024. Large language models are inconsistent and biased evaluators. Preprint, arXiv:2405.01724.
  32. Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large language models for data annotation and synthesis: A survey. Preprint, arXiv:2402.13446.
  33. Reut Tsarfaty, Tzuf Paz-Argaman, and Itai Mondshine. 2025. Beyond english: The impact of prompt translation strategies across languages and tasks in multilingual llms. Preprint, arXiv:2502.09331. find
  34. Yu-Min Tseng, Wei-Lin Chen, Chung-Chi Chen, and Hsin-Hsi Chen. 2024. Are expert-level language models expert-level annotators? Preprint, arXiv:2410.03254. — [arXiv:2410.03254](https://arxiv.org/abs/2410.03254) find
  35. Vijay Viswanathan, Xiang Yue, Alisa Liu, Yizhong Wang, and Graham Neubig. Synthetic data in the era of large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 5: Tutorial Abstracts). Association for Computational Linguistics. find
  36. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large language models are not fair evaluators. Preprint, arXiv:2305.17926. find
  37. Saizhuo Wang, Jian Guo, Wei Li, Wen Gao, Kun Zhang, Shengjie Ma, Chengjin Xu, Xuhui Jiang, Yinghan Shen, Yuanzhuo Wang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Jiawei Gu, Honghao Liu, and Lionel Ni. 2025. A survey on llm-as-a-judge. Preprint, arXiv:2411.15594. find
  38. Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want to reduce labeling cost? gpt-3 can help. Preprint, arXiv:2108.13487.
  39. Chenxi Whitehouse, Monojit Choudhury, and Alham Fikri Aji. 2023. LLM-powered data augmentation for enhanced cross-lingual performance. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 671– 686, Singapore. Association for Computational Linguistics. find
  40. Sachin Yadav, Tejaswi Choppa, and Dominik Schlechtweg. 2024. Towards automating text annotation: A case study on semantic proximity annotation using gpt-4. Preprint, arXiv:2407.04130. — [arXiv:2407.04130](https://arxiv.org/abs/2407.04130) find
  41. Shunyu Yao, Thomas L. Griffiths, Dan Friedman, R. Thomas McCoy, and Matthew Hardy. 2023. Embers of autoregression: Understanding large language models through the problem they are trained to solve. Preprint, arXiv:2309.13638. find
  42. Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen. 2024. Prosa: Assessing and understanding the prompt sensitivity of llms. Preprint, arXiv:2410.12405. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/panahi-2026-when-llms-annotate-reliability-challenges/