Elamur Research⌕
☰
home / papers / 2026 / Annotating Indian Regional Biases using Large Language Models: Evaluation and Analysis

Annotating Indian Regional Biases using Large Language Models: Evaluation and Analysis

Summary

Regional bias, meaning prejudice tied to someone's geographic origin, is common in Indian social media discussions. Labeled data for it is scarce because human annotation is costly, often inconsistent, and complicated by code-mixing and transliteration. The authors ask whether open-source LLMs can serve as automatic annotators for this task.

They use the IndRegBias corpus of YouTube and Reddit comments, each labeled as regional bias (RB) or non-regional bias (NRB). Nine open-source LLMs were tested in two settings:

  • Zero-shot: prompting based on chain-of-thought reasoning.
  • Fine-tuned: low-rank adaptation on a balanced half of the data, evaluated on the other half.

Agreement with the human labels was measured with Cohen's κ, broken down by writing style (English, code-mixed, transliterated).

Zero-shot agreement was weak for nearly all models. Fine-tuning raised κ for every model. On 500 freshly collected YouTube comments, most fine-tuned models also beat their zero-shot versions. Models remained noticeably weaker on code-mixed and transliterated text than on English. The study suggests that targeted domain adaptation can make LLMs practical tools for scaling up bias corpora in multilingual, low-resource settings.

Key points

  • Dataset: IndRegBias contains 25,000 comments (13,015 RB, 11,985 NRB).
    • By writing style: 18,222 English, 6,315 code-mixed and 463 transliterated.
    • Hinglish makes up about 87% of the code-mixed portion, and Hindi about 86% of the transliterated portion.
  • Zero-shot results: Almost all nine LLMs scored κ below 0.49 against the human labels.
  • Out-of-distribution test (500 new YouTube comments):
    • Krutrim-2 performed best after fine-tuning, with κ rising from 0.5447 to 0.6950.
    • MNemo-12B and Llama-8B climbed from about 0.46 and 0.41 to above 0.60.
    • Qwen-8B, Gemma-27B and DSeek-14B did not improve over their zero-shot results, so six of nine models gained overall.
  • Writing-style gap: Even after fine-tuning, the best code-mixed κ was 0.694, well below the best English κ of 0.833 (both from Mixtral-MoE). The authors attribute this to the heavy skew toward English and to gaps in the models' pretraining.
  • Resources: Models were selected to fit on a single H200 GPU (about 141 GB), and the authors have released their code on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.starsem-conference.16

@inproceedings{panda2026annotating,
  title = {{Annotating Indian Regional Biases using Large Language Models: Evaluation and Analysis}},
  author = {Panda, Debasmita and Anil, Akash and Shukla, Neelesh Kumar},
  booktitle = {Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.starsem-conference.16},
  url = {https://doi.org/10.18653/v1/2026.starsem-conference.16}
}

References

21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Mohammadmasiha Zahedivafa, Juan Diego Bermeo, Maria Korobeynikova, and Fabrizio Gilardi. 2025. Open-source LLMs for text annotation: A practical guide for model setting and fine-tuning. Journal of Computational Social Science, 8(1):17. find
  2. Tatiana Anikina, Jan Cegin, Jakub Simko, and Simon Ostermann. 2025. A rigorous evaluation of LLM data generation strategies for low-resource languages. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1142–1158. Association for Computational Li… find
  3. Shaily Bhatt, Sunipa Dev, Partha Talukdar, Shachi Dave, and Vinodkumar Prabhakaran. 2022. Re-contextualizing fairness in NLP: The case of India. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conferen… find
  4. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. find
  5. Sunipa Dev, Jaya Goyal, Dinesh Tewari, Shachi Dave, and Vinodkumar Prabhakaran. 2023. Building socio-culturally inclusive stereotype resources with community engagement. In Advances in Neural Information Processing Systems (NeurIPS), volume 36. find
  6. Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023. Is GPT-3 a good data annotator? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11173–11195. Association for Computational Li… find our card
  7. Fahim Faisal and Antonios Anastasopoulos. 2023. Geographic and geopolitical biases of language models. In Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL), pages 139–163. find
  8. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e2305016120. find our card
  9. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. find
  10. Suramya Jadhav, Abhay Shanbhag, Amogh Thakurdesai, Ridhima Sinare, and Raviraj Joshi. 2025. On limitations of LLM as annotator for low resource languages. In Proceedings of the International Conference on Natural Language and Speech Processing (ICNLSP 2025). find
  11. Khyati Khandelwal, Manuel Tonneau, Andrew M. Bean, Hannah Rose Kirk, and Scott A. Hale. 2024. Indian-BhED: A dataset for measuring India-centric biases in large language models. In Proceedings of the 2024 International Conference on Information Technology for Social Good (GoodIT ’24), pages 231–239.… find
  12. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916. find our card
  13. Arbi Haza Nasution and Aytuğ Onan. 2024. Chatgpt label: Comparing the quality of human-generated and llm-generated annotations in low-resource language nlp tasks. IEEE Access, 12:71876–71900. find
  14. Debasmita Panda, Akash Anil, and Neelesh Kumar Shukla. 2026. IndRegBias: A dataset for studying indian regional biases in english and code-mixed social media comments. arXiv preprint arXiv:2601.06477. — [arXiv:2601.06477](https://arxiv.org/abs/2601.06477) find
  15. Maja Pavlović and Massimo Poesio. 2024. The effectiveness of LLMs as annotators: A comparative overview and empirical analysis of direct representation. In Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) @ LREC-COLING 2024, pages 100–110. ELRA and ICCL. find
  16. Aditya Tomar, Nihar Ranjan Sahoo, and Pushpak Bhattacharyya. 2025. BharatBBQ: A multilingual bias benchmark for question answering in the indian context. arXiv preprint arXiv:2508.07090. find
  17. Petter Törnberg. 2023. Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning. arXiv preprint arXiv:2304.06588. find our card
  18. Muhammad Uzair Ul Haq, Hussain Ahmad Zafar, and David Broneske. 2025. LLMs as data annotators: How close are we to human performance? arXiv preprint arXiv:2504.15022. — [arXiv:2504.15022](https://arxiv.org/abs/2504.15022) find
  19. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 24824–24837. find our card
  20. Lingling Xu, Haoran Xie, S. Joe Qin, Xiaohui Tao, and Fu Lee Wang. 2026. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. IEEE Transactions on Pattern Analysis and Machine Intelligence, PP(99):1–20. find
  21. Mingyue Yuan, Jieshan Chen, Zhenchang Xing, and Gelareh Mohammadi. 2025. A case study of scalable content annotation using multi-LLM consensus and human review. In Proceedings of the Generative AI and HCI Workshop at CHI 2025. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/panda-2026-annotating-indian-regional-biases-large/