Elamur Research⌕
☰
home / papers / 2026 / Evaluating Linguistic Knowledge of LLMs in Tamil: The ILAKKANAM Benchmark

Evaluating Linguistic Knowledge of LLMs in Tamil: The ILAKKANAM Benchmark

Paper · 2026 · Jeyarajalingam Varsha, Menan Velayuthan, Sumirtha Karunakaran et al.

Summary

Tamil is agglutinative and computationally low-resource, yet LLM evaluation for it relies mostly on multilingual suites that are often translated from English or cover few phenomena. To fill this gap, the authors built ILAKKANAM, the first benchmark designed specifically to test LLMs' linguistic knowledge of Tamil. Its 820 questions were drawn by hand from Sri Lankan Tamil-subject school exam papers covering Grades 1–13. Trained linguists tagged each question as one of five linguistic levels or as factual knowledge. Several closed- and open-source models were run through a common evaluation pipeline and compared overall, by category and by grade.

Gemini 2.5 achieved the best answer accuracy, and open-source systems lagged behind. All models handled early-grade items well but degraded as grade level and linguistic complexity increased. In a separate task, models were asked to label each question's linguistic category. Success at this did not track answer accuracy, which the authors read as a sign of data exposure rather than genuine grammatical understanding. The resource offers a culturally grounded, native-sourced test bed for Tamil LLM work.

Key points

  • Dataset: 820 questions from Sri Lankan exam papers, with two recent papers per grade across Grades 1–13.
    • Question labels: phonetics, phonology, morphology, syntax, semantics (L1–L5), plus a factual category (F).
    • Pragmatics and stylistics were merged into semantics because they were rare.
  • Models evaluated: GPT-5, Claude Sonnet 4.5, Gemini 2.5, LLaMA 4, DeepSeek V3, Qwen 2.5 72B and Grok 4.
  • Category-tagging task:
    • GPT-5 scored highest at 65.61% (538/820).
    • Gemini 2.5, the top question-answering model, correctly tagged 111 fewer items than GPT-5.
  • Grade-wise trend: accuracy is high on lower grades and declines for higher-grade, more complex questions.
  • Availability: code and data are on GitHub (LTG-UoJ/ILAKKANAM-Public).
    • Only the extracted questions are released, to limit data leakage.
    • The source materials are licensed CC BY-NC-SA 4.0.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Utrecht UniversityUtrecht · Netherlands39 works · 1,603 citations🎓University of JaffnaJaffna · Sri Lanka7 works · 70 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Jeyarajalingam Varsha, Menan Velayuthan, Sumirtha Karunakaran, Rasan Nivethiga and Kengatharaiyer Sarveswaran. 2026. Evaluating Linguistic Knowledge of LLMs in Tamil: The ILAKKANAM Benchmark. Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/23ob532xgibu

@inproceedings{varsha2026evaluating,
  title = {{Evaluating Linguistic Knowledge of LLMs in Tamil: The ILAKKANAM Benchmark}},
  author = {Jeyarajalingam Varsha, Menan Velayuthan, Sumirtha Karunakaran, Rasan Nivethiga and Kengatharaiyer Sarveswaran},
  booktitle = {Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)},
  year = {2026},
  doi = {10.63317/23ob532xgibu},
  url = {https://doi.org/10.63317/23ob532xgibu}
}

References

35 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. 11 — The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants https://sites.google.com/univ.jfn.ac. lk/talkcorpus/home The belebele benchmark: a parallel reading comprehension dataset i… ↗
  2. Gasper Begus, Maksymilian Dabkowski, and Ryan Rhodes. 2025. Large linguistic models: Investigating llms’ metalinguistic abilities. IEEE Transactions on Artificial Intelligence, page 1–15. find
  3. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2023. A survey on evaluation of large language models.
  4. Claude. 2025. Introducing Claude Sonnet 4.5 — anthropic.com. https://www.anthropic.com/news/claude-sonnet-4-5. Accessed 2025-10-03. ↗
  5. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, ..., and Wesley Helmholz. 2025. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long C…
  6. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, ..., and Zizheng Pan. 2025. Deepseek-v3 technical report. find
  7. Xiyan Fu and Wei Liu. 2025. How Reliable is Multilingual LLM-as-a-Judge? find
  8. Omid Ghahroodi, Marzia Nouri, Mohammad V. Sanian, Alireza Sahebi, Doratossadat Dastgheib, Ehsaneddin Asgari, Mahdieh Soleymani Baghshah, and Mohammad Hossein Rohban. 2024. Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language? ArXiv, abs/2404.06644. — [arXiv:2404.06644](https://arxiv.org/abs/2404.06644) find
  9. Grok4. 2025. Grok 4. https://x.ai/news/ grok-4. Accessed 2025-10-05. ↗
  10. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A Survey on LLM-as-a-Judge. find
  11. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR).
  12. Meng Ji, Meng Ji, Pierrette Bouillon, and Mark Seligman. 2023. Cultural and Linguistic Bias of Neural Machine Translation Technology, Studies in Natural Language Processing, page 100– 128. Cambridge University Press. find
  13. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Comp…
  14. Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. 2023. Large language models only pass primary school exams in Indonesia: A comprehensive test on IndoMMLU. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12359–12374, Singapore. Association fo… find
  15. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of th…
  16. Wei Qi Leong, Jian Gang Ngui, Yosephine Susanto, Hamsawardhini Rengarajan, Kengatharaiyer Sarveswaran, and William Chandra Tjhi. 2023. BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Models.
  17. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2024. CMMLU: Measuring massive multitask language understanding in Chinese. In Findings of the Association for Computational Linguistics: ACL 2024, pages 11260– 11285, Bangkok, Thailand. Association…
  18. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hu… find
  19. IM Kamala Liyanage. 2014. Education system of sri lanka: strengths and weaknesses. Educ Syst Sri Lanka, pages 116–40. find
  20. Llama4Herd. 2025. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https://ai.meta.com/blog/ llama-4-multimodal-intelligence/. Accessed 2025-10-01. ↗
  21. OpenAI. 2022. ChatGPT: Optimizing Language Models for Dialogue. Accessed: 2024-08-15. find
  22. OpenAI. 2025. OpenAI Introducing https://openai.com/index/ GPT5. introducing-gpt-5/. Accessed 2025-10-22. ↗
  23. Ashmari Pramodya, Nirasha Nelki, Heshan Shalinda, Chamila Liyanage, Yusuke Sakai, Randil Pushpananda, Ruvan Weerasinghe, Hidetaka Kamigaito, and Taro Watanabe. 2025. SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala. find
  24. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng… find
  25. Nitin Rane, Saurabh Choudhary, and Jayesh Rane. 2024. Gemini versus chatgpt: applications, performance, architecture, capabilities, and implementation. Journal of Applied Artificial Intelligence, 5:69–93. find
  26. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Chosh… — Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation
  27. Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. 2025. KMMLU: Measuring massive multitask language understanding in Korean. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Ass…
  28. Yosephine Susanto, Adithya Venkatadri Hulagadri, Jann Railey Montalan, Jian Gang Ngui, Xianbin Yong, Wei Qi Leong, Hamsawardhini Rengarajan, Peerat Limkonchotiwat, Yifan Mai, and William Chandra Tjhi. 2025. SEA-HELM: Southeast Asian holistic evaluation of language models. In Findings of the Associat… find
  29. Ashima Suvarna, Harshita Khandelwal, and Nanyun Peng. 2024. PhonologyBench: Evaluating phonological skills of large language models. In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pages 1–14, Bangkok, Thailand. Association for Computational Linguistics. find
  30. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32.
  31. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,…
  32. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377–392.
  33. Ishaan Watts, Varun Gumma, Aditya Yadavalli, Vivek Seshadri, Manohar Swaminathan, and Sunayana Sitaram. 2024. PARIKSHA: A large-scale investigation of human-LLM evaluator agreement on multilingual and multi-cultural data. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language… find
  34. Arda Yüksel, Abdullatif Köksal, Lütfi Kerem Senel, Anna Korhonen, and Hinrich Schuetze. 2024. TurkishMMLU: Measuring massive multitask language understanding in Turkish. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7035–7055, Miami, Florida, USA. Association for Co… find
  35. Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298–1308, M…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/varsha-2026-evaluating-linguistic-knowledge-llms-tamil/