Elamur Research⌕
☰
home / papers / 2026 / NeCCo: Nepali Cultural Commonsense Benchmark for Large Language Model Evaluation

NeCCo: Nepali Cultural Commonsense Benchmark for Large Language Model Evaluation

Paper · 2026 · Sanket Shrestha, Raunak Regmi, Sadikshya Ghimire et al.

Summary

Large language models can score well on multilingual benchmarks yet miss the everyday cultural knowledge Nepali speakers rely on without stating it. Examples include kinship terms that separate paternal from maternal and elder from younger relatives, honorific choice, ritual etiquette, idioms, and food and farming practices. Existing commonsense and Indic benchmarks are mostly English-centric, translation-based or pooled across many languages, so this gap stays hidden.

The authors built NeCCo, a native-speaker-authored and cross-reviewed benchmark of four-option questions. Each question has one correct answer and culturally plausible distractors, and the set spans five cultural domains. Items are provided in Devanagari, Romanized Nepali and, where meaning survives translation, English.

Several state-of-the-art LLMs were tested with a fixed 2-shot prompt. Models did better on globally documented knowledge but faltered on relational and implicit reasoning such as extended kinship and proverbs, and they hallucinated more in culturally dense categories. Devanagari input generally beat English, while Romanized input lowered accuracy. The authors conclude that culturally representative data matters more than model scale or translation for low-resource cultural competence.

Key points

  • Size and format: NeCCo contains 1,295 multiple-choice questions (options A–D), giving 3,460 released items in total:
    • 1,295 in Devanagari
    • 1,295 in Romanized Nepali
    • 870 in English
  • Domains:
    • family relationships and social rank
    • geography, along with rituals and festivals
    • proverbs, idioms and metaphors
    • everyday life and commonsense
    • food, farming and the natural world
  • Withheld translations: the kinship and idioms categories have no English version, to avoid losing semantic nuance.
  • Script effect: native Devanagari prompts usually outperform English, and Romanized Nepali consistently degrades results. Agriculture, kinship and local geography are among the weakest areas.
  • Limitations:
    • annotation was done only by the authors, with no inter-annotator agreement reported
    • answer option order was not randomized
    • evaluation used a single 2-shot setup with default inference settings
  • Availability: data and code are on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Birmingham City UniversityBirmingham · United Kingdom5 works · 26 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksEvaluation & benchmarks · 1,220 worksIdioms & multiword expressions · 217 works

Cite

Cite as Sanket Shrestha, Raunak Regmi, Sadikshya Ghimire, Satyam Rana and Supriya Khadka. 2026. NeCCo: Nepali Cultural Commonsense Benchmark for Large Language Model Evaluation. Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3zsmcs7pxtgi

@inproceedings{shrestha2026necco,
  title = {{NeCCo: Nepali Cultural Commonsense Benchmark for Large Language Model Evaluation}},
  author = {Sanket Shrestha, Raunak Regmi, Sadikshya Ghimire, Satyam Rana and Supriya Khadka},
  booktitle = {Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)},
  year = {2026},
  doi = {10.63317/3zsmcs7pxtgi},
  url = {https://doi.org/10.63317/3zsmcs7pxtgi}
}

References

38 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Co… — gpt-oss-120b & gpt-oss-20b Model Card
  2. Samuel Cahyawijaya. 2024. Llm for everyone: Representing the underrepresented in large language models. Hong Kong University of Science and Technology (Hong Kong). find
  3. Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. 2025. Culturalbench: A robust, diverse, and challenging cultural benchmark by human-ai culturalteaming. find
  4. Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. Tydi qa: A benchmark for information-seeking question answering in ty pologically di verse languages. Transactions of the Association for Computational Linguistics, 8:454–47… find
  5. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. The flores evaluation datasets for low-resource machine translation: Nepali–english and sinhala–english. In Proceedings of the 2019 conference on empirical method…
  6. Alaa Hesham and Abeer Hamdy. 2024. Fine-tuning GPT-4o-Mini for programming questions generation. In Proceedings of the 2024 International Conference on Computer and Applications (ICCA), pages 1–6. find
  7. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International conference on machine learning, pages 4411–4421. PMLR. find
  8. Neha Joshi, Pamir Gogoi, AasimBaig Mirza, Aayush Jansari, Aditya Yadavalli, Ayushi Pandey, Arunima Shukla, Deepthi Sudharsan, Kalika Bali, and Vivek Seshadri. 2025. Elr-1000: A community-generated dataset for endangered indic indigenous languages. In Proceedings of the 14th International Joint Confe… find
  9. Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaibhav Mehta, Lalit K Jain, Virginia Aglietti, Disha Jindal, Yuanzhu Peter Chen, et al. 2025. Big-bench extra hard. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisti…
  10. Supriya Khadka and Bijayan Bhattarai. 2025. Gender bias in nepali-english machine translation: A comparison of llms and existing mt systems. In Proceedings of the 6th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 75–82. find
  11. Supriya Khadka, Ranju G.C., Prabin Paudel, Rahul Shah, and Basanta Joshi. 2023. Nepali text-to-speech synthesis using tacotron2 for mel-spectrogram generation. In SIGUL 2023, 2nd Annual Meeting of the Special Interest Group on Under-resourced Languages: a Satellite Workshop of Interspeech 2023. find
  12. Ju-Chun Ko. 2026. Bilingual bias in large language models: A taiwan sovereignty benchmark study. arXiv preprint arXiv:2602.06371. — [arXiv:2602.06371](https://arxiv.org/abs/2602.06371) find
  13. Britt Krause. 1980. Kinship, hierarchy and equality in north western nepal. Contributions to Indian Sociology, 14(2):169–194. find
  14. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, et al. 2020. Xglue: A new benchmark dataset for cross-lingual pre-training, understanding and generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Langua…
  15. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. 2025. Deepseek-v3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556.
  16. Xingjun Ma, Yixu Wang, Hengyuan Xu, Yutao Wu, Yifan Ding, Yunhan Zhao, Zilong Wang, Jiabin Hua, Ming Wen, Jianan Liu, Ranjie Duan, Yifeng Gao, Yingshui Tan, Yunhao Chen, Hui Xue, Xin Wang, Wei Cheng, Jingjing Chen, Zuxuan Wu, Bo Li, and Yu-Gang Jiang. 2026. A safety report on GPT-5.2, Gemini 3 Pro,… — [arXiv:2601.10527](https://arxiv.org/abs/2601.10527) find
  17. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. Bbq: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105.
  18. Yash Madhani, Sushane Parthan, Priyanka Bedekar, Gokul Nc, Ruchi Khapra, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh M Khapra. 2023. Aksharantar: Open indic-language transliteration datasets and models for the next billion users. In Findings of the association for computational linguistics: Emnlp… find
  19. Uwe Peters, Alexander Krauss, and Oliver Bra-ganza. 2022. Generalization bias in science. Cognitive science, 46(9):e13188. find
  20. Arijit Maji, Raghvendra Kumar, Akash Ghosh, Anushka Anushka, and Sriparna Saha. 2025. Sanskriti: A comprehensive benchmark for evaluating language models’ knowledge of indian culture. In Findings of the Association for Computational Linguistics: ACL 2025, pages 4434– 4451. find
  21. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. Stereoset: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long…
  22. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel Bowman. 2020. Crows-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 1953– 1967.
  23. Jinu Nyachhyon, Mridul Sharma, Prajwal Thapa, and Bal Krishna Bal. 2025. Consolidating and developing benchmarking datasets for the nepali natural language understanding tasks. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia… find
  24. OpenRouter. n.d. Llm rankings. https://openrouter.ai/rankings. Accessed: 2026-02-21. ↗
  25. Shubham Parashar, Blake Olson, Sambhav Khurana, Eric Li, Hongyi Ling, James Caverlee, and Shuiwang Ji. 2025. Inference-time computations for llm reasoning and planning: A benchmark and insights. arXiv preprint arXiv:2502.12521. — [arXiv:2502.12521](https://arxiv.org/abs/2502.12521) find
  26. Pouya Pezeshkpour and Estevam Hruschka. 2024. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017.
  27. Mengyao Piao, Lingqi Miao, Yilun Liu, Minggui He, Hongxia Ma, Li Zhang, Daimeng Wei, and Shimin Tao. Beyond the west: A survey of cultural datasets for culturally-grounded llms. find
  28. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. Xcopa: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376.
  29. Greeshma Prabha, PV Jyothsna, KK Shahina, B Premjith, and KP Soman. 2018. A deep learning approach for part-of-speech tagging in nepali language. In 2018 international conference on advances in computing, communications and informatics (ICACCI), pages 1132– 1136. IEEE. find
  30. Ashish Pradhan and Archit Yajnik. 2024. Parts-of-speech tagging of nepali texts with bidirectional lstm, conditional random fields and hmm. Multimedia Tools and Applications, 83(4):9893– 9909. find
  31. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processi… find
  32. Tej Bahadur Shahi and Chiranjibi Sitaula. 2022. Natural language processing for nepali text: A review. Artificial Intelligence Review, 55(4):3401–3429. find
  33. Oyesh Mann Singh, Ankur Padia, and Anupam Joshi. 2019. Named entity recognition for nepali language. In 2019 IEEE 5th international conference on collaboration and internet computing (cic), pages 184–190. IEEE. find
  34. Madhavendra Thakur. 2025. Culturally-grounded chain-of-thought (cg-cot): Enhancing llm performance on culturally-specific tasks in arXiv preprint low-resource languages. arXiv:2506.01190. — [arXiv:2506.01190](https://arxiv.org/abs/2506.01190) find
  35. Surendrabikram Thapa, Kritesh Rauniyar, Hariram Veeramani, Surabhi Adhikari, Imran Razzak, and Usman Naseem. 2025. Probing the limits of multilingual language understanding: Low-resource language proverbs as llm benchmark for ai wisdom. In Proceedings of the 6th Workshop on Computational Approaches… find
  36. Sulav Timilsina, Milan Gautam, and Binod Bhattarai. 2022. Nepberta: Nepali language model trained in a large corpus. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Proce… find
  37. Armin Toroghi, Ali Pesaranghader, Tanmana Sadhu, and Scott Sanner. 2025. Llm-based typed hyperresolution for commonsense reasoning with knowledge bases. In The Thirteenth International Conference on Learning Representations. find
  38. Sshubam Verma, Mohammed Safi Ur Rahman Khan, Vishwajeet Kumar, Rudra Murthy, and Jaydeep Sen. 2025. Milu: A multi-task indic language understanding benchmark. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shrestha-2026-necco-nepali-cultural-commonsense-benchmark/