Elamur Research⌕
☰
home / papers / 2026 / The Riddle of Reflection: Evaluating Reasoning and Self-Awareness in Multilingual LLMs Using Indian Riddles

The Riddle of Reflection: Evaluating Reasoning and Self-Awareness in Multilingual LLMs Using Indian Riddles

Summary

Large language models are mostly benchmarked in English, so it is unclear how well they handle culturally rooted reasoning in other languages. To explore this, the paper turns to riddles in seven languages spoken in India, namely Tamil, Telugu, Hindi, Bengali, Gujarati, Kannada and Malayalam. The authors built a new dataset that pairs traditional riddles with context-reconstructed variants, following the RISCORE idea of keeping a riddle's logic while changing its theme. To decide which models would generate these variants, native-speaker experts blind-rated outputs from seven LLMs. Gemini 2.5 Pro and Gemini 2.5 Flash were chosen, and humans curated the riddles they produced.

Five models were tested on solving the riddles under seven prompting setups. They were then asked to judge whether their own answers were correct. Gemini 2.5 Pro solved the most riddles, but few-shot prompting helped only slightly, and accuracy differed sharply between languages. The models that solved the most riddles were the worst at spotting their own errors. This points to a gap between reasoning skill and self-awareness that matters for building reliable multilingual systems.

Key points

  • Models and prompts: Five models were evaluated: Gemini 2.5 Pro, Gemini 2.5 Flash, Mistral-Saba, LLaMA-4-Scout and LLaMA-4-Maverick. The seven prompting strategies were zero-shot, plus 2- and 4-shot versions with random, semantically similar or context-reconstructed exemplars. Solving was scored with accuracy and BERTScore.
  • Riddle-solving accuracy: Gemini 2.5 Pro was the strongest model, ranging from about 37% to 42% depending on the prompt. Its best score was 42.11% with 4-shot random exemplars. Averaged over all models, accuracy per strategy stayed within roughly 21–23%.
  • Differences by language: Averaged over strategies and models, Gujarati was easiest (36.01%) and Hindi came next (32.59%). Telugu (10.62%) and Kannada (12.98%) were the hardest.
  • Self-evaluation: Models gave a Yes/No verdict on their own answers, scored with TPR, TNR and balanced accuracy. Gemini 2.5 Pro caught very few of its own mistakes (4.34% True Negative Rate), while LLaMA-4-Scout caught far more (42.09%). Initial accuracy and the ability to detect one's own errors were inversely related.
  • Availability and limitations: Data and code are released at github.com/ltrc/Riddle_of_reflection_paper. The authors note several limitations: limited dialectal coverage, possible annotator subjectivity, heavy reliance on manual curation, and the risk that traditional riddles appeared in model training data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

2 papers of our library cite it (of the 140673 whose bibliographies we read).

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2pgfbjkdofoe

@inproceedings{m2026riddle,
  title = {{The Riddle of Reflection: Evaluating Reasoning and Self-Awareness in Multilingual LLMs Using Indian Riddles}},
  author = {M, Abhinav P and Saxena, Ojasva and C, Oswald and Krishnamurthy, Parameswari},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2pgfbjkdofoe},
  url = {https://doi.org/10.63317/2pgfbjkdofoe}
}

References

24 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. find our card
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  3. Alexander R Fabbri, Diego Mares, Jorge Flores, Meher Mankikar, Ernesto Hernandez, Dean Lee, Bing Liu, and Chen Xing. 2025. Multinrc: A challenging and native multilingual reasoning evaluation benchmark for llms. arXiv preprint arXiv:2507.17476. find
  4. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic bert sentence embedding. arXiv preprint arXiv:2007.01852. find our card
  5. Akash Ghosh, Debayan Datta, Sriparna Saha, and Chirag Agarwal. 2025. The multilingual mind: A survey of multilingual reasoning in language models. arXiv preprint arXiv:2502.09457. find
  6. Panagiotis Giadikiaroglou, Maria Lymperaiou, Giorgos Filandrianos, and Giorgos Stamou. 2024. Puzzle solving using reasoning of large language models: A survey. arXiv preprint arXiv:2402.11291. find
  7. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. find our card
  8. Hui Huang, Yingqi Qu, Jing Liu, Muyun Yang, Bing Xu, Tiejun Zhao, and Wenpeng Lu. 2024. Self-evaluation of large language model based on glass-box features. arXiv preprint arXiv:2403.04222. find
  9. Yifan Jiang, Filip Ilievski, Kaixin Ma, and Zhivar Sourati. 2023. BRAINTEASER: Lateral thinking puzzles for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14317–14332, Singapore. Association for Computational Linguistics. find
  10. Belinda Z Li, Been Kim, and Zi Wang. 2025. Questbench: Can llms ask the right question to acquire information in reasoning tasks? arXiv preprint arXiv:2503.22674. — [arXiv:2503.22674](https://arxiv.org/abs/2503.22674) find
  11. Bill Yuchen Lin, Ziyi Wu, Yichi Yang, Dong-Ho Lee, and Xiang Ren. 2021. Riddlesense: Reasoning about riddle questions featuring linguistic creativity and commonsense knowledge. arXiv preprint arXiv:2101.00376. find
  12. Chen Cecilia Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2023. Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings. arXiv preprint arXiv:2309.08591. find
  13. Zhendong Liu, Changhong Xia, Wei He, and Chongjun Wang. 2024. Trustworthiness and self-awareness in large language models: An exploration through the think-solve-verify framework. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluatio… find
  14. Jord Nguyen, Khiem Hoang, Carlo Leonardo Attubato, and Felix Hofstätter. 2025. Probing evaluation awareness of language models. arXiv preprint arXiv:2507.01786. — [arXiv:2507.01786](https://arxiv.org/abs/2507.01786) find
  15. Ioannis Panagiotopoulos, Giorgos Filandrianos, Maria Lymperaiou, and Giorgos Stamou. 2024. Riscore: Enhancing in-context riddle solving in language models through context-reconstructed example augmentation. arXiv preprint arXiv:2409.16383. find
  16. Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. 2022. Reasoning with language model prompting: A survey. arXiv preprint arXiv:2212.09597. find our card
  17. Jie Ren, Yao Zhao, Tu Vu, Peter J Liu, and Balaji Lakshminarayanan. 2023. Self-evaluation improves selective generation in large language models. In Proceedings on, pages 49–64. PMLR. find
  18. Lisa Schut, Yarin Gal, and Sebastian Farquhar. 2025. Do multilingual llms think in english? arXiv preprint arXiv:2502.15603. — [arXiv:2502.15603](https://arxiv.org/abs/2502.15603) find
  19. Harman Singh, Nitish Gupta, Shikhar Bharadwaj, Dinesh Tewari, and Partha Talukdar. 2024. Indicgenbench: A multilingual benchmark to evaluate generation capabilities of llms on indic languages. arXiv preprint arXiv:2404.16816. find
  20. Nemika Tyagi, Mihir Parmar, Mohith Kulkarni, Aswin Rrv, Nisarg Patel, Mutsumi Nakamura, Arindam Mitra, and Chitta Baral. 2024. Step-by-step reasoning to solve grid puzzles: Where do llms falter? arXiv preprint arXiv:2407.14790. find
  21. Vijay Viswanathan, Kiril Gashteovski, Kiril Gashteovski, Carolin Lawrence, Tongshuang Wu, and Graham Neubig. 2024. Large language models enable few-shot clustering. Transactions of the Association for Computational Linguistics, 12:321–333. find
  22. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. find our card
  23. Zheng-Xin Yong, M Farid Adilazuarda, Jonibek Mansurov, Ruochen Zhang, Niklas Muennighoff, Carsten Eickhoff, Genta Indra Winata, Julia Kreutzer, Stephen H Bach, and Alham Fikri Aji. 2025. Crosslingual reasoning through test-time scaling. arXiv preprint arXiv:2505.05408. — [arXiv:2505.05408](https://arxiv.org/abs/2505.05408) find
  24. Yunxiang Zhang and Xiaojun Wan. 2022. Birdqa: A bilingual dataset for question answering on tricky riddles. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11748–11756. find

↑ back to top