Elamur Research⌕
☰
home / papers / 2026 / A Comprehensive Evaluation of Chain-of-Thought Faithfulness in Persian Classification Tasks

A Comprehensive Evaluation of Chain-of-Thought Faithfulness in Persian Classification Tasks

Paper · 2026 · Shakib Yazdani, Cristina España-Bonet, Eleftherios Avramidis et al.

Summary

Chain-of-thought (CoT) output is often treated as an explanation of how a language model reached its answer. However, research on whether these traces are faithful has focused almost entirely on English. This paper presents the first broad study of CoT faithfulness for Persian. The authors took 15 classification datasets from the FaMTEB benchmark: seven pre-existing human-annotated sets and eight synthetic, chatbot-oriented ones. Together they cover sentiment, emotion, topic, tone and satisfaction labels. Six open models in three classes (small, large, reasoning) were run with a fixed zero-shot CoT template, prompted in both Persian and English, and their reasoning and predicted labels were stored. A separate, larger model then judged whether each trace actually justifies its label, and a manual check on one dataset validated the judge. Faithfulness varied considerably and depended mostly on the dataset and model class, with prompt language mattering much less. Small models were never more faithful than larger ones. The findings caution against reading CoT as trustworthy explanation in low-resource languages.

Key points

  • Models and setup: qwen3:4b and gemma3:4b (SLMs), qwen3:30b in non-thinking mode and gemma3:27b (LLMs), and deepseek-r1:32b and gpt-oss:20b (LRMs). All were served through Ollama with temperature 0.5, top-p 0.9 and seed 42.
  • Judge: Llama-3.3:70b was used as the judge. It is larger than every evaluated model and comes from a different model family. The choice follows Shen et al. (2025), who report that LLM-as-a-judge outperforms counterfactual- and logit-based methods.
  • Task performance: LLMs and LRMs achieved the strongest results. gpt-oss:20b showed the largest gains on the synthetic SynPer* datasets.
  • Human validation: The first author, a native Persian speaker, annotated 100 sampled traces from PersianTextEmotion as Faithful or Non-Faithful. Agreement with the judge was described as fair to moderate.
  • Error analysis: Unfaithful traces fell into four patterns: truncated reasoning, reasoning unrelated to the candidate labels, insufficient evidence, and post-hoc justification. The authors also show that a faithful trace can still end in a wrong prediction. Code and prompts are available on GitHub (shakibyzn/persian-faithful-cot).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,685 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Shakib Yazdani, Cristina España-Bonet, Eleftherios Avramidis, Yasser Hamidullah and Josef van Genabith. 2026. A Comprehensive Evaluation of Chain-of-Thought Faithfulness in Persian Classification Tasks. Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.loreslm-1.27

@inproceedings{yazdani2026comprehensive,
  title = {{A Comprehensive Evaluation of Chain-of-Thought Faithfulness in Persian Classification Tasks}},
  author = {Shakib Yazdani, Cristina España-Bonet, Eleftherios Avramidis, Yasser Hamidullah and Josef {van Genabith}},
  booktitle = {Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loreslm-1.27},
  url = {https://doi.org/10.18653/v1/2026.loreslm-1.27}
}

References

42 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy. 2025. Chain-of-thought reasoning in the wild is not always faithful. In Workshop on Reasoning and Planning for Large Language Models. find
  2. Fazl Barez, Tung-Yu Wu, Iván Arcuschin, and 1 others. 2025. Chain-of-Thought is not explainability. arXiv preprint arXiv:2501.xxxx. find
  3. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Sur… — LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
  4. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. DeepSeek-R1: Incentivizing reasoning capability in LL…
  5. Mehrdad Farahani, Mohammad Gharachorloo, Marzieh Farahani, and Mohammad Manthouri. 2020. Pars-BERT: Transformer-based model for Persian language understanding. Neural Processing Letters, 53:3831–3847. find
  6. Xiyan Fu and Wei Liu. 2025. How reliable is multilingual LLM-as-a-judge? In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 11040–11053, Suzhou, China. Association for Computational Linguistics. find
  7. Omid Ghahroodi, Marzia Nouri, Mohammad Vali Sanian, Alireza Sahebi, Doratossadat Dastgheib, Ehsaneddin Asgari, Mahdieh Soleymani Baghshah, and Mohammad Hossein Rohban. 2024. Khayyam challenge (persianMMLU): Is your LLM truly wise to the persian language? In First Conference on Language Modeling. find
  8. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  9. Sara Bourbour Hosseinbeigi, Behnam Rohani, Mostafa Masoudi, Mehrnoush Shamsfard, Zahra Saaberi, Mostafa Karimi Manesh, and Mohammad Amin Abbasi. 2025a. Advancing Persian LLM evaluation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2711–2727, Albuquerque, New Mexico… find
  10. Sara Bourbour Hosseinbeigi, MohammadAli SeifKashani, Javad Seraj, Fatemeh Taherinezhad, Ali Nafisi, Fatemeh Nadi, Iman Barati, Hosein Hasani, Mostafa Amiri, and Mostafa Masoudi. 2025b. Matina: A culturally-aligned Persian language model using multiple LoRA experts. In Findings of the Association for… find
  11. Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198–4205, Online. Association for Computational Linguistics.
  12. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems.
  13. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, an… — Measuring Faithfulness in Chain-of-Thought Reasoning
  14. Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of-thought reasoning. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter…
  15. Andreas Madsen, Sarath Chandar, and Siva Reddy. 2024. Are self-explanations from large language models faithful? In Findings of the Association for Computational Linguistics: ACL 2024, pages 295–337, Bangkok, Thailand. Association for Computational Linguistics. find
  16. Ollama. 2024. Openai compatibility. Accessed: 2025-12-22. find
  17. Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh, and 49 others. 20… — Olmo 3
  18. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, and 108 others. 2025. gpt-oss-120b & gpt-oss-20b model…
  19. Letitia Parcalabescu and Anette Frank. 2024. On measuring faithfulness or self-consistency of natural language explanations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6048– 6089, Bangkok, Thailand. Association for Comput… find
  20. Debjit Paul, Robert West, Antoine Bosselut, and Boi Faltings. 2024. Making reasoning matter: Measuring and improving faithfulness of chain-of-thought reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15012– 15032, Miami, Florida, USA. Association for Computat…
  21. Pedram Rostami, Ali Salemi, and Mohammad Javad Dousti. 2024. PersianMind: A cross-lingual Persian-English Large Language Model. arXiv preprint arXiv:2401.06466. — [arXiv:2401.06466](https://arxiv.org/abs/2401.06466) find
  22. Javad Pourmostafa Roshan Sharami, Parsa Abbasi Sarabestani, and Seyed Abolghasem Mirroshandel. 2020. DeepSentiPers: Novel deep learning models trained over proposed augmented Persian sentiment corpus. arXiv preprint arXiv:2004.05328. — [arXiv:2004.05328](https://arxiv.org/abs/2004.05328) find
  23. Aryan Shekarlaban and Pooya Mohammadi Kazaj. 2023. Hezar: The all-in-one ai library for persian. https://github.com/hezarai/hezar. ↗
  24. Xu Shen, Song Wang, Zhen Tan, and 1 others. 2025. FaithCoT-Bench: Benchmarking instance-level faithfulness of Chain-of-Thought reasoning. arXiv preprint arXiv:2510.04040. — [arXiv:2510.04040](https://arxiv.org/abs/2510.04040) find
  25. Noah Siegel, Oana-Maria Camburu, Nicolas Heess, and Maria Perez-Ortiz. 2024. The probabilities also matter: A more faithful metric for faithfulness of free-text explanations in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2… find
  26. Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2025. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations. find
  27. Sree Harsha Tanneru, Dan Ley, Chirag Agarwal, and Himabindu Lakkaraju. 2024. On the hardness of faithful chain-of-thought reasoning in large language models. arXiv preprint arXiv:2406.10625. — [arXiv:2406.10625](https://arxiv.org/abs/2406.10625) find
  28. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  29. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. In Thirty-seventh Conference on Neural Information Processing Systems.
  30. Martin Tutek, Fateme Hashemi Chaleshtori, Ana Marasovic, and Yonatan Belinkov. 2025. Measuring chain of thought faithfulness by unlearning reasoning steps. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 9946–9971, Suzhou, China. Association for Compu…
  31. Fali Wang, Minhua Lin, Yao Ma, Hui Liu, Qi He, Xianfeng Tang, Jiliang Tang, Jian Pei, and Suhang Wang. 2025. A survey on small language models in the era of large language models: Architecture, capabilities, and trustworthiness. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery… find
  32. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
  33. Xuezhi Wang and Denny Zhou. 2024. Chain-of-thought reasoning without prompting. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  34. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
  35. Xumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye, Zhirong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, Ziming Miao, Jiang Bian, and Mao Yang. 2025. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms. arXiv preprint arXiv:2506.14245. find
  36. Zidi Xiong, Shan Chen, Zhenting Qi, and Himabindu Lakkaraju. 2025. Measuring the faithfulness of thinking drafts in large reasoning models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  37. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  38. Sohee Yang, Sang-Woo Lee, Nora Kassner, Daniela Gottesman, Sebastian Riedel, and Mor Geva. 2025b. How well can reasoning models identify and recover from unhelpful thoughts? In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 7030–7047, Suzhou, China. Association for Comp… find
  39. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems. find
  40. Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovec, Percy Liang, Ed H. Chi, and Denny Zhou. 2024. Large language models as analogical reasoners. In The Twelfth International Conference on Learning Representations.
  41. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. 2025. Justice or prejudice? quantifying biases in LLM-as-a-judge. In The Thirteenth International Conference on Learning Representations.
  42. Erfan Zinvandi, Morteza Alikhani, Mehran Sarmadi, Zahra Pourbahman, Sepehr Arvin, Reza Kazemi, and Arash Amini. 2025. FaMTEB: Massive text embedding benchmark in Persian language. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 11441–11468, Suzhou, China. Association… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yazdani-2026-comprehensive-evaluation-chain-thought-faithfulness/