Elamur Research⌕
☰
home / papers / 2026 / Can LLMs Faithfully Explain Themselves in Low-Resource Languages? A Case Study on Emotion Detection in Persian

Can LLMs Faithfully Explain Themselves in Low-Resource Languages? A Case Study on Emotion Detection in Persian

Summary

The paper asks whether the explanations LLMs give for their own decisions reflect how they actually decide, when the language is low-resource. The test case is emotion classification in Persian. The authors use a Persian emotion-labeled corpus containing both formal and informal texts. Models assign each text to one of six emotions and name the top-k input words behind that choice. Two prompt orders are compared: Predict-then-Explain (P-E) and Explain-then-Predict (E-P).

Faithfulness is measured by re-classifying each text with only the chosen words kept and with those words masked. Changes in prediction and confidence show whether the words carried the decision. Confidence comes from token-level log-probabilities rather than self-reported certainty. Because the raw scores were overconfident, they are corrected with temperature scaling. A two-stage human annotation, built to match the model output format, supplies reference influential words.

The models classify emotions well, but their explanations often do not match their actual reasoning. They also agree more with one another than with human annotators. The authors conclude that current explanation methods and metrics need improvement before LLMs can be trusted in multilingual, low-resource settings.

Key points

  • Six emotion classes are used: Sadness, Happiness, Anger, Surprise, Hatred, Fear. Labels are returned as numeric codes 0–5.
  • Models evaluated include GPT-4-turbo, GPT-4o, Llama3.3-70B-instruct and DeepSeek-V3, all accessed via API.
  • There are three input conditions: Full-text, Top-k only, and Top-k removed, where the chosen words are replaced by a [removed] placeholder. Each condition runs in a fresh conversation.
  • Overconfidence is reduced with post-hoc temperature scaling (Guo et al., 2017), tuned on a held-out calibration set by minimizing Expected Calibration Error. Reliability diagrams are reported per model and paradigm.
  • The authors present the work as the first to combine log-probability-based, calibrated confidence with self-explanation evaluation. It is also among the first such studies on Persian emotion detection.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4tds2kffmg7j

@inproceedings{mehrazar2026can,
  title = {{Can LLMs Faithfully Explain Themselves in Low-Resource Languages? A Case Study on Emotion Detection in Persian}},
  author = {Mehrazar, Mobina and Yousefi, Mohammad Amin and Beygi, Parisa and Bahrak, Behnam},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4tds2kffmg7j},
  url = {https://doi.org/10.63317/4tds2kffmg7j}
}

References

26 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-SNLI: Natural Language Inference with Natural Language Explanations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31,… find our card
  2. George Chrysostomou and Nikolaos Aletras. 2021. Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text Classification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on… find
  3. Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020. ERASER: A Benchmark to Evaluate Rationalized NLP Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4443–4458, Online. Asso… find
  4. DeepSeek-AI et al. 2025. DeepSeek-V3 Technical Report. find
  5. OpenAI et al. 2024a. GPT-4 Technical Report. find our card
  6. OpenAI et al. 2024b. GPT-4o System Card. find our card
  7. Aaron Grattafiori, Abhimanyu Dubey, Hugo Touvron, Angela Fan, Gabriel Synnaeve, Sergey Edunov, Armen Aghajanyan, Thomas Scialom, Roberta Raileanu, Soumith Chintala, and Meta AI. 2024. The Llama 3 Herd of Models. find
  8. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 1321–1330. JMLR.org. find
  9. Henning Heyen, Amy Widdicombe, Noah Yamamoto Siegel, Philip Colin Treleaven, and Maria Perez-Ortiz. 2024. The Effect of Model Size on LLM post-hoc Explainability via LIME. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models. find
  10. Miriam Horovicz and Roni Goldshmidt. 2024. TokenSHAP: Interpreting Large Language Models with Monte Carlo Shapley Value Estimation. In Proceedings of the 1st Workshop on NLP for Science (NLP4Science), pages 1–8, Miami, FL, USA. Association for Computational Linguistics. find
  11. Shiyuan Huang, Siddarth Mamidanna, Shreedhar Jangam, Yilun Zhou, and Leilani H. Gilpin. 2023. Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations. find
  12. Pieter-Jan Kindermans, Kristof Schütt, Klaus-Robert Müller, and Sven Dähne. 2016. Investigating the Influence of Noise and Distractors on the Interpretation of Neural Networks. find
  13. Satyapriya Krishna, Tessa Han, Alex Gu, Steven Wu, Shahin Jabbari, and Himabindu Lakkaraju. 2024. The Disagreement Problem in Explainable Machine Learning: A Practitioner’s Perspective. Transactions on Machine Learning Research. find
  14. Jiwei Li, Will Monroe, and Dan Jurafsky. 2017. Understanding Neural Networks through Representation Erasure. find
  15. Chen Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2024. Are Multilingual LLMs Culturally-diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human L… find
  16. Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. find our card
  17. Andreas Madsen, Sarath Chandar, and Siva Reddy. 2024. Are Self-explanations from Large Language Models Faithful? In Findings of the Association for Computational Linguistics: ACL 2024, pages 295–337, Bangkok, Thailand. Association for Computational Linguistics. find
  18. Hossein Mirzaee, Javad Peymanfard, Hamid Habibzadeh Moshtaghin, and Hossein Zeinali. 2025. ArmanEmo: a Persian Dataset for Text-based Emotion Detection. Language Resources and Evaluation. find
  19. Korbinian Randl, John Pavlopoulos, Aron Henriksson, and Tony Lindgren. 2025. Evaluating the Reliability of Self-explanations in Large Language Models. In Discovery Science, pages 36–51, Cham. Springer Nature Switzerland. find
  20. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why Should I Trust You?": Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 1135–1144, New York, NY, USA. Association for Co… find our card
  21. Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning Important Features Through Propagating Activation Differences. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 3145–3153. JMLR.org. find our card
  22. Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. 2020. Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, AIES ’20, page 180–186, New York, NY, USA. Association for Computing… find
  23. Jeevithashree Divya Venkatesh, Aparajita Jaiswal, and Gaurav Nanda. 2024. Comparing Human Text Classification Performance and Explainability with Large Language and Machine Learning Models Using Eye-tracking. Scientific Reports, 14:14295. find
  24. Boyang Xue, Hongru Wang, Rui Wang, Sheng Wang, Zezhong Wang, Yiming Du, Bin Liang, and Kam-Fai Wong. 2024. A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models. find
  25. Kaitlyn Zhou, Jena Hwang, Xiang Ren, and Maarten Sap. 2024. Relying on the Unreliable: The Impact of Language Models’ Reluctance to Express Uncertainty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3623–3643, Bangkok, Thail… find
  26. Input Format Average Reduction P-E E-P Full-text 16.60 12.17 GPT-4o Top-k only 8.77 7.07 Top-k removed 12.32 9.85 Full-text 39.03 20.65 GPT-4-turbo Top-k only 17.85 10.95 Top-k removed 9.67 5.03 Full-text 36.82 35.22 Llama3.3 70B-instruct Top-k only 18.71 14.99 Top-k removed 17.71 22.71 Full-text 33… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/mehrazar-2026-can-llms-faithfully-explain-themselves/