Elamur Research⌕
☰
home / papers / 2026 / LIBERTy: A Causal Framework for Benchmarking Concept-Based Explanations of LLMs with Structural Counterfactuals

LIBERTy: A Causal Framework for Benchmarking Concept-Based Explanations of LLMs with Structural Counterfactuals

Paper · 2026 · Gilat Toker, Nitay Calderon, Ohad Amosy et al.

Summary

Judging whether a concept-based explanation of an LLM is faithful requires the true causal effect of a concept, such as gender or experience, on the model's output. That effect is normally unknown. Earlier benchmarks such as CEBaB approximate it with human-written counterfactuals of short restaurant reviews, using a small, flat causal graph. The authors propose LIBERTy, a framework that defines an explicit structural causal model (SCM) of text generation. In it, an LLM turns concept values into text, and templated real-world text and author personas act as exogenous noise. A concept is intervened on and the change is propagated through the SCM, with noise held fixed and decoding at temperature zero. This produces structural counterfactuals and reference causal effects without human annotation. The authors release three datasets and a new metric, then benchmark many explanation methods on five models. Matching on representations from a dataset-specific fine-tuned model performs best, but large room for improvement remains. The resource offers a scalable, causally grounded testbed for building more faithful explainability methods.

Key points

  • Datasets: three domains (disease detection, CV screening, workplace violence prediction), each built from a hand-specified causal graph grounded in domain knowledge.
  • New metric: order-faithfulness checks whether a method ranks concept-intervention effects correctly. This makes it usable for methods whose importance scores are on arbitrary scales.
  • Methods compared: approaches that remove concepts with linear projections, produce counterfactual texts, pair similar examples, or assign importance scores to concepts, all evaluated against individual-level (ICaCE) and population-level (CaCE) effects.
  • Model sensitivity: zero-shot LLMs (Llama-3.1-8B, GPT-4o) barely react to interventions on Race, Gender and Age, plausibly due to post-training alignment. Among fine-tuned models, Qwen2.5-1.5B tracks the gold effects most closely, though a gap remains.
  • Data quality and code: a human evaluation found the synthetic texts coherent and fluent, with counterfactuals differing in only one concept. Code is available at github.com/GilatToker/Liberty-benchmark.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technion – Israel Institute of TechnologyHaifa · Israel49 works · 5,306 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 works

Cite

Cite as Gilat Toker, Nitay Calderon, Ohad Amosy and Roi Reichart. 2026. LIBERTy: A Causal Framework for Benchmarking Concept-Based Explanations of LLMs with Structural Counterfactuals. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.399

@inproceedings{toker2026liberty,
  title = {{LIBERTy: A Causal Framework for Benchmarking Concept-Based Explanations of LLMs with Structural Counterfactuals}},
  author = {Gilat Toker, Nitay Calderon, Ohad Amosy and Roi Reichart},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.399},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.399}
}

References

71 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eldar David Abraham, Karel D’Oosterlinck, Amir Feder, Yair Ori Gat, Atticus Geiger, Christopher Potts, Roi Reichart, and Zhengxuan Wu. 2022. Cebab: Estimating the causal effects of real-world concepts on NLP model behavior. CoRR, abs/2205.14140. find
  2. Ahmed Alqaraawi, Martin Schuessler, Philipp Weiß, Enrico Costanza, and Nadia Berthouze. 2020. Evaluating saliency map explanations for convolutional neural networks: A user study. CoRR, abs/2002.00772. — [arXiv:2002.00772](https://arxiv.org/abs/2002.00772) find
  3. Esma Balkir, Svetlana Kiritchenko, Isar Nejadgholi, and Kathleen C. Fraser. 2022. Challenges in applying explainability methods to improve the fairness of NLP models. CoRR, abs/2206.03945. find
  4. Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. 2023. LEACE: perfect linear concept erasure in closed form. CoRR, abs/2306.03819. — [arXiv:2306.03819](https://arxiv.org/abs/2306.03819) find
  5. Jin Du, Li Chen, Xun Xian, An Luo, Fangqiao Tian, Ganghua Wang, Charles Doss, Xiaotong Shen, and Jie Ding. 2025. Ice cream doesn’t cause drowning: Benchmarking llms against statistical pitfalls in causal inference. CoRR, abs/2505.13770. — [arXiv:2505.13770](https://arxiv.org/abs/2505.13770) find
  6. Kenza Benkirane, Jackie Kay, and María Pérez-Ortiz. 2024. How can we diagnose and treat bias in large language models for clinical decision-making? CoRR, abs/2410.16574. find
  7. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 82 others. 2024… — The Llama 3 Herd of Models
  8. Diane Bouchacourt and Ludovic Denoyer. 2019. EDUCE: explaining model decisions through unsupervised concepts extraction. CoRR, abs/1905.11852. — [arXiv:1905.11852](https://arxiv.org/abs/1905.11852) find
  9. Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. 2021. Causalm: Causal model explanation through counterfactual language models. Comput. Linguistics, 47(2):333–386.
  10. Roger K Cady and Curtis P Schreiber. 2002. Sinus headache or migraine? considerations in making a differential diagnosis. Neurology, 58(9_suppl_6):S10–S14. find
  11. Yair Ori Gat, Nitay Calderon, Amir Feder, Alexander Chapanin, Amit Sharma, and Roi Reichart. 2023. Faithful explanations of black-box NLP models using llm-generated counterfactuals. CoRR, abs/2310.00603. — [arXiv:2310.00603](https://arxiv.org/abs/2310.00603) find
  12. Nitay Calderon, Liat Ein-Dor, and Roi Reichart. 2025. Multi-domain explainability of preferences. CoRR, abs/2505.20088. find
  13. Nitay Calderon and Roi Reichart. 2024. On behalf of the stakeholders: Trends in NLP model interpretability in the era of llms. CoRR, abs/2407.19200. find
  14. Fateme Hashemi Chaleshtori, Atreya Ghosal, Alexander Gill, Purbid Bambroo, and Ana Marasovic. 2024. On evaluating explanation utility for human-ai decision making in NLP. CoRR, abs/2407.03545. find
  15. Ivi Chatzi, Nina L. Corvelo Benz, Eleni Straitouri, Stratis Tsirtsis, and Manuel Gomez-Rodriguez. 2025. Counterfactual token generation in large language models. In Causal Learning and Reasoning, Lausanne, Switzerland, 7-9 May 2025, volume 275 of Proceedings of Machine Learning Research, pages 1291–… find
  16. Giandomenico Cornacchia, Vito Walter Anelli, Fedelucio Narducci, Azzurra Ragone, and Eugenio Di Sciascio. 2023. Counterfactual reasoning for bias evaluation and detection in a fairness under unawareness setting. CoRR, abs/2302.08204. — [arXiv:2302.08204](https://arxiv.org/abs/2302.08204) find
  17. Bo Cowgill, Fabrizio Dell’Acqua, Samuel Deng, Daniel Hsu, Nakul Verma, and Augustin Chaintreau. 2020. Biased programmers? or biased data? A field experiment in operationalizing AI ethics. CoRR, abs/2012.02394. — [arXiv:2012.02394](https://arxiv.org/abs/2012.02394) find
  18. Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Durrani, Jia Xu, and Hassan Sajjad. 2022. Discovering latent concepts learned in BERT. CoRR, abs/2205.07237. — [arXiv:2205.07237](https://arxiv.org/abs/2205.07237) find
  19. Jeffrey Dastin. 2018. Amazon scrapped ‘ai’ recruiting tool that showed bias against women. find
  20. S Gerberich, T Church, P McGovern, and et al. 2004. An epidemiological study of the magnitude and consequences of work related violence: the minnesota nurses’ study. Occupational and Environmental Medicine, 61(6):495–503. find
  21. Amirata Ghorbani, James Wexler, James Y. Zou, and Been Kim. 2019. Towards automatic concept-based explanations. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 92… find
  22. Yash Goyal, Uri Shalit, and Been Kim. 2019. Explaining classifiers with causal concept effect (cace). CoRR, abs/1907.07165. — [arXiv:1907.07165](https://arxiv.org/abs/1907.07165) find
  23. Riccardo Guidotti, Anna Monreale, Franco Turini, Dino Pedreschi, and Fosca Giannotti. 2018. A survey of methods for explaining black box models. CoRR, abs/1802.01933. — [arXiv:1802.01933](https://arxiv.org/abs/1802.01933) find
  24. Sai Gurrapu, Ajay Kulkarni, Lifu Huang, Ismini Lourentzou, and Feras A. Batarseh. 2023. Rationalization for explainable NLP: a survey. Frontiers Artif. Intell., 6. find
  25. Peter Hase and Mohit Bansal. 2020. Evaluating explainable AI: which algorithmic explanations help users predict model behavior? CoRR, abs/2005.01831.
  26. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced BERT with disentangled attention. CoRR, abs/2006.03654.
  27. Anna Hedström, Philine Lou Bommer, Kristoffer Knutsen Wickstrøm, Wojciech Samek, Sebastian Lapuschkin, and Marina M.-C. Höhne. 2023. The meta-evaluation problem in explainable AI: identifying reliable estimators with metaquantus. find
  28. Trans. Mach. Learn. Res., 2023. find
  29. XinYue Jiang, Jingsong He, and Li Gu. 2025. MTCR: method for matching texts against causal relationship. Neural Process. Lett., 57(3):58. find
  30. Enkelejda Kasneci, Kathrin Sessler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, Stephan Krusche, Gitta Kutyniok, Tilman Michaeli, Claudia Nerdel, Jürgen Pfeffer, Oleksandra Poquet, Michael Sailer, Albrecht Schmidt, T… — ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
  31. Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres. 2018. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Proceedings of the 35th International Conference on Machine Learning, ICML… find
  32. Sunnie S. Y. Kim, Elizabeth Anne Watkins, Olga Russakovsky, Ruth Fong, and Andrés Monroy-Hernández. 2022. "help me help the ai": Understanding how explainability can support human-ai interaction. CoRR, abs/2210.03735. — [arXiv:2210.03735](https://arxiv.org/abs/2210.03735) find
  33. Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. 2020. Concept bottleneck models. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Res… find
  34. Sören R. Künzel, Jasjeet S. Sekhon, Peter J. Bickel, and Bin Yu. 2019. Meta-learners for estimating heterogeneous treatment effects using machine learning. arXiv. find
  35. Jun Rui Lee, Sadegh Emami, Michael David Hollins, Timothy C. H. Wong, Carlos Ignacio Villalobos Sánchez, Francesca Toni, Dekai Zhang, and Adam Dejl. 2025. Xai-units: Benchmarking explainability methods with unit tests. CoRR, abs/2506.01059. — [arXiv:2506.01059](https://arxiv.org/abs/2506.01059) find
  36. Yongqi Li, Mayi Xu, Xin Miao, Shen Zhou, and Tieyun Qian. 2024. Prompting large language models for counterfactual generation: An empirical study. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2… find
  37. Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. CoRR, abs/1705.07874.
  38. Siwen Luo, Hamish Ivison, Soyeon Caren Han, and Josiah Poon. 2024. Local interpretations for explainable natural language processing: A survey. ACM Comput. Surv., 56(9):232:1–232:36. find
  39. Qing Lyu, Marianna Apidianaki, and Chris Callison-Burch. 2022. Towards faithful model explanation in NLP: A survey. CoRR, abs/2209.11326.
  40. Arnold S Monto, Stefan Gravenstein, Michael Elliott, Michael Colopy, and Jo Schweinle. 2000. Clinical signs and symptoms predicting influenza infection. Archives of internal medicine, 160(21):3243–3247. find
  41. Yuqi Nie, Yaxuan Kong, Xiaowen Dong, John M. Mulvey, H. Vincent Poor, Qingsong Wen, and Stefan Zohren. 2024. A survey of large language models for financial applications: Progress, prospects and challenges. CoRR, abs/2406.11903. — [arXiv:2406.11903](https://arxiv.org/abs/2406.11903) find
  42. OpenAI. 2024. Gpt-4o technical report. arXiv preprint. find
  43. Judea Pearl. 2009. Causality: Models, Reasoning, and Inference, 2 edition. Cambridge University Press. find
  44. Judea Pearl. 2013. Structural counterfactuals: A brief introduction. Cognitive science, 37(6):977–985. find
  45. Lotem Peled-Cohen, Maya Zadok, Nitay Calderon, Hila Gonen, and Roi Reichart. 2025. Dementia through different eyes: Explainable modeling of human and LLM perceptions for early awareness. CoRR, abs/2505.13418. find
  46. Eleonora Poeta, Gabriele Ciravegna, Eliana Pastor, Tania Cerquitelli, and Elena Baralis. 2023. Concept-based explainable artificial intelligence: A survey. CoRR, abs/2312.12936. — [arXiv:2312.12936](https://arxiv.org/abs/2312.12936) find
  47. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67.
  48. Manish Raghavan, Solon Barocas, Jon M. Kleinberg, and Karen Levy. 2020. Mitigating bias in algorithmic hiring: evaluating claims and practices. In FAT* ’20: Conference on Fairness, Accountability, and Transparency, Barcelona, Spain, January 27-30, 2020, pages 469–481. ACM. find
  49. Shauli Ravfogel, Anej Svete, Vésteinn Snæbjarnarson, and Ryan Cotterell. 2025. Gumbel counterfactual generation from language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. find
  50. Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan Cotterell. 2022. Linear adversarial concept erasure. CoRR, abs/2201.12091. — [arXiv:2201.12091](https://arxiv.org/abs/2201.12091) find
  51. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, Ch…
  52. Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should I trust you?": Explaining the predictions of any classifier. CoRR, abs/1602.04938.
  53. Marcel Robeer, Floris Bex, and Ad Feelders. 2021. Generating realistic natural language counterfactuals. In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pages 3611–3625. Association for Computational Ling… find
  54. Wojciech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher J. Anders, and Klaus-Robert Müller. 2021. Explaining deep neural networks and beyond: A review of methods and applications. Proc. IEEE, 109(3):247–278. find
  55. Pratinav Seth and Vinay Kumar Sankarapu. 2025. Bridging the gap in xai-why reliable metrics matter for explainability and compliance. CoRR, abs/2502.04695. — [arXiv:2502.04695](https://arxiv.org/abs/2502.04695) find
  56. Ruihao Shui, Yixin Cao, Xiang Wang, and Tat-Seng Chua. 2023. A comprehensive evaluation of large language models on legal judgment prediction. CoRR, abs/2310.11761. find
  57. Qwen Team. 2023. Qwen: The official repo of qwen chat. find
  58. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019. Generating token-level explanations for natural language inference. CoRR, abs/1904.10717. find
  59. Victor Veitch, Dhanya Sridhar, and David M. Blei. 2020. Adapting text embeddings for causal inference. In Proceedings of the Thirty-Sixth Conference on Uncertainty in Artificial Intelligence, UAI 2020, virtual online, August 3-6, 2020, volume 124 of Proceedings of Machine Learning Research, pages 91… find
  60. Lijie Wang, Yaozong Shen, Shuyuan Peng, Shuai Zhang, Xinyan Xiao, Hao Liu, Hongxuan Tang, Ying Chen, Hua Wu, and Haifeng Wang. 2022. A fine-grained interpretability evaluation benchmark for neural NLP. CoRR, abs/2205.11097. find
  61. Yongjie Wang, Xiaoqi Qiu, Yu Yue, Xu Guo, Zhiwei Zeng, Yuhong Feng, and Zhiqi Shen. 2024. A survey on natural language counterfactual generation. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 4798–4818. Association for Comp… find
  62. Zach Wood-Doughty, Ilya Shpitser, and Mark Dredze. 2018. Challenges of using text classifiers for causal inference. CoRR, abs/1810.00956. find
  63. Tongshuang Wu, Marco Túlio Ribeiro, Jeffrey Heer, and Daniel S. Weld. 2021. Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference o…
  64. Zhengxuan Wu, Karel D’Oosterlinck, Atticus Geiger, Amir Zur, and Christopher Potts. 2022. Causal proxy models for concept-based model explanations. CoRR, abs/2209.14279. find
  65. Zhengxuan Wu, Atticus Geiger, Christopher Potts, and Noah D. Goodman. 2023. Interpretability at scale: Identifying causal mechanisms in alpaca. CoRR, abs/2305.08809.
  66. Fan Yang, Mengnan Du, and Xia Hu. 2019. Evaluating explanation without ground truth in interpretable machine learning. CoRR, abs/1907.06831. — [arXiv:1907.06831](https://arxiv.org/abs/1907.06831) find
  67. Chih-Kuan Yeh, Been Kim, Sercan Ömer Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. 2020. On completeness-aware concept-based explanations in deep neural networks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, Ne… find
  68. Wei Jie Yeo, Ranjan Satapathy, and Erik Cambria. 2024. Towards faithful natural language explanations: A study using activation patching in large language models. CoRR, abs/2410.14155. find
  69. Xuemin Yu, Fahim Dalvi, Nadir Durrani, and Hassan Sajjad. 2024. Latent concept-based explanation of NLP models. CoRR, abs/2404.12545. find
  70. Raymond Zhang, Neha Nayak Kennard, Daniel Scott Smith, Daniel A. McFarland, Andrew McCallum, and Katherine Keith. 2023. Causal matching with text embeddings: A case study in estimating the causal effects of peer review policies. In Findings of the Association for Computational Linguistics: ACL 2023,… find
  71. Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. 2024. Explainability for large language models: A survey. ACM Trans. Intell. Syst. Technol., 15(2):20:1–20:38.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/toker-2026-liberty-causal-framework-benchmarking-concept/