Elamur Research
☰
home / papers / 2023 / Explainability for Large Language Models: A Survey

Explainability for Large Language Models: A Survey

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2309.01029

Summary

Large language models are widely deployed but remain opaque. This makes it hard to anticipate harmful outputs, hallucinations or hidden biases. This survey introduces no new data or models. Instead, it organizes the literature on explaining Transformer-based language models into a single taxonomy. Methods are split by how a model is adapted to tasks. One route is classic pre-training followed by fine-tuning, typical of encoder models under roughly a billion parameters. The other is prompting, used by much larger base and assistant (chat) models. Within each route, the authors separate local explanations, which account for one specific prediction, from global ones, which describe the knowledge a model holds overall. They also review how explanations are evaluated and how they can help debug models and improve performance. The paper ends with open problems specific to the LLM era. It argues that interpretability must keep pace with model scale if these systems are to be safe, fair and trustworthy. For the library, it serves as a map of the field rather than a source of benchmarks.

Key points

  • Fine-tuning paradigm: covers models such as BERT, RoBERTa, ELECTRA and DeBERTa, adapted to GLUE tasks like SST-2, MNLI and QQP.
  • Prompting paradigm: has two subtypes.
    • Base models: GPT-3, OPT, LLaMA-1, LLaMA-2 (up to 70B parameters) and Falcon.
    • Assistant models built with instruction tuning and RLHF: GPT-3.5, GPT-4, Claude, LLaMA-2-Chat, Alpaca and Vicuna.
  • Local explanation families:
    • Feature attribution, which includes perturbation, gradient, surrogate-model and decomposition variants.
    • Attention-based explanations.
    • Example-based explanations.
    • Natural-language explanations.
  • Reported findings on model behavior:
    • Models exploit shortcuts. Fine-tuned NLI models rely on lexical, overlap, position and style biases. Prompted models handle information placed mid-context worse than information at the start or end.
    • Attention heads are often redundant, both in BERT-base and in OPT-66B, which points to pruning opportunities.
  • Emerging directions:
    • Analyzing training dynamics across checkpoints rather than only finished models, for example the abrupt emergence of Syntactic Attention Structure during pre-training.
    • Using explainability tools in safety audits.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

135 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 133 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2309.01029 · arXiv 2309.01029

@misc{zhao2023explainability,
  title = {{Explainability for Large Language Models: A Survey}},
  author = {Zhao, Haiyan and Chen, Hanjie and Yang, Fan and Liu, Ninghao and Deng, Huiqi and Cai, Hengyi and Wang, Shuaiqiang and Yin, Dawei and Du, Mengnan},
  year = {2023},
  doi = {10.48550/arxiv.2309.01029},
  eprint = {2309.01029},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.01029}
}

References

207 references, 48 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, Guilherme Penedo, et al. Falcon-40b: an open large language model with state-of-the-… find
  2. Anthropic. Decomposing language models into understandable components? https://www.anthropic. com/index/decomposing-language-models-into-understandable-components, 2023. ↗
  3. AnthropicAI. Introducing claude, 2023. URL https://www.anthropic.com/index/introducing-claude. ↗
  4. Omer Antverg and Yonatan Belinkov. On the Pitfalls of Analyzing Individual Neurons in Language Models, August 2022. URL http://arxiv.org/abs/2110.07483. arXiv:2110.07483 [cs]. — [arXiv:2110.07483](https://arxiv.org/abs/2110.07483) ↗
  5. Marianna Apidianaki and Aina Garí Soler. ALL Dolphins Are Intelligent and SOME Are Friendly: Probing BERT for Nouns’ Semantic Properties and their Prototypicality, October 2021. URL http://arxiv.org/ abs/2110.06376. arXiv:2110.06376 [cs]. ↗
  6. Ines Arous, Ljiljana Dolamic, Jie Yang, Akansha Bhardwaj, Giuseppe Cuccu, and Philippe Cudré-Mauroux. Marta: Leveraging human rationales for explainable text classification. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 5868–5876, 2021. find
  7. Pepa Atanasova, Oana-Maria Camburu, Christina Lioma, Thomas Lukasiewicz, Jakob Grue Simonsen, and Isabelle Augenstein. Faithfulness Tests for Natural Language Explanations, June 2023. URL http: //arxiv.org/abs/2305.18029. arXiv:2305.18029 [cs]. find
  8. Bing Bai, Jian Liang, Guanhua Zhang, Hao Li, Kun Bai, and Fei Wang. Why attentions may not be interpretable? In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 25–34, 2021. find
  9. Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, and Dan Roth. Rethinking the role of scale for in-context learning: An interpretability-based case study at 66 billion scale. arXiv preprint arXiv:2212.09095, 2022. find
  10. Oren Barkan, Edan Hauon, Avi Caciularu, Ori Katz, Itzik Malkiel, Omri Armstrong, and Noam Koenigstein. Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 2882–2887, Virtual Event Queen… — [doi:10.1145/3459637.3482126](https://doi.org/10.1145/3459637.3482126) find
  11. Jasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm, and Katja Filippova. "Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification, November 2022. URL http://arxiv.org/abs/2111.07367. arXiv:2111.07367 [cs]. ↗
  12. Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Identifying and Controlling Important Neurons in Neural Machine Translation, November 2018. URL http://arxiv. org/abs/1811.01157. arXiv:1811.01157 [cs]. find
  13. Yonatan Belinkov. Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics, 48(1):207–219, April 2022. ISSN 0891-2017, 1530-9312. doi: 10.1162/coli\_a\_00422. URL https://direct.mit.edu/coli/article/48/1/207/107571/ Probing-Classifiers-Promises-Shortcomings-and. ↗ our card
  14. Yonatan Belinkov, Lluís Màrquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Vol… find our card
  15. Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A", September 2023. URL http://arxiv.org/abs/2309.12288. arXiv:2309.12288 [cs]. ↗
  16. Yuchen Bian, Jiaji Huang, Xingyu Cai, Jiahong Yuan, and Kenneth Church. On attention redundancy: A comprehensive study. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), 2021. find
  17. Terra Blevins, Omer Levy, and Luke Zettlemoyer. Deep RNNs Encode Soft Hierarchical Syntax. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 14–19, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi:… find
  18. Terra Blevins, Hila Gonen, and Luke Zettlemoyer. Prompting language models for linguistic structure. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2023. URL https://arxiv.org/abs/2211.07830. ↗
  19. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tris… find
  20. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 2020. find
  21. Gino Brunner, Yang Liu, Damian Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Wattenhofer. On identifiability in transformers. arXiv preprint arXiv:1908.04211, 2019. — [arXiv:1908.04211](https://arxiv.org/abs/1908.04211) find
  22. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. find our card
  23. Captum. Testing with concept activation vectors (tcav) on sensitivity classification examples and a convnet model trained on imdb dataset. https://github.com/pytorch/captum/blob/master/tutorials/TCAV\_ NLP.ipynb. Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao… ↗
  24. Aaron Chan, Maziar Sanjabi, Lambert Mathias, Liang Tan, Shaoliang Nie, Xiaochang Peng, Xiang Ren, and Hamed Firooz. Unirex: A unified learning framework for language model rationale extraction. In International Conference on Machine Learning, pp. 2867–2889. PMLR, 2022a. find
  25. Chun Sik Chan, Huanqi Kong, and Liang Guanqing. A comparative study of faithfulness metrics for model interpretability methods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5029–5038, 2022b. find
  26. Hila Chefer, Shir Gur, and Lior Wolf. Transformer Interpretability Beyond Attention Visualization. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 782–791, Nashville, TN, USA, June 2021. IEEE. ISBN 978-1-66544-509-2. doi: 10.1109/CVPR46437.2021.00084. URL https: //… — [doi:10.1109/cvpr46437.2021.00084](https://doi.org/10.1109/cvpr46437.2021.00084) find
  27. Angelica Chen, Ravid Schwartz-Ziv, Kyunghyun Cho, Matthew L Leavitt, and Naomi Saphra. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in mlms. arXiv preprint arXiv:2309.07311, 2023a. — [arXiv:2309.07311](https://arxiv.org/abs/2309.07311) find
  28. Boli Chen, Yao Fu, Guangwei Xu, Pengjun Xie, Chuanqi Tan, Mosha Chen, and Liping Jing. Probing BERT in Hyperbolic Spaces, April 2021. URL http://arxiv.org/abs/2104.03869. arXiv:2104.03869 [cs]. — [arXiv:2104.03869](https://arxiv.org/abs/2104.03869) ↗
  29. Hanjie Chen and Yangfeng Ji. Adversarial training for improving model robustness? look at both prediction and interpretation. In The 36th AAAI Conference on Artificial Intelligence (AAAI), 2022. find
  30. Hanjie Chen, Faeze Brahman, Xiang Ren, Yangfeng Ji, Yejin Choi, and Swabha Swayamdipta. Rev: information-theoretic evaluation of free-text rationales. The 61th Annual Meeting of the Association for Computational Linguistics (ACL), 2023b. find
  31. Hugh Chen, Ian C. Covert, Scott M. Lundberg, and Su-In Lee. Algorithms to estimate Shapley value feature attributions. Nature Machine Intelligence, 5(6):590–601, May 2023c. ISSN 2522-5839. doi: 10.1038/ s42256-023-00657-x. URL https://www.nature.com/articles/s42256-023-00657-x. ↗
  32. Yanda Chen, Ruiqi Zhong, Narutatsu Ri, Chen Zhao, He He, Jacob Steinhardt, Zhou Yu, and Kathleen McKeown. Do models explain themselves? counterfactual simulatability of natural language explanations, 2023d. find
  33. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/ 2023-03-30-vicuna… ↗
  34. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… find
  35. George Chrysostomou and Nikolaos Aletras. Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text Classification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natur… find
  36. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What Does BERT Look at? An Analysis of BERT’s Attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 276–286, Florence, Italy, August 2019. Association for Computa… find
  37. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. International Conference on Learning Representations (ICLR), 2020. find
  38. Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):6309–6317, July 2019. ISSN 2374-3468, 2159-5399.… — [doi:10.1609/aaai.v33i01.33016309](https://doi.org/10.1609/aaai.v33i01.33016309) find
  39. Sanjoy Dasgupta, Nave Frost, and Michal Moshkovitz. Framework for Evaluating Faithfulness of Local Explanations. 2022. find
  40. Joseph F. DeRose, Jiayao Wang, and Matthew Berger. Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Models, September 2020. URL http://arxiv.org/abs/2009.07053. arXiv:2009.07053 [cs]. — [arXiv:2009.07053](https://arxiv.org/abs/2009.07053) ↗
  41. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. North American Chapter of the Association for Computational Linguistics (NAACL), 2019a. find
  42. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics (NAACL), 2019b. find
  43. Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. ERASER: A Benchmark to Evaluate Rationalized NLP Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4443–4458, Online, July 2020. A… find
  44. Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608, 2017. find our card
  45. Timothy Dozat and Christopher D. Manning. Deep Biaffine Attention for Neural Dependency Parsing, March 2017. URL http://arxiv.org/abs/1611.01734. arXiv:1611.01734 [cs]. ↗ our card
  46. Mengnan Du, Ninghao Liu, and Xia Hu. Techniques for interpretable machine learning. Communications of the ACM, 63(1):68–77, 2019a. find
  47. Mengnan Du, Ninghao Liu, Fan Yang, Shuiwang Ji, and Xia Hu. On Attribution of Recurrent Neural Network Predictions via Additive Decomposition, March 2019b. URL http://arxiv.org/abs/1903. 11245. arXiv:1903.11245 [cs]. — [arXiv:1903.11245](https://arxiv.org/abs/1903.11245) ↗
  48. Mengnan Du, Varun Manjunatha, Rajiv Jain, Ruchi Deshpande, Franck Dernoncourt, Jiuxiang Gu, Tong Sun, and Xia Hu. Towards interpreting and mitigating shortcut learning behavior of nlu models. North American Chapter of the Association for Computational Linguistics (NAACL), 2021. find
  49. Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. Shortcut learning of large language models in natural language understanding. Communications of the ACM (CACM), 2023. find
  50. Jinhao Duan, Hao Cheng, Shiqi Wang, Chenan Wang, Alex Zavalny, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the uncertainty estimation of large language models. arXiv preprint arXiv:2307.01379, 2023. find
  51. Nouha Dziri, Sivan Milton, Mo Yu, Osmar Zaiane, and Siva Reddy. On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models? In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Cha… find
  52. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Ja… find
  53. Joseph Enguehard. Sequential Integrated Gradients: a simple but effective method for explaining language models, May 2023. URL http://arxiv.org/abs/2305.15853. arXiv:2305.15853 [cs]. ↗
  54. Kawin Ethayarajh and Dan Jurafsky. Attention Flows are Shapley Value Explanations, May 2021. URL http://arxiv.org/abs/2105.14652. arXiv:2105.14652 [cs]. ↗
  55. Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. Pathologies of Neural Models Make Interpretations Difficult. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3719–3728, Brussels, Belgium, October 2018. Asso… find
  56. Siddhant Garg and Goutham Ramakrishnan. BAE: BERT-based Adversarial Examples for Text Classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6174–6181, 2020. doi: 10.18653/v1/2020.emnlp-main.498. URL http://arxiv.org/abs/ 2004.01970. ar… ↗ our card
  57. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913, 2020. find
  58. Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint arXiv:2203.14680, 2022. find our card
  59. Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International conference on machine learning, pp. 2242–2251. PMLR, 2019. find
  60. Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. Roscoe: A suite of metrics for scoring step-by-step reasoning. arXiv preprint arXiv:2212.07919, 2022. find our card
  61. Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296, 2023. find
  62. Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary llms. arXiv preprint arXiv:2305.15717, 2023. find our card
  63. Han Guo, Nazneen Fatema Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. Fastif: Scalable influence functions for efficient model interpretation and debugging. arXiv preprint arXiv:2012.15781, 2020. find
  64. Wes Gurnee and Max Tegmark. Language models represent space and time. arXiv preprint arXiv:2310.02207, 2023. find
  65. Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Self-Attention Attribution: Interpreting Information Interactions Inside Transformer. Proceedings of the AAAI Conference on Artificial Intelligence, 35(14):12963–12971, May 2021. ISSN 2374-3468. doi: 10.1609/aaai.v35i14.17533. URL https://ojs.aaai.org/index.ph… — [doi:10.1609/aaai.v35i14.17533](https://doi.org/10.1609/aaai.v35i14.17533) ↗
  66. Satoshi Hara, Atsushi Nitanda, and Takanori Maehara. Data cleansing for models trained with sgd. Advances in Neural Information Processing Systems, 32, 2019. find
  67. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations, 2021. find our card
  68. Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, Scott Johnston, Ben Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCandlish. Scaling Laws and Interpretabilit… find
  69. John Hewitt and Percy Liang. Designing and Interpreting Probes with Control Tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2733–2743, Hong Kong, China, Nove… find
  70. John Hewitt and Christopher D. Manning. A Structural Probe for Finding Syntax in Word Representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4129–4138, M… find our card
  71. Benjamin Hoover, Hendrik Strobelt, and Sebastian Gehrmann. exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 187–196, Online, July 2020. Assoc… find
  72. Yuheng Huang, Jiayang Song, Zhijie Wang, Huaming Chen, and Lei Ma. Look before you leap: An exploratory study of uncertainty measurement for large language models. arXiv preprint arXiv:2307.10236, 2023. — [arXiv:2307.10236](https://arxiv.org/abs/2307.10236) find
  73. Alon Jacovi and Yoav Goldberg. Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4198–4205, Online, July 2020. Association for Computational Linguistics. doi: 1… find our card
  74. Sarthak Jain and Byron C. Wallace. Attention is not Explanation, May 2019. URL http://arxiv.org/ abs/1902.10186. arXiv:1902.10186 [cs]. ↗ our card
  75. Theo Jaunet, Corentin Kervadec, Romain Vuillemot, Grigory Antipov, Moez Baccouche, and Christian Wolf. VisQA: X-raying Vision and Language Reasoning in Transformers, July 2021. URL http://arxiv.org/ abs/2104.00926. arXiv:2104.00926 [cs]. — [arXiv:2104.00926](https://arxiv.org/abs/2104.00926) ↗
  76. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What Does BERT Learn about the Structure of Language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3651–3657, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19… find our card
  77. Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? natural language attack on text classification and entailment. AAAI Conference on Artificial Intelligence (AAAI), 2020. find
  78. Brihi Joshi, Aaron Chan, Ziyi Liu, Shaoliang Nie, Maziar Sanjabi, Hamed Firooz, and Xiang Ren. Er-test: Evaluating explanation regularization methods for language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 3315–3336, 2022. find
  79. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large Language Models Struggle to Learn Long-Tail Knowledge, July 2023. URL http://arxiv.org/abs/2211.08411. arXiv:2211.08411 [cs]. ↗ our card
  80. Cheongwoong Kang and Jaesik Choi. Impact of Co-occurrence on Factual Knowledge of Large Language Models, October 2023. URL http://arxiv.org/abs/2310.08256. arXiv:2310.08256 [cs] version: 1. ↗
  81. Divyansh Kaushik, Eduard Hovy, and Zachary C Lipton. Learning the Difference that Makes a Difference with Counterfactually-Augmented Data. 2020. find
  82. Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV), June 2018. URL http://arxiv.org/abs/1711.11279. arXiv:1711.11279 [stat]. — [arXiv:1711.11279](https://arxiv.org/abs/1711.11279) ↗
  83. Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (Un)reliability of saliency methods, November 2017. URL http: //arxiv.org/abs/1711.00867. arXiv:1711.00867 [cs, stat]. — [arXiv:1711.00867](https://arxiv.org/abs/1711.00867) find
  84. Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885–1894. find
  85. Enja Kokalj, Blaž Škrlj, Nada Lavrač, Senja Pollak, and Marko Robnik-Šikonja. BERT meets Shapley: Extending SHAP Explanations to Transformer-based Classifiers. In Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation, pp. 16–21, Online, April 2021. Associat… find
  86. Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the Dark Secrets of BERT, September 2019. URL http://arxiv.org/abs/1908.08593. arXiv:1908.08593 [cs, stat]. ↗ our card
  87. Nicholas Kroeger, Dan Ley, Satyapriya Krishna, Chirag Agarwal, and Himabindu Lakkaraju. Are large language models post hoc explainers? arXiv preprint arXiv:2310.05797, 2023. — [arXiv:2310.05797](https://arxiv.org/abs/2310.05797) find
  88. Jenny Kunz and Marco Kuhlmann. Classifier Probes May Just Learn from Linear Context Features. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 5136–5146, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 10.18653… find
  89. Faisal Ladhak, Esin Durmus, Mirac Suzgun, Tianyi Zhang, Dan Jurafsky, Kathleen McKeown, and Tatsunori Hashimoto. When Do Pre-Training Biases Propagate to Downstream Tasks? A Case Study in Text Summarization. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the… find
  90. Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. Can language models learn from explanations in context? In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 537–563, 2022. find our card
  91. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702, 2023. find our card
  92. Dong-Ho Lee, Akshen Kadakia, Brihi Joshi, Aaron Chan, Ziyi Liu, Kiran Narahari, Takashi Shibuya, Ryosuke Mitani, Toshiyuki Sekiya, Jay Pujara, and Xiang Ren. Xmd: An end-to-end framework for interactive explanation-based debugging of nlp models, 2022. find
  93. Bai Li, Zining Zhu, Guillaume Thomas, Yang Xu, and Frank Rudzicz. How is BERT surprised? Layerwise detection of linguistic anomalies, May 2021a. URL http://arxiv.org/abs/2105.07452. arXiv:2105.07452 [cs]. ↗
  94. Dianqi Li, Yizhe Zhang, Hao Peng, Liqun Chen, Chris Brockett, Ming-Ting Sun, and Bill Dolan. Contextualized Perturbation for Textual Adversarial Attack. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  95. Jiaoda Li, Ryan Cotterell, and Mrinmaya Sachan. Probing via Prompting, July 2022. URL http://arxiv. org/abs/2207.01736. arXiv:2207.01736 [cs]. — [arXiv:2207.01736](https://arxiv.org/abs/2207.01736) find
  96. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and Understanding Neural Models in NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 681–691, San Diego, California, June 2016.… find our card
  97. Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding Neural Networks through Representation Erasure, January 2017. URL http://arxiv.org/abs/1612.08220. arXiv:1612.08220 [cs]. — [arXiv:1612.08220](https://arxiv.org/abs/1612.08220) ↗
  98. Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. A survey on fairness in large language models. arXiv preprint arXiv:2308.10149, 2023a. — [arXiv:2308.10149](https://arxiv.org/abs/2308.10149) find
  99. Zongxia Li, Paiheng Xu, Fuxiao Liu, and Hyemi Song. Towards understanding in-context learning with contrastive demonstrations and saliency maps, 2023b. find
  100. Tom Lieberum, Matthew Rahtz, János Kramár, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. arXiv preprint arXiv:2307.09458, 2023. find
  101. Yongjie Lin, Yi Chern Tan, and Robert Frank. Open sesame: getting inside bert’s linguistic knowledge. arXiv preprint arXiv:1906.01698, 2019. find our card
  102. Jiaxiang Liu, Tianxiang Hu, Yan Zhang, Xiaotang Gai, Yang Feng, and Zuozhu Liu. A chatgpt aided explainable framework for zero-shot medical image diagnosis, 2023a. find
  103. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172, 2023b. find our card
  104. Ninghao Liu, Yunsong Meng, Xia Hu, Tie Wang, and Bo Long. Are interpretations fairly evaluated? a definition driven pipeline for post-hoc interpretability. arXiv preprint arXiv:2009.07494, 2020. — [arXiv:2009.07494](https://arxiv.org/abs/2009.07494) find
  105. Yibing Liu, Haoliang Li, Yangyang Guo, Chenqi Kong, Jing Li, and Shiqi Wang. Rethinking Attention-Model Explainability through Faithfulness Violation Test. 2022. find
  106. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. find
  107. Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017a. find our card
  108. Scott M Lundberg and Su-In Lee. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017b. URL https://proceedings.neurips.cc/paper\_files/paper/2017/hash/ 8a20a8621978632d76c43dfd28b67767-Abstract.html. ↗ our card
  109. Daniel D. Lundstrom, Tianjian Huang, and Meisam Razaviyayn. A Rigorous Study of Integrated Gradients Method and Extensions to Internal Neuron Attributions. In Proceedings of the 39th International Conference on Machine Learning, pp. 14485–14508. PMLR, June 2022. URL https://proceedings.mlr. press/v1… ↗
  110. Siwen Luo, Hamish Ivison, Caren Han, and Josiah Poon. Local Interpretations for Explainable Natural Language Processing: A Survey, October 2022. URL http://arxiv.org/abs/2103.11072. arXiv:2103.11072 [cs]. — [arXiv:2103.11072](https://arxiv.org/abs/2103.11072) ↗
  111. Jiaqi Ma, Dylan Slack, Asma Ghandeharioun, Sameer Singh, Himabindu Lakkaraju, et al. Post hoc explanations of language models can improve language models. arXiv preprint arXiv:2305.11426, 2023. — [arXiv:2305.11426](https://arxiv.org/abs/2305.11426) find
  112. Aman Madaan and Amir Yazdanbakhsh. Text and patterns: For effective chain of thought, it takes two to tango. arXiv preprint arXiv:2209.07686, 2022. find
  113. Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824, 2023. find our card
  114. Sammy Martin. Ten Levels of AI Alignment Difficulty, 2023. URL https://www.lesswrong.com/posts/ EjgfreeibTXRx9Ham/ten-levels-of-ai-alignment-difficulty. [Accessed 21-08-2023]. ↗
  115. Rebecca Marvin and Tal Linzen. Targeted Syntactic Evaluation of Language Models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1192–1202, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1151. URL https:… find
  116. Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 14867–14875, 2021. find
  117. Rowan Hall Maudslay and Ryan Cotterell. Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing, June 2021. URL http://arxiv.org/abs/2106.02559. arXiv:2106.02559 [cs]. ↗
  118. Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations. arXiv preprint arXiv:2307.15771, 2023. find
  119. Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Javad Hosseini, Mark Johnson, and Mark Steedman. Sources of Hallucination by Large Language Models on Inference Tasks, October 2023. URL http: //arxiv.org/abs/2305.14552. arXiv:2305.14552 [cs]. find
  120. Yusuf Mehdi. Reinventing search with a new AI-powered Microsoft Bing and Edge, your copilot for the web, 2023. URL https://blogs.microsoft.com/blog/2023/02/07/ reinventing-search-with-a-new-ai-powered-microsoft-bing-and-edge-your-copilot-for-the-web/. ↗
  121. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372, 2022. find
  122. Vivek Miglani, Narine Kokhlikyan, Bilal Alsallakh, Miguel Martin, and Orion Reblitz-Richardson. Investigating Saturation Effects in Integrated Gradients, October 2020. URL http://arxiv.org/abs/2010.12697. arXiv:2010.12697 [cs]. — [arXiv:2010.12697](https://arxiv.org/abs/2010.12697) ↗
  123. Akash Kumar Mohankumar, Preksha Nema, Sharan Narasimhan, Mitesh M. Khapra, Balaji Vasan Srinivasan, and Balaraman Ravindran. Towards Transparent and Explainable Attention Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4206–4216, Online, July 2… find
  124. Hosein Mohebbi, Ali Modarressi, and Mohammad Taher Pilehvar. Exploring the Role of BERT Token Representations to Explain Sentence Probing Results, September 2021. URL http://arxiv.org/abs/ 2104.01477. arXiv:2104.01477 [cs]. ↗
  125. Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojciech Samek, and Klaus-Robert Müller. Layer-wise relevance propagation: an overview. Explainable AI: interpreting, explaining and visualizing deep learning, pp. 193–209, 2019. find
  126. Grégoire Montavon, Sebastian Bach, Alexander Binder, Wojciech Samek, and Klaus-Robert Müller. Explaining NonLinear Classification Decisions with Deep Taylor Decomposition, December 2015. URL http://arxiv.org/abs/1512.02479. arXiv:1512.02479 [cs, stat]. — [arXiv:1512.02479](https://arxiv.org/abs/1512.02479) ↗
  127. Pooya Moradi, Nishant Kambhatla, and Anoop Sarkar. Measuring and improving faithfulness of attention in neural machine translation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2791–2802, 2021. find
  128. Jesse Mu and Jacob Andreas. Compositional Explanations of Neurons, February 2021. URL http://arxiv. org/abs/2006.14032. arXiv:2006.14032 [cs, stat]. — [arXiv:2006.14032](https://arxiv.org/abs/2006.14032) find
  129. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707, 2023. find our card
  130. Michael Neely, Stefan F Schouten, Maurits JR Bleeker, and Ana Lucic. Order in the court: Explainable ai methods prone to disagreement. arXiv preprint arXiv:2105.03287, 2021. — [arXiv:2105.03287](https://arxiv.org/abs/2105.03287) find
  131. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021. find our card
  132. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom In: An Introduction to Circuits. https://distill.pub/2020/circuits/zoom-in/, March 2020a. [Accessed 24-11-2023]. ↗
  133. Chris Olah, Nick Cammarata, Chelsea Voss, Ludwig Schubert, and Gabriel Goh. Naturally Occurring Equivariance in Neural Networks — distill.pub. https://distill.pub/2020/circuits/equivariance/, December 2020b. [Accessed 27-11-2023]. ↗
  134. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amode… find
  135. OpenAI. Language models can explain neurons in language models, 2023a. URL https://openai.com/ research/language-models-can-explain-neurons-in-language-models?s=09. ↗
  136. OpenAI. Gpt-4 technical report, 2023b. find our card
  137. Cheonbok Park, Inyoup Na, Yongjang Jo, Sungbok Shin, Jaehyo Yoo, Bum Chul Kwon, Jian Zhao, Hyungjong Noh, Yeonsoo Lee, and Jaegul Choo. SANVis: Visual Analytics for Understanding Self-Attention Networks, September 2019. URL http://arxiv.org/abs/1909.09595. arXiv:1909.09595 [cs]. — [arXiv:1909.09595](https://arxiv.org/abs/1909.09595) ↗
  138. Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting Contextual Word Embeddings: Architecture and Representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1499–1509, Brussels, Belgium, October 2018. Association for Compu… find our card
  139. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. Language Models as Knowledge Bases?, September 2019. URL http://arxiv.org/ abs/1909.01066. arXiv:1909.01066 [cs]. ↗ our card
  140. Michael Petrov, Chelsea Voss, Ludwig Schubert, Nick Cammarata, Gabriel Goh, and Chris Olah. Weight Banding — distill.pub. https://distill.pub/2020/circuits/weight-banding/, April 2021. [Accessed 27-11-2023]. ↗
  141. Archiki Prasad, Swarnadeep Saha, Xiang Zhou, and Mohit Bansal. Receval: Evaluating reasoning chains via correctness and informativeness. arXiv preprint arXiv:2304.10703, 2023. find
  142. Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920–19930, 2020. find
  143. Luyu Qiu, Yi Yang, Caleb Chen Cao, Jing Liu, Yueyuan Zheng, Hilary Hei Ting Ngai, Janet Hsiao, and Lei Chen. Resisting Out-of-Distribution Data Problem in Perturbation of XAI, July 2021. URL http: //arxiv.org/abs/2107.14000. arXiv:2107.14000 [cs]. — [arXiv:2107.14000](https://arxiv.org/abs/2107.14000) find
  144. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PMLR, 2021… find our card
  145. Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, et al. Question decomposition improves the faithfulness of model-generated reasoning. arXiv preprint arXiv:2307.11768, 2023. — [arXiv:2307.11768](https://arxiv.org/abs/2307.11768) find
  146. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020. find
  147. Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain Yourself! Leveraging Language Models for Commonsense Reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4932–4942, Florence, Italy, July 2019. Association for Compu… find our card
  148. Abhilasha Ravichander, Eduard Hovy, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Cheung. On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT. In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pp. 88–102, Barcelon… find
  149. Ruiyang Ren, Yuhao Wang, Yingqi Qu, Wayne Xin Zhao, Jing Liu, Hao Tian, Hua Wu, Ji-Rong Wen, and Haifeng Wang. Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation, July 2023. URL http://arxiv.org/abs/2307.11019. arXiv:2307.11019 [cs]. ↗
  150. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should I Trust You?": Explaining the Predictions of Any Classifier, August 2016. URL http://arxiv.org/abs/1602.04938. arXiv:1602.04938 [cs, stat]. ↗ our card
  151. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8:842–866, 2021. find our card
  152. Alexis Ross, Ana Marasović, and Matthew E Peters. Explaining nlp models via minimal contrastive editing (mice). In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3840–3852, 2021. find
  153. Soumya Sanyal and Xiang Ren. Discretized Integrated Gradients for Explaining Language Models, August 2021. URL http://arxiv.org/abs/2108.13654. arXiv:2108.13654 [cs]. ↗
  154. Naomi Saphra. Interpretability Creationism. https://nsaphra.github.io/post/creationism/, 2022. [Accessed 22-10-2023]. ↗
  155. Sofia Serrano and Noah A. Smith. Is Attention Interpretable?, June 2019. URL http://arxiv.org/abs/ 1906.03731. arXiv:1906.03731 [cs]. ↗ our card
  156. Lloyd S Shapley et al. A value for n-person games. 1953. find
  157. Yaozong Shen, Lijie Wang, Ying Chen, Xinyan Xiao, Jing Liu, and Hua Wu. An Interpretability Evaluation Benchmark for Pre-trained Language Models, July 2022. URL http://arxiv.org/abs/2207.13948. arXiv:2207.13948 [cs]. — [arXiv:2207.13948](https://arxiv.org/abs/2207.13948) ↗
  158. Sandipan Sikdar, Parantapa Bhattacharya, and Kieran Heese. Integrated Directional Gradients: Feature Interaction Attribution for Neural NLP Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Langua… find
  159. Chandan Singh, Aliyah R Hsu, Richard Antonello, Shailee Jain, Alexander G Huth, Bin Yu, and Jianfeng Gao. Explaining black box text modules in natural language with language models. arXiv preprint arXiv:2305.09863, 2023. — [arXiv:2305.09863](https://arxiv.org/abs/2305.09863) find
  160. Ionut-Teodor Sorodoc, Kristina Gulordava, and Gemma Boleda. Probing for Referential Information in Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4177–4189, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v… find
  161. Joe Stacey, Yonatan Belinkov, and Marek Rei. Supervising model attention with human explanations for robust natural language inference. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 11349–11357, 2022. find
  162. Hendrik Strobelt, Sebastian Gehrmann, Michael Behrisch, Adam Perer, Hanspeter Pfister, and Alexander M. Rush. Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models, October 2018. URL http://arxiv.org/abs/1804.09299. arXiv:1804.09299 [cs]. — [arXiv:1804.09299](https://arxiv.org/abs/1804.09299) ↗
  163. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. International Conference on Machine Learning (ICML), 2017. find
  164. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html, 3(6):7, 202… find
  165. Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT Rediscovers the Classical NLP Pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4593–4601, Florence, Italy, July 2019a. Association for Computational Linguistics. doi: 10.18653/v1/P19-1452. URL… find our card
  166. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? Probing for sentence structure in contextualized word representations. 2019b. find our card
  167. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Generating token-level explanations for natural language inference. arXiv preprint arXiv:1904.10717, 2019. find
  168. Lucas Torroba Hennigen, Adina Williams, and Ryan Cotterell. Intrinsic Probing through Dimension Selection. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 197–216, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v… find
  169. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. find
  170. Hugo Touvron, Louis Martin, Kevin Stone, et al. Llama-2: Open foundation and fine-tuned chat models. 2023b. URL https://ai.meta.com/research/publications/ llama-2-open-foundation-and-fine-tuned-chat-models/. ↗
  171. Marcos Treviso, Alexis Ross, Nuno M Guerreiro, and André FT Martins. Crest: A joint framework for rationalization and counterfactual text generation. The 61th Annual Meeting of the Association for Computational Linguistics (ACL), 2023. find
  172. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. arXiv preprint arXiv:2305.04388, 2023. find our card
  173. Betty Van Aken, Benjamin Winter, Alexander Löser, and Felix A. Gers. How Does BERT Answer Questions?: A Layer-Wise Analysis of Transformer Representations. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 1823–1832, Beijing China, November 2019. AC… — [doi:10.1145/3357384.3358028](https://doi.org/10.1145/3357384.3358028) find
  174. Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances in neural information processing systems, 29, 2016. find
  175. Jesse Vig. BertViz: A Tool for Visualizing Multi-Head Self-Attention in the BERT Model. 2019. find
  176. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5797–5808, Florence, Italy, Ju… find our card
  177. Elena Voita, Rico Sennrich, and Ivan Titov. Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processin… find
  178. Chelsea Voss, Gabriel Goh, Nick Cammarata, Michael Petrov, Ludwig Schubert, and Chris Olah. Branch Specialization — distill.pub. https://distill.pub/2020/circuits/branch-specialization/, April 2021. [Accessed 26-11-2023]. ↗
  179. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. International Conference on Learning Representations (ICLR), 2019. find our card
  180. Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards understanding chain-of-thought prompting: An empirical study of what matters. arXiv preprint arXiv:2212.10001, 2022a. find our card
  181. Boxin Wang, Chejian Xu, Xiangyu Liu, Yu Cheng, and Bo Li. SemAttack: Natural Textual Attacks via Different Semantic Spaces. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 176–205, Seattle, United States, 2022b. Association for Computational Linguistics. doi: 10.18653/v… find
  182. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022. find our card
  183. Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V. Le. Simple synthetic data reduces sycophancy in large language models, August 2023a. URL http://arxiv.org/abs/2308.03958. arXiv:2308.03958 [cs]. ↗
  184. Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846, 2023b. find our card
  185. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021. find our card
  186. Sarah Wiegreffe and Yuval Pinter. Attention is not not Explanation, September 2019. URL http://arxiv. org/abs/1908.04626. arXiv:1908.04626 [cs]. find our card
  187. Skyler Wu, Eric Meng Shen, Charumathi Badrinath, Jiaqi Ma, and Himabindu Lakkaraju. Analyzing chain-of-thought prompting in large language models via gradient-based feature attributions. 2023a. find
  188. Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural… find
  189. Weiqi Wu, Chengyue Jiang, Yong Jiang, Pengjun Xie, and Kewei Tu. Do PLMs Know and Understand Ontological Knowledge? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3080–3101, Toronto, Canada, 2023b. Association for Computational… find
  190. Xuansheng Wu, Wenlin Yao, Jianshu Chen, Xiaoman Pan, Xiaoyang Wang, Ninghao Liu, and Dong Yu. From language modeling to instruction following: Understanding the behavior shift in llms after instruction tuning. arXiv preprint arXiv:2310.00492, 2023c. find
  191. Zhengxuan Wu and Desmond C. Ong. On Explaining Your Explanations of BERT: An Empirical Study with Sequence Classification, January 2021. URL http://arxiv.org/abs/2101.00196. arXiv:2101.00196 [cs]. — [arXiv:2101.00196](https://arxiv.org/abs/2101.00196) ↗
  192. Zhengxuan Wu, Thanh-Son Nguyen, and Desmond Ong. Structured Self-Attention Weights Encode Semantics in Sentiment Analysis. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 255–264, Online, November 2020a. Association for Computational Lingui… find
  193. Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4166–4176, Online, July 2020b. Association for Computational Linguistics. doi: 10… find
  194. Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13063, 2023. find
  195. Xi Ye and Greg Durrett. The unreliability of explanations in few-shot prompting for textual reasoning. Advances in neural information processing systems, 35:30378–30392, 2022. find our card
  196. Xi Ye and Greg Durrett. Explanation selection using unlabeled data for chain-of-thought prompting, 2023. find
  197. Catherine Yeh, Yida Chen, Aoyu Wu, Cynthia Chen, Fernanda Viégas, and Martin Wattenberg. Atten-tionViz: A Global View of Transformer Attention, May 2023. URL http://arxiv.org/abs/2305.03210. arXiv:2305.03210 [cs]. — [arXiv:2305.03210](https://arxiv.org/abs/2305.03210) ↗
  198. Chih-Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. Advances in neural information processing systems, 31, 2018. find
  199. Fan Yin, Jesse Vig, Philippe Laban, Shafiq Joty, Caiming Xiong, and Chien-Sheng Jason Wu. Did you read the instructions? rethinking the effectiveness of task definitions in instruction learning. arXiv preprint arXiv:2306.01150, 2023. find
  200. Yordan Yordanov, Vid Kocijan, Thomas Lukasiewicz, and Oana-Maria Camburu. Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations in a Label-Abundant Setup. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 3486–3501, Abu Dhabi, United Arab Emirates, Dec… find
  201. Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc Concept Bottleneck Models, February 2023. URL http://arxiv.org/abs/2205.15480. arXiv:2205.15480 [cs, stat]. — [arXiv:2205.15480](https://arxiv.org/abs/2205.15480) ↗
  202. Lining Zhang, Mengchen Wang, Liben Chen, and Wenxin Zhang. Probing GPT-3’s Linguistic Knowledge on Semantic Tasks. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 297–304, Abu Dhabi, United Arab Emirates (Hybrid), December 2022a. Associatio… find
  203. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022b. find
  204. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models, September 2023. URL http: //arxiv.org/a… find our card
  205. Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual Probing Is [MASK]: Learning vs. Learning to Recall, December 2021. URL http://arxiv.org/abs/2104.05240. arXiv:2104.05240 [cs]. ↗ our card
  206. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023. find
  207. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023. find our card