Elamur Research⌕
☰
home / papers / 2023 / Analyzing Chain-of-Thought Prompting in Large Language Models via Gradient-based Feature Attributions

Analyzing Chain-of-Thought Prompting in Large Language Models via Gradient-based Feature Attributions

Paper · 2023 · Skyler Wu, Eric Meng Shen, Charumathi Badrinath et al.

Summary

Chain-of-thought (CoT) prompting is known to improve LLM accuracy on reasoning questions, but it is still unclear why it works. The authors, from Harvard, study this without reading the generated explanations, which can be wrong or misleading. Instead, they use gradient-based saliency scores, which estimate how much each input token influences the token carrying the final answer. They compare standard few-shot prompting with CoT prompting on open-source models of 1.5–6B parameters, mainly GPT-J. The setup uses four attribution variants, 50-question samples from four question-answering datasets, hand-labelled sets of semantically relevant tokens, and manually rephrased question versions. Contrary to their hypothesis, CoT did not make the models weight relevant tokens more heavily; those saliency magnitudes actually dropped. Under CoT, however, the scores changed much less when questions were reworded or when generation was repeated. The study shows that CoT changes how models process input even below the scale where accuracy gains appear. It also offers an interpretability approach that does not depend on the model's own explanations.

Key points

  • Models: GPT-J (6B) for the main results, plus GPT-2 XL (1.5B) and GPT-Neo (2.7B). The four saliency methods are the regular and contrastive versions of the gradient × input and L1 norm methods from Yin & Neubig (2022).
  • Data: 50 questions each from SST (Stanford Sentiment Treebank v2), GSM8K, CSQA (CommonsenseQA) and a manually written CoinFlip set. CoinFlip and GSM8K also have reworded versions with the same meaning.
  • Accuracy: CoT gave no general improvement at this scale. The exception was SST, where accuracy rose from 0.5 to 0.64.
  • Across all datasets and attribution methods, CoT lowered the saliency magnitudes of relevant tokens. The authors attribute this to the longer prompt spreading the gradient over more tokens.
  • CoT made relevant-token saliencies more stable across original and reworded GSM8K/CoinFlip questions. It also produced tighter clusters over 20 repeated generation runs. The work was presented at an ICML 2023 workshop.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harvard UniversityCambridge · United States112 works · 42,023 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksInterpretability · 130 works

Cite

Cite as Skyler Wu, Eric Meng Shen, Charumathi Badrinath, Jiaqi Ma and Himabindu Lakkaraju. 2023. Analyzing Chain-of-Thought Prompting in Large Language Models via Gradient-based Feature Attributions. arXiv:2307.13339.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.13339 · arXiv 2307.13339

@misc{wu2023analyzing,
  title = {{Analyzing Chain-of-Thought Prompting in Large Language Models via Gradient-based Feature Attributions}},
  author = {Skyler Wu, Eric Meng Shen, Charumathi Badrinath, Jiaqi Ma and Himabindu Lakkaraju},
  year = {2023},
  doi = {10.48550/arxiv.2307.13339},
  eprint = {2307.13339},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.13339}
}

References

25 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Madaan, A. and Yazdanbakhsh, A. Text and patterns: For effective chain of thought, it takes two to tango. arXiv preprint arXiv:2209.07686, 2022. find
  2. Black, S., Leo, G., Wang, P., Leahy, C., and Biderman, S. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. URL https://doi.org/10.5281/zenodo.5297715. If you use this software, please cite it using these metadata. — [doi:10.5281/zenodo.5297715](https://doi.org/10.5281/zenodo.5297715) ↗
  3. Madsen, A., Reddy, S., and Chandar, S. Post-hoc interpretability for neural NLP: A survey. ACM Computing Surveys, 55(8):1–42, dec 2022. doi: 10.1145/3546577. URL https://doi.org/10.1145%2F3546577. find
  4. Bolukbasi, T., Pearce, A., Yuan, A., Coenen, A., Reif, E., Viégas, F., and Wattenberg, M. An interpretability illusion for bert, 2021. find
  5. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  6. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  7. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631– 1642, Seattle, Washington, USA, Oct…
  8. Golovneva, O., Chen, M., Poff, S., Corredor, M., Zettlemoyer, L., Fazel-Zarandi, M., and Celikyilmaz, A. Roscoe: A suite of metrics for scoring step-by-step reasoning, 2022.
  9. Sundararajan, M., Taly, A., and Yan, Q. Axiomatic attribution for deep networks, 2017. URL https://arxiv.org/abs/1703.01365. ↗
  10. Jacovi, A., Swayamdipta, S., Ravfogel, S., Elazar, Y., Choi, Y., and Goldberg, Y. Contrastive explanations for model interpretability. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1597–1611, Online and Punta Cana, Dominican Republic, November 2021. A… find
  11. Talmor, A., Herzig, J., Lourie, N., and Berant, J. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Shor…
  12. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners, 2022. URL https://arxiv.org/abs/2205.11916. ↗
  13. Turpin, M., Michael, J., Perez, E., and Bowman, S. R. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting, 2023. — Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
  14. Krishna, S., Han, T., Gu, A., Pombra, J., Jabbari, S., Wu, S., and Lakkaraju, H. The disagreement problem in explainable machine learning: A practitioner’s perspective, 2022. find
  15. Wallace, E., Tuyls, J., Wang, J., Subramanian, S., Gardner, M., and Singh, S. Allennlp interpret: A framework for explaining predictions of nlp models. arXiv preprint arXiv:1909.09251, 2019.
  16. Lampinen, A. K., Dasgupta, I., Chan, S. C., Matthewson, K., Tessler, M. H., Creswell, A., McClelland, J. L., Wang, J. X., and Hill, F. Can language models learn from explanations in context? arXiv preprint arXiv:2204.02329, 2022.
  17. Wang, B. and Komatsuzaki, A. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/ mesh-transformer-jax, May 2021. ↗
  18. Wang, B., Min, S., Deng, X., Shen, J., Wu, Y., Zettlemoyer, L., and Sun, H. Towards understanding chain-of-thought prompting: An empirical study of what matters. arXiv preprint arXiv:2212.10001, 2022.
  19. Li, J., Chen, X., Hovy, E., and Jurafsky, D. Visualizing and understanding neural models in NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 681–691, San Diego, California, June 2016. Associati…
  20. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reasoning in language models, 2023.
  21. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL http…
  22. Yin, K. and Neubig, G. Interpreting language models with contrastive explanations. arXiv preprint arXiv:2202.10419, 2022. find
  23. Zelikman, E., Wu, Y., Mu, J., and Goodman, N. D. Star: Bootstrapping reasoning with reasoning, 2022a. URL https://arxiv.org/abs/2203.14465. ↗
  24. Zelikman, E., Wu, Y., Mu, J., and Goodman, N. D. Star: Bootstrapping reasoning with reasoning. mar 2022b.
  25. Zhang, Z., Zhang, A., Li, M., and Smola, A. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2023-analyzing-chain-thought-prompting-large/