Elamur Research⌕
☰
home / papers / 2023 / Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs

Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs

Paper · 2023 · Angelica Chen, Jason Phang, Alicia Parrish et al.

Summary

Evaluations of LLMs usually check whether answers are correct, not whether a model's answers agree with each other. This paper argues that agreement matters for multi-step reasoning, where a final answer depends on answers to sub-steps. The authors formally define two new kinds of self-consistency. Hypothetical consistency asks whether a model can predict what it would output for a prompt when asked indirectly. Compositional consistency asks whether the final output stays the same when an intermediate sub-question is replaced by the model's own answer to it.

For hypothetical consistency, the authors use a multiple-choice setup, which avoids having to judge semantic equivalence automatically. For compositional consistency, they test on arithmetic expressions and GeoQuery semantic parsing.

Several GPT-3 and GPT-4 variants show low consistency of both kinds. Larger models do somewhat better. The results suggest that LLM outputs on complex compositional tasks should not be trusted without careful empirical checking.

Key points

  • Hypothetical consistency setup:
    • Prompts are sampled from Wikipedia and DailyDialog.
    • Each prompt is rewritten with four hypothetical templates, and results are averaged across templates to reduce sensitivity to wording.
    • The model chooses among five options: the original dataset continuation, its own greedy completion, and three completions from other models.
  • Arithmetic errors: about 90% of compositional inconsistencies happen when the sub-expression is answered correctly but the final answer does not match the answer to the substituted version.
  • GeoQuery errors:
    • Roughly 59% of inconsistencies occur because the subtree's parse is not reused inside the parent parse.
    • The remaining ~41% come from an incorrect parse of the child query.
  • Correctness vs. consistency on arithmetic: across four model sizes, the two are roughly linearly related. All models except davinci-001 are slightly more correct than consistent.
  • Relation to prior work: the study builds on earlier work by Elazar et al. (2021) and Jang et al. (2022) that measured consistency across paraphrases. It also ties into claims by Dziri et al. (2023) that Transformers are inherently limited on compositional tasks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States430 works · 424,363 citations🎓New York UniversityNew York · United States201 works · 100,180 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Angelica Chen, Jason Phang, Alicia Parrish, Vishakh Padmakumar, Chen Zhao, Samuel R. Bowman and Kyunghyun Cho. 2023. Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs. arXiv:2305.14279 · Transactions on Machine Learning Research (2024).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.14279 · arXiv 2305.14279

@misc{chen2023two,
  title = {{Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs}},
  author = {Angelica Chen, Jason Phang, Alicia Parrish, Vishakh Padmakumar, Chen Zhao, Samuel R. Bowman and Kyunghyun Cho},
  year = {2023},
  doi = {10.48550/arxiv.2305.14279},
  eprint = {2305.14279},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.14279}
}

References

20 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  2. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. ↗
  3. OpenAI Help Center. Do the openai api models have knowledge of current events? URL — Faith and Fate: Limits of Transformers on Compositionality https://help.openai.com/en/articles/ 6639781-do-the-openai-api-models-have-knowledge-of-current-events. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhaga… ↗
  4. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and Improving Consistency in Pretrained Language Models. Transactions of the Association for Computational Linguistics, 9:1012–1031, December 2021. ISSN 2307-387X. doi: 10.1… find
  5. Constanza Fierro and Anders Søgaard. Factual consistency of multilingual pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 3046–3052, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-acl.240. UR… find
  6. Myeongjun Jang, Deuk Sin Kwon, and Thomas Lukasiewicz. Accurate, yet inconsistent? consistency analysis on language understanding models. 2021. URL https://arxiv.org/abs/2108.06665. — [arXiv:2108.06665](https://arxiv.org/abs/2108.06665) ↗
  7. Myeongjun Jang, Deuk Sin Kwon, and Thomas Lukasiewicz. BECEL: Benchmark for consistency evaluation of language models. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 3680–3696, Gyeongju, Republic of Korea, October 2022. International Committee on Computational… find
  8. Nora Kassner, Oyvind Tafjord, Hinrich Schütze, and Peter Clark. BeliefBank: Adding memory to a pre-trained language model for a systematic notion of belief. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8849–8861, Online and Punta Cana, Dominican Repu… find
  9. Rohit J. Kate, Yuk Wah Wong, and Raymond J. Mooney. Learning to transform natural to formal languages. In Proceedings of the 20th National Conference on Artificial Intelligence - Volume 3, AAAI’05, pp. 1062–1068. AAAI Press, 2005. ISBN 157735236x. find
  10. Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9332–9346, Online, November 2020. Association for Computa…
  11. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 986–995, Taipei, Taiwan, November 2017. Asian F…
  12. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3?, 2021.
  13. Eric Mitchell, Joseph J. Noh, Siyan Li, William S. Armstrong, Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher D. Manning. Enhancing self-consistency and performance of pretrained language models with nli. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Proce… find
  14. Benjamin Newman, Prafulla Kumar Choubey, and Nazneen Rajani. P-adapters: Robustly extracting factual information from language models with diverse prompts. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=DhzIU48OcZh. ↗
  15. OpenAI. Model index model-index-for-researchers. for researchers. OpenAI. Gpt-4 technical report, 2023. https://platform.openai.com/docs/ ↗
  16. Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in multiple-choice questions, 2023.
  17. Harsh Raj, Domenic Rosati, and Subhabrata Majumdar. Measuring reliability of large language models through semantic consistency. In NeurIPS ML Safety Workshop, 2022. URL https://openreview.net/ forum?id=SgbpddeEV-C. ↗
  18. Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2655–2671, Seattle, United States, July 2022.…
  19. Wikimedia Foundation. Wikimedia downloads. URL — Learning to Parse Database Queries Using Inductive Logic Programming https://dumps.wikimedia.org. John M. Zelle and Raymond J. Mooney. Learning to parse database queries using inductive logic programming. In Proceedings of the Thirteenth National Conference on Artificial Intelligence - Volume 2, AAAI’96, pp. 1050–1055. A… ↗
  20. Chunting Zhou, Junxian He, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Prompt consistency for zero-shot task generalization. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 2613–2626, Abu Dhabi, United Arab Emirates, December 2022. Association for Computation… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2023-two-failures-self-consistency-multi/