Elamur Research⌕
☰
home / papers / 2023 / Decomposed Prompting: A Modular Approach for Solving Complex Tasks

Decomposed Prompting: A Modular Approach for Solving Complex Tasks

Paper · 2023 · Tushar Khot, Harsh Trivedi, Matthew Finlayson et al.

Summary

DecomP (AI2, ICLR 2023) is a few-shot prompting scheme in which a large task is broken down by a separate LLM "decomposer": it writes a "program" of steps of the form (function, sub-query, answer), and each sub-query goes to its own handler from a library. A handler can be another prompt, a further decomposition (hierarchy), a call to the decomposer itself on a smaller input (recursion), or a symbolic function such as Elasticsearch retrieval, a calculator, or a trained model. Execution is managed by a simple imperative controller with operators such as foreach and references #1, #2 to previous answers; the [EOQ] marker terminates the program. The authors compare against CoT and least-to-most on GPT-3 (text-davinci-002, Codex) and Flan-T5: k-th letter concatenation, list reversal, CommaQA-E, open-domain multi-hop QA (HotpotQA, 2WikiMultihopQA, MuSiQue), and two math datasets. In math the contribution is modest but telling: separating answer extraction from the reasoning chain into a dedicated module yields +14–17 points. For us, this is a general architecture for a tree of subtasks: the decomposer defines the nodes, and the leaves are handed off to specialized handlers, including an exact core for arithmetic.

Key points

  • Features: no hand-crafted features; the structure is specified by example decompositions in the prompt; subtasks can be taught with separate, richer sets of examples than fit into a single CoT.
  • Architecture: decomposer LLM → controller → handlers (prompts, recursive calls, symbolic APIs); greedy decoding; modules are swappable and can be debugged individually.
  • Numbers (math, Fig. 16, test subsamples — 300 GSM8K, 200 MultiArith): GSM8K — CoT 36 → DecomP 50.6; MultiArith — 78 → 95 (a separate GPT-3 module extracts the answer instead of the "answer is" template).
  • Other: on k-th letter concatenation DecomP stays near 100 % on longer inputs where CoT drops; recursive reversal gives davinci-001 generalization to longer lists; on CommaQA DecomP outperforms CoT and does not degrade on the compositional split; in open-domain QA with retrieval it surpasses the single-step retrieval baseline everywhere except Codex on HotpotQA.
  • Limitations: only large models of GPT-3 size reliably write structured programs; errors occur mostly in subtask execution; API calls are expensive, so evaluation is done on subsamples.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom381 works · 80,747 citations🎓Stony Brook UniversityStony Brook · United States46 works · 3,024 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark and Ashish Sabharwal. 2023. Decomposed Prompting: A Modular Approach for Solving Complex Tasks. ICLR 2023 (arXiv 2210.02406).

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2210.02406

@misc{khot2023decomposed,
  title = {{Decomposed Prompting: A Modular Approach for Solving Complex Tasks}},
  author = {Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark and Ashish Sabharwal},
  year = {2023},
  eprint = {2210.02406},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.02406}
}

References

34 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In CVPR, 2016.
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  3. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. ↗
  4. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  6. Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. In ICLR, 2023. URL https://openreview.net/forum?id=3Pf3Wg6o-A4. ↗
  7. David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A Saurous, Jascha Sohl-dickstein, et al. Language model cascades. arXiv preprint arXiv:2207.10342, 2022.
  8. Dheeru Dua, Shivanshu Gupta, Sameer Singh, and Matt Gardner. Successive prompting for decomposing complex questions. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1251–1265, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational… find
  9. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: Program-aided language models. ArXiv, abs/2211.10435, 2022. find
  10. Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. Neural module networks for reasoning over text. In ICLR, 2020. find
  11. Xanh Ho, A. Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In COLING, 2020.
  12. Yichen Jiang and Mohit Bansal. Self-assembling modular networks for interpretable multi-hop reasoning. In EMNLP, 2019. find
  13. Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explanations. In EMNLP, 2022.
  14. Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Text modular networks: Learning to decompose tasks in the language of existing models. In NAACL, 2021.
  15. Tushar Khot, Kyle Richardson, Daniel Khashabi, and Ashish Sabharwal. Hey AI, can you solve complex tasks by talking to agents? In Findings of ACL, 2022. find
  16. Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev. Internet-augmented language models through few-shot prompting for open-domain question answering. ArXiv, abs/2203.05115, 2022.
  17. Yoav Levine, Itay Dalmedigos, Ori Ram, Yoel Zeldes, Daniel Jannai, Dor Muhlgay, Yoni Osin, Opher Lieber, Barak Lenz, Shai Shalev-Shwartz, et al. Standing on the shoulders of giant frozen language models. arXiv preprint arXiv:2204.10019, 2022. — [arXiv:2204.10019](https://arxiv.org/abs/2204.10019) find
  18. Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. Multi-hop reading comprehension through question decomposition and rescoring. In ACL, 2019.
  19. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models. ArXiv, abs/2112.00114, 2021.
  20. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Low… — Training language models to follow instructions with human feedback
  21. Aaron Parisi, Yao Zhao, and Noah Fiedel. TALM: Tool augmented language models. arXiv preprint arXiv:2205.12255, 2022.
  22. Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. Unsupervised question decomposition for question answering. In EMNLP, 2020.
  23. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. In NeurIPS, 2021.
  24. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022. find
  25. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100, 000+ questions for machine comprehension of text. In EMNLP, 2016. — SQuAD: 100,000+ Questions for Machine Comprehension of Text
  26. Subhro Roy and Dan Roth. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1743–1752, 2015. doi: 10.18653/v1/D15-1202. URL https://aclanthology.org/D15-1202. ↗
  27. Timo Schick, Jane Dwivedi-Yu, Roberto Dessı̀, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. ArXiv, abs/2302.04761, 2023.
  28. Alon Talmor and Jonathan Berant. The web as a knowledge-base for answering complex questions. In NAACL, 2018.
  29. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multi-hop questions via single-hop question composition. TACL, 2022. find
  30. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR, 2023.
  31. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In NeurIPS, 2022. — Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  32. Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. Break it down: A question understanding benchmark. TACL, 2020.
  33. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP, 2018.
  34. Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. In ICLR, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/khot-2022-decomposed-prompting-modular-approach/