Elamur Research⌕
☰
home / papers / 2023 / Large Language Model Programs

Large Language Model Programs

Paper · 2023 · Imanol Schlag, Sainbayar Sukhbaatar, Asli Celikyilmaz et al.

Summary

Large language models can pick up new tasks from instructions or a few examples. They still handle algorithmic work such as searching or sorting poorly, and their fixed context window limits how much material they can use at once. Fine-tuning on demonstration traces is expensive and often fails outside the training distribution. Schlag and co-authors (IDSIA, Meta AI, KAUST) propose "LLM programs." In this approach, an ordinary program such as Python code breaks a task into simpler steps. It then queries a frozen LLM once per step with a step-specific prompt. The program, not the model, tracks the overall state, so each call sees only the information that step requires. The authors demonstrate the idea on evidence-supported question answering built on StrategyQA. First, the LLM helps discard paragraphs unrelated to the question. Then the program builds a reasoning chain step by step, using a tree search to decide which paragraph to put in context for each step. Without any fine-tuning, this design beats a chain-of-thought baseline by 6.4%. The approach offers a cheaper, more interpretable and more testable alternative to retraining, especially when knowledge sources change often.

Key points

  • The method keeps the LLM frozen. Control flow, output parsing and prompt construction live in an external program, which also handles generalisation to more input paragraphs.
  • The QA example has two stages: LLM-based filtering of a large paragraph set, then tree search over paragraph choice for each reasoning step, expanding the highest-ranked chain.
  • The QA data is derived from StrategyQA (Geva et al., 2021), a binary yes/no benchmark for implicit multi-step reasoning. The model does best when all required facts are in context (Table 2).
  • The main reported result is a 6.4% gain over chain of thought, with no fine-tuning. The construction details and exact setup are in a section missing from the provided extract.
  • The paper places the idea alongside related work: Decomposed Prompting (Khot et al., 2023), language model cascades (Dohan et al., 2022), and Schuurmans (2023), who shows that an LLM inside a recurrent loop with external memory is computationally universal.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Imanol Schlag, Sainbayar Sukhbaatar, Asli Celikyilmaz, Wen-tau Yih, Jason Weston, Jurgen Schmidhuber and Xian Li. 2023. Large Language Model Programs. arXiv:2305.05364.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.05364 · arXiv 2305.05364

@misc{schlag2023large,
  title = {{Large Language Model Programs}},
  author = {Imanol Schlag, Sainbayar Sukhbaatar, Asli Celikyilmaz, Wen-tau Yih, Jason Weston, Jurgen Schmidhuber and Xian Li},
  year = {2023},
  doi = {10.48550/arxiv.2305.05364},
  eprint = {2305.05364},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.05364}
}

References

81 references, 41 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., et al. Do as i can, not as i say: Grounding language in robotic affordances. In 6th Annual Conference on Robot Learning, 2022. URL https://openreview.net/forum?id=bdHkMjBJG_w. ↗
  2. Anil, C., Wu, Y., Andreassen, A. J., Lewkowycz, A., Misra, V., Ramasesh, V. V., Slone, A., Gur-Ari, G., Dyer, E., and Neyshabur, B. Exploring length generalization in large language models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems…
  3. Brockman, G., Murati, M., and Welinder, P., Sept 2018. URL https://openai.com/blog/ openai-api/. ↗
  4. Brown, T. B. et al. Language models are few-shot learners. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Virtual only, December 2020.
  5. Chase, H. Langchain. https://github.com/ hwchase17/langchain, 2023. [Accessed 09-Jan-2023]. ↗
  6. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  7. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  8. Cohen, R., Geva, M., Berant, J., and Globerson, A. Crawling the internal knowledge-base of language models. arXiv preprint arXiv:2301.12810, 2023.
  9. Creswell, A. and Shanahan, M. Faithful reasoning using large language models. arXiv preprint arXiv:2208.14271, 2022.
  10. Csordás, R., Irie, K., and Schmidhuber, J. The devil is in the detail: Simple tricks improve systematic generalization of transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 619–634, Online and Punta Cana, Dominican Republic, November 2021. A… find
  11. Dalvi, B., Tafjord, O., and Clark, P. Towards teachable reasoning systems. arXiv preprint arXiv:2204.13074, 2022. find
  12. Deletang, G., Ruoss, A., Grau-Moya, J., Genewein, T., Wenliang, L. K., Catt, E., Cundy, C., Hutter, M., Legg, S., Veness, J., and Ortega, P. A. Neural networks and the chomsky hierarchy. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=WbxHAzkeQcn. ↗
  13. Dev, S., Li, T., Phillips, J. M., and Srikumar, V. On measuring and mitigating biased inferences of word embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7659–7666, 2020. find
  14. Dohan, D., Xu, W., Lewkowycz, A., Austin, J., Bieber, D., Lopes, R. G., Wu, Y., Michalewski, H., Saurous, R. A., Sohl-Dickstein, J., et al. Language model cascades. arXiv preprint arXiv:2207.10342, 2022.
  15. Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., a… find
  16. Fan, A., Lavril, T., Grave, E., Joulin, A., and Sukhbaatar, S. Addressing some limitations of transformers with feedback memory, 2021. URL https://openreview.net/forum?id=OCm0rwa1lx1. ↗
  17. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. find
  18. Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. Pal: Program-aided language models. arXiv preprint arXiv:2211.10435, 2022. find
  19. Gers, F. A. and Schmidhuber, E. Lstm recurrent networks learn simple context-free and context-sensitive languages. IEEE transactions on neural networks, 12(6):1333–1340, 2001. find
  20. Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., and Berant, J. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9: 346–361, 2021. find
  21. Giannou, A., Rajput, S., Sohn, J.-y., Lee, K., Lee, J. D., and Papailiopoulos, D. Looped transformers as programmable computers. arXiv preprint arXiv:2301.13196, 2023. find
  22. Kim, N., Pavlick, E., Ayan, B. K., and Ramachandran, D. Which linguist invented the lightbulb? presupposition verification for question-answering. In Annual Meeting of the Association for Computational Linguistics, 2021.
  23. Gupta, P., Jiao, C., Yeh, Y.-T., Mehri, S., Eskenazi, M., and Bigham, J. P. Improving zero and few-shot generalization in dialogue through instruction tuning. arXiv preprint arXiv:2205.12673, 2022. — [arXiv:2205.12673](https://arxiv.org/abs/2205.12673) find
  24. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. In ICML 2022 Workshop on Knowledge Retrieval and Language Models, 2022. URL https://openreview.net/ forum?id=6p3AuaHAFiN. ↗
  25. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  26. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J. W., and Sifre, L.… find
  27. Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., Sermanet, P., Jackson, T., Brown, N., Luu, L., Levine, S., Hausman, K., and brian ichter. Inner monologue: Embodied reasoning through planning with language models. In 6th Annual Confe… find
  28. Hutchins, D., Schlag, I., Wu, Y., Dyer, E., and Neyshabur, B. Block-recurrent transformers. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=uloenYmLCAo. ↗
  29. Karpas, E., Abend, O., Belinkov, Y., Lenz, B., Lieber, O., Ratner, N., Shoham, Y., Bata, H., Levine, Y., Leyton-Brown, K., et al. Mrkl systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning. arXiv preprint arXiv:2205.004… — [arXiv:2205.00445](https://arxiv.org/abs/2205.00445) find
  30. Kazemi, S. M., Kim, N., Bhatia, D., Xu, X., and Ramachandran, D. Lambada: Backward chaining for automated reasoning in natural language. arXiv preprint arXiv:2212.13894, 2022.
  31. Khot, T., Trivedi, H., Finlayson, M., Fu, Y., Richardson, K., Clark, P., and Sabharwal, A. Decomposed prompting: A modular approach for solving complex tasks. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum? id=_nGgzQjzaRy. ↗
  32. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., and Petrov, S. Natural questions: A benchmark for question answering research. T…
  33. Lazaridou, A., Gribovskaya, E., Stokowiec, W. J., and Grigorev, N. Internet-augmented language models through few-shot prompting for open-domain question answering, 2023. URL https://openreview.net/forum? id=hFCUPkSSRE. ↗
  34. Lee, A., Wu, D., Dinan, E., and Lewis, M. Improving chess commentaries by combining language models with symbolic reasoning engines. arXiv preprint arXiv:2212.08195, 2022. — [arXiv:2212.08195](https://arxiv.org/abs/2212.08195) find
  35. Li, J., Zhang, Z., and Zhao, H. Self-prompting large language models for open-domain qa. arXiv preprint arXiv:2212.08635, 2022. — [arXiv:2212.08635](https://arxiv.org/abs/2212.08635) find
  36. Liu, B., Ash, J. T., Goel, S., Krishnamurthy, A., and Zhang, C. Transformers learn shortcuts to automata. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum? id=De4FYqjFueZ. ↗
  37. Liu, J. GPT Index, 11 2022. URL https://github.com/jerryjliu/gpt_index. ↗
  38. Liu, J., Hallinan, S., Lu, X., He, P., Welleck, S., Hajishirzi, H., and Choi, Y. Rainier: Reinforced knowledge introspector for commonsense question answering. arXiv preprint arXiv:2210.03078, 2022a.
  39. Liu, J., Liu, A., Lu, X., Welleck, S., West, P., Le Bras, R., Choi, Y., and Hajishirzi, H. Generated knowledge prompting for commonsense reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3154– 3169, Dublin, Ireland, May… find
  40. Medress, M. F., Cooper, F. S., Forgie, J. W., Green, C., Klatt, D. H., O’Malley, M. H., Neuburg, E. P., Newell, A., Reddy, D., Ritea, B., et al. Speech understanding systems: Report of a steering committee. Artificial Intelligence, 9 (3):307–316, 1977. find
  41. Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021.
  42. Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., Sutton, C., and Odena, A. Show your work: Scratchpads for intermediate computation with language models, 2022. URL https://openreview.net/ forum?id=iedYJm92o0a. ↗
  43. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Gray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with h… — Training language models to follow instructions with human feedback
  44. Paranjape, B., Michael, J., Ghazvininejad, M., Hajishirzi, H., and Zettlemoyer, L. Prompting contrastive explanations for commonsense reasoning tasks. In Workshop on Commonsense Reasoning and Knowledge Bases, 2021. URL https://openreview.net/forum? id=KFcjxSNdMBq. ↗
  45. Saparov, A. and He, H. Language models can (kind of) reason: A systematic formal analysis of chain-of-thought. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum? id=qFVVBzXxR2V. ↗
  46. Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. find
  47. Schmidhuber, J. Optimal ordered problem solver. Machine Learning, 54(3):211–254, 2004. find
  48. Schmidhuber, J. Gödel machines: Fully self-referential optimal universal self-improvers. In Artificial general intelligence, pp. 199–226. Springer, 2007. find
  49. Schmidhuber, J. On learning to think: Algorithmic information theory for novel combinations of reinforcement learning controllers and recurrent neural world models. arXiv preprint arXiv:1511.09249, 2015. — [arXiv:1511.09249](https://arxiv.org/abs/1511.09249) find
  50. Schulman, J., Zoph, B., Kim, C., Hilton, J., Menick, J., Weng, J., Ceron Uribe, J. F., Fedus, L., Metz, L., Pokorny, M., Gontijo Lopes, R., Zhao, S., Vijayvergiya, A., Sigler, E., Perelman, A., Voss, C., Heaton, M., Parish, J., Cummings, D., Nayak, R., Balcom, V., Schnurr, D., Kaftan, T., Hallacy, C… find
  51. Schuurmans, D. Memory augmented large language models are computationally universal. arXiv preprint arXiv:2301.04589, 2023.
  52. Patel, P., Mishra, S., Parmar, M., and Baral, C. Is a question decomposition unit all we need? arXiv preprint arXiv:2205.12538, 2022. find
  53. Shanahan, M. Talking about large language models. arXiv preprint arXiv:2212.03551, 2022. — [arXiv:2212.03551](https://arxiv.org/abs/2212.03551) find
  54. Perez, E., Lewis, P., Yih, W.-t., Cho, K., and Kiela, D. Unsupervised question decomposition for question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8864–8880, Online, November 2020. Association for Computational Linguistics. doi…
  55. Sheng, E., Chang, K.-W., Natarajan, P., and Peng, N. Societal biases in language generation: Progress and challenges. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long…
  56. Reppert, J., Rachbach, B., George, C., Byun, L. S. J., Appleton, M., and Stuhlmüller, A. Iterated decomposition: Improving science q&a by supervising reasoning processes. arXiv preprint arXiv:2301.01751, 2023. — [arXiv:2301.01751](https://arxiv.org/abs/2301.01751) find
  57. Shuster, K., Komeili, M., Adolphs, L., Roller, S., Szlam, A., and Weston, J. Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion. arXiv preprint arXiv:2203.13224, 2022a.
  58. Shuster, K., Xu, J., Komeili, M., Ju, D., Smith, E. M., Roller, S., Ung, M., Chen, M., Arora, K., Lane, J., et al. Blenderbot 3: a deployed conversational agent that continually learns to responsibly engage. arXiv preprint arXiv:2208.03188, 2022b.
  59. Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022a. URL https://openreview.net/forum?id=gEZrGCozdqR. ↗
  60. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.
  61. Wei, J., Wang, X., Schuurmans, D., Bosma, M., brian ichter, Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022b. URL h…
  62. Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022.
  63. Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022.
  64. Valmeekam, K., Olmo, A., Sreedharan, S., and Kambhampati, S. Large language models still can’t plan (a benchmark for LLMs on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022. URL https://openreview.net/forum?id=wUU-7XTL5XO. ↗
  65. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In Proc. Advances in Neural Information Processing Systems (NIPS), pp. 5998–6008, Long Beach, CA, USA, December 2017.
  66. Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=1PL1NIMMrw. ↗
  67. Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., Pathak, E., Karamanolakis, G., Lai, H., Purohit, I., Mondal, I., Anderson, J., Kuznia, K., Doshi, K., Pal, K. K., Patel, M., Moradshahi, M., Parmar, M., Purohit, M.,… find
  68. Wu, J., Ouyang, L., Ziegler, D. M., Stiennon, N., Lowe, R., Leike, J., and Christiano, P. F. Recursively summarizing books with human feedback. CoRR, abs/2109.10862, 2021a. URL https://arxiv.org/abs/2109. 10862. ↗
  69. Wu, J., Ouyang, L., Ziegler, D. M., Stiennon, N., Lowe, R., Leike, J., and Christiano, P. F. Recursively summarizing books with human feedback. CoRR, abs/2109.10862, 2021b. URL https://arxiv.org/abs/2109. 10862. ↗
  70. Wu, T., Terry, M., and Cai, C. J. Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1–22, 2022a. find
  71. Wu, T. S., Jiang, E., Donsbach, A., Gray, J., Molina, A., Terry, M., and Cai, C. J. Promptchainer: Chaining large language model prompts through visual programming. CHI Conference on Human Factors in Computing Systems Extended Abstracts, 2022b. find
  72. Xu, J., Szlam, A. D., and Weston, J. Beyond goldfish memory: Long-term open-domain conversation. In Annual Meeting of the Association for Computational Linguistics, 2021. find
  73. Yang, J., Jiang, H., Yin, Q., Zhang, D., Yin, B., and Yang, D. SEQZERO: Few-shot compositional semantic parsing with sequential prompts and zero-shot models. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 49–60, Seattle, United States, July 2022a. Association for Compu…
  74. Yang, K., Peng, N., Tian, Y., and Klein, D. Re3: Generating longer stories with recursive reprompting and revision. arXiv preprint arXiv:2210.06774, 2022b. find
  75. Yang, Z., Dong, L., Du, X., Cheng, H., Cambria, E., Liu, X., Gao, J., and Wei, F. Language models as inductive reasoners. arXiv preprint arXiv:2212.10923, 2022c.
  76. Yao, S., Chen, H., Yang, J., and Narasimhan, K. R. Webshop: Towards scalable real-world web interaction with grounded language agents. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=R9KnuFlvnU… ↗
  77. Zelikman, E., Wu, Y., Mu, J., and Goodman, N. STar: Bootstrapping reasoning with reasoning. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum? id=_3ELRdg2sgI. ↗
  78. Zeng, A., Attarian, M., brian ichter, Choromanski, K. M., Wong, A., Welker, S., Tombari, F., Purohit, A., Ryoo, M. S., Sindhwani, V., Lee, J., Vanhoucke, V., and Florence, P. Socratic models: Composing zero-shot multimodal reasoning with language. In International Conference on Learning Representati…
  79. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
  80. Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q. V., and Chi, E. H. Least-to-most prompting enables complex reasoning in large language models. In International Conference on Learning Representations, 2023a. URL https://openreview.net/for… ↗
  81. Zhou, H., Nova, A., Courville, A., Larochelle, H., Neyshabur, B., and Sedghi, H. Teaching algorithmic reasoning via in-context learning, 2023b. URL https://openreview.net/forum?id=6dlC7E1H_9. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schlag-2023-large-language-model-programs/