Elamur Research⌕
☰
home / papers / 2023 / Collaborating with language models for embodied reasoning

Collaborating with language models for embodied reasoning

Paper · 2023 · Ishita Dasgupta, Christine Kaeser-Chen, Kenneth Marino et al.

Summary

Large language models reason well and adapt to new tasks through in-context learning, but they cannot perceive or act in an environment. RL agents can act, yet they need large amounts of data and generalize poorly. The authors combine the two in a three-module architecture. A pre-trained Chinchilla model acts as the Planner and turns a task description into simple text instructions. An RL-trained Actor carries these out in a partially observable 2D grid-world (PycoLab). A Reporter converts the outcomes back into text for the Planner. The tasks require explicit information gathering: the agent examines objects to uncover a hidden "secret property" and then decides which object to pick up. With five few-shot demonstrations and only pre-trained components, the system solves these tasks zero-shot, while a pure RL baseline does poorly. The 70B Planner is much more accurate than the 7B one and copes better with noisy reports. The authors also show that a Reporter can be learned from scratch with RL using episode reward. This is an early step toward replacing hand-coded feedback when grounding language models in embodied settings.

Key points

  • Architecture: The Planner is Chinchilla (7B or 70B), prompted with 5 solved example dialogues. The Actor is pre-trained only on "pick up X" and "examine X" instructions, sees an egocentric RGB view limited to 5 squares, and can examine or pick up objects.
  • Conditional task: Given complete information, the 7B Planner chooses the right object 58% of the time and the 70B Planner 96% of the time. A pure RL baseline stays weak even after 100M learner frames.
  • Search task: Here the agent must examine several objects in turn. The agents surprisingly score higher than on the single-examine task, likely because the flexible number of steps lets the Planner recover from Actor mistakes.
  • Noisy Reporter: Irrelevant movement messages were injected 20% of the time. This reduced accuracy only moderately for 70B, which coped by repeating or cycling instructions. Showing a repeat strategy in the prompt fully restored 70B performance, while 7B recovered only partly.
  • Trained Reporter: On a visual conditional task (is an object near the wall?), a Reporter with a binary report head was trained from scratch with RL and improved task performance. The work appeared at the NeurIPS 2022 LaReL workshop.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksEngineering · 206 worksControl and Systems Engineering · 72 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksReasoning · 1,948 works

Cite

Cite as Ishita Dasgupta, Christine Kaeser-Chen, Kenneth Marino, Arun Ahuja, Sheila Babayan, Felix Hill and Rob Fergus. 2023. Collaborating with language models for embodied reasoning. arXiv:2302.00763.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2302.00763 · arXiv 2302.00763

@misc{dasgupta2023collaborating,
  title = {{Collaborating with language models for embodied reasoning}},
  author = {Ishita Dasgupta, Christine Kaeser-Chen, Kenneth Marino, Arun Ahuja, Sheila Babayan, Felix Hill and Rob Fergus},
  year = {2023},
  doi = {10.48550/arxiv.2302.00763},
  eprint = {2302.00763},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2302.00763}
}

References

13 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
  2. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. J. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C.… — Language Models are Few-Shot Learners
  3. L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. CoRR, abs/1802.01561, 2018. URL http://arxiv.org/ abs/1802.01561. ↗
  4. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre. Training compute-opt… — Training Compute-Optimal Large Language Models
  5. W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022. find
  6. R. Kirk, A. Zhang, E. Grefenstette, and T. Rocktaschel. A survey of generalisation in deep reinforcement learning. ArXiv, abs/2111.09794, 2021. — [arXiv:2111.09794](https://arxiv.org/abs/2111.09794) find
  7. T. D. Kulkarni, K. Narasimhan, A. Saeedi, and J. Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29, 2016. find
  8. M. Mitchell. Abstraction and analogy-making in artificial intelligence. Annals of the New York Academy of Sciences, 1505(1):79–101, 2021.
  9. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019. find
  10. J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P.-S. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. F.… find
  11. J. Russin, R. C. O’Reilly, and Y. Bengio. Deep learning needs a prefrontal cortex. Work Bridging AI Cogn Sci, 107:603–616, 2020. find
  12. R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181–211, 1999. find
  13. A. Zeng, A. Wong, S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani, J. Lee, V. Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dasgupta-2023-collaborating-language-models-embodied-reasoning/