Elamur Research⌕
☰
home / papers / 2023 / CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization

CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization

Paper · 2023 · Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Peter Jansen et al.

Summary

Agents built on LLMs can act in text-based simulated worlds without costly reinforcement learning. Reflection-based methods such as Reflexion can retry a single task better, but their lessons tend to be tied to one setting and do not accumulate. CLIN (continual learning from interactions) addresses this. The model's parameters stay frozen. Instead, the agent keeps a persistent textual memory of causal abstractions, which are short statements about which actions are necessary for, or irrelevant to, certain outcomes. After every trial, a memory generator rewrites this memory. Across episodes, a "meta-memory" summarizes the best memories so they can be reused in new environments or tasks. The agent itself has three parts: a controller proposes the next goal, an executor turns it into a valid action, and the memory supplies guidance. In ScienceWorld, CLIN improves steadily over repeated trials and beats Reflexion by about 23 points. It also transfers what it has learned to unseen environments and tasks. The work suggests that frozen-model agents can keep learning through structured memory rather than fine-tuning.

Key points

  • The evaluation uses 18 ScienceWorld tasks, with up to 10 test variants per task. Electricity tasks are excluded because they do not follow standard electrical conventions. Tasks are labeled Short or Long depending on whether their oracle length falls below or at/above a 37-step median.
  • In the adaptation setting, the average scores are:
    • ReAct: 29.6
    • Reflexion: 39.4
    • CLIN BASE: 48.6
    • CLIN ADAPT: 62.2
  • CLIN's advantage is largest on Long tasks, where it scores 61.6 versus 28.9 for Reflexion.
  • Transferring meta-memory raises zero-shot performance by 4 points in new environments and 13 points on new tasks. Continued memory updates then add a further 17 and 7 points, respectively.
  • Memory entries are phrased with causal markers such as "SHOULD BE NECESSARY", "MAY CONTRIBUTE" and "DOES NOT CONTRIBUTE". This format can be seen as a linguistic form of action model learning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PennsylvaniaPhiladelphia · United States221 works · 63,234 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States275 works · 97,634 citations🎓University of ArizonaTucson · United States38 works · 10,603 citations🔬Allen InstituteSeattle · United States7 works · 4,351 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

Cite

Cite as Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Peter Jansen, Oyvind Tafjord, Niket Tandon, Li Zhang, Chris Callison-Burch and Peter Clark. 2023. CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization. arXiv:2310.10134.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.10134 · arXiv 2310.10134

@misc{majumder2023clin,
  title = {{CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization}},
  author = {Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Peter Jansen, Oyvind Tafjord, Niket Tandon, Li Zhang, Chris Callison-Burch and Peter Clark},
  year = {2023},
  doi = {10.48550/arxiv.2310.10134},
  eprint = {2310.10134},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.10134}
}

References

25 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adaptive-Agent-Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal M. P. Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks-Thompson, Hannah Openshaw, Ja… find
  2. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Jayant… — Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
  3. Diego Aineto, Sergio Jiménez, and Eva Onaindía. Learning strips action models with classical planning. In International Conference on Automated Planning and Scheduling, 2018. URL https://api.semanticscholar.org/CorpusID:49405691. ↗
  4. Prithviraj Ammanabrolu and Matthew J. Hausknecht. Graph constrained reinforcement learning for natural language action spaces. In ICLR, 2020. find
  5. Prithviraj Ammanabrolu, Jack Urbanek, Margaret Li, Arthur Szlam, Tim Rocktaschel, and Jason Weston. How to motivate your dragon: Teaching goal-driven agents to speak and act in fantasy worlds. In North American Chapter of the Association for Computational Linguistics, 2020. URL https://api.semantics… ↗
  6. Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Joshua Tobin, P. Abbeel, and Wojciech Zaremba. Hindsight experience replay. ArXiv, abs/1707.01495, 2017. URL https://api.semanticscholar.org/CorpusID:3532908. ↗
  7. Ankuj Arora, Humbert Fiorino, Damien Pellier, Marc Métivier, and Sylvie Pesty. A review of learning planning action models. The Knowledge Engineering Review, 33, 2018. URL https://api.semanticscholar.org/CorpusID:56483203. ↗
  8. Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, P. Abbeel, A. Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Neural Information Processing Systems, 2021. URL https://api.semanticscholar.org/CorpusID:235294299. ↗
  9. Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Ya… find
  10. Bhavana Dalvi, Oyvind Tafjord, and Peter Clark. Towards teachable reasoning systems: Using a dynamic memory of user feedback for continual system improvement. In EMNLP, 2022. find
  11. Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. Deep reinforcement learning with a natural language action space. arXiv: Artificial Intelligence, 2015. URL https://api.semanticscholar.org/CorpusID:15986631. ↗
  12. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pp. 9118–9147. PMLR, 2022.
  13. Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Nicolaus Foerster, Edward Grefenstette, and Tim Rocktaschel. Replay-guided adversarial environment design. In Neural Information Processing Systems, 2021. URL https://api.semanticscholar.org/CorpusID:238408352. ↗
  14. Bill Yuchen Lin, Yicheng Fu, Karina Yang, Prithviraj Ammanabrolu, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. ArXiv, abs/2305.17390, 2023. URL https://api.semanticscholar.org/Cor… ↗
  15. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. ArXiv, ab… find
  16. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602, 2013. URL https://api.semanticscholar.org/CorpusID:15238391. ↗
  17. Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023. find
  18. Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley D. Edwards, Nicolas Manfred Otto Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, M… find
  19. Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https://arxiv.org/abs/1908. 10084. ↗
  20. Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023.
  21. Niket Tandon, Aman Madaan, Peter Clark, and Yiming Yang. Memory-assisted prompt editing to improve GPT-3 after deployment. In ACL Workshop on Commonsense Representation and Reasoning (CSRR’22), 2022. (also arxiv:2201.06009). find
  22. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi (Jim) Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. ArXiv, abs/2305.16291, 2023. URL https://api.semanticscholar.org/CorpusID: 258887849. ↗
  23. Ruoyao Wang, Peter Alexander Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than a 5th grader? In Conference on Empirical Methods in Natural Language Processing, 2022. URL https://api.semanticscholar.org/CorpusID:247451124. ↗
  24. Shunyu Yao, Rohan Rao, Matthew J. Hausknecht, and Karthik Narasimhan. Keep calm and explore: Language models for action generation in text-based games. ArXiv, abs/2010.02903, 2020. URL https://api.semanticscholar.org/CorpusID:222142129. ↗
  25. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ArXiv, abs/2210.03629, 2022. URL https://api.semanticscholar.org/CorpusID:252762395. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/majumder-2023-clin-continually-learning-language-agent/