Elamur Research⌕
☰
home / papers / 2026 / Self-Guided Plan Extraction for Instruction-Following Tasks with Goal-Conditional Reinforcement Learning

Self-Guided Plan Extraction for Instruction-Following Tasks with Goal-Conditional Reinforcement Learning

Paper · 2026 · Zoya Volovikova, Nikita Sorokin, Dmitriy Lukashevskiy et al.

Summary

Reinforcement-learning agents that follow natural-language instructions struggle to link free-form text to long-horizon behavior when reward arrives only after the whole task is finished. Prior LLM-planning approaches typically assume a fixed set of subgoals that the environment can verify, and often rely on very large models. SuperIgor drops these assumptions by co-training a language-model planner and an RL policy without manually annotated data.

In the first cycle, the LLM builds three things: a normalized subtask vocabulary, a goal-level plan for each instruction, and an ontology of hypothesized prerequisites. It then expands these into several candidate plans. A PPO agent learns to execute plan steps, aided by a skill curriculum and a DONE action for switching subtasks. Each plan is scored by the agent's success rate, and these scores become preferences for DPO fine-tuning of the planner. In later cycles, the planner reranks candidates by NLL.

On the dynamic, partially observable CrafText benchmark, the authors report state-of-the-art out-of-distribution performance. Their ablations indicate that the gains come from planner–policy co-adaptation rather than from extra compute.

Key points

  • Each training cycle has four stages: LLM plan generation, PPO policy learning, plan validation by execution success, and DPO fine-tuning of the planner. An SFT-pretrained plan model is also used.
  • Ablation on the Atom subset of CrafText (success rate, Cycle-1 → Cycle-2):
    • Full system: 0.35 → 0.45.
    • Without ontology-based plans: 0.06 in Cycle-1.
    • Without curriculum: 0.08 in Cycle-1.
    • Without DPO: 0.39 in Cycle-2.
    • Without SFT: 0.25 → 0.13.
  • Compute-matched tests with a fixed planner:
    • Extending PPO to 15B environment steps plateaus around 0.30–0.31 (peak of 0.32 at 5B).
    • At 5B steps (train / New Objects): longer training scores 0.30/0.11, reward filtering 0.34/0.10, and SuperIgor (SI-DPO) 0.36/0.17.
  • Limitations: the automatically induced ontology may contain redundant or cyclic dependencies, and training failures are hard to attribute to the policy versus the plan space. The authors note that several mid-sized LLMs already produce stable plan graphs.
  • Code and dataset are publicly released via the project site (sites.google.com/view/super-igor). The paper appeared in Findings of ACL 2026.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Zoya Volovikova, Nikita Sorokin, Dmitriy Lukashevskiy, Aleksandr Panov and Alexey Skrynnik. 2026. Self-Guided Plan Extraction for Instruction-Following Tasks with Goal-Conditional Reinforcement Learning. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1691

@inproceedings{volovikova2026self,
  title = {{Self-Guided Plan Extraction for Instruction-Following Tasks with Goal-Conditional Reinforcement Learning}},
  author = {Zoya Volovikova, Nikita Sorokin, Dmitriy Lukashevskiy, Aleksandr Panov and Alexey Skrynnik},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1691},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1691}
}

References

31 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. 2022. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639–24654. find
  2. Adrian Bolton, Alexander Lerchner, Alexandra Cordell, Alexandre Moufarek, Andrew Bolt, Andrew Lampinen, Anna Mitenkova, Arne Olav Hallingstad, Bojan Vujatovic, Bonnie Li, et al. 2025. Sima 2: A generalist embodied agent for virtual worlds. arXiv preprint arXiv:2512.04797. — [arXiv:2512.04797](https://arxiv.org/abs/2512.04797) find
  3. Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. 2018. Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprint arXiv:1810.08272. find
  4. Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. 2022. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35:18343– 18362. find
  5. Jonathan Gray, Kavya Srinet, Yacine Jernite, Haonan Yu, Zhuoyuan Chen, Demi Guo, Siddharth Goyal, C. Lawrence Zitnick, and Arthur Szlam. 2019. Craftassist: A framework for dialogue-enabled interactive agents. find
  6. Dongge Han, Trevor McInroe, Adam Jelley, Stefano V. Albrecht, Peter Bell, and Amos Storkey. 2025. LLM-personalize: Aligning LLM planners with human preferences via reinforced self-training for housekeeping robots. In Proceedings of the 31st International Conference on Computational Linguistics, page… find
  7. Austin W. Hanjie, Victor Zhong, and Karthik Narasimhan. 2021. Grounding language to entities and dynamics for generalization in reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 4051–4062. PM… find
  8. Felix Hill, Olivier Tieleman, Tamara von Glehn, Nathaniel Wong, Hamza Merzic, and Stephen Clark. 2021. Grounded language learning fast and slow. In International Conference on Learning Representations. find
  9. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference on machine learning, page 9118–9147. PMLR.
  10. Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, et al. 2023. Do as i can, not as i say: Grounding language in robotic affordances. In Proceedings of The 6th Conference on Robot Learning, volume 205 of Proce…
  11. Peter Jansen. 2020. Visually-grounded planning without vision: Language models infer detailed plans from high-level instructions. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4412–4417, Online. Association for Computational Linguistics. find
  12. Muyao Li, Zihao Wang, Kaichen He, Xiaojian Ma, and Yitao Liang. 2025. JARVIS-VLA: Post-training large-scale vision language models to play visual games with keyboards and mouse. In Findings of the Association for Computational Linguistics: ACL 2025, pages 17878–17899, Vienna, Austria. Association fo… find
  13. Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, and Sheila McIlraith. 2023. Steve-1: A generative model for text-to-behavior in minecraft. Advances in Neural Information Processing Systems, 36:69900– 69929. find
  14. Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan. 2024. Learning to model the world with language. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 29992–30017. PMLR. find
  15. Lajanugen Logeswaran, Yao Fu, Moontae Lee, and Honglak Lee. 2022. Few-shot subgoal planning with language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5493–5506. find
  16. Aryan Mathur and Asaduddin Ahmed. 2025. Adapting interleaved encoders with ppo for language-guided reinforcement learning in babyai. arXiv preprint arXiv:2510.23148. — [arXiv:2510.23148](https://arxiv.org/abs/2510.23148) find
  17. Fabian Paischer, Thomas Adler, Markus Hofmarcher, and Sepp Hochreiter. 2023. Semantic helm: A human-readable memory for reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems. find
  18. Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. 2018. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
  19. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741.
  20. Manolis Savva, Jitendra Malik, Devi Parikh, Dhruv Batra, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, and Vladlen Koltun. 2019. Habitat: A platform for embodied AI research. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 201… find
  21. Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, page…
  22. Alessandro Suglia, Qiaozi Gao, Jesse Thomason, Govind Thattai, and Gaurav S. Sukhatme. 2021. Embodied bert: A transformer model for embodied, language-guided visual task completion. In EMNLP 2021 Workshop on Novel Ideas in Learning-to-Learn through Interaction. find
  23. Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. 2024. True knowledge comes from practice: Aligning large language models with embodied environments via reinforcement learning. In International Conference on Learning Representations. find
  24. Zoya Volovikova, Gregory Gorbov, Petr Kuderov, Aleksandr Panov, and Alexey Skrynnik. 2025. CrafText benchmark: Advancing instruction following in complex multimodal open-ended world. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p… find
  25. Zoya Volovikova, Alexey Skrynnik, Petr Kuderov, and Aleksandr I. Panov. 2024. Instruction following with goal-conditioned reinforcement learning in virtual environments. In Proceedings of ECAI 2024, the 27th European Conference on Artificial Intelligence, volume 392 of Frontiers in Artificial Intell… find
  26. Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. 2023. Describe, explain, plan and select: Interactive planning with llms enables open-world multi-task agents. In Advances in Neural Information Processing Systems, volume 36. find
  27. Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. 2022. Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection. Advances in Neural Information Processing Systems, 35:9125–9138. find
  28. Victor Zhong, Austin W. Hanjie, Sida I. Wang, Karthik Narasimhan, and Luke Zettlemoyer. 2021. Silg: The multi-environment symbolic interactive language grounding benchmark. In Advances in Neural Information Processing Systems, volume 34, pages 21505– 21519. find
  29. Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. 2020. Rtfm: Generalising to new environment dynamics via reading. In International Conference on Learning Representations. find
  30. Enshen Zhou, Yiran Qin, Zhenfei Yin, Yuzhou Huang, Ruimao Zhang, Lu Sheng, Yu Qiao, and Jing Shao. 2024. Minedreamer: Learning to follow instructions via chain-of-imagination for simulated-world control. arXiv preprint arXiv:2403.12037. — [arXiv:2403.12037](https://arxiv.org/abs/2403.12037) find
  31. Zhiyuan Zhou, Pranav Atreya, Abraham Lee, Homer Rich Walke, Oier Mees, and Sergey Levine. 2025. Autonomous improvement of instruction following skills via foundation models. In Proceedings of The 8th Conference on Robot Learning, volume 270 of Proceedings of Machine Learning Research, pages 4805–482… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/volovikova-2026-self-guided-plan-extraction-instruction/