Elamur Research⌕
☰
home / papers / 2026 / Lightweight and Faithful Visual Condition Checking in Behavior Trees via Expert-Regularized Reinforcement Learning

Lightweight and Faithful Visual Condition Checking in Behavior Trees via Expert-Regularized Reinforcement Learning

Paper · 2026 · Hyosik Moon and Eldan Cohen

Summary

Behavior trees (BTs) make expert-designed policies modular and readable. However, they need symbolic condition checks, which are hard to define over raw images or text. Large VLMs can answer natural-language condition queries directly on perceptual inputs, but calling them at every decision step is slow and expensive. Moon and Cohen propose CERL (Condition-node Expert-regularized Reinforcement Learning), which trains a small shared binary classifier to evaluate each image–query condition node. The method has three parts:

  • It casts BT execution as an MDP and derives a policy gradient for it.
  • A limited budget of expert labels (e.g., from a VLM) initializes the classifier and then regularizes RL training, so node outputs keep their intended meaning instead of drifting toward pure reward-seeking.
  • A factorized policy treats all condition decisions in one root-to-leaf traversal as a single decision unit, which eases credit assignment under sparse rewards.

Evaluated on seven tasks spanning the GymCards, FrozenLake and BabyAIText benchmark collections, CERL outperforms both pure imitation learning and pure RL baselines. It stays highly consistent with expert judgments while being far smaller and faster than the expert. This makes interpretable, language-specified BT policies feasible in real-time settings.

Key points

  • Evaluation covers seven tasks drawn from the GymCards, FrozenLake and BabyAIText suites. All are sparse-reward settings, where feedback arrives only at episode end.
  • The approach changes only condition nodes. Action and control-flow nodes stay untouched, which preserves the original BT structure and its transparency.
  • In EZPoints, the expert (GPT-4.1-mini) struggles with arithmetic visual conditions. RL lets the distilled model outperform it there, while agreement with the expert on the other conditions stays around 99%.
  • In the card tasks, CERL learns when to stop drawing in high-risk states, which raises win rates over the expert's fixed decision logic.
  • The authors note two limitations:
    • CERL depends on a well-designed BT and does not learn its structure or queries.
    • It needs an accurate expert labeler. Noisy labels can be mitigated with tricks such as prompt repetition.
  • Code is published at github.com/HyosikMoon/CERL.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TorontoToronto · Canada110 works · 179,061 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Hyosik Moon and Eldan Cohen. 2026. Lightweight and Faithful Visual Condition Checking in Behavior Trees via Expert-Regularized Reinforcement Learning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1935

@inproceedings{moon2026lightweight,
  title = {{Lightweight and Faithful Visual Condition Checking in Behavior Trees via Expert-Regularized Reinforcement Learning}},
  author = {Hyosik Moon and Eldan Cohen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1935},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1935}
}

References

38 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fernando Acero and Zhibin Li. 2024. Distilling reinforcement learning policies for interpretable robot locomotion: Gradient boosting machines and symbolic regression. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6840–6847. IEEE. find
  2. Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. 2009. A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5):469–483. find
  3. Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. 2018. Verifiable reinforcement learning via policy extraction. Advances in Neural Information Processing Systems, 31:2499–2509. find
  4. Aude Billard, Sylvain Calinon, Ruediger Dillmann, and Stefan Schaal. 2008. Robot programming by demonstration. In Springer Handbook of Robotics, pages 1371–1394. Springer. find
  5. Leo Breiman, Jerome Friedman, Richard A Olshen, and Charles J Stone. 2017. Classification and regression trees. Chapman and Hall/CRC. find
  6. Michele Colledanchise and Petter Ögren. 2018. Behavior trees in robotics and AI: an introduction. CRC Press. find
  7. Michele Colledanchise, Ramviyas Parasuraman, and Petter Ögren. 2018. Learning of behavior trees for autonomous agents. IEEE Transactions on Games, 11(2):183–189. find
  8. Akash Ghosh, Arkadeep Acharya, Sriparna Saha, Vinija Jain, and Aman Chadha. 2024. Exploring the frontier of vision-language models: a survey of current methodologies and future directions. arXiv preprint arXiv:2404.07214. — [arXiv:2404.07214](https://arxiv.org/abs/2404.07214) find
  9. Ryan Hoque, Ashwin Balakrishna, Ellen Novoseller, Albert Wilcox, Daniel S Brown, and Ken Goldberg. 2022. ThriftyDAgger: budget-aware novelty and risk gating for interactive imitation learning. In Proceedings of the 5th Conference on Robot Learning (CoRL), pages 598–608. PMLR. find
  10. Hao Hu, Xiaoliang Jia, Kuo Liu, and Bingyang Sun. 2021. Self-adaptive traffic control model with behavior trees and reinforcement learning for AGV in industry 4.0. IEEE Transactions on Industrial Informatics, 17(12):7968–7979. find
  11. Yu Huang, Ziji Wu, Kexin Ma, and Ji Wang. 2025. Differentiable synthesis of behavior tree architectures and execution nodes. In Proceedings of the 2nd International Conference on Neuro-Symbolic Systems (NeuS 2025), volume 288. PMLR. find
  12. Matteo Iovino, Edvards Scukins, Jonathan Styrud, Petter Ögren, and Christian Smith. 2022. A survey of behavior trees in robotics and AI. Robotics and Autonomous Systems, 154:104096. find
  13. Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbell, and Mykel J Kochenderfer. 2019. HG-DAgger: interactive imitation learning with human experts. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 8077–8083. IEEE. find
  14. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2025. Prompt repetition improves non-reasoning llms. arXiv preprint arXiv:2512.14982. find
  15. Xianglong Li, Yuan Li, Jieyuan Zhang, Xinhai Xu, and Donghong Liu. 2024. Embedding multi-agent reinforcement learning into behavior trees with unexpected interruptions. Complex & Intelligent Systems, 10(3):3273–3282. find
  16. Jianlan Luo, Perry Dong, Yuexiang Zhai, Yi Ma, and Sergey Levine. 2024. RLIF: interactive imitation learning as reinforcement learning. In Proceedings of the 12th International Conference on Learning Representations (ICLR). find
  17. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2023. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594. find
  18. Ryan Marcotte and Howard J Hamilton. 2017. Behavior trees for modelling artificial intelligence in games: a tutorial. The Computer Games Journal, 6(3):171– 184. find
  19. W James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. 2019. Definitions, methods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences, 116(44):22071–22080. find
  20. Petter Ögren and Christopher I Sprague. 2022. Behavior trees in robot control systems. Annual Review of Control, Robotics, and Autonomous Systems, 5(1):81– 107. find
  21. Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński, Lerrel Pinto, Rob Fergus, and 1 others. 2025. BALROG: benchmarking agentic LLM and VLM reasoning on games. In Proceedings of the 13th International Conference on Lea… find
  22. Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS), pages 627–635. JMLR Workshop and Conference Proce… find
  23. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2015. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438. find
  24. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  25. Andrew Silva, Taylor Killian, Ivan Jimenez, Sung-Hyun Son, and Matthew Gombolay. 2020. Optimization methods for interpretable differentiable decision trees applied to reinforcement learning. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), page… find
  26. Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460–9471. find
  27. Daniel Stonier and 1 others. 2025. py_trees: Pythonic behaviour trees for python. https://github.com/ splintered-reality/py_trees. ↗
  28. Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12:1057–1063. find
  29. Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulao, Andreas Kallinteris, Markus Krimmel, Arjun KG, and 1 others. 2024. Gymnasium: a standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032.
  30. Naoki Wake, Atsushi Kanehira, Jun Takamatsu, Kazuhiro Sasabuchi, and Katsushi Ikeuchi. 2025. VLM-driven behavior tree for context-aware task planning. arXiv preprint arXiv:2501.03968. — [arXiv:2501.03968](https://arxiv.org/abs/2501.03968) find
  31. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In Proceedings of the 13th International Conference on Learning Representations (ICLR).
  32. Yongyan Wen, Siyuan Li, Rongchang Zuo, Lei Yuan, Hangyu Mao, and Peng Liu. 2025. Skilltree: Explainable skill-based deep reinforcement learning for long-horizon control tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 21491–21500. find
  33. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229–256. find
  34. Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. 2021. Policy finetuning: bridging sample-efficient offline and online reinforcement learning. Advances in Neural Information Processing Systems, 34:27395–27407. find
  35. Jiahua Xu, Yunhan Lin, Haotian Zhou, and Huasong Min. 2022. Generating manipulation sequences using reinforcement learning and behavior trees for peg-in-hole task. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 2715–2720. IEEE. find
  36. Zhenghai Xue, Zhenghao Peng, Quanyi Li, Zhihan Liu, and Bolei Zhou. 2023. Guarded policy optimization with imperfect online demonstrations. In Proceedings of the 11th International Conference on Learning Representations (ICLR). find
  37. Maryam Zare, Parham M Kebria, Abbas Khosravi, and Saeid Nahavandi. 2024. A survey of imitation learning: algorithms, recent developments, and challenges. IEEE Transactions on Cybernetics. find
  38. Simon Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Peter Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, and 1 others. 2024. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. Advances in Neural Information Processing Systems, 37:110935–110971. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/moon-2026-lightweight-faithful-visual-condition-checking/