Elamur Research⌕
☰
home / papers / 2026 / MoEC: A Memory-Routed Mixture-of-Experts Controller for Adaptive Minecraft Control

MoEC: A Memory-Routed Mixture-of-Experts Controller for Adaptive Minecraft Control

Paper · 2026 · Hui Wu, Chao Xu, Jianghui Wang et al.

Summary

Minecraft agents usually pair an LLM planner with a single low-level controller. The authors show this setup is fragile. STEVE-1 receiving identical subgoals yields very different results from trial to trial, because one shared parameter set must cover conflicting behaviours. They propose MoEC, which replaces the single controller with a growing library of lightweight expert policy heads on a frozen backbone, trained online with reinforcement learning. Routing is retrieval-based rather than learned: a non-parametric memory keyed by subgoal first filters experts by subgoal and then picks one by context similarity. If no expert fits, a base policy is used. When execution fails, an LLM judge decides whether to replan or to train a new expert. Redundant experts are merged so memory and inference cost stay bounded. On JARVIS-1 task groups, MoEC is best or second-best, beats language and action baselines, and nearly matches Optimus-1 under a stricter protocol. This suggests that adaptive modularity in the control layer improves robustness without retraining the backbone.

Key points

  • Motivating probe: with fixed subgoals, STEVE-1 collected between 2 and 7 seeds and between 8 and 13 dirt across trials.
  • Expert memory ablation: gains level off beyond about 60% of experts retained.
    • white_carpet scores 0.82 at 20% memory and 0.86 at full memory.
    • chain rises from 0.35 to 0.47.
  • Sequential fine-tuning of a monolithic controller: performance drops on both earlier and newly added tasks.
    • P4 is worse than P2 on Dirt.
    • MoEC reaches 15.4 on Dirt versus 12.2 for frozen STEVE-1.
  • Efficiency and the judge:
    • Head-only adaptation matches or beats full fine-tuning on seed and dirt tasks and takes about 28.4 s for 3 epochs.
    • Judge-labelled failures shift from mostly control (≥65%) early in training to mostly planning (≥90%) later.
  • Open-weight LLMs: replacing GPT-4o with LLaMA2-70B as planner and judge lowers success only slightly.
    • wooden_pickaxe: 96.77% → 92.52%.
    • stone_pickaxe: 94.52% → 90.84%.
    • The method has not been evaluated outside Minecraft.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China246 works · 24,952 citations🎓Jilin UniversityChangchun · China30 works · 5,252 citations🏢Advanced Micro DevicesSunnyvale · United States3 works🎓Aerospace Information Research InstituteBeijing · China3 works

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Hui Wu, Chao Xu, Jianghui Wang, Ziqiong Liu, Dong Li, Yiwei Dai and Emad Barsoum. 2026. MoEC: A Memory-Routed Mixture-of-Experts Controller for Adaptive Minecraft Control. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1027

@inproceedings{wu2026moec,
  title = {{MoEC: A Memory-Routed Mixture-of-Experts Controller for Adaptive Minecraft Control}},
  author = {Hui Wu, Chao Xu, Jianghui Wang, Ziqiong Liu, Dong Li, Yiwei Dai and Emad Barsoum},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1027},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1027}
}

References

42 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Artemij Amiranashvili, Nicolai Dorka, Wolfram Burgard, Vladlen Koltun, and Thomas Brox. 2020. Scaling imitation learning in minecraft. arXiv preprint arXiv:2007.02701. — [arXiv:2007.02701](https://arxiv.org/abs/2007.02701) find
  2. Abrar Anwar, John Welsh, Joydeep Biswas, Soha Pouya, and Yan Chang. 2025. Remembr: Building and reasoning over long-horizon spatio-temporal memory for robot navigation. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 2838–2845. IEEE. find
  3. Pierre-Luc Bacon, Jean Harb, and Doina Precup. 2017. The option-critic architecture. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17), pages 1726–1734. AAAI Press. find
  4. Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. 2022. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639–24654. find
  5. Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. 2018. End-to-end driving via conditional imitation learning. In 2018 IEEE international conference on robotics and automation (ICRA), pages 4693–4700. IEEE. find
  6. Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, and 1 others. 2022. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547–5569. PMLR.
  7. Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. 2022. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35:18343– 18362.
  8. William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39. find
  9. William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. 2019. Minerl: A large-scale dataset of minecraft demonstrations. Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI. find
  10. William Hebgen Guss, Stephanie Milani, Nicholay Topin, Brandon Houghton, Sharada Mohanty, Andrew Melnik, Augustin Harter, Benoit Buschmaas, Bjarne Jaster, Christoph Berganski, and 1 others. 2021. Towards robust and domain agnostic reinforcement learning competitions: Minerl 2020. In NeurIPS 2020 Com… find
  11. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference on machine learning, pages 9118–9147. PMLR.
  12. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, and 1 others. 2023. Inner monologue: Embodied reasoning through planning with language models. In Conference on Robot Learning, pages 1769–1782. PMLR. find
  13. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991. Adaptive mixtures of local experts. Neural computation, 3(1):79–87. find
  14. Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. 2016. The malmo platform for artificial intelligence experimentation. In Ijcai, volume 16, pages 4246–4247. find
  15. Daehee Lee, Minjong Yoo, Woo Kyung Kim, Wonje Choi, and Honguk Woo. 2024. Incremental learning of retrievable skills for efficient continual task adaptation. Advances in Neural Information Processing Systems, 37:17286–17312. find
  16. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668. find
  17. Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. 2016. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1–40. find
  18. Hao Li, Xue Yang, Zhaokai Wang, Xizhou Zhu, Jie Zhou, Yu Qiao, Xiaogang Wang, Hongsheng Li, Lewei Lu, and Jifeng Dai. 2024a. Auto mc-reward: Automated dense reward design with large language models for minecraft. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p… find
  19. Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Weili Guan, Dongmei Jiang, and Liqiang Nie. 2025. Optimus-3: Towards generalist multimodal minecraft agents with scalable task experts. arXiv preprint arXiv:2506.10357. — [arXiv:2506.10357](https://arxiv.org/abs/2506.10357) find
  20. Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. 2024b. Optimus-1: Hybrid multimodal memory empowered agents excel in long-horizon tasks. arXiv preprint arXiv:2408.03615. — [arXiv:2408.03615](https://arxiv.org/abs/2408.03615) find
  21. Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, and Sheila McIlraith. 2023. Steve-1: A generative model for text-to-behavior in minecraft. Advances in Neural Information Processing Systems, 36:69900– 69929. find
  22. Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, and Wei Yang. 2022. Juewu-mc: Playing minecraft with sample-efficient hierarchical reinforcement learning. In In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI), pages 3257–3263. find
  23. Hangyu Mao, Chao Wang, Xiaotian Hao, Yihuan Mao, Yiming Lu, Chengjie Wu, Jianye Hao, Dong Li, and Pingzhong Tang. 2022. Seihai: A sample-efficient hierarchical ai for the minerl competition. In Distributed Artificial Intelligence: Third International Conference, DAI 2021, Shanghai, China, December 1… find
  24. Stephanie Milani, Nicholay Topin, Brandon Houghton, William H Guss, Sharada P Mohanty, Keisuke Nakata, Oriol Vinyals, and Noboru Sean Kuno. 2020. Retrospective analysis of the 2019 minerl competition on sample efficient reinforcement learning. In NeurIPS 2019 competition and demonstration track, pag… find
  25. Anusha Nagabandi, Chelsea Finn, and Sergey Levine. 2018. Deep online learning via meta-learning: Continual adaptation for model-based rl. arXiv preprint arXiv:1812.07671. — [arXiv:1812.07671](https://arxiv.org/abs/1812.07671) find
  26. Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, and 1 others. 2024. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In 2024 IEEE International Conf… find
  27. Junyeong Park, Junmo Cho, and Sungjin Ahn. 2025. Mrsteve: Instruction-following agents in minecraft with what-where-when memory. In The Thirteenth International Conference on Learning Representations. find
  28. Yiran Qin, Enshen Zhou, Qichang Liu, Zhenfei Yin, Lu Sheng, Ruimao Zhang, Yu Qiao, and Jing Shao. 2024. Mp5: A multi-modal open-ended embodied system in minecraft via active perception. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16307–16316. IEEE. find
  29. Rouhollah Rahmatizadeh, Pooya Abolghasemi, Ladis-lau Bölöni, and Sergey Levine. 2018. Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3758–3765. IEEE. find
  30. Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. 2019. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning, pages 5331–5340. PMLR. find
  31. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634–8652.
  32. Wenxuan Song, Han Zhao, Pengxiang Ding, Can Cui, Shangke Lyu, Yaning Fan, and Donglin Wang. 2024. find
  33. Germ: A generalist robotic model with mixture-of-experts for quadruped robot. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11879–11886. IEEE. find
  34. Weikang Wan, Yifeng Zhu, Rutav Shah, and Yuke Zhu. 2024. Lotus: Continual imitation learning for robot manipulation through unsupervised skill discovery. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 537–544. IEEE. find
  35. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023a. Voyager: An open-ended embodied agent with large language models. Comment: Project website and open-source codebase: https://voyager.minedojo. org/Cited on, page 33. ↗
  36. Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, Yitao Liang, and Team CraftJarvis. 2023b. Describe, explain, plan and select: interactive planning with large language models enables open-world multi-task agents. In Proceedings of the 37th International Conference on Neural Information…
  37. Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, and 1 others. 2024. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. IEEE Transactions on Pattern Analysis and Machine Intelligence. find
  38. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629.
  39. Jesse Zhang, Jiahui Zhang, Karl Pertsch, Ziyi Liu, Xiang Ren, Minsuk Chang, Shao-Hua Sun, and Joseph J Lim. 2023. Bootstrap your own skills: Learning to solve new tasks with large language model guidance. arXiv preprint arXiv:2310.10021. — [arXiv:2310.10021](https://arxiv.org/abs/2310.10021) find
  40. Jinman Zhao, Xueyan Zhang, Jiaru Li, Jingcheng Niu, Yulan Hu, Erxue Min, and Gerald Penn. 2025. Tiny budgets, big gains: Parameter placement strategy in parameter super-efficient fine-tuning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 6326–6344.
  41. Wang Bill Zhu, Miaosen Chai, Ishika Singh, Robin Jia, and Jesse Thomason. 2025. Psalm-v: Automating symbolic planning in interactive visual environments with large language models. arXiv preprint arXiv:2506.20097. — [arXiv:2506.20097](https://arxiv.org/abs/2506.20097) find
  42. Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, and 1 others. 2023. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2026-moec-memory-routed-mixture-experts/