Elamur Research⌕
☰
home / papers / 2026 / AVA: Attentive VLM Agent for Mastering StarCraft II

AVA: Attentive VLM Agent for Mastering StarCraft II

Paper · 2026 · Weiyu Ma, Yuqian Fu, Zecheng Zhang et al.

Summary

Most StarCraft II testbeds, such as SMAC and SMACv2, give agents abstract feature vectors and support only reinforcement learning. Vision-language models therefore cannot be tested on fine-grained unit control. The authors build AVACraft, a benchmark that offers RGB screen and minimap images, textual battlefield descriptions and structured unit data in one POMDP setting with Fog of War. Training-based and zero-shot agents can thus be compared under identical conditions. The benchmark keeps full unit abilities and uses adaptive opponents: the built-in AI plus scripted behavior-tree policies that are drawn at random. Baselines cover six MARL algorithms with Swin-Transformer encoders, each trained for 5M steps, and several proprietary and open VLMs used without training. VLM agents reach 75–90% win rates zero-shot, while the best MARL result is only 19.3%. Blinded expert raters also judge VLM behavior as more coherent and human-like. The work maps out trade-offs in sample efficiency, interpretability and cost, and motivates hybrid RL–VLM agents.

Key points

  • AVACraft has 21 scenarios: 12 micromanagement, 5 coordination and 4 strategic. All support PvE and PvP, use sparse ±1/0 rewards and end at a 300-second time limit.
  • MARL baselines are IQL, QMIX, QTRAN, VDN, MAPPO and IPPO. VLMs include GPT-4o, GPT-4-Turbo, GPT-4o-mini, Qwen3-VL-8B and Qwen3-VL-30B.
  • The MARL agents did not benefit from added text input (IPPO Vision+Text did worse than vision alone). Every VLM scored 0% on the hardest maps.
  • Seven raters, ranging from a professional player to a spectator, preferred VLM agents on all metrics (Mann-Whitney U, p < 0.01).
  • On cost, MARL training takes about 65–80 h on two A100s, with ~8–10 ms inference per step. GPT-4o needs ~2.3 s and ~$0.027 per decision, about $4.05 per episode. The Qwen3-VL models run on a single A100 with no API fees.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Weiyu Ma, Yuqian Fu, Zecheng Zhang, Bernard Ghanem and Guohao Li. 2026. AVA: Attentive VLM Agent for Mastering StarCraft II. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.208

@inproceedings{ma2026ava,
  title = {{AVA: Attentive VLM Agent for Mastering StarCraft II}},
  author = {Weiyu Ma, Yuqian Fu, Zecheng Zhang, Bernard Ghanem and Guohao Li},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.208},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.208}
}

References

32 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2024. Claude 3 model card. find
  2. Jinze Bai et al. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966.
  3. Anthony Brohan et al. 2023. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818. find
  4. Shaofei Cai et al. 2023. Groot: Learning to follow instructions by watching gameplay videos. arXiv preprint arXiv:2310.08235. — [arXiv:2310.08235](https://arxiv.org/abs/2310.08235) find
  5. Yue Deng, Weiyu Ma, Yuxin Fan, Yin Zhang, Haifeng Zhang, and Jian Zhao. 2024a. A new approach to solving smac task: Generating decision tree code from large language models. Preprint, arXiv:2410.16024. — [arXiv:2410.16024](https://arxiv.org/abs/2410.16024) find
  6. Yue Deng, Yan Yu, Weiyu Ma, Zirui Wang, Wenhui Zhu, Jian Zhao, and Yin Zhang. 2024b. Smac-hard: Enabling mixed opponent strategy script and self-play on smac. Preprint, arXiv:2412.17707. — [arXiv:2412.17707](https://arxiv.org/abs/2412.17707) find
  7. Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob N. Foerster, and Shimon Whiteson. 2023. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. Preprint, arXiv:2212.07489. — [arXiv:2212.07489](https://arxiv.org/abs/2212.07489) find
  8. Lei Han, Jiechao Xiong, Peng Sun, Xinghai Sun, Meng Fang, Qingwei Guo, Qiaobo Chen, Tengfei Shi, Hongsheng Yu, Xipeng Wu, et al. 2020. Tstarbot-x: An open-sourced and comprehensive study for efficient league training in starcraft ii full game. arXiv preprint arXiv:2011.13729. — [arXiv:2011.13729](https://arxiv.org/abs/2011.13729) find
  9. Hongliang He et al. 2024. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919.
  10. Ruozi Huang, Xipeng Wu, Hongsheng Yu, Zhong Fan, Haobo Fu, QIANG FU, and Yang Wei. 2023. A robust and opponent-aware league training method for starcraft ii. In Thirty-seventh Conference on Neural Information Processing Systems. find
  11. Zongyuan Li, Yanan Ni, Runnan Qi, Lumin Jiang, Chang Lu, Xiaojie Xu, Xiangbei Liu, Pengfei Li, Yunzheng Guo, Zhe Ma, et al. 2024. Llm-pysc2: Starcraft ii learning environment for large language models. arXiv preprint arXiv:2411.05348. — [arXiv:2411.05348](https://arxiv.org/abs/2411.05348) find
  12. Michael Mathieu, Sherjil Ozair, Srivatsan Srinivasan, Caglar Gulcehre, Shangtong Zhang, Ray Jiang, Tom Le Paine, Konrad Zolna, Richard Powell, Julian Schrittwieser, et al. 2021. Starcraft ii unplugged: Large scale offline reinforcement learning. In Deep RL Workshop NeurIPS 2021. find
  13. OpenAI. 2023a. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  14. OpenAI. 2023b. Gpt-4v(ision) system card. find
  15. Davide Paglieri et al. 2025. Balrog: Benchmarking agentic llm and vlm reasoning on games. ICLR. find
  16. Alec Radford et al. 2021. Learning transferable visual models from natural language supervision. In ICML.
  17. Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. 2019. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043. — [arXiv:1902.04043](https://arxiv.org/abs/1902.04043) find
  18. DI star Contributors. 2021. Di-star: An open-sourse reinforcement learning framework for starcraftii. https://github.com/opendilab/DI-star. ↗
  19. Weihao Tan et al. 2024. Cradle: Empowering foundation agents towards general computer control. arXiv preprint arXiv:2403.03186. — [arXiv:2403.03186](https://arxiv.org/abs/2403.03186) find
  20. Gemini Team et al. 2023. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  21. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  22. Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. 2019. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350–354. find
  23. Zihao Wang et al. 2025. Jarvis-vla: Post-training large-scale vision language models to play visual games with keyboards and mouse. arXiv preprint arXiv:2503.16365. find
  24. Haotian Liu et al. 2023. Visual instruction tuning. NeurIPS.
  25. Xinrun Xu et al. 2024. Mcu: An evaluation framework for open-ended game agents. arXiv preprint arXiv:2310.08367. — [arXiv:2310.08367](https://arxiv.org/abs/2310.08367) find
  26. Weiyu Ma, Qirui Mi, Yongcheng Zeng, Xue Yan, Yuqiao Wu, Runji Lin, Haifeng Zhang, and Jun Wang. 2024. Large language models play starcraft ii: Benchmarks and a chain of summarization approach. Preprint, arXiv:2312.11865. — [arXiv:2312.11865](https://arxiv.org/abs/2312.11865) find
  27. Zelai Xu, Zican Xu, Xuanhan Yi, Hongru Yuan, Xin Chen, Yi Wu, Chao Yu, and Yu Wang. 2025. Vs-bench: Evaluating vlms for strategic reasoning and decision-making in multi-agent environments. arXiv preprint arXiv:2506.02387. — [arXiv:2506.02387](https://arxiv.org/abs/2506.02387) find
  28. Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. 2022. The surprising effectiveness of PPO in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611–24624. find
  29. Alex L. Zhang et al. 2025a. Videogamebench: Can vision-language models complete popular video games? arXiv preprint arXiv:2505.18134. — [arXiv:2505.18134](https://arxiv.org/abs/2505.18134) find
  30. Chen Zhang, Qiang He, Zhou Yuan, Elvis S. Liu, Hong Wang, Jian Zhao, and Yang Wang. 2024. Advancing drl agents in commercial fighting games: Training, integration, and agent-human alignment. Preprint, arXiv:2406.01103. — [arXiv:2406.01103](https://arxiv.org/abs/2406.01103) find
  31. Chen Zhang, Huan Hu, Yuan Zhou, Xu Wang, and Elvis S. Liu. 2025b. Hifas: A hybrid interactive fps agent system for large game maps. IEEE Transactions on Games, pages 1–13. find
  32. Zhonghan Zhao et al. 2024. Steve: See and think: Embodied agent in virtual environment. In ECCV. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2026-ava-attentive-vlm-agent-mastering/