Elamur Research⌕
☰
home / papers / 2026 / DIXITWORLD: Evaluating Multimodal Abductive Reasoning in Vision-Language Models with Multi-Agent Dixit Gameplay

DIXITWORLD: Evaluating Multimodal Abductive Reasoning in Vision-Language Models with Multi-Agent Dixit Gameplay

Paper · 2026 · Yunxiang MO, Tianshi Zheng, Qing Zong et al.

Summary

The paper asks how well vision-language models (VLMs) handle multimodal abductive reasoning, meaning proposing and choosing the most plausible explanation from incomplete visual evidence. Earlier benchmarks such as VAR and Sherlock test this only in static, single-agent settings. The authors built DIXITWORLD, based on the board game Dixit, which has two parts:

  • DixitArena, a four-player game environment. A Storyteller writes a cryptic clue for an image, which is hypothesis generation. Listeners then pick the matching image from decoys, which is hypothesis selection.
  • DixitBench, a static multiple-choice benchmark that tests only the listener role.

Six VLMs were compared in a round-robin tournament. The main finding is an asymmetry between roles. Larger proprietary models are the strongest listeners. Storytelling quality does not grow with model size, and most clues end up either too obvious or too obscure. DixitBench scores track arena listener performance very closely, so it can serve as a cheap substitute for the full game. The authors argue that VLMs need explicit modeling of ambiguity, intention and theory-of-mind to become balanced abductive agents.

Key points

  • Models tested: Qwen2.5-VL-7B/32B, Gemma3-12B/27B, GPT-4o and Gemini-2.5-Flash, all at temperature 0.7.
    • Gemini-2.5-Flash had the best Overall arena score (36.52%).
    • Gemma3-27B was the best storyteller (32.14%).
  • DixitBench design: 252 items, built from 84 images × 3 difficulty tiers. Each item has 6 candidate images, and distractors are chosen by caption semantic similarity. Human validation agreement was κ > 0.8.
  • Storyteller failures: over 78% of storyteller rounds scored zero, because either every listener guessed correctly or every listener guessed wrong.
  • Listener results: GPT-4o was the top listener on DixitBench (75.6%). DixitBench results correlate with arena listener results at r = 0.947.
  • Robustness checks:
    • A 72B scaling test showed the asymmetry persists at larger scale.
    • All models were poorly calibrated (ECE > 0.17).
    • Results varied by less than ±3.0pp across random seeds.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hong Kong University of Science and TechnologyHong Kong · China142 works · 25,136 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksVision & language · 657 works

Cite

Cite as Yunxiang MO, Tianshi Zheng, Qing Zong, Jiayu Liu, Baixuan Xu, Yauwai Yim, Chunkit Chan, Jiaxin Bai and Yangqiu Song. 2026. DIXITWORLD: Evaluating Multimodal Abductive Reasoning in Vision-Language Models with Multi-Agent Dixit Gameplay. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-short.37

@inproceedings{mo2026dixitworld,
  title = {{DIXITWORLD: Evaluating Multimodal Abductive Reasoning in Vision-Language Models with Multi-Agent Dixit Gameplay}},
  author = {Yunxiang MO, Tianshi Zheng, Qing Zong, Jiayu Liu, Baixuan Xu, Yauwai Yim, Chunkit Chan, Jiaxin Bai and Yangqiu Song},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-short.37},
  url = {https://doi.org/10.18653/v1/2026.acl-short.37}
}

References

25 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Saaket Agashe, Yue Fan, Anthony Reyna, and Xin Eric Wang. 2025. Llm-coordination: Evaluating and analyzing multi-agent coordination abilities in large language models. Preprint, arXiv:2310.03903. find
  2. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. Preprint, arXiv:2502.13…
  3. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. Preprint, arXiv:1908.05739.
  4. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. Piqa: Reasoning about physical commonsense in natural language. Preprint, arXiv:1911.11641.
  5. Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. 2019. On the utility of learning about humans for human-ai coordination. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. find
  6. Harry Frankfurt. 1958. Peirce’s notion of abduction. Journal of Philosophy, 55:593–596. find
  7. Google. 2025. Gemini: A family of highly capable multimodal models. Preprint, arXiv:2312.11805.
  8. Jiaxian Guo, Bo Yang, Paul Yoo, Bill Yuchen Lin, Yusuke Iwasawa, and Yutaka Matsuo. 2024. Suspicion-agent: Playing imperfect information games with theory of mind aware gpt-4. Preprint, arXiv:2309.17277. — [arXiv:2309.17277](https://arxiv.org/abs/2309.17277) find
  9. Jack Hessel, Jena D. Hwang, Jae Sung Park, Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, and Yejin Choi. 2022. The abduction of sherlock holmes: A dataset for visual abductive reasoning. Preprint, arXiv:2202.04800. — [arXiv:2202.04800](https://arxiv.org/abs/2202.04800) find
  10. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 911…
  11. Huao Li, Yu Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Charles Lewis, and Katia Sycara. 2023. Theory of mind for multi-agent collaboration via large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational L…
  12. Chen Liang, Wenguan Wang, Tianfei Zhou, and Yi Yang. 2022. Visual abductive reasoning. Preprint, arXiv:2203.14040. — [arXiv:2203.14040](https://arxiv.org/abs/2203.14040) find
  13. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Preprint, arXiv:2304.08485.
  14. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others.… — GPT-4 Technical Report
  15. Jae Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi, Ali Farhadi, and Yejin Choi. 2020. Visualcomet: Reasoning about the dynamic context of a still image. Preprint, arXiv:2004.10796. find
  16. Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685.
  17. Tianshi Zheng, Jiayang Cheng, Chunyang Li, Haochen Shi, Zihao Wang, Jiaxin Bai, Yangqiu Song, Ginny Y. Wong, and Simon See. 2025. Logidynamics: Unraveling the dynamics of logical inference in large language model reasoning. Preprint, arXiv:2502.11176. — [arXiv:2502.11176](https://arxiv.org/abs/2502.11176) find
  18. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, New York, NY, USA. Ass… find
  19. Charles Sanders Peirce. 1931. Collected Papers of Charles Sanders Peirce, Vol. 5: Pragmatism and Pragmaticism. Harvard University Press. find
  20. Christoph Riedl, Young Ji Kim, Pranav Gupta, Thomas W. Malone, and Anita Williams Woolley. 2021. Quantifying collective intelligence in human groups. Proceedings of the National Academy of Sciences, 118. find
  21. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  22. Rose E. Wang, Sarah A. Wu, James A. Evans, David C. Parkes, Joshua B. Tenenbaum, and Max Kleiman-Weiner. 2021. 152too many cooks: Bayesian inference for coordinating multi-agent collaboration. In Human-Like Machine Intelligence. Oxford University Press. find
  23. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. Preprint, arXiv:22…
  24. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Preprint, arXiv:2305.10601. find
  25. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mo-2026-dixitworld-evaluating-multimodal-abductive-reasoning/