Elamur Research⌕
☰
home / papers / 2026 / DART: Leveraging Multi-Agent Disagreement for Tool Recruitment in Multimodal Reasoning

DART: Leveraging Multi-Agent Disagreement for Tool Recruitment in Multimodal Reasoning

Paper · 2026 · Nithin Sivakumaran, Justin Chen, David Wan et al.

Summary

Debates between vision-language models (VLMs) often stall because the agents share similar perceptual blind spots and bring no new evidence. Meanwhile, single-LLM tool-calling systems plan their tool use from the question alone, without looking at the image. DART combines both ideas for visual question answering. Several VLM agents first answer with chain-of-thought reasoning and a confidence score. A recruiter LLM then finds where their reasoning conflicts and calls matching vision tools to settle those conflicts. An agreement scorer measures how closely each agent matches the tool outputs. The agents discuss and revise using this evidence, and an aggregator picks the final answer. Across A-OKVQA, MMMU, NaturalBench and the medical M3D set, DART beats both multi-agent debate and tool-calling baselines such as ViperGPT and Chameleon. It also adapts to a new domain when a specialist tool is added. The work suggests that disagreement is a useful signal for deciding when external perception is needed, though the approach costs more compute than single-agent setups.

Key points

  • Pipeline and tools: DART runs in five steps: initial answers, disagreement-driven tool recruitment, tool-based agreement scoring, discussion, and aggregation.
    • Grounding, object detection, OCR and spatial reasoning are handled by fine-tuned expert models.
    • A distinct general-purpose VLM takes care of captioning and attribute detection.
  • Gains over the strongest baseline: Compared with multi-agent debate plus a judge, DART improves accuracy by 3.4% on A-OKVQA, 2.4% on MMMU and 1.2% on NaturalBench.
  • Gains over tool-calling systems: Against ViperGPT/Chameleon, DART leads by 19.0%, 11.5% and 5.3% on the same three benchmarks.
  • Medical domain (M3D): With MedGemma 4B added as a tool, DART gains 1.3% over Qwen2.5-VL, the strongest single-agent baseline. Naive debate, by contrast, falls 2.7% below Qwen2.5-VL.
  • Ablations on A-OKVQA:
    • Using only Ovis2 instead of diverse agents lowers accuracy from 68.9% to 65.3%.
    • Removing tool-based agreement scores costs 1.1%.
    • Removing expert tool outputs costs 3.2%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at AustinAustin · United States109 works · 56,942 citations🎓University of North Carolina at Chapel HillChapel Hill · United States67 works · 16,474 citations🎓Nanyang Technological UniversitySingapore · Singapore156 works · 25,644 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksVision & language · 657 works

Cite

Cite as Nithin Sivakumaran, Justin Chen, David Wan, Yue Zhang, Jaehong Yoon, Elias Stengel-Eskin and Mohit Bansal. 2026. DART: Leveraging Multi-Agent Disagreement for Tool Recruitment in Multimodal Reasoning. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.253

@inproceedings{sivakumaran2026dart,
  title = {{DART: Leveraging Multi-Agent Disagreement for Tool Recruitment in Multimodal Reasoning}},
  author = {Nithin Sivakumaran, Justin Chen, David Wan, Yue Zhang, Jaehong Yoon, Elias Stengel-Eskin and Mohit Bansal},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.253},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.253}
}

References

58 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV). find
  2. Fan Bai, Yuxin Du, Tiejun Huang, Max Q. H. Meng, and Bo Zhao. 2024. M3d: Advancing 3d medical image analysis with multi-modal large language models. Preprint, arXiv:2404.00578. — [arXiv:2404.00578](https://arxiv.org/abs/2404.00578) find
  3. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. arXiv preprint arXiv:25…
  4. Edward Y. Chang. 2025. Evince: Optimizing multi-llm dialogues using conditional statistics and information theory. Preprint, arXiv:2408.14575. — [arXiv:2408.14575](https://arxiv.org/abs/2408.14575) find
  5. Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. 2024a. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14455–14465. find
  6. Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. 2024b. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. Preprint, arXiv:2309.13007.
  7. Ming-Chang Chiu, Fuxiao Liu, Karan Sapra, Andrew Tao, Yaser Jacoob, Xuezhe Ma, Zhiding Yu, and Guilin Liu. 2025. Aide: Agentically improve visual language model with domain experts. Preprint, arXiv:2502.09051. — [arXiv:2502.09051](https://arxiv.org/abs/2502.09051) find
  8. Chung-En, Yu, Hsuan-Chih, Chen, Brian Jalaian, and Nathaniel D. Bastian. 2025. Hydra: An agentic reasoning approach for enhancing adversarial robustness and mitigating hallucinations in vision-language models. Preprint, arXiv:2504.14395. — [arXiv:2504.14395](https://arxiv.org/abs/2504.14395) find
  9. Debrup Das, Debopriyo Banerjee, Somak Aditya, and Ashish Kulkarni. 2024. Mathsensei: A tool-augmented large language model for mathematical reasoning. Preprint, arXiv:2402.17231. find
  10. Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. 2024. Naturalbench: Evaluating vision-language models on natural adversarial samples. In The Thirty-eight Conference on Neural Information Processing Sy… find
  11. Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate. Preprint, arXiv:2305.14325.
  12. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages…
  13. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2024. Improving factuality and reasoning in language models through multiagent debate. In International Conference on Machine Learning, pages 11733–11763. PMLR.
  14. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. Encouraging divergent thinking in large language models through multi-agent debate. Preprint, arXiv:2305.19118.
  15. Andrew Estornell and Yang Liu. 2024. Multi-LLM debate: Framework, principals, and interventions. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  16. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  17. Andrew Estornell, Jean-Francois Ton, Yuanshun Yao, and Yang Liu. 2025a. ACC-collab: An actor-critic approach to multi-agent LLM collaboration. In The Thirteenth International Conference on Learning Representations. find
  18. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. Lla-vanext: Improved reasoning, ocr, and world knowledge. find
  19. Andrew Estornell, Jean-Francois Ton, Yuanshun Yao, and Yang Liu. 2025b. Acc-collab: An actor-critic approach to multi-agent llm collaboration. Preprint, arXiv:2411.00053. — [arXiv:2411.00053](https://arxiv.org/abs/2411.00053) find
  20. Google. 2025. Medgemma hugging face. https://huggingface.co/collections/google/ medgemma-release-680aade845f90bec6a3f60c4. ↗
  21. Tanmay Gupta and Aniruddha Kembhavi. 2022. Visual programming: Compositional visual reasoning without training. Preprint, arXiv:2211.11559. find
  22. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2024. MetaGPT: Meta programming for a multi-agent collaborative framework. In The T…
  23. Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez. 2024. Debating with more persuasive llms leads to more truthful answers. Preprint, arXiv:2402.06782. find
  24. Rahima Khanam and Muhammad Hussain. 2024. Yolov11: An overview of the key architectural enhancements. Preprint, arXiv:2410.17725. — [arXiv:2410.17725](https://arxiv.org/abs/2410.17725) find
  25. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and 1 others. 2023. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499.
  26. Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-play compositional reasoning with large language models. In The 37th Conference on Neural Information Processing Systems (NeurIPS).
  27. Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. 2024. Ovis: Structural embedding alignment for multimodal large language model. arXiv:2405.20797. — [arXiv:2405.20797](https://arxiv.org/abs/2405.20797) find
  28. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. In… find
  29. Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. 2023. Augmented language models: a survey. Preprint, arXiv:2302.07842.
  30. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. Webgpt: Browser-assis…
  31. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. Gpt-4o system card.…
  32. OpenAI. 2025. GPT-5 system card. Technical report, OpenAI. OpenAI system card, August 13, 2025. find
  33. Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1). find
  34. Aaron Parisi, Yao Zhao, and Noah Fiedel. 2022. Talm: Tool augmented language models. Preprint, arXiv:2205.12255.
  35. Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. Chatdev: Communicative agents for software development. Preprint, arXiv:2307.07924.
  36. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM: Facilitating large language models to master 16000+ real…
  37. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. Preprint, arXiv:2302.04761.
  38. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VIII, pages 146…
  39. Ruoyue Shen, Nakamasa Inoue, and Koichi Shinoda. 2024. Pyramid coder: Hierarchical code generator for compositional visual question answering. Preprint, arXiv:2407.20563. — [arXiv:2407.20563](https://arxiv.org/abs/2407.20563) find
  40. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. In Advances in Neural Information Processing Systems. find
  41. Dídac Surís, Sachit Menon, and Carl Vondrick. 2023. Vipergpt: Visual inference via python execution for reasoning. Proceedings of IEEE International Conference on Computer Vision (ICCV).
  42. Xunzhu Tang, Kisub Kim, Yewei Song, Cedric Lothritz, Bei Li, Saad Ezzini, Haoye Tian, Jacques Klein, and Tegawende F. Bissyande. 2024. Codeagent: Autonomous communicative agents for code review. Preprint, arXiv:2402.02172. find
  43. Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song. 2024a. Rethinking the bounds of llm reasoning: Are multi-agent discussions the key? In Long Papers, Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 6106–6131. Association for Computational Li… find
  44. Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai. 2024b. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442.
  45. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
  46. Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. 2024c. Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. Preprint, arXiv:2307.05300.
  47. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
  48. Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. 2023. Visual chatgpt: Talking, drawing and editing with visual foundation models. Preprint, arXiv:2303.04671.
  49. Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. 2025. Expertprompting: Instructing large language models to be distinguished experts. Preprint, arXiv:2305.14688. find
  50. Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint. find
  51. Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, and 1 others. 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800.
  52. Xiangyu Yin, Chuqiao Shi, Yimo Han, and Yi Jiang. 2024. Pear: A robust and flexible automation framework for ptychography enabled by multiple large language model agents. Preprint, arXiv:2410.09034. — [arXiv:2410.09034](https://arxiv.org/abs/2410.09034) find
  53. Shoubin Yu, Jaehong Yoon, and Mohit Bansal. 2025a. CREMA: Generalizable and efficient video-language reasoning via multimodal modular fusion. In The Thirteenth International Conference on Learning Representations. find
  54. Shoubin Yu, Yue Zhang, Ziyang Wang, Jaehong Yoon, and Mohit Bansal. 2025b. Mexa: Towards general multimodal reasoning with dynamic multi-expert aggregation. arXiv preprint arXiv:2506.17113. find
  55. Shoubin Yu, Yue Zhang, Ziyang Wang, Jaehong Yoon, and Mohit Bansal. 2025c. Mexa: Towards general multimodal reasoning with dynamic multi-expert aggregation. Preprint, arXiv:2506.17113. find
  56. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others. 2024. Mmmu: A massive multi-discipline multimodal u…
  57. Ceyao Zhang, Kaijie Yang, Siyi Hu, Zihao Wang, Guanghe Li, Yihang Sun, Cheng Zhang, Zhaowei Zhang, Anji Liu, Song-Chun Zhu, Xiaojun Chang, Junge Zhang, Feng Yin, Yitao Liang, and Yaodong Yang. 2024. Proagent: Building proactive cooperative agents with large language models. Preprint, arXiv:2308.1133… — [arXiv:2308.11339](https://arxiv.org/abs/2308.11339) find
  58. Xiutian Zhao, Ke Wang, and Wei Peng. 2024. An electoral approach to diversify llm-based multi-agent collective decision-making. Preprint, arXiv:2410.15168. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sivakumaran-2026-dart-leveraging-multi-agent-disagreement/