Elamur Research⌕
☰
home / papers / 2026 / Agentic Oversight via Dialectic Reasoning

Agentic Oversight via Dialectic Reasoning

Paper · 2026 · Leonardo Ranaldi and Federico Ranaldi

Summary

The paper asks how weaker models can reliably supervise stronger LLMs. Debate is a known answer, but it offers little verifiable justification for its verdicts and has rarely been tested beyond English text. The authors propose an Agentic Oversight framework built on Dialectic Argumentation. First, they collect an Argumentation Set: instances where two expert models give different answers. Each expert can see the context (an image, retrieved documents or extra knowledge) and defends its own answer over several rounds. The rounds cover premises, arguments, agreements, challenges and counter-observations. A third, weaker judge never sees the context. It decides the winner by weighing claims and objections for grounding, relevance and consistency, and it writes an auditable explanation. The protocol is compared with a single-expert Consultancy mode across six multilingual, multimodal and knowledge-intensive tasks. Debate judged dialectically consistently beats single-expert baselines. The judge's reasoning traces can also be used to fine-tune the experts without gold labels, yielding gains comparable to supervised tuning. This points to a label-free route to weak-to-strong oversight.

Key points

  • Setup: two sighted experts argue only for belief-consistent answers on instances where they disagree. A context-blind judge decides from the arguments alone, separating perception from judgement.
  • Models: vision-language models evaluated include Llama-3.2-V, Qwen-2.5-V, DeepSeek-V and Molmo-D. The illustrative example comes from World-Cuisines, where the judge picks Sfogliatella over Choux pastry.
  • Tuning on Debate traces: Llama-3.2-V reaches 72.6 on BorderLines and 58.4 on Real-MM-RAG. Molmo-D reaches 33.4 on Real-MM-RAG. Consultancy traces generally yield lower scores.
  • Other comparisons: the authors report advantages over self-correction and CoT heuristics in both accuracy and generation length.
  • Limitations: the approach requires models that follow complex instructions and describe images faithfully. Fluent but deceptive or hallucinated arguments remain a risk, so stronger grounding checks against visual evidence are proposed as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations🔬Idiap Research InstituteMartigny-Combe · Switzerland32 works · 1,954 citations🎓University of Rome Tor VergataRome · Italy15 works · 1,321 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Leonardo Ranaldi and Federico Ranaldi. 2026. Agentic Oversight via Dialectic Reasoning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1143

@inproceedings{ranaldi2026agentic,
  title = {{Agentic Oversight via Dialectic Reasoning}},
  author = {Leonardo Ranaldi and Federico Ranaldi},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1143},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1143}
}

References

27 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ashutosh Adhikari and Mirella Lapata. 2025. Debating for better reasoning: An unsupervised multimodal approach. Preprint, arXiv:2505.14627. — [arXiv:2505.14627](https://arxiv.org/abs/2505.14627) find
  2. Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilė Lukošiūtė, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, and 27 others. 20… — Measuring Progress on Scalable Oversight for Large Language Models
  3. Hyeong Kyu Choi, Xiaojin Zhu, and Sharon Li. 2025. Debate or vote: Which yields better decisions in Preprint, multi-agent large language models? arXiv:2508.17536. — [arXiv:2508.17536](https://arxiv.org/abs/2508.17536) find
  4. Mark Yatskar, Chris Callison-Burch, and 32 others. 2024. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146.
  5. Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate. Preprint, arXiv:2305.14325.
  6. Andrew Estornell and Yang Liu. 2024. Multi-llm debate: Framework, principals, and interventions. In Advances in Neural Information Processing Systems, volume 37, pages 28938–28964. Curran Associates, Inc. find
  7. Andrew Estornell, Jean-Francois Ton, Yuanshun Yao, and Yang Liu. 2025. ACC-collab: An actor-critic approach to multi-agent LLM collaboration. In The Thirteenth International Conference on Learning Representations. find
  8. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Onur Çelebi, Licheng Yu, Liron Moshkovich, and Zhiyu Ma. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  9. Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate. Preprint, 2018. arXiv:1805.00899. find
  10. Zachary Kenton, Noah Y Siegel, János Kramár, Jonah Brown-Cohen, Samuel Albanie, Jannis Bulian, Rishabh Agarwal, David Lindner, Yunhao Tang, Noah D Goodman, and Rohin Shah. 2024. On scalable oversight with weak llms judging strong llms. In Advances in Neural Information Processing Systems (NeurIPS). find
  11. Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez. 2024. Debating with more persuasive llms leads to more truthful answers. Preprint, arXiv:2402.06782. find
  12. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. Deepseek-r1: Incentivizing reasoning capability in ll…
  13. Bryan Li, Samar Haider, and Chris Callison-Burch. 2024. This land is Your, My land: Evaluating geopolitical bias in language models through territorial disputes. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn… find
  14. Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Shayne Longpre, Yi Lu, and Joachim Daiber. 2021. MKQA: A l… find
  15. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. Preprint, arXiv:2310.02255.
  16. Karlinsky. 2025. REAL-MM-RAG: A real-world multi-modal retrieval benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 31660–31683, Vienna, Austria. Association for Computational Linguistics.
  17. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others.… — GPT-4 Technical Report
  18. Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, David Anugraha, Rifki Afina Putri, Wang Yutong, Adam Nohejl, Ubaidillah Ariq Prathama, Nedjma Ousidhoum, Afifa Amriani, Anar Rzayev, Anirban Das, Ashmari Pramodya, Aulia Adila, Bryan Wilie, Candy Olivia Mawalim, Cheng Ching Lam, Daud Abola… — WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines
  19. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train…
  20. Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman. 2022. QuALITY: Question answering with long input texts, yes! In Proceedings of the 2022 Conference of the North American Chapter of…
  21. Leonardo Ranaldi, Federico Ranaldi, Fabio Massimo Zanzotto, Barry Haddow, and Alexandra Birch. 2025. Improving multilingual retrieval-augmented language models through dialectic reasoning argumentations. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages…
  22. Andries Smit, Paul Duckworth, Nathan Grinsztajn, Thomas D. Barrett, and Arnu Pretorius. 2024. Should we be going mad? a look at multi-agent debate strategies for llms. Preprint, arXiv:2311.17371. find
  23. Vighnesh Subramaniam, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch. 2025. Multiagent finetuning: Self improvement with diverse reasoning chains. In The Thirteenth International Conference on Learning Representations. find
  24. Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. 2023. Principle-driven self-alignment of language models from scratch Preprint, with minimal human supervision. arXiv:2305.03047.
  25. Navve Wasserman, Roi Pony, Oshri Naparstek, Adi Raz Goldfarb, Eli Schwartz, Udi Barzelay, and Leonid Andrea Wynn, Harsh Satija, and Gillian Hadfield. 2025. Talk isn’t always cheap: Understanding failure modes in multi-agent debate. Preprint, arXiv:2509.05396. — [arXiv:2509.05396](https://arxiv.org/abs/2509.05396) find
  26. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  27. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others. 2024. Mmmu: A massive multi-discipline multimodal u…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ranaldi-2026-agentic-oversight-dialectic-reasoning/