Elamur Research⌕
☰
home / papers / 2026 / DialDefer: A Framework for Detecting and Mitigating LLM Dialogic Deference

DialDefer: A Framework for Detecting and Mitigating LLM Dialogic Deference

Paper · 2026 · Parisa Rabbani, Priyam Sahoo, Ruben Mathew et al.

Summary

LLMs increasingly act as outside arbiters of conversations, but their verdicts can depend on presentation rather than substance. The authors introduce DialDefer, a framework that pairs each claim in two forms: as a bare statement to verify (C1) and as a speaker's utterance to evaluate (C2). The content stays the same and only the framing changes. They also propose the Dialogic Deference Score (DDS), which measures whether a model leans toward agreeing with speakers (positive, deference) or rejecting them (negative, skepticism). Plain accuracy can hide this shift because gains and losses offset each other. The test set covers a Unified Benchmark of nine Q/A datasets plus r/AIO, a set of real Reddit interpersonal disputes, for 3,244 items in total. Five models were evaluated. Framing alone moves verdicts substantially while accuracy barely changes, and the effect is stronger on real conversations. Mitigation through prompting or fine-tuning helps only partly and sometimes overshoots into skepticism. This matters for anyone using LLMs as judges, moderators or dispute arbiters.

Key points

  • Models tested: GPT-4o, GPT-4o-mini, GPT-5-mini, Gemma-3-12B and Qwen-2.5-7B.
  • Core result: mean |DDS| is 15.9 pp (p < .0001), while accuracy moves by less than 2 pp. DDS ranges from −53 to +87 pp depending on model and domain.
  • Open-weight models defer most: Qwen-2.5-7B scores +33.8 and Gemma-3-12B +29.5, versus −1.1 for GPT-4o.
  • Real conversations amplify the effect: on r/AIO (N=280), deference grows 2–5×. Gemma rises from +23 to +87 pp.
  • What drives it: labeling a speaker as human rather than an LLM causes a 17.7 pp swing, while demographic cues matter little.
  • Why judgments flip: analysis of 2,410 flips found two main failure types. Internal Incoherence accounts for 29.7%, and Social Framing, where the model validates feelings over facts, accounts for 27%.
  • Mitigation:
    • SFT on Qwen cut DDS by 20.2 pp and raised accuracy by 18.9 pp.
    • SFT and DPO increased DDS for Gemma.
    • Neither fine-tuning method transferred to r/AIO.
    • The "Dehumanizing" prompt lowered r/AIO DDS by 8.9–26.1 pp across the three models tested.
  • Availability: code and data are released at github.com/LadyPary/DialDefer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Parisa Rabbani, Priyam Sahoo, Ruben Mathew, Aishee Mondal, Harshita Ketharaman, Nimet Beyza Bozdag and Dilek Hakkani-Tür. 2026. DialDefer: A Framework for Detecting and Mitigating LLM Dialogic Deference. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2067

@inproceedings{rabbani2026dialdefer,
  title = {{DialDefer: A Framework for Detecting and Mitigating LLM Dialogic Deference}},
  author = {Parisa Rabbani, Priyam Sahoo, Ruben Mathew, Aishee Mondal, Harshita Ketharaman, Nimet Beyza Bozdag and Dilek Hakkani-Tür},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2067},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2067}
}

References

47 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Myra Cheng, Sunny Yu, Cinoo Lee, Pranav Khadpe, Lujain Ibrahim, and Dan Jurafsky. 2025a. Elephant: Measuring and understanding social sycophancy in llms. Preprint, arXiv:2505.13995.
  2. Myra Cheng, Sunny Yu, Cinoo Lee, Pranav Khadpe, Lujain Ibrahim, and Dan Jurafsky. 2025b. Social sycophancy: A broader understanding of LLM sycophancy. Preprint, arXiv:2505.13995v1.
  3. Robert B. Cialdini. 2001. Influence: Science and Practice, 4th edition. Allyn & Bacon. find
  4. Yann Dubois, Percy Liang, and Tatsunori B. Hashimoto. 2024. Length-controlled AlpacaEval: A simple way to debias automatic evaluators. In Proceedings of the Conference on Language Modeling. find
  5. Editorial. 2025. Can AI chatbots validate delusional thinking? BMJ. find
  6. Aaron Fanous, Jacob Goldberg, Ank A. Agarwal, Joanna Lin, Anson Zhou, Sonnet Xu, Vasiliki Bikia, Roxana Daneshjou, and Sanmi Koyejo. 2025. SycEval: Evaluating LLM sycophancy. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 893–900. find
  7. Gemma Team. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786. find
  8. Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2023. ROSCOE: A suite of metrics for scoring step-by-step reasoning. In Proceedings of the International Conference on Learning Representations.
  9. Jiseung Hong, Grace Byun, Seungone Kim, Kai Shu, and Jinho D. Choi. 2025. Measuring sycophancy of language models in multi-turn dialogues. In Findings of the Association for Computational Linguistics: EMNLP 2025. find
  10. Minbeom Kim, Hwanhee Lee, Joonsuk Park, Hwaran Lee, and Kyomin Jung. 2025. AdvisorQA: Towards helpful and harmless advice-seeking question answering with collective intelligence. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguis… find
  11. Mahi Kolla, Siddharth Salunkhe, Eshwar Chandrasekharan, and Koustuv Saha. 2024. LLM-mod: Can large language models assist content moderation? In Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1–8, Honolulu, HI, USA. Association for Computing Machinery. find
  12. Satyapriya Krishna, Andy Zou, Rahul Gupta, Eliot Krzysztof Jones, Nick Winter, Dan Hendrycks, J. Zico Kolter, Matt Fredrikson, and Spyros Matsoukas. 2025. D-rex: A Benchmark for detecting deceptive reasoning in large language models. Preprint, arXiv:2509.17938. — [arXiv:2509.17938](https://arxiv.org/abs/2509.17938) find
  13. Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. HaluEval: A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.
  14. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics…
  15. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Lin…
  16. Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2024. Calibrating LLM-based evaluator. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pag…
  17. Pedro Henrique Luz de Araujo, Paul Röttger, Dirk Hovy, and Benjamin Roth. 2025. Principled personas: Defining and measuring the intended effects of persona prompting on task performance. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 26845–26874, Suz… find
  18. Alexander Meinke, Bronson Schoen, Jérémy Scheurer, Mikita Balesni, Rusheb Shah, and Marius Hobbhahn. 2024. Frontier models are capable of in-context scheming. Preprint, arXiv:2412.04984. — [arXiv:2412.04984](https://arxiv.org/abs/2412.04984) find
  19. Stanley Milgram. 1963. Behavioral study of obedience. The Journal of Abnormal and Social Psychology, 67(4):371–378. find
  20. Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt LLM evaluation. Transactions of the Association for Computational Linguistics, 12:933–949. find
  21. Jamshid Mozafari, Abdelrahman Abdallah, Bhawna Piryani, and Adam Jatowt. 2025. Wrong answers can also be useful: PlausibleQA – a large-scale QA dataset with answer plausibility scores. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval,… find
  22. OpenAI. 2024. GPT-4o system card. arXiv:2410.21276.
  23. Preprint, OpenAI. 2025. OpenAI GPT-5 System Card. Preprint, arXiv:2601.03267.
  24. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Tra… — Training language models to follow instructions with human feedback
  25. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. LLM evaluators recognize and favor their own generations. In Advances in Neural Information Processing Systems, volume 37, pages 68772–68802, Vancouver, Canada. find
  26. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105, Dublin, Ireland. Asso…
  27. Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, and 44 others. 2023. Dis… — Discovering Language Model Behaviors with Model-Written Evaluations
  28. Adrian Preda. 2025. AI-induced psychosis: A new frontier in mental health. Psychiatric News. find
  29. Parisa Rabbani, Nimet Beyza Bozdag, and Dilek Hakkani-Tur. 2026. From fact to judgment: Investigating the impact of task framing on LLM conviction in dialogue systems. In Proceedings of the 16th International Workshop on Spoken Dialogue System Technology, pages 193–204, Trento, Italy. Association fo…
  30. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level Google-proof Q&A benchmark. In Proceedings of the First Conference on Language Modeling.
  31. Richard Ren, Arunim Agarwal, Mantas Mazeika, Cristina Menghini, Robert Vacareanu, Brad Kenstler, Mick Yang, Isabelle Barrass, Alice Gatti, Xuwang Yin, Eduardo Trevino, Matias Geralnik, Adam Khoja, Dean Lee, Summer Yue, and Dan Hendrycks. 2025. The MASK benchmark: Disentangling honesty from accuracy… — [arXiv:2503.03750](https://arxiv.org/abs/2503.03750) find
  32. Susanne Rust and Justin Chang. 2025. AIs gave scarily specific self-harm advice to users expressing suicidal intent, researchers find. Los Angeles Times. find
  33. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processi… find
  34. Jérémy Scheurer, Mikita Balesni, and Marius Hobbhahn. 2023. Large language models can strategically deceive their users when put under pressure. Preprint, arXiv:2311.07590. — [arXiv:2311.07590](https://arxiv.org/abs/2311.07590) find
  35. Annika M. Schoene and Cansu Canca. 2025. Jailbreaking LLMs in suicide and self-harm contexts. Preprint, arXiv:2507.02990. — [arXiv:2507.02990](https://arxiv.org/abs/2507.02990) find
  36. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations. find
  37. John R. Searle. 1969. Speech Acts: An Essay in the Philosophy of Language. Cambridge University Press. find
  38. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, Shauna M Kravec, Timothy Maxwell, Sam Mc-Candlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. 2024. Towards under…
  39. Richard Susskind. 2023. The implications of ChatGPT for legal services and society. Harvard Center on the Legal Profession. find
  40. Fiona Anting Tan, Gerard Christopher Yeo, Kokil Jaidka, Fanyou Wu, Weijie Xu, Vinija Jain, Aman Chadha, Yang Liu, and See-Kiong Ng. 2025. PHAn-ToM: Persona-based prompting has an effect on theory-of-mind reasoning in large language models. In Proceedings of the International AAAI Conference on Web a… find
  41. Qwen Team. 2024. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  42. Keyu Wang, Jin Li, Shu Yang, Zhuoran Zhang, and Di Wang. 2025. When truth is overridden: Uncovering the internal origins of sycophancy in large language models. Preprint, arXiv:2508.02087. — [arXiv:2508.02087](https://arxiv.org/abs/2508.02087) find
  43. Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V. Le. 2024. Simple synthetic data reduces sycophancy in large language models. Preprint, arXiv:2308.03958. find
  44. Ying Xiao, Jie Huang, Ruijuan He, Jing Xiao, Mohammad Reza Mousavi, Yepang Liu, Kezhi Li, Zhenpeng Chen, and Jie M. Zhang. 2025. AMQA: An adversarial dataset for benchmarking bias of LLMs in medicine and healthcare. Preprint, arXiv:2505.19562. — [arXiv:2505.19562](https://arxiv.org/abs/2505.19562) find
  45. Albert S. Yue, Lovish Madaan, Ted Moskovitz, DJ Strouse, and Aaditya K. Singh. 2024. HARP: A challenging human-annotated math reasoning benchmark. Preprint, arXiv:2412.08819. — [arXiv:2412.08819](https://arxiv.org/abs/2412.08819) find
  46. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Proceedings of the Advances in Neural Information Processing…
  47. Notation Note. In the appendix, we use Speaker A/B interchangeably with Speaker 1/2 from the main text, where Speaker B (= Speaker 2) denotes the original poster and Speaker A (= Speaker 1) denotes the other party. Similarly, B→A corresponds to 2→1. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rabbani-2026-dialdefer-framework-detecting-mitigating-llm/