Elamur Research⌕
☰
home / papers / 2024 / Debating with More Persuasive LLMs Leads to More Truthful Answers

Debating with More Persuasive LLMs Leads to More Truthful Answers

Paper · 2024 · Akbir Khan, John Hughes, Dan Valentine et al.

Summary

The paper asks whether weaker overseers can reliably check stronger AI systems, a problem that grows as models outpace human expertise and ground-truth labels become unavailable. The authors simulate this gap with information asymmetry on the QuALITY reading-comprehension benchmark. "Expert" LLMs can read a long science-fiction story, while "non-expert" judges, either LLMs or humans, see only the question and two candidate answers. Three protocols are compared. In debate, two experts defend opposite answers. Interactive debate is the same, but the judge may also pose questions. In consultancy, a single expert advocates one answer. Experts can cite passages, and an automatic tool marks each citation as verified or not. The authors also make debaters more persuasive without labels by using inference-time selection guided by judge approval. Debate produced clearly more accurate judgements than both consultancy and no help. Stronger debaters further increased judge accuracy, while stronger consultants reduced it. This is early empirical evidence that adversarial debate could support scalable oversight.

Key points

  • Data: HARD questions from the Project Gutenberg sci-fi portion of QuALITY, with stories around 7,000 tokens at the median. Each question is reduced to the correct answer plus the strongest distractor.
  • Accuracy with the most persuasive debaters:
    • Human judges: 88% with debate, 78% with consultancy, 60% naive. Annotators who read the text score 89%.
    • LLM judges: 76% with debate, 54% with consultancy, 48% naive.
  • Models and setup:
    • Debaters and consultants: GPT-4-Turbo, GPT-3.5-Turbo, Claude 2.1, Claude 1.3 and Mixtral 8x7B.
    • Persuasiveness is boosted with best-of-N (up to best-of-16) and critique-and-refinement.
    • Each run has three rounds and a 900-word transcript cap.
  • Persuasiveness and truth: Optimising for judge approval helps debaters more when they argue the true answer than when they argue the false one. This benefit carries over to judges that were not used during optimisation.
  • Human study and code: The human study used over 1,000 annotator-hours, and judges were better calibrated under debate than under consultancy. The code is available at github.com/ucl-dark/llm_debate (ICML 2024).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktaschel and Ethan Perez. 2024. Debating with More Persuasive LLMs Leads to More Truthful Answers. arXiv:2402.06782.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.06782 · arXiv 2402.06782

@misc{khan2024debating,
  title = {{Debating with More Persuasive LLMs Leads to More Truthful Answers}},
  author = {Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktaschel and Ethan Perez},
  year = {2024},
  doi = {10.48550/arxiv.2402.06782},
  eprint = {2402.06782},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.06782}
}

References

43 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016.
  2. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022.
  3. Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., and Weld, D. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–16, 2021. find
  4. Barnes, B. Debate update: Obfuscated arguments problem. AI Alignment Forum, 2020. find
  5. Bowman, S. R., Hyun, J., Perez, E., Chen, E., Pettit, C., Heiner, S., Lukosuite, K., Askell, A., Jones, A., Chen, A., et al. Measuring progress on scalable oversight for large language models. arXiv preprint arXiv:2211.03540, 2022.
  6. Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., Sutskever, I., and Wu, J. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023.
  7. Christiano, P., Shlegeris, B., and Amodei, D. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575, 2018. find
  8. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. find
  9. Chromik, M., Eiband, M., Buchner, F., Krüger, A., and Butz, A. I think i get your point, ai! the illusion of explanatory depth in explainable ai. In 26th International Conference on Intelligent User Interfaces, pp. 307–317, 2021. find
  10. Cotra, A. The case for aligning narrowly superhuman models. AI Alignment Forum, 2021. find
  11. Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mordatch, I. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023.
  12. Elo, A. The Rating of Chessplayers: Past and Present. Ishi Press International, 1978. ISBN 9780923891275. URL https://books.google.com/books?id=syjcPQAACAAJ. ↗
  13. Gemini Team, Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  14. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
  15. Goswami, M., Sanil, V., Choudhry, A., Srinivasan, A., Udompanyawit, C., and Dubrawski, A. Aqua: A benchmarking tool for label quality assessment. arXiv preprint arXiv:2306.09467, 2023. — [arXiv:2306.09467](https://arxiv.org/abs/2306.09467) find
  16. Greenblatt, R., Shlegeris, B., Sachan, K., and Roger, F. Ai control: Improving safety despite intentional subversion. arXiv preprint arXiv:2312.06942, 2023. — [arXiv:2312.06942](https://arxiv.org/abs/2312.06942) find
  17. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
  18. Herbrich, R., Minka, T., and Graepel, T. Trueskill™: a bayesian skill rating system. Advances in neural information processing systems, 19, 2006. find
  19. Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., Lanham, T., Ziegler, D. M., Maxwell, T., Cheng, N., et al. Sleeper agents: Training deceptive llms that persist through safety training. arXiv preprint arXiv:2401.05566, 2024.
  20. Irving, G., Christiano, P., and Amodei, D. AI safety via debate. arXiv preprint arXiv:1805.00899, 2018.
  21. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. find
  22. Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214– 3252, 2022.
  23. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023. find
  24. Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., et al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147, 2022.
  25. Michael, J., Mahdi, S., Rein, D., Petty, J., Dirani, J., Padmakumar, V., and Bowman, S. R. Debate helps supervise unreliable experts. arXiv preprint arXiv:2311.08702, 2023. find
  26. Nikola. Jailbreaking gpt-4’s code interpreter. LessWrong, July 2023. URL https://www.lesswrong.com/posts/KSroBnxCHodGmPPJ8/ jailbreaking-gpt-4-s-code-interpreter. ↗
  27. Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. Show your work: Scratchpads for intermediate computation with language models. In Deep Learning for Code Workshop, 2022.
  28. OpenAI. GPT-4 technical report. arXiv:2303.08774, 2023. arXiv preprint Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Proce…
  29. Pang, R. Y., Parrish, A., Joshi, N., Nangia, N., Phang, J., Chen, A., Padmakumar, V., Ma, J., Thompson, J., He, H., et al. QuALITY: Question answering with long input texts, yes! In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hum…
  30. Parrish, A., Trivedi, H., Nangia, N., Phang, J., Padmakumar, V., Saimbhi, A. S., and Bowman, S. R. Two-turn debate does not help humans answer hard reading comprehension questions. In NeurIPS ML Safety Workshop, 2022a. find
  31. Parrish, A., Trivedi, H., Perez, E., Chen, A., Nangia, N., Phang, J., and Bowman, S. Single-turn debate does not help humans answer hard reading-comprehension questions. In Proceedings of the First Workshop on Learning with Natural Language Supervision, pp. 17–28, 2022b. find
  32. Perez, E., Karamcheti, S., Fergus, R., Weston, J., Kiela, D., and Cho, K. Finding generalizable evidence by learning to convince Q&A models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proces…
  33. Perez, E., Kiela, D., and Cho, K. True few-shot learning with language models. Advances in neural information processing systems, 34:11054–11070, 2021.
  34. Radhakrishnan, A. Anthropic fall 2023 debate progress update. find
  35. AI Alignment Forum, 2023. find
  36. Radhakrishnan, A., Nguyen, K., Chen, A., Chen, C., Denison, C., Hernandez, D., Durmus, E., Hubinger, E., Kernion, J., Lukošiūtė, K., et al. Question decomposition improves the faithfulness of model-generated reasoning. arXiv preprint arXiv:2307.11768, 2023. find
  37. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023.
  38. Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., and Leike, J. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802, 2022.
  39. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33: 3008–3021, 2020. find
  40. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824–24837, 2022.
  41. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023. find
  42. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging LLM-as-a-judge with MT-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023.
  43. Debaters: Consultancy Judge Accuracy (%) Aggregate Win Rate (%) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/khan-2024-debating-more-persuasive-llms-leads/