Elamur Research
☰
home / papers / 2022 / Self-critiquing models for assisting human evaluators

Self-critiquing models for assisting human evaluators

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2206.05802

Summary

Training models from human feedback works only when people can reliably judge model outputs, and that becomes harder as tasks grow more complex. This is the scalable oversight problem. Saunders et al. (OpenAI) test whether AI-written critiques can help human evaluators. Using supervised fine-tuning (behavioral cloning), they train large language models on several tasks at once: topic-based summarization, writing critiques of summaries, judging whether an answer has flaws (critiqueability), judging whether a critique is valid (helpfulness), and refining answers. Data were gathered over several rounds, with labelers responding to outputs from progressively improved models. When labelers had model critiques, they found substantially more flaws in both model-written and human-written summaries, including summaries deliberately written to mislead. Larger models produced more useful critiques, could critique their own harder-to-fault outputs, and could improve their summaries using self-critiques. However, the gap between the models' ability to discriminate good from bad answers and their ability to critique them did not clearly shrink with scale. This suggests models know more than they put into words. The work is an early proof of concept for AI-assisted supervision.

Key points

  • With critique assistance, labelers identified roughly 50% more flaws. On intentionally misleading summaries, they caught the planted flaws about 50% more often.
  • The topic-based summarization dataset has over 6,000 queries on over 2,000 passages drawn from short stories, Wikipedia, and web/news articles. Most passages are under 2,048 GPT-2 tokens. The released data include tens of thousands of critiques, refinements, and helpfulness labels.
  • Refinement conditioned on critiques, especially critiques chosen by rejection sampling, outperformed direct refinement. Small models showed no gain from either approach.
  • The authors propose generator–discriminator–critique gap measurements. They found no clear evidence that critique ability converges with discrimination ability as models scale.
  • Limitations include roughly 75% labeler agreement on comparisons, a base task that is relatively easy for humans, and training with SFT only, without RL.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

174 papers of our library cite it (of the 140673 whose bibliographies we read); 24 of them have a page here, 150 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2206.05802 · arXiv 2206.05802

@misc{saunders2022self,
  title = {{Self-critiquing models for assisting human evaluators}},
  author = {Saunders, William and Yeh, Catherine and Wu, Jeff and Bills, Steven and Ouyang, Long and Ward, Jonathan and Leike, Jan},
  year = {2022},
  doi = {10.48550/arxiv.2206.05802},
  eprint = {2206.05802},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2206.05802}
}

References

30 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. find our card
  2. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016. find our card
  3. Cem Anil, Guodong Zhang, Yuhuai Wu, and Roger Grosse. Learning to give checkable answers with prover-verifier games. arXiv preprint arXiv:2108.12099, 2021. — [arXiv:2108.12099](https://arxiv.org/abs/2108.12099) find
  4. Elizabeth Barnes, Paul Christiano, Long Ouyang, and Geoffrey Irving. Progress on AI safety via debate. URL https://www.alignmentforum.org/posts/Br4xDbYu4Frwrb64a/writeup-progress-on-ai-safety-via-debate-1, 2020. ↗
  5. Elizabeth Barnes, Paul Christiano, William Saunders, Joe Collman, Mark Xu, Chris Painter, Mihnea Maftei, and Ronny Fernandez. Debate update: Obfuscated arguments problem. URL https://www.alignmentforum.org/posts/PJLABqQ962hZEqhdB/debate-update-obfuscated-arguments-problem, 2020. ↗
  6. Tom Bosc, Elena Cabrio, and Serena Villata. DART: A dataset of arguments and their relations on Twitter. In Proceedings of the 10th edition of the Language Resources and Evaluation Conference, pages 1258–1263, 2016. find
  7. László Babai, Lance Fortnow, and Carsten Lund. Non-deterministic exponential time has two-prover interactive protocols. Computational complexity, 1(1):3–40, 1991. find
  8. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. find our card
  9. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova Das-Sarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. find
  10. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  11. Aidan Clark, Diego de las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. arXiv preprint arXiv:2202.01169, 2022. — [arXiv:2202.01169](https://arxiv.org/abs/2202.01169) find
  12. Paul Christiano, Ajeya Cotra, and Mark Xu. Eliciting latent knowledge: How to tell if your eyes deceive you. https://www.alignmentforum.org/posts/qHCDysDnvhteW7kRd/arc-s-first-technical-report-eliciting-latent-knowledge, 2021. ↗
  13. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pages 4299–4307, 2017. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam… find
  14. Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation. arXiv preprint arXiv:2201.05955, 2022. find our card
  15. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683, 2017. find our card
  16. David Manheim and Scott Garrabrant. Categorizing variants of Goodhart’s Law. arXiv preprint arXiv:1803.04585, 2018. — [arXiv:1803.04585](https://arxiv.org/abs/1803.04585) find
  17. Shahbuland Matiana, JR Smith, Ryan Teehan, Louis Castricato, Stella Biderman, Leo Gao, and Spencer Frazier. Cut the carp: Fishing for zero-shot story evaluation. arXiv preprint arXiv:2110.03111, 2021. — [arXiv:2110.03111](https://arxiv.org/abs/2110.03111) find
  18. Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147, 2022. find our card
  19. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. WebGPT: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021. find our card
  20. Khanh X Nguyen, Dipendra Misra, Robert Schapire, Miroslav Dudík, and Patrick Shafto. Interactive learning from activity description. In International Conference on Machine Learning, pages 8096–8108. PMLR, 2021. find
  21. Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, page 2, 2000. find
  22. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. find
  23. John Pougué-Biyong, Valentina Semenova, Alexandre Matton, Rachel Han, Aerin Kim, Renaud Lambiotte, and Doyne Farmer. DEBAGREEMENT: A comment-reply dataset for (dis)agreement detection in online debates. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track… find
  24. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286, 2022. find our card
  25. Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela, and Kyunghyun Cho. Finding generalizable evidence by learning to convince q&a models. arXiv preprint arXiv:1909.05863, 2019. find
  26. Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt. Can OpenAI Codex and Other Large Language Models Help Us Fix Security Bugs? arXiv preprint arXiv:2112.02125, 2021. — [arXiv:2112.02125](https://arxiv.org/abs/2112.02125) find
  27. Alicia Parrish, Harsh Trivedi, Ethan Perez, Angelica Chen, Nikita Nangia, Jason Phang, and Samuel R Bowman. Single-turn debate does not help humans answer hard reading-comprehension questions. arXiv preprint arXiv:2204.05212, 2022. find
  28. Christian Rupprecht, Iro Laina, Nassir Navab, Gregory D Hager, and Federico Tombari. Guide me: Interacting with deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8551–8561, 2018. find
  29. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language\_understanding\_paper.pdf, 2018. ↗
  30. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9, 2019. [SB22] Andreas Stuhlmüller and Jungwon Byun. Supervise Process, not Outcomes. https://ought.org/updates/2022-04-06-process, 2022. + [SB… ↗