Elamur Research
☰
home / papers / 2022 / Improving alignment of dialogue agents via targeted human judgements

Improving alignment of dialogue agents via targeted human judgements

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2209.14375

Summary

DeepMind presents Sparrow, a dialogue agent for information-seeking conversations. It is designed to be more helpful, correct and harmless than a prompted language model. The base model is Dialogue Prompted Chinchilla 70B (DPC), which the authors refine with reinforcement learning from human feedback (RLHF). They add two ideas that make human evaluation easier. First, the broad goals are broken into explicit natural-language rules, such as no threats, no financial advice and no claiming to be human. Raters judge each rule separately and also try to provoke violations on purpose. This produces targeted labels and a rule-conditional reward model. Second, the agent retrieves web snippets and shows them as evidence for factual claims, which extends GopherCite to multi-turn dialogue. The policy is trained with A2C against preference and rule reward models, and responses are also reranked at test time. Raters prefer it to prompted baselines, and it breaks rules much less often. The study suggests that splitting feedback into targeted judgements is a workable route to alignment, though rule-following can amplify distributional biases.

Key points

  • Pipeline: it starts from DPC (Chinchilla 70B) and trains reward models for preferences and for rules from human ratings. RL uses A2C to optimise both reward models together. Rounds of data collection and retraining repeat, and test-time reranking is added on top. The comparison baselines are DPC with prompting only, reranking, and supervised fine-tuning (SFT).
  • Evidence: on factual questions, the cited evidence supports a plausible answer 78% of the time. This is clearly better than the prompted baselines and close to GopherCite on single-turn QA.
  • Robustness: when humans deliberately try to provoke rule violations, Sparrow breaks the targeted rule in only 8% of conversations. In three-way comparisons against two prompted baselines (one that always searches and one that never does), raters still prefer it more often.
  • Comparison with related systems: unlike LaMDA, the Anthropic assistant, SeeKeR and BlenderBot 3, Sparrow combines per-rule labels, multi-objective RL and evidence that raters can see.
  • Limitation: the methods target only instance-level harms. The authors' analysis shows they can make distributional fairness problems worse. They also note the approach is dual-use, since the same machinery could enforce harmful rules.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

215 papers of our library cite it (of the 140673 whose bibliographies we read); 41 of them have a page here, 174 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2209.14375 · arXiv 2209.14375

@misc{glaese2022improving,
  title = {{Improving alignment of dialogue agents via targeted human judgements}},
  author = {Glaese, Amelia and McAleese, Nat and Trębacz, Maja and Aslanides, John and Firoiu, Vlad and Ewalds, Timo and Rauh, Maribeth and Weidinger, Laura and Chadwick, Martin and Thacker, Phoebe and Campbell-Gillingham, Lucy and Uesato, Jonathan and Huang, Po-Sen and Comanescu, Ramona and Yang, Fan and See, Abigail and Dathathri, Sumanth and Greig, Rory and Chen, Charlie and Fritz, Doug and Elias, Jaume Sanchez and Green, Richard and Mokrá, Soňa and Fernando, Nicholas and Wu, Boxi and Foley, Rachel and Young, Susannah and Gabriel, Iason and Isaac, William and Mellor, John and Hassabis, Demis and Kavukcuoglu, Koray and Hendricks, Lisa Anne and Irving, Geoffrey},
  year = {2022},
  doi = {10.48550/arxiv.2209.14375},
  eprint = {2209.14375},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2209.14375}
}

References

102 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318, 2016. find
  2. A. Abdolmaleki, S. Huang, L. Hasenclever, M. Neunert, F. Song, M. Zambelli, M. Martins, N. Heess, R. Hadsell, and M. Riedmiller. A distributional view on multi-objective policy optimization. In International Conference on Machine Learning, pages 11–22. PMLR, 2020. find
  3. L. Adolphs, K. Shuster, J. Urbanek, A. Szlam, and J. Weston. Reason first, then respond: Modular generation for knowledge-infused dialogue. arXiv preprint arXiv:2111.05204, 2021. URL https: //arxiv.org/abs/2111.05204 . find
  4. A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, J. Kernion, K. Ndousse, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, and J. Kaplan. A general language assistant as a laboratory… find our card
  5. Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J.… find
  6. M. Banko, B. MacKeen, and L. Ray. A unified taxonomy of harmful content. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 125–137, Online, Nov. 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.alw-1.16. URL https://aclanthology. org/2020.alw-1.16 . find
  7. E. M. Bender, T. Gebru, A. McMillan-Major, and M. Mitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, 2021. find
  8. A. Berditchevskaia, E. Malliaraki, and K. Peach. Participatory AI for humanitarian innovation, 2021. find
  9. A. Birhane, V. Prabhakaran, M. Diaz, I. Gabriel, M. C. Elish, S. Mohamed, and W. S. Isaac. Power to the people? opportunities and challenges for participatory ai. Proceedings of the ACM conference on Equity and Access in Algorithms, Mechanisms, and Optimization, 2022. find
  10. S. L. Blodgett, S. Barocas, H. Daumé III, and H. Wallach. Language (technology) is power: A critical survey of" bias" in nlp. arXiv preprint arXiv:2005.14050, 2020. URL https://arxiv.org/abs/ 2005.14050 . ↗ our card
  11. S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. van den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, D. de Las Casas, A. Guy, J. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. Simo… find
  12. J. Boyd-Graber, B. Satinoff, H. He, and H. Daumé III. Besting the Quiz Master: Crowdsourcing incremental classification games. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1290–1301, Jeju Island, Ko… find
  13. R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. find
  14. L. Breitfeller, E. Ahn, D. Jurgens, and Y. Tsvetkov. Finding microaggressions in the wild: A case for locating elusive phenomena in social media posts. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lang… find
  15. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner… find
  16. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf . ↗
  17. P. Christiano, B. Shlegeris, and D. Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575, 2018. URL https://arxiv.org/abs/1810.08575 . ↗
  18. P. Christiano, A. Cotra, and M. Xu. Eliciting latent knowledge: How to tell if your eyes deceive you, Dec 2021. URL https://docs.google.com/document/d/1WwsnJQstPq91\_Yh-Ch2XRL8H\_ EpsnjrC1dwZXR37PC8 . ↗
  19. A. Creswell, M. Shanahan, and I. Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712, 2022. URL https://arxiv. org/abs/2205.09712 . find
  20. B. Dang, M. J. Riedl, and M. Lease. But who protects the moderators? the case of crowdsourced image moderation. arXiv preprint arXiv:1804.10999, 2018. URL https://arxiv.org/abs/ 1804.10999 . — [arXiv:1804.10999](https://arxiv.org/abs/1804.10999) ↗
  21. E. Denton, A. Hanna, R. Amironesei, A. Smart, H. Nicole, and M. K. Scheuerman. Bringing the people back in: Contesting benchmark machine learning datasets. arXiv preprint arXiv:2007.07399, 2020. URL https://arxiv.org/abs/2007.07399 . — [arXiv:2007.07399](https://arxiv.org/abs/2007.07399) ↗
  22. E. Dinan, S. Humeau, B. Chintagunta, and J. Weston. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… find our card
  23. D. Dohan, W. Xu, A. Lewkowycz, J. Austin, D. Bieber, R. Gontijo Lopes, Y. Wu, H. Michalewski, R. A. Saurous, J. Sohl-dickstein, K. Murphy, and C. Sutton. Language model cascades. arXiv preprint arXiv:2207.10342, 2022. URL https://arxiv.org/abs/2207.10342 . ↗
  24. N. Dziri, S. Milton, M. Yu, O. Zaiane, and S. Reddy. On the origin of hallucinations in conversational models: Is it the datasets or the models? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 52… find
  25. R. El-Yaniv and Y. Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(5), 2010. find
  26. N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah. A mathematic… find
  27. A. E. Elo. The Rating of Chessplayers, Past and Present. Arco Pub., New York, 1978. ISBN 0668047216 9780668047210. URL http://www.amazon.com/Rating-Chess-Players-Past-Present/ dp/0668047216 . ↗
  28. O. Evans, O. Cotton-Barratt, L. Finnveden, A. Bales, A. Balwit, P. Wills, L. Righetti, and W. Saunders. Truthful ai: Developing and governing ai that does not lie. arXiv preprint arXiv:2110.06674, 2021. — [arXiv:2110.06674](https://arxiv.org/abs/2110.06674) find
  29. A. Fan, Y. Jernite, E. Perez, D. Grangier, J. Weston, and M. Auli. Eli5: Long form question answering. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567, 2019. find our card
  30. T. Gao, A. Fisch, and D. Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723, 2020. find our card
  31. Y. Geifman and R. El-Yaniv. Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017. find
  32. Y. Geifman and R. El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In International Conference on Machine Learning, pages 2151–2159. PMLR, 2019. find
  33. S. Gershman. How to never be wrong. Psychonomic Bulletin & Review, 26:13–28, 2019. find
  34. M. Gor, K. Webster, and J. Boyd-Graber. Toward deconfounding the effect of entity demographics for question answering accuracy. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5457–5473, Online and Punta Cana, Dominican Republic, Nov. 2021. Associatio… find
  35. M. Goulden, M. A. Mason, and K. Frasch. Keeping women in the science pipeline. The Annals of the American Academy of Political and Social Science, 638:141–162, 2011. ISSN 00027162, 15523349. URL http://www.jstor.org/stable/41328583 . ↗
  36. H. P. Grice. Logic and conversation. In Speech acts, pages 41–58. Brill, 1975. find
  37. S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan. Deep learning with limited numerical precision. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, page 1737–1746. JMLR.org, 2015. find
  38. D. Hadfield-Menell, S. J. Russell, P. Abbeel, and A. Dragan. Cooperative inverse reinforcement learning. Advances in neural information processing systems, 29:3909–3917, 2016. find
  39. A. Halfaker and R. S. Geiger. Ores: Lowering barriers with participatory machine learning in wikipedia. Proceedings of the ACM Conference on Human-Computer Interaction, 4(CSCW2):1–37, 2020. find
  40. D. Hendrycks, C. Burns, S. Basart, A. Critch, J. Li, D. Song, and J. Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021a. find our card
  41. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021b. find our card
  42. G. E. Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771–1800, 2002. find
  43. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. v. d. Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre. Training compute-optima… find
  44. A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019. URL https://arxiv.org/abs/1904.09751 . ↗
  45. G. Irving and A. Askell. AI safety needs social scientists. Distill, 2019. doi: 10.23915/distill.00014. URL https://doi.org/10.23915/distill.00014 . https://distill.pub/2019/safety-needs-social-scientists. — [doi:10.23915/distill.00014](https://doi.org/10.23915/distill.00014) ↗
  46. G. Irving, P. Christiano, and D. Amodei. AI safety via debate. arXiv preprint arXiv:1805.00899, 2018. URL https://arxiv.org/abs/1805.00899 . ↗
  47. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada, July 2017.… find our card
  48. A. Kamath, R. Jia, and P. Liang. Selective question answering under domain shift. arXiv preprint arXiv:2006.09462, 2020. URL https://arxiv.org/abs/2006.09462 . ↗
  49. A. Kasirzadeh and I. Gabriel. In conversation with Artificial Intelligence: towards a theory of ideal speech for humans and language technologies. Forthcoming, 2022. find
  50. D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. URL https://arxiv.org/abs/1412.6980 . ↗
  51. N. Kotonya, A. Vlachos, M. Yazdani, L. Mathias, and M. Saeidi. Policy compliance detection via expression tree inference. arXiv preprint arXiv:2205.12259, 2022. URL https://arxiv.org/ abs/2205.12259 . — [arXiv:2205.12259](https://arxiv.org/abs/2205.12259) ↗
  52. K. Krippendorff. Computing Krippendorff ’s alpha-reliability, 2011. URL https://repository. upenn.edu/asc\_papers/43 . find
  53. Z. Kunda. The case for motivated reasoning. psychological bulletin, 108.3:480, 1990. find
  54. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research. Transactions of th… find our card
  55. A. Lazaridou, E. Gribovskaya, W. Stokowiec, and N. Grigorev. Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115, 2022. URL https://arxiv.org/abs/2203.05115 . ↗ our card
  56. M. K. Lee, D. Kusbit, A. Kahng, J. T. Kim, X. Yuan, A. Chan, D. See, R. Noothigattu, S. Lee, A. Psomas, and A. D. Procaccia. WeBuildAI: Participatory framework for algorithmic governance. Proceedings of the ACM Conference on Human-Computer Interaction, 3(CSCW):1–35, 2019. URL https://doi. org/10.114… — [doi:10.1145/3359283](https://doi.org/10.1145/3359283) find
  57. J. Leike, D. Krueger, T. Everitt, M. Martic, V. Maini, and S. Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871, 2018. URL https: //arxiv.org/abs/1811.07871 . find
  58. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020. find
  59. A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858, 2022. URL https: //arxiv.org/abs/22… find our card
  60. S. Lin, J. Hilton, and O. Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10… find our card
  61. A. Liška, T. Kočiský, E. Gribovskaya, T. Terzi, E. Sezener, D. Agrawal, C. de Masson d’Autume, T. Scholtes, M. Zaheer, S. Young, E. Gilsenan-McMahon, S. Austin, P. Blunsom, and A. Lazaridou. StreamingQA: A benchmark for adaptation to new knowledge over time in question answering models. In Internati… find
  62. K. McGuffie and A. Newhouse. The radicalization risks of GPT-3 and advanced neural language models. arXiv preprint arXiv:2009.06807, 2020. URL https://arxiv.org/abs/2009.06807 . — [arXiv:2009.06807](https://arxiv.org/abs/2009.06807) ↗
  63. J. Menick, M. Trebacz, V. Mikulik, J. Aslanides, F. Song, M. Chadwick, M. Glaese, S. Young, L. Campbell-Gillingham, G. Irving, and N. McAleese. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147, 2022. URL https://arxiv.org/abs/2203.11147 . ↗ our card
  64. V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Harley, T. P. Lillicrap, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, page 1928–1937. find
  65. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021… find our card
  66. Open Ended Learning Team, A. Stooke, A. Mahajan, C. Barros, C. Deck, J. Bauer, J. Sygnowski, M. Trebacz, M. Jaderberg, M. Mathieu, N. McAleese, N. Bradley-Schmieg, N. Wong, N. Porcel, R. Raileanu, S. Hughes-Fitt, V. Dalibard, and W. M. Czarnecki. Open-ended learning leads to generally capable agents… — [arXiv:2107.12808](https://arxiv.org/abs/2107.12808) find
  67. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. ar… find
  68. A. Parrish, A. Chen, N. Nangia, V. Padmakumar, J. Phang, J. Thompson, P. M. Htut, and S. R. Bowman. Bbq: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193, 2021. URL https://arxiv.org/abs/2110.08193 . ↗ our card
  69. A. Parrish, H. Trivedi, E. Perez, A. Chen, N. Nangia, J. Phang, and S. R. Bowman. Single-turn debate does not help humans answer hard reading-comprehension questions. arXiv preprint arXiv:2204.05212, 2022. URL https://arxiv.org/abs/2204.05212 . ↗
  70. E. Perez, S. Karamcheti, R. Fergus, J. Weston, D. Kiela, and K. Cho. Finding generalizable evidence by learning to convince Q&A models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing… find
  71. E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286, 2022. find our card
  72. J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P.-S. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mel… find
  73. G. Ras, N. Xie, M. van Gerven, and D. Doran. Explainable deep learning: A field guide for the uninitiated. Journal of Artificial Intelligence Research, 73:329–397, 2022. find
  74. M. Rauh, J. Mellor, J. Uesato, P.-S. Huang, J. Welbl, L. Weidinger, S. Dathathri, A. Glaese, G. Irving, I. Gabriel, W. Isaac, and L. A. Hendricks. Characteristics of harmful text: Towards rigorous benchmarking of language models. arXiv preprint arXiv:2206.08325, 2022. URL https://arxiv. org/abs/2206… — [arXiv:2206.08325](https://arxiv.org/abs/2206.08325) find
  75. R. Rudinger, J. Naradowsky, B. Leonard, and B. Van Durme. Gender bias in coreference resolution. arXiv preprint arXiv:1804.09301, 2018. URL https://arxiv.org/abs/1804.09301 . ↗ our card
  76. S. Russell. Human Compatible. Penguin, 2020. find
  77. M. Saeidi, M. Yazdani, and A. Vlachos. Cross-policy compliance detection via question answering. arXiv preprint arXiv:2109.03731, 2021. URL https://arxiv.org/abs/2109.03731 . ↗
  78. W. Saunders, C. Yeh, J. Wu, S. Bills, L. Ouyang, J. Ward, and J. Leike. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802, 2022. URL https://arxiv.org/ abs/2206.05802 . ↗ our card
  79. A. Schopenhauer. The art of being right, 1831. URL https://en.wikisource.org/wiki/The\_ Art\_of\_Being\_Right . ↗
  80. N. Shazeer and M. Stern. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. In Proceedings of the 35th International Conference on Machine Learning, pages 4603–4611. PMLR, 2018. find
  81. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2019. URL http://arxiv. org/abs/1909.08053 . cite arxiv:1909.08053. find
  82. K. Shuster, M. Komeili, L. Adolphs, S. Roller, A. Szlam, and J. Weston. Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion. arXiv preprint arXiv:2203.13224, 2022a. URL https://arxiv.org/abs/2203.13224 . ↗
  83. K. Shuster, J. Xu, M. Komeili, D. Ju, E. M. Smith, S. Roller, M. Ung, M. Chen, K. Arora, J. Lane, M. Behrooz, W. Ngan, S. Poff, N. Goyal, A. Szlam, Y.-L. Boureau, M. Kambadur, and J. Weston. BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage. arXiv preprint a… find our card
  84. M. Sloane, E. Moss, O. Awomolo, and L. Forlano. Participation is not a design fix for machine learning. arXiv preprint arXiv:2007.02423, 2020. URL https://arxiv.org/abs/2007.02423 . — [arXiv:2007.02423](https://arxiv.org/abs/2007.02423) ↗
  85. H. F. Song, A. Abdolmaleki, J. T. Springenberg, A. Clark, H. Soyer, J. W. Rae, S. Noury, A. Ahuja, S. Liu, D. Tirumala, N. Heess, D. Belov, M. Riedmiller, and M. M. Botvinick. V-mpo: On-policy maximum a posteriori policy optimization for discrete and continuous control. arXiv preprint arXiv:1909.122… — [arXiv:1909.12238](https://arxiv.org/abs/1909.12238) find
  86. M. Steiger, T. J. Bharucha, S. Venkatagiri, M. J. Riedl, and M. Lease. The psychological well-being of content moderators: the emotional labor of commercial moderation and avenues for improving support. In Proceedings of the 2021 CHI conference on human factors in computing systems, pages 1–14, 2021… find
  87. N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. find
  88. R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL http://incompleteideas.net/book/the-book-2nd.html . ↗
  89. R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, Y. Li, H. Lee, H. S. Zheng, A. Ghafouri, M. Menegali, Y. Huang, M. Krikun, D. Lepikhin, J. Qin, D. Chen, Y. Xu, Z. Chen, A. Roberts, M. Bosma, V. Zhao, Y. Zhou, C.-C. Chang, I. Krivokon,… find our card
  90. N. Tomasev, K. R. McKee, J. Kay, and S. Mohamed. Fairness for unobserved characteristics: Insights from technological impacts on queer communities. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 254–265, 2021. find
  91. B. Vidgen, A. Harris, D. Nguyen, R. Tromble, S. Hale, and H. Margetts. Challenges and frontiers in abusive content detection. In Proceedings of the Third Workshop on Abusive Language Online, pages 80–93, Florence, Italy, Aug. 2019. Association for Computational Linguistics. doi: 10.18653/v1/ W19-350… find
  92. O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky,… find
  93. J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021. URL https: //arxiv.org/abs/2109.01652 . find
  94. L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh, Z. Kenton, S. Brown, W. Hawkins, T. Stepleton, C. Biles, A. Birhane, J. Haas, L. Laura Rimell, L. A. Hendricks, W. Isaac, S. Legassick, G. Irving, and I. Gabriel. Ethical and social ri… find our card
  95. J. Welbl, A. Glaese, J. Uesato, S. Dathathri, J. Mellor, L. A. Hendricks, K. Anderson, P. Kohli, B. Coppin, and P.-S. Huang. Challenges in detoxifying language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2447–2469, Punta Cana, Dominican Republic, Nov. 2021… find
  96. J. Wu, L. Ouyang, D. M. Ziegler, N. Stiennon, R. Lowe, J. Leike, and P. Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862, 2021. URL https: //arxiv.org/abs/2109.10862 . find our card
  97. A. Xu, E. Pathak, E. Wallace, S. Gururangan, M. Sap, and D. Klein. Detoxifying language models risks marginalizing minority voices. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2390–2397, Onli… find
  98. J. Xu, D. Ju, M. Li, Y.-L. Boureau, J. Weston, and E. Dinan. Bot-adversarial dialogue for safe conversational agents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2950–2968, 2021b. find
  99. H. Zamani, J. R. Trippas, J. Dalton, and F. Radlinski. Conversational information seeking. arXiv preprint arXiv:2201.08808, 2022. URL https://arxiv.org/abs/2201.08808 . — [arXiv:2201.08808](https://arxiv.org/abs/2201.08808) ↗
  100. Z. Zhang and L. Luo. Hate speech detection: A solved problem? the challenging case of long tail on twitter. Semantic Web, 10(5):925–945, 2019. find
  101. J. Zhao, T. Wang, M. Yatskar, V. Ordonez, and K.-W. Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804.06876, 2018. URL https://arxiv. org/abs/1804.06876 . find our card
  102. D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. URL https://arxiv.org/abs/1909.08593 . ↗