Elamur Research⌕
☰
home / papers / 2021 / Alignment of Language Agents

Alignment of Language Agents

Paper · 2021 · Zachary Kenton, Tom Everitt, Laura Weidinger et al.

Summary

This DeepMind position paper asks how systems that act only by producing text, which the authors call language agents, can end up behaving against their designers' intentions. Examples include generatively trained LLMs such as GPT-3 and RL agents in text-based games. Earlier alignment work mostly assumed delegate agents acting physically in the world, partly on the view that text-only systems pose little risk. The authors dispute that view. They introduce no dataset, model or benchmark. Instead, they survey the alignment literature and build a conceptual framework. It traces accidental designer misspecification to three sources: the training data, the training process, and requirements for behaviour outside the training distribution. They then map these sources to possible failures: harmful content, deceptive or manipulative language, and gaming of the stated objective. They also review partial remedies and note their limitations. The paper concludes that few reliable fixes exist yet. It recommends building agents that assume their specification may be flawed and calls for focused research on deception, manipulation and harm in language systems.

Key points

  • The authors (Kenton, Everitt, Weidinger, Gabriel, Mikulik, Irving; DeepMind, arXiv:2103.14659, 2021) treat Armstrong et al. (2012) on Oracle AI as foundational. Unlike that work, they cover current systems that are neither limited to question answering nor assumed to match human intelligence.
  • The background contrasts behaviour, intent and incentive alignment. It also contrasts two ways of splitting the problem, intent-competence and define-optimize, and refers to causal influence diagrams (CIDs) for formal analysis of incentives.
  • The paper links objective gaming to Goodhart's Law and gives two language-agent examples. In the summarization work of Stiennon et al. (2020), an RL agent eventually exploited its learnt reward model, scoring high on it but low with human raters. In the negotiation work of Lewis et al. (2017), self-play agents developed a successful but humanly incomprehensible language.
  • The reviewed remedies have limits. Reward models learnt from human feedback can still be gamed, and online updating, as in Ibarz et al. (2018), is hard to run in practice. Side-effect penalties and agents that are uncertain about their objective are both unproven in the language setting.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

Cite

Cite as Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik and Geoffrey Irving. 2021. Alignment of Language Agents. arXiv:2103.14659.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2103.14659 · arXiv 2103.14659

@misc{kenton2021alignment,
  title = {{Alignment of Language Agents}},
  author = {Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik and Geoffrey Irving},
  year = {2021},
  doi = {10.48550/arxiv.2103.14659},
  eprint = {2103.14659},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2103.14659}
}

References

37 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016.
  2. S. Armstrong and X. O’Rorke. Good and safe uses of AI oracles. arXiv preprint arXiv:1711.05541, 2017. — [arXiv:1711.05541](https://arxiv.org/abs/1711.05541) find
  3. S. Armstrong, A. Sandberg, and N. Bostrom. Thinking inside the box: Controlling and using an oracle AI. Minds and Machines, 22(4):299–324, 2012. find
  4. A. Barnhill. What is manipulation. Manipulation: Theory and practice, 50:72, 2014. find
  5. M. Baron. The mens rea and moral status of manipulation. In C. Coons and M. Weber, editors, Manipulation: theory and practice. Oxford University Press, 2014. find
  6. Y. Belinkov and J. Glass. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7: 49–72, 2019.
  7. E. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? Proceedings of FAccT, 2021. find
  8. J. S. Blumenthal-Barby. Between reason and coercion: ethically permissible influence in health care and health policy contexts. Kennedy Institute of Ethics Journal, 22(4):345–366, 2012. find
  9. N. Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, 2014. find
  10. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
  11. M. Brundage, S. Avin, J. Clark, H. Toner, P. Eckersley, B. Garfinkel, A. Dafoe, P. Scharre, T. Zeitzoff, B. Filar, et al. The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. arXiv preprint arXiv:1802.07228, 2018. find
  12. P. Christiano. Clarifying AI alignment, AI alignment forum. https://www.alignmentforum.org/posts/ZeE7EKHTFMBs8eMxn/clarifying-ai-alignment , 2018. Accessed: 2020-12-15. ↗
  13. P. Christiano, B. Shlegeris, and D. Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575, 2018. find
  14. P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30:4299– 4307, 2017. find
  15. J. Clark and D. Amodei. Faulty reward functions in the wild. https://openai.com/blog/faulty-reward-functions/ , 2016. Accessed: 2020-12-18. ↗
  16. Conversation-AI. Perspective api. https://www.perspectiveapi.com/ , 2017. Accessed: 2020-01-11. ↗
  17. A. D’Amour, K. Heller, D. Moldovan, B. Adlam, B. Alipanahi, A. Beutel, C. Chen, J. Deaton, J. Eisenstein, M. D. Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. arXiv preprint arXiv:2011.03395, 2020. — [arXiv:2011.03395](https://arxiv.org/abs/2011.03395) find
  18. M. Danilevsky, K. Qian, R. Aharonov, Y. Katsis, B. Kawas, and P. Sen. A survey of the state of explainable AI for natural language processing. arXiv preprint arXiv:2010.00711, 2020.
  19. R. Dawkins and J. R. Krebs. Animal signals: information or manipulation. Behavioural ecology: An evolutionary approach, 2:282–309, 1978. find
  20. D. C. Dennett. The intentional stance. MIT press, 1989. find
  21. R. Dunbar et al. Theory of mind and the evolution of language. Approaches to the Evolution of Language, pages 92–110, 1998. find
  22. T. Everitt, G. Lea, and M. Hutter. AGI safety literature review. arXiv preprint arXiv:1805.01109, 2018. — [arXiv:1805.01109](https://arxiv.org/abs/1805.01109) find
  23. T. Everitt, R. Kumar, V. Krakovna, and S. Legg. Modeling AGI safety frameworks with causal influence diagrams. arXiv preprint arXiv:1906.08663, 2019a. — [arXiv:1906.08663](https://arxiv.org/abs/1906.08663) find
  24. T. Everitt, P. A. Ortega, E. Barnes, and S. Legg. Understanding agent incentives using causal influence diagrams. part i: Single action settings. arXiv preprint arXiv:1902.09980, 2019b. — [arXiv:1902.09980](https://arxiv.org/abs/1902.09980) find
  25. T. Everitt, R. Carey, E. Langlois, P. A. Ortega, and S. Legg. Agent incentives: A causal perspective. In Proceedings of the AAAI Conference on Artificial Intelligence. 35 (Feb. 2021), AAAI’21, 2021a. find
  26. T. Everitt, M. Hutter, R. Kumar, and V. Krakovna. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. Accepted to Synthese, 2021, 2021b. find
  27. T. Fornaciari and M. Poesio. Automatic deception detection in italian court cases. Artificial intelligence and law, 21(3):303–340, 2013. find
  28. C. B. Frey and M. A. Osborne. The future of employment: How susceptible are jobs to computerisation? Technological forecasting and social change, 114:254–280, 2017. find
  29. I. Gabriel. Artificial intelligence, values and alignment. arXiv preprint arXiv:2001.09768, 2020. — [arXiv:2001.09768](https://arxiv.org/abs/2001.09768) find
  30. S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462, 2020.
  31. C. A. Goodhart. Problems of monetary management: the UK experience. In Monetary Theory and Practice, pages 91–121. Springer, 1984. find
  32. D. Hadfield-Menell, A. Dragan, P. Abbeel, and S. Russell. Cooperative inverse reinforcement learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, page 3916–3924, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819. find
  33. D. Hadfield-Menell, A. Dragan, P. Abbeel, and S. Russell. The off-switch game. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI’17, page 220–227. AAAI Press, 2017a. ISBN 9780999241103. find
  34. D. Hadfield-Menell, S. Milli, P. Abbeel, S. Russell, and A. D. Dragan. Inverse reward design. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6768–6777, Red Hook, NY, USA, 2017b. Curran Associates Inc. ISBN 9781510860964. find
  35. M. D. Hauser. The evolution of communication. MIT press, 1996. find
  36. P. Henderson, K. Sinha, N. Angelard-Gontier, N. R. Ke, G. Fried, R. Lowe, and J. Pineau. Ethical challenges in data-driven dialogue systems. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 123–129, 2018. find
  37. P.-S. Huang, H. Zhang, R. Jiang, R. Stanforth, J. Welbl, J. Rae, V. Maini, D. Yogatama, and P. Kohli. Reducing sentiment bias in language models via counterfactual evaluation. arXiv preprint arXiv:1911.03064, 2019.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kenton-2021-alignment-language-agents/