Elamur Research
☰
home / papers / 2020 / Aligning AI With Shared Human Values

Aligning AI With Shared Human Values

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2008.02275

Summary

Machine ethics has had no shared way to measure whether a model understands everyday human values. Earlier work on fairness, safety, empathy or preference learning each covered only a narrow, task-specific slice. This paper introduces ETHICS, a benchmark of over 130,000 English text scenarios. Each scenario links facts about the world to a moral judgement. The tasks follow five normative perspectives: justice (impartiality and desert), deontology, virtue ethics, utilitarianism and commonsense morality. Scenarios were written and relabelled by qualified MTurk workers from the US, Canada and Great Britain. Low-agreement items were removed, so the examples are clear-cut rather than real dilemmas. Contrast-set style counterfactual augmentation and adversarial filtration were used to build a separate Hard Test split. Pretrained language models fine-tuned on ETHICS, BERT-base among them, do better than chance but remain far from reliable, and they degrade sharply on the Hard Test. The authors present the dataset as a tool for measuring ethical knowledge, not for training it into a model. They see it as a step toward steering chatbots or regularizing open-ended RL agents.

Key points

  • Five tasks: Justice, Virtue, Deontology, Utilitarianism and Commonsense, each with Dev, Test and Hard Test splits (Table 1). The Dev splits are, for example, about 21.8k examples for Justice and 28.2k for Virtue.
  • Annotation: writers had to pass qualification tests, examples were relabelled by several annotators, and contentious items were dropped. Agreement with Indian annotators is estimated in Appendix C.
  • Long examples: the Commonsense task includes long, Reddit-style "am I in the wrong" posts alongside short first-person statements.
  • Results: Figure 1 shows a BERT-base model confidently misjudging some harmful scenarios, such as one based on Bostrom's paperclip maximizer. Overall, current models show partial but incomplete ethical understanding. I can't give per-model scores because the results section was missing from the text provided.
  • Availability: the data and the MTurk collection forms are public at github.com/hendrycks/ethics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

289 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 268 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2008.02275 · arXiv 2008.02275

@misc{hendrycks2020aligning,
  title = {{Aligning AI With Shared Human Values}},
  author = {Hendrycks, Dan and Burns, Collin and Basart, Steven and Critch, Andrew and Li, Jerry and Song, Dawn and Steinhardt, Jacob},
  year = {2020},
  doi = {10.48550/arxiv.2008.02275},
  eprint = {2008.02275},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2008.02275}
}

References

67 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Abadi, L. Lamport, and P. Wolper. Realizable and unrealizable specifications of reactive systems. In ICALP, 1989. find
  2. Aristotle. Nicomachean Ethics. 340 BC. S. Armstrong. General purpose intelligence: Arguing the orthogonality thesis. 2013. find
  3. I. Asimov. I, Robot. Gnome Press, 1950. find
  4. I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. ArXiv, abs/2004.05150, 2020. find
  5. J. Bentham. An Introduction to the Principles of Morals and Legislation. Batoche Books, 1781. Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: Reasoning about physical commonsense in natural language. In AAAI, 2019. find our card
  6. N. Bostrom. Superintelligence: Paths, dangers, strategies. 2014. find
  7. R. L. Bras, S. Swayamdipta, C. Bhagavatula, R. Zellers, M. E. Peters, A. Sabharwal, and Y. Choi. Adversarial filters of dataset biases, 2020. find
  8. T. B. Brown, B. P. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krüger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. J. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C… find
  9. C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. In ICML, 2005. find our card
  10. P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In NIPS, 2017. find
  11. S. Corbett-Davies and S. Goel. The measure and mismeasure of fairness: A critical review of fair machine learning. ArXiv, abs/1808.00023, 2018. — [arXiv:1808.00023](https://arxiv.org/abs/1808.00023) find
  12. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805, 2019. find
  13. C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. S. Zemel. Fairness through awareness. ArXiv, abs/1104.3913, 2011. — [arXiv:1104.3913](https://arxiv.org/abs/1104.3913) find
  14. European Commission. Ethics guidelines for trustworthy artificial intelligence. 2019. find
  15. M. Gardner, Y. Artzi, V. Basmova, J. Berant, B. Bogin, S. Chen, P. Dasigi, D. Dua, Y. Elazar, A. Gottumukkala, N. Gupta, H. Hajishirzi, G. Ilharco, D. Khashabi, K. Lin, J. Liu, N. F. Liu, P. Mulcaire, Q. Ning, S. Singh, N. A. Smith, S. Subramanian, R. Tsarfaty, E. Wallace, A. Q. Zhang, and B. Zhou.… find our card
  16. T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. Daumeé III, and K. Crawford. Datasheets for datasets. arXiv preprint arXiv:1803.09010, 2018. find
  17. S. Gillen, C. Jung, M. Kearns, and A. Roth. Online learning with an unknown fairness metric. In NeurIPS, 2018. find
  18. I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014. find our card
  19. J. Haidt et al. The moral emotions. Handbook of affective sciences, 11(2003):852–870, 2003. find
  20. M. Hardt, E. Price, and N. Srebro. Equality of opportunity in supervised learning. In NIPS, 2016. find our card
  21. M. Hausknecht, P. Ammanabrolu, C. Marc-Alexandre, and Y. Xingdi. Interactive fiction games: A colossal adventure. CoRR, abs/1909.05398, 2019. — [arXiv:1909.05398](https://arxiv.org/abs/1909.05398) find
  22. D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. ICLR, 2019. find
  23. D. Hendrycks and K. Gimpel. Gaussian error linear units (GELUs). arXiv preprint 1606.08415, 2016. find
  24. D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. ICLR, 2017. find
  25. F. Hill, S. Mokra, N. Wong, and T. Harley. Human instruction-following with deep reinforcement learning via transfer-learning from text. ArXiv, abs/2005.09382, 2020. — [arXiv:2005.09382](https://arxiv.org/abs/2005.09382) find
  26. D. Hume. A Treatise of Human Nature. 1739. F. Hutcheson. Inquiry into the Original of Our Ideas of Beauty and Virtue. 1725. A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov. Bag of tricks for efficient text classification. ArXiv, abs/1607.01759, 2017. find
  27. Justinian I. The Institutes of Justinian. 533. S. Kagan. The Limits of Morality. Oxford: Clarendon Press, 1991. find
  28. I. Kant. Groundwork of the Metaphysics of Morals. 1785. D. Kaushik, E. H. Hovy, and Z. C. Lipton. Learning the difference that makes a difference with counterfactually-augmented data. ArXiv, abs/1909.12434, 2020. find
  29. M. P. Kim, O. Reingold, and G. N. Rothblum. Fairness through computationally-bounded awareness. In NeurIPS, 2018. find
  30. N. Kitaev, L. Kaiser, and A. Levskaya. Reformer: The efficient transformer. ArXiv, abs/2001.04451, 2020. find our card
  31. J. M. Kleinberg, S. Mullainathan, and M. Raghavan. Inherent trade-offs in the fair determination of risk scores. ArXiv, abs/1609.05807, 2017. — [arXiv:1609.05807](https://arxiv.org/abs/1609.05807) find
  32. Y. Koren. Factorization meets the neighborhood: a multifaceted collaborative filtering model. In KDD, 2008. find
  33. B. Krause, A. D. Gotmare, B. McCann, N. Keskar, S. R. Joty, R. Socher, and N. F. Rajani. Gedi: Generative discriminator guided sequence generation. ArXiv, abs/2009.06367, 2020. find our card
  34. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. ArXiv, abs/1909.11942, 2020. find
  35. S. Lazar. Duty and doubt. Journal of Practical Ethics, 2020. find
  36. K. d. Lazari-Radek and P. Singer. Utilitarianism: a very short introduction. Oxford Univ. Press, 2017. find
  37. M. A. Lewis. A comparison of three models for determining test fairness. 1978. find
  38. Z. C. Lipton and J. Steinhardt. Troubling trends in machine learning scholarship. ACM Queue, 17:80, 2018. find
  39. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692, 2019. find
  40. J. Lygeros, C. Tomlin, and S. Sastry. Controllers for reachability specifications for hybrid systems. Automatica, 35(3):349–370, 1999. find
  41. J. S. Mill. Utilitarianism. Batoche Books, 1863. find
  42. J. H. Moor. The nature, importance, and difficulty of machine ethics. IEEE Intelligent Systems, 21 (4):18–21, 2006. find
  43. Mozi. Mozi. 5th century BC. V. C. Müller. Ethics of artificial intelligence and robotics. In The Stanford Encyclopedia of Philosophy, chapter 2.8 Machine Ethics. 2020. find
  44. J. V. Neumann and O. Morgenstern. Theory of games and economic behavior. Journal of the American Statistical Association, 40:263, 1944. find
  45. A. Y. Ng and S. J. Russell. Algorithms for inverse reinforcement learning. In ICML, 2000. find
  46. M. Nussbaum. Capabilities as fundamental entitlements: Sen and social justice. Feminist Economics, 9:33 – 59, 2003. find
  47. D. Parfit. Reasons and Persons. Oxford: Clarendon Press, 1987. find
  48. J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. In EMNLP, 2014. find
  49. H. Rashkin, E. M. Smith, M. Li, and Y.-L. Boureau. Towards empathetic open-domain conversation models: A new benchmark and dataset. In ACL, 2019. find
  50. J. Rawls. A Theory of Justice. Harvard University Press, 1999. find
  51. A. Ray, J. Achiam, and D. Amodei. Benchmarking safe exploration in deep reinforcement learning. 2019. find
  52. T. Reid. Essays on the Active Powers of Man. Edinburgh University Press, 1788. find
  53. I. Robeyns. Wellbeing, Freedom and Social Justice: The Capability Approach Re-Examined. 2017. find
  54. S. Roller, E. Dinan, N. Goyal, D. Y. Ju, M. F. Williamson, Y. Liu, J. Xu, M. Ott, K. Shuster, E. M. Smith, Y.-L. Boureau, and J. Weston. Recipes for building an open-domain chatbot. ArXiv, abs/2004.13637, 2020. find our card
  55. W. D. Ross. The Right and the Good. 1930. find
  56. G. N. Rothblum and G. Yona. Probably approximately metric-fair learning. In ICML, 2018. find
  57. V. Sanh, L. Debut, J. Chaumond, and T. Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, 2019. find
  58. P. Scharre. Army of None: Autonomous Weapons and the Future of War. Tantor Audio, Old Saybrook, CT, 2018. ISBN 1541469682. find
  59. H. Sidgwick. The Methods of Ethics. 1907. find
  60. N. Soares, B. Fallenstein, S. Armstrong, and E. Yudkowsky. Corrigibility. In AAAI Workshop, 2015. find
  61. R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013. find
  62. D. Tang, B. Qin, and T. Liu. Learning semantic representations of users and products for document level sentiment classification. In ACL, 2015. find
  63. White House. Big data: A report on algorithmic systems, opportunity, and civil rights. 2016. find
  64. J. Wieting, M. Bansal, K. Gimpel, and K. Livescu. Towards universal paraphrastic sentence embeddings. CoRR, abs/1511.08198, 2016. find our card
  65. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, and J. Brew. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771, 2019. find our card
  66. M. B. Zafar, I. Valera, M. Gomez-Rodriguez, and K. P. Gummadi. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. WWW, 2017. find
  67. B. Zhou, D. Khashabi, Q. Ning, and D. Roth. “going on a vacation” takes longer than “going for a walk”: A study of temporal commonsense understanding. EMNLP-IJCNLP, 2019. find