Elamur Research
☰
home / papers / 2024 / A Roadmap to Pluralistic Alignment

A Roadmap to Pluralistic Alignment

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2402.05070

Summary

This ICML 2024 position paper argues that aligning AI to an "average" human preference ignores real disagreement among people. Using LLMs as a testbed, the authors define three ways a model can be pluralistic:

  • Overton pluralism: covering every reasonable answer.
  • Steerable pluralism: reliably adopting a requested perspective or value.
  • Distributional pluralism: matching how a target population's answers are spread.

They also propose three kinds of benchmarks:

  • multi-objective,
  • trade-off steerable, which rewards hitting arbitrary balances between objectives,
  • jury-pluralistic, which models individual raters explicitly.

To test the claim that current alignment narrows diversity, they compare base and aligned models against human answer distributions on GlobalQA and MPI using Jensen-Shannon distance. Aligned models generally drift further from human distributions, suggesting RLHF-style methods may reduce distributional pluralism. The paper is a framework and research agenda, not a finished method. It calls for finer-grained pluralistic evaluations, normative debate on alignment targets and customization limits, and new alignment techniques.

Key points

  • Three model-level notions of pluralism: Overton, steerable and distributional. The authors note they can conflict, for example Overton versus distributional pluralism.
  • Three benchmark classes: multi-objective, trade-off steerable and jury-pluralistic. Each is suited to different evaluation settings.
  • Experiment setup: human versus model answer distributions are compared on GlobalQA (Japan, US, Germany targets) and MPI. Model families are LLaMA (base vs. Alpaca and Tulu), LLaMA2 (13B), Gemma (7B) and GPT-3.
  • Example result: on GlobalQA (Japan), JS distance is about 0.40 for base LLaMA, compared with 0.45 for Alpaca and 0.54 for Tulu. The only noted exception is GPT-3 on MPI, which the authors ask readers to treat cautiously because of uncertainty over OpenAI model versions.
  • Code: released on GitHub at jfisher52/AI_Pluralistic_Alignment.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

121 papers of our library cite it (of the 140673 whose bibliographies we read).

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.05070 · arXiv 2402.05070

@misc{sorensen2024roadmap,
  title = {{A Roadmap to Pluralistic Alignment}},
  author = {Sorensen, Taylor and Moore, Jared and Fisher, Jillian and Gordon, Mitchell and Mireshghallah, Niloofar and Rytting, Christopher Michael and Ye, Andre and Jiang, Liwei and Lu, Ximing and Dziri, Nouha and Althoff, Tim and Choi, Yejin},
  year = {2024},
  doi = {10.48550/arxiv.2402.05070},
  eprint = {2402.05070},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.05070}
}

References

132 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. theoretical, we believe that the positive impact to discussions around pluralism in alignment outweigh any marginal potential for dual use, which we believe to be minimal. Oxford English Dictionary, s.v. “Overton window (n.)”, July 2023. URL https://doi.org/10.1093/OED/ 1985277434. \#\# Acknowledgment… — [doi:10.1093/oed/](https://doi.org/10.1093/oed/) find
  2. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosuite,… find our card
  3. Aher, G. V., Arriaga, R. I., and Kalai, A. T. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, pp. 337–371. PMLR, 2023. find
  4. Bakker, M. A., Chadwick, M. J., Sheahan, H. R., Tessler, M. H., Campbell-Gillingham, L., Balaguer, J., McAleese, N., Glaese, A., Aslanides, J., Botvinick, M. M., and Summerfield, C. Fine-tuning language models to find agreement among humans with diverse preferences, 2022. find
  5. Anthropic. Introducing claude, 2023. URL https://ww w.anthropic.com/index/introducing-cla ude. find
  6. Berlin, I. Two concepts of liberty. In Four Essays on Liberty, pp. 118–172. Oxford University Press, Oxford, 1969. find
  7. Argyle, L., Busby, E., Fulda, N., Gubler, J., Rytting, C., and Wingate, D. Out of one, many: Using language models to simulate human samples. Political Analysis, 31:1–15, 02 2023. doi: 10.1017/pan.2023.2. find our card
  8. Bobu, A., Peng, A., Agrawal, P., Shah, J., and Dragan, A. D. Aligning robot and human representations. arXiv preprint arXiv:2302.01928, 2023. — [arXiv:2302.01928](https://arxiv.org/abs/2302.01928) find
  9. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N. S., Chen, A. S., Creel, K. A., Davis, J., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E., Ermo… find our card
  10. Arnesen, S. and Peters, Y. The legitimacy of representation: How descriptive, formal, and responsiveness representation affect the acceptability of political decisions. Comparative Political Studies, 51(7):868–899, 2018. doi: 10.1177/0010414017720702. URL https: //doi.org/10.1177/0010414017720702. — [doi:10.1177/0010414017720702](https://doi.org/10.1177/0010414017720702) find
  11. Aroyo, L., Taylor, A. S., Diaz, M., Homan, C. M., Parrish, A., Serapio-Garcia, G., Prabhakaran, V., and Wang, D. Dices dataset: Diversity in conversational ai evaluation for safety, 2023. find
  12. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., Das-Sarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., and Kaplan, J. A general language assi… find our card
  13. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das-Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C.… find
  14. Bowman, S. R., Hyun, J., Perez, E., Chen, E., Pettit, C., Heiner, S., Lukošiūtė, K., Askell, A., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Olah, C., Amodei, D., Amodei, D., Drain, D., Li, D., Tran-Johnson, E., Kernion, J., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse… find
  15. Cotra, A. Why ai alignment could be hard with modern deep learning. https://www.cold-takes.com/why -ai-alignment-could-be-hard-with-mod ern-deep-learning/, 2021. ↗
  16. Crenshaw, K. Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. The University of Chicago Legal Forum, 140:139–167, 1989. find
  17. Boykoff, M. T. and Boykoff, J. M. Balance as bias: global warming and the us prestige press. Global Environmental Change, 14(2):125–136, 2004. ISSN 0959-3780. doi: https://doi.org/10.1016/j.gloenvcha.2003.10.001. URL https://www.sciencedirect.com/scienc e/article/pii/S0959378003000669. — [doi:10.1016/j.gloenvcha.2003.10.001](https://doi.org/10.1016/j.gloenvcha.2003.10.001) ↗
  18. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T. J., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S.,… find
  19. Buttrick, N. Studying large language models as compression algorithms for human culture. Trends in Cognitive Sciences, S1364-6613(24):00001–9, 2024. doi: 10.1016/j.tics.2024.01.001. Epub ahead of print. — [doi:10.1016/j.tics.2024.01.001](https://doi.org/10.1016/j.tics.2024.01.001) find
  20. Danry, V., Pataranutaporn, P., Mao, Y., and Maes, P. Don’t just tell me, ask me: Ai systems that intelligently frame explanations as questions improve human logical discernment accuracy over causal ai explanations. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI… — [doi:10.1145/3544548.3580672](https://doi.org/10.1145/3544548.3580672) find
  21. Durmus, E., Nyugen, K., Liao, T. I., Schiefer, N., Askell, A., Bakhtin, A., Chen, C., Hatfield-Dodds, Z., Hernandez, D., Joseph, N., Lovitt, L., McCandlish, S., Sikder, O., Tamkin, A., Thamkul, J., Kaplan, J., Clark, J., and Ganguli, D. Towards measuring the representation of subjective global opini… find our card
  22. Ethayarajh, K. and Jurafsky, D. Utility is in the eye of the user: A critique of nlp leaderboard design. In Conference on Empirical Methods in Natural Language Processing, 2020. URL https://api.semanticscholar. org/CorpusID:235408131. ↗
  23. Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., Wang, T., Marks, S., Ségerie, C.-R., Carroll, M., Peng, A., Christoffersen, P. J., Damani, M., Slocum, S., Anwar, U., Siththaranjan, A., Nadeau, M., Michaud, E. J., Pfau, J.,… find
  24. Ethayarajh, K. and Jurafsky, D. The authenticity gap in human evaluation. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6056–6070, Abu Dhabi, United Arab Emirates, December 2022. Association for Comput… find
  25. Chen, J., Liu, Z., Huang, X., Wu, C., Liu, Q., Jiang, G., Pu, Y., Lei, Y., Chen, X., Wang, X., Lian, D., and Chen, E. When large language models meet personalization: Perspectives of challenges and opportunities, 2023. find
  26. Feng, S., Park, C. Y., Liu, Y., and Tsvetkov, Y. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair nlp models, 2023. find our card
  27. Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De-Flanigan, B., Gölz, P., Gupta, A., Hennig, B., and Procaccia, A. D. Fair algorithms for selecting citizens’assemblies. Nature, 596(7873):548–552, 2021. doi: 10.1038/s41586 -021-03788-6. U… — [doi:10.1038/s41586](https://doi.org/10.1038/s41586) find
  28. Hayati, S. A., Lee, M., Rajagopal, D., and Kang, D. How far can we extract diverse perspectives from large language models? criteria-based diversity prompting! ArXiv, abs/2311.09799, 2023. URL https://api.semant icscholar.org/CorpusID:265220883. ↗
  29. Fleisig, E., Abebe, R., and Klein, D. When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks, November 2023. URL http://arxiv.org/ abs/2305.06626. arXiv:2305.06626 [cs]. ↗
  30. Hayes, C. F., Rădulescu, R., Bargiacchi, E., Källström, J., Macfarlane, M., Reymond, M., Verstraeten, T., Zintgraf, L. M., Dazeley, R., Heintz, F., Howley, E., Irissappane, A. A., Mannion, P., Nowé, A., Ramos, G., Restelli, M., Vamplew, P., and Roijers, D. M. A practical guide to multi-objective re… — [doi:10.1007/s10458-022-09552-y](https://doi.org/10.1007/s10458-022-09552-y) find
  31. Gabriel, I. Artificial intelligence, values, and alignment. Minds and Machines, 30(3):411–437, 2020. doi: 10.100 7/s11023-020-09539-2. URL https://doi.org/ 10.1007/s11023-020-09539-2. — [doi:10.1007/s11023-020-09539-2](https://doi.org/10.1007/s11023-020-09539-2) find
  32. Girotra, K., Meincke, L., Terwiesch, C., and Ulrich, K. T. Ideas are dimes a dozen: Large language models for idea generation in innovation. https:// ssrn.com/abstract=4526071, July 2023. Available at SSRN: https://ssrn.com/abstract=4526071 or http://dx.doi.org/10.2139/ssrn.4526071. — [doi:10.2139/ssrn.4526071](https://doi.org/10.2139/ssrn.4526071) ↗
  33. Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., and Steinhardt, J. Aligning ai with shared human values. arXiv preprint arXiv:2008.02275, 2020. find our card
  34. Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., and Steinhardt, J. Aligning ai with shared human values, 2023. find
  35. Glaese, A., McAleese, N., Tr˛ebacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., Campbell-Gillingham, L., Uesato, J., Huang, P.-S., Comanescu, R., Yang, F., See, A., Dathathri, S., Greig, R., Chen, C., Fritz, D., Elias, J. S., Green, R., Mokrá, S., F… find our card
  36. Henrich, J., Heine, S. J., and Norenzayan, A. The weirdest people in the world? Behavioral and Brain Sciences, 33 (2-3):61–83, 2010. URL http://www2.psych.ubc .ca/~henrich/audiofiles/WEIRD1.mp3. ↗
  37. Hosking, T., Blunsom, P., and Bartolo, M. Human feedback is not gold standard. ArXiv, abs/2309.16349, 2023. URL https://api.semanticscholar.org/Corp usID:263134280. — [arXiv:2309.16349](https://arxiv.org/abs/2309.16349) ↗
  38. Gordon, M. L., Lam, M. S., Park, J. S., Patel, K., Hancock, J., Hashimoto, T., and Bernstein, M. S. Jury learning: Integrating dissenting voices into machine learning models. In CHI Conference on Human Factors in Computing Systems, CHI ’22. ACM, April 2022. doi: 10.1145/3491102.3502004. URL http://d… — [doi:10.1145/3491102.3502004](https://doi.org/10.1145/3491102.3502004) find
  39. Hsieh, N.-h. and Andersson, H. Incommensurable Values. In Zalta, E. N. (ed.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Fall 2021 edition, 2021. find
  40. Hwang, E., Majumder, B. P., and Tandon, N. Aligning Language Models to User Opinions. 2023. doi: 10.48550 /ARXIV.2305.14929. URL https://arxiv.org/ abs/2305.14929. Publisher: arXiv Version Number: 1. ↗
  41. Guerreiro, A. P., Fonseca, C. M., and Paquete, L. The hypervolume indicator. ACM Computing Surveys (CSUR), 54:1–42, 2020. URL https://api.semanticsc holar.org/CorpusID:218470181. ↗
  42. Imundo, M. and Rapp, D. When fairness is flawed: Effects of false balance reporting and weight-of-evidence statements on beliefs and perceptions of climate change. Journal of Applied Research in Memory and Cognition, 11, 10 2021. doi: 10.1016/j.jarmac.2021.10.002. — [doi:10.1016/j.jarmac.2021.10.002](https://doi.org/10.1016/j.jarmac.2021.10.002) find
  43. Haraway, D. Situated knowledges: The science question in feminism and the privilege of partial perspective. Feminist Studies, 14(3):575–599, 1988. ISSN 00463663. URL http://www.jstor.org/stable/3178066. ↗
  44. Harsanyi, J. C., Selten, R., et al. A general theory of equilibrium selection in games. MIT Press Books, 1, 1988. find
  45. Jakesch, M., Bhat, A., Buschek, D., Zalmanson, L., and Naaman, M. Co-writing with opinionated language models affects users’ views. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 97814503942… find
  46. Hartmann, J., Schwenzow, J., and Witte, M. The political ideology of conversational ai: Converging evidence on chatgpt’s pro-environmental, left-libertarian orientation, 2023. doi: 10.1145/3544548.3581196. URL https: //doi.org/10.1145/3544548.3581196. find
  47. Krügel, S., Ostermaier, A., and Uhl, M. Chatgpt’s inconsistent moral advice influences users’judgment. Scientific Reports, 13(1):4569, Apr 2023. ISSN 2045-2322. doi: 10.1038/s41598-023-31341-0. URL https: //doi.org/10.1038/s41598-023-31341-0. — [doi:10.1038/s41598-023-31341-0](https://doi.org/10.1038/s41598-023-31341-0) find
  48. Jang, J., Kim, S., Lin, B. Y., Wang, Y., Hessel, J., Zettlemoyer, L., Hajishirzi, H., Choi, Y., and Ammanabrolu, P. Personalized soups: Personalized large language model alignment via post-hoc parameter merging, 2023. find
  49. Landemore, H. and Page, S. E. Deliberation and disagreement: Problem solving, prediction, and positive dissensus. Politics, philosophy & economics, 14(3):229–254, 2015. find
  50. Ji, J., Qiu, T., Chen, B., Zhang, B., Lou, H., Wang, K., Duan, Y., He, Z., Zhou, J., Zhang, Z., Zeng, F., Ng, K. Y., Dai, J., Pan, X., O’Gara, A., Lei, Y., Xu, H., Tse, B., Fu, J., McAleer, S., Yang, Y., Wang, Y., Zhu, S.-C., Guo, Y., and Gao, W. Ai alignment: A comprehensive survey, 2024. find
  51. Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., and Legg, S. Scalable agent alignment via reward modeling: a research direction, 2018. find
  52. Ji, Z., Li, J. D., and Telgarsky, M. Early-stopped neural networks are consistent, 2021. find
  53. Li, C., Zhang, M., Mei, Q., Wang, Y., Hombaiah, S. A., Liang, Y., and Bendersky, M. Teach llms to personalize – an approach inspired by writing education, 2023a. find
  54. Jiang, G., Xu, M., Zhu, S.-C., Han, W., Zhang, C., and Zhu, Y. Evaluating and inducing personality in pre-trained language models, 2023. URL https://api.sema nticscholar.org/CorpusID:258865158. ↗
  55. Li, J., Mehrabi, N., Peris, C., Goyal, P., Chang, K.-W., Galstyan, A., Zemel, R., and Gupta, R. On the steerability of large language models toward data-driven personas. 2023b. doi: 10.48550/ARXIV.2311.04978. URL ht tps://arxiv.org/abs/2311.04978. Publisher: arXiv Version Number: 1. find
  56. Jiang, H., Beeferman, D., Roy, B., and Roy, D. Communitylm: Probing partisan worldviews from language models, 2022. find
  57. Jung, J., Qin, L., Welleck, S., Brahman, F., Bhagavatula, C., Bras, R. L., and Choi, Y. Maieutic prompting: Logically consistent reasoning with recursive explanations, 2022. find our card
  58. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., Newman, B., Yuan, B., Yan, B., Zhang, C., Cosgrove, C., Manning, C. D., Ré, C., Acosta-Navas, D., Hudson, D. A., Zelikman, E., Durmus, E., Ladhak, F., Rong, F., Ren, H., Yao, H., Wan… find
  59. Kant, I. Kant: Critique of Practical Reason. Cambridge Texts in the History of Philosophy. Cambridge University Press, 2 edition, 1788. doi: 10.1017/CBO97813161364 78. Kasirzadeh, A. and Gabriel, I. In conversation with artificial intelligence: aligning language models with human values, 2022. — [doi:10.1017/cbo97813161364](https://doi.org/10.1017/cbo97813161364) find
  60. Kekes, J. The Morality of Pluralism. Princeton University Press, Princeton, 1993. find
  61. Liu, A., Sap, M., Lu, X., Swayamdipta, S., Bhagavatula, C., Smith, N. A., and Choi, Y. Dexperts: Decoding-time controlled text generation with experts and anti-experts. In Annual Meeting of the Association for Computational Linguistics, 2021. URL https://api.semantic scholar.org/CorpusID:235313967. ↗ our card
  62. Kim, J. and Lee, B. Ai-augmented surveys: Leveraging large language models for opinion prediction in nationally representative surveys. arXiv preprint arXiv:2305.09620, 2023. — [arXiv:2305.09620](https://arxiv.org/abs/2305.09620) find
  63. Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., and Raileanu, R. Understanding the effects of rlhf on llm generalisation and diversity, 2024. find our card
  64. Liu, A., Swayamdipta, S., Smith, N. A., and Choi, Y. Wanli: Worker and ai collaboration for natural language inference dataset creation, 2022. find our card
  65. Liu, A., Han, X., Wang, Y., Tsvetkov, Y., Choi, Y., and Smith, N. A. Tuning language models by proxy, 2024. find
  66. Koster, R., Balaguer, J., Tacchetti, A., Weinstein, A., Zhu, T., Hauser, O., Williams, D., Campbell-Gillingham, L., Thacker, P., Botvinick, M., and Summerfield, C. Human-centred mechanism design with democratic ai. Nature Human Behaviour, 6(10):1398–1407, 2022. doi: 10.103 8/s41562-022-01383-x. URL… — [doi:10.1038/s41562-022-01383-x](https://doi.org/10.1038/s41562-022-01383-x) find
  67. Liu, N. F., Kumar, A., Liang, P., and Jia, R. Are sample-efficient nlp models more robust?, 2023. find
  68. Long, J. Large language model guided tree-of-thought. arXiv preprint arXiv:2305.08291, 2023. find
  69. Lu, X., West, P., Zellers, R., Bras, R. L., Bhagavatula, C., and Choi, Y. Neurologic decoding: (un)supervised neural text generation with predicate logic constraints. ArXiv, abs/2010.12884, 2020. URL https://api.semant icscholar.org/CorpusID:225067055. ↗
  70. Page, S. The difference: How the power of diversity creates better groups, firms, schools, and societies-new edition. Princeton University Press, 2008. find
  71. Page, S. E. The diversity bonus: How great teams pay off in the knowledge economy. Princeton University Press, 2019. find
  72. Lu, X., Welleck, S., Hessel, J., Jiang, L., Qin, L., West, P., Ammanabrolu, P., and Choi, Y. Quark: Controllable text generation with reinforced unlearning, 2022. find our card
  73. Pan, A., Chan, J. S., Zou, A., Li, N., Basart, S., Woodside, T., Ng, J., Zhang, H., Emmons, S., and Hendrycks, D. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark, 2023. find
  74. Ma, X., Mishra, S., Liu, A., Su, S., Chen, J., Kulkarni, C., Cheng, H.-T., Le, Q., and Chi, E. Beyond chatbots: Ex-plorellm for structured thoughts and personalized model responses, 2023. find
  75. MacAskill, W. Normative Uncertainty as a Voting Problem. Mind, 125(500):967–1004, October 2016. ISSN 0026-4423. doi: 10.1093/mind/fzv169. URL https://doi. org/10.1093/mind/fzv169. — [doi:10.1093/mind/fzv169](https://doi.org/10.1093/mind/fzv169) find
  76. Michael, J., Mahdi, S., Rein, D., Petty, J., Dirani, J., Padmakumar, V., and Bowman, S. R. Debate helps supervise unreliable experts. arXiv preprint arXiv:2311.08702, 2023. — [arXiv:2311.08702](https://arxiv.org/abs/2311.08702) find
  77. Park, J. S., Popowski, L., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Social simulacra: Creating populated prototypes for social computing systems, 2022. find
  78. Park, J. S., O’Brien, J., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp. 1–22, 2023. find
  79. Peng, A., Netanyahu, A., Ho, M. K., Shu, T., Bobu, A., Shah, J., and Agrawal, P. Diagnosis, feedback, adaptation: A human-in-the-loop framework for test-time policy adaptation. In Proceedings of the 40th International Conference on Machine Learning, 2023. find
  80. Min, S., Michael, J., Hajishirzi, H., and Zettlemoyer, L. AmbigQA: Answering ambiguous open-domain questions. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5783–5797, Online, November 2020. Ass… find
  81. Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251, pp. 13387–13434, July 2022. doi: 10.18653/v1/2023.finding s-acl.847.… find our card
  82. Mishra, A. Ai alignment and social choice: Fundamental limitations and policy implications, 2023. find
  83. Moulin, H. Fair Division and Collective Welfare. MIT Press, 2004. find
  84. Pew Research Center. The state of online harassment. Technical report, Washington, D.C., January 2021. URL https://www.pewresearch.org/internet /2021/01/13/the-state-of-online-haras sment/. ↗
  85. Nagel, T. The fragmentation of value. In Mortal Questions. Cambridge University Press, Cambridge, 1979. find
  86. OpenAI. Openai davinci-002 model. https://www.op enai.com, 2023a. Accessed on Date 06/2023. ↗
  87. Qin, L., Welleck, S., Khashabi, D., and Choi, Y. Cold decoding: Energy-based constrained text generation with langevin dynamics. ArXiv, abs/2202.11705, 2022. URL https://api.semanticscholar.org/Corp usID:247058662. — [arXiv:2202.11705](https://arxiv.org/abs/2202.11705) ↗
  88. OpenAI. Openai gpt3.5-turbo. https://www.openai .com, 2023b. Accessed on Date 06/2023. ↗
  89. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with… find
  90. Ramezani, A. and Xu, Y. Knowledge of cultural moral norms in large language models, 2023. find
  91. Ramé, A., Couairon, G., Shukor, M., Dancette, C., Gaya, J.-B., Soulier, L., and Cord, M. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards, 2023. find
  92. Ovadya, A. Reimagining democracy for ai. Journal of Democracy, 34(4):162–170, Oct 2023. find
  93. Ratcliffe, S. Albert einstein, 2016. URL https://www. oxfordreference.com/view/10.1093/acr ef/9780191826719.001.0001/q-oro-ed4-0 0003988. — [doi:10.1093/acr](https://doi.org/10.1093/acr) find
  94. Rawls, J. A Theory of Justice: Original Edition. Harvard University Press, 1971. ISBN 9780674880108. URL ht tp://www.jstor.org/stable/j.ctvjf9z6v. find
  95. Sharma, A., Rushton, K., Lin, I. W., Nguyen, T., and Althoff, T. Facilitating self-guided mental health interventions through human-language model interaction: A case study of cognitive restructuring. ArXiv, abs/2310.15461, 2023c. URL https://api.semanticscholar.org/ CorpusID:264439507. — [arXiv:2310.15461](https://arxiv.org/abs/2310.15461) ↗
  96. Rawls, J. Political Liberalism. Columbia University Press, New York, 1996. find
  97. Sharma, A., Rushton, K., Lin, I. W., Wadden, D., Lucas, K. G., Miner, A. S., Nguyen, T., and Althoff, T. Cognitive reframing of negative thoughts through human-language model interaction, 2023d. find
  98. Raz, J. Engaging Reason: On the Theory of Value and Action. Oxford University Press, Oxford, 1999. find
  99. Sher, G. On the possibility of a substantive theory of truth. Synthese, 117:133–172, 1998. find
  100. Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., and Hashimoto, T. Whose opinions do language models reflect?, 2023. find our card
  101. Simmons, G. Moral mimicry: Large language models produce moral rationalizations tailored to political identity. In Padmakumar, V., Vallejo, G., and Fu, Y. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pp. 282–29… find
  102. Santy, S., Liang, J. T., Bras, R. L., Reinecke, K., and Sap, M. Nlpositionality: Characterizing design biases of datasets and models, 2023. find
  103. Scherrer, N., Shi, C., Feder, A., and Blei, D. M. Evaluating the moral beliefs encoded in llms, 2023. find
  104. Siththaranjan, A., Laidlaw, C., and Hadfield-Menell, D. Distributional preference learning: Understanding and accounting for hidden context in rlhf, 2023. find
  105. Shajalal, M., Atabuzzaman, M., Baby, M. B., Karim, M. R., and Boden, A. Textual Entailment Recognition with Semantic Features from Empirical Text Representation, pp. 183–195. Springer International Publishing, 2023. ISBN 9783031332319. doi: 10.1007/978-3-031-33231-9\_12. URL http://dx.doi.org/10.1007… — [doi:10.1007/978-3-031-33231-9\_12](https://doi.org/10.1007/978-3-031-33231-9_12) find
  106. Solaiman, I. and Dennison, C. Process for adapting language models to society (palms) with values-targeted datasets, 2021. find
  107. Song, I., Pendse, S. R., Kumar, N., and Choudhury, M. D. The typing cure: Experiences with large language model chatbots for mental health support, 2024. find
  108. Shanahan, M. and Clarke, C. Evaluating large language model creativity from a literary perspective, 2023. find
  109. Sorensen, T., Jiang, L., Hwang, J., Levine, S., Pyatkin, V., West, P., Dziri, N., Lu, X., Rao, K., Bhagavatula, C., Sap, M., Tasioulas, J., and Choi, Y. Value kaleidoscope: Engaging ai with pluralistic human values, rights, and duties, 2023. find
  110. Shanahan, M., McDonell, K., and Reynolds, L. Role-play with large language models, 2023. find our card
  111. Sharma, A., Lin, I. W., Miner, A. S., Atkins, D. C., and Althoff, T. Human–ai collaboration enables more empathic conversations in text-based peer-to-peer mental health support. Nature Machine Intelligence, 5(1):46–57, 2023a. doi: 10.1038/s42256-022-00593-2. URL https: //doi.org/10.1038/s42256-022-0… — [doi:10.1038/s42256-022-00593-2](https://doi.org/10.1038/s42256-022-00593-2) find
  112. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., Kluska, A., Lewkowycz, A., Agarwal, A., Power, A., Ray, A., Warstadt, A., Kocurek, A. W., Safaya, A., Tazarv, A., Xiang, A., Parrish, A., Nie, A., Hussain, A., Askell… find
  113. Stein, B., Ekmekci, B., Lin, B. Y., Howald, B., Orinion, B., Diao, C., Dour, C., Stinson, C., Argueta, C., Ramírez, C. F., Singh, C., Rathkopf, C., Meng, C., Baral, C., Wu, C., Callison-Burch, C., Waites, C., Voigt, C., Manning, C. D., Potts, C., Ramirez, C., Rivera, C. E., Siro, C., Raffel, C., Ash… find our card
  114. Sumers, T. R., Yao, S., Narasimhan, K., and Griffiths, T. L. Cognitive architectures for language agents, 2023. find
  115. Swamy, G., Dann, C., Kidambi, R., Wu, Z. S., and Agarwal, A. A minimaximalist approach to reinforcement learning from human feedback, 2024. find
  116. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model. https:// github.com/tatsu-lab/stanford\_alpaca, 2023. find
  117. Tasioulas, J. Artificial Intelligence, Humanistic Ethics. Daedalus, 151(2):232–243, 05 2022. ISSN 0011-5266. doi: 10.1162/daed\_a\_01912. URL https://doi.or g/10.1162/daed\_a\_01912. — [doi:10.1162/daed\_a\_01912](https://doi.org/10.1162/daed_a_01912) ↗
  118. Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., Tafti, P., Hussenot, L., Chowdhery, A., Roberts, A., Barua, A., Botev, A., Castro-Ros, A., Slone, A., Héliou, A., Tacchetti, A., Bulanova, A., Paterson, A., Tsai, B., Shahriari… find
  119. I., and Hajishirzi, H. How far can camels go? exploring the state of instruction tuning on open resources. ArXiv, abs/2306.04751, 2023a. URL https://api.sema nticscholar.org/CorpusID:259108263. ↗ our card
  120. Wang, Y., Zhong, W., Li, L., Mi, F., Zeng, X., Huang, W., Shang, L., Jiang, X., and Liu, Q. Aligning large language models with human: A survey, 2023b. find our card
  121. Wojcik, S., Hilgard, S., Judd, N., Mocanu, D., Ragain, S., Hunzaker, M. B. F., Coleman, K., and Baxter, J. Birdwatch: Crowd wisdom and bridging algorithms can inform understanding and reduce the spread of misinformation, 2022. find
  122. Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, 2022. find our card
  123. Wright, C. Truth and Objectivity. Harvard University Press, Cambridge, MA, 1992. find
  124. Yang, R., Sun, X., and Narasimhan, K. A generalized algorithm for multi-objective reinforcement learning and policy adaptation, 2019. find
  125. Törnberg, P., Valeeva, D., Uitermark, J., and Bail, C. Simulating social media using large language models to evaluate alternative news feed algorithms. arXiv preprint arXiv:2310.05984, 2023. — [arXiv:2310.05984](https://arxiv.org/abs/2310.05984) find
  126. Zhao, S., Dang, J., and Grover, A. Group Preference Optimization: Few-Shot Alignment of Large Language Models. 2023. doi: 10.48550/ARXIV.2310.11523. URL https://arxiv.org/abs/2310.11523. Publisher: arXiv Version Number: 1. — [doi:10.48550/arxiv.2310.11523](https://doi.org/10.48550/arxiv.2310.11523) ↗
  127. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hossei… find
  128. Zhou, K., Hwang, J. D., Ren, X., and Sap, M. Relying on the unreliable: The impact of language models’ reluctance to express uncertainty, 2024. find
  129. Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., and Yang, D. Can large language models transform computational social science? arXiv preprint arXiv:2305.03514, 2023. find our card
  130. Zintgraf, L. M., Kanters, T. V., Roijers, D. M., Oliehoek, F. A., and Beau, P. Quality assessment of morl algorithms: A utility-based approach. 2015. URL https://api. semanticscholar.org/CorpusID:15373186. find
  131. Tozer, B., Mazzuchi, T., and Sarkani, S. Many-objective stochastic path finding using reinforcement learning. Expert Systems with Applications, 72:371–382, 2017. ISSN 0957-4174. doi: https://doi.org/10.1016/j.eswa.2016.10 .045. URL https://www.sciencedirect.com/ science/article/pii/S0957417416305863… — [doi:10.1016/j.eswa.2016.10](https://doi.org/10.1016/j.eswa.2016.10) ↗
  132. Wang, Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K. R., Wadden, D., MacMillan, K., Smith, N. A., Beltagy, find