Elamur Research⌕
☰
home / papers / 2024 / The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models

The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models

Paper · 2024 · Hannah Rose Kirk, Alexander Whitefield, Paul Rottger et al.

Summary

LLM alignment depends on human feedback, yet most existing preference datasets offer only binary comparisons, draw on narrow crowdworker pools and record little about who the raters are. PRISM addresses these gaps by linking each rater's profile to their actual feedback. Its 1,500 English-speaking participants, born in 75 countries, first completed a survey covering demographics, LLM familiarity, a self-written "constitution" (system string) and importance ratings for behaviour attributes. They then held live multi-turn conversations with 21 LLMs, scoring responses on a cardinal scale and adding fine-grained feedback. Many dialogues were deliberately steered toward value-laden or controversial topics, where disagreement between people and cultures is expected. Neutral, unguided chats serve as a baseline. The resource includes census-representative samples for the UK and US, and every rating carries a pseudonymous ID tied to a full participant profile. Three case studies show that who supplies alignment data shapes the results. This matters for personalised alignment, pluralistic aggregation and the calibration of LLM-as-judge setups.

Key points

  • Scale: 8,011 conversations, 27,172 interactions and 68,371 rated utterances, spread across 21 models (12 commercial API, 9 open-access).
  • Conversation design: In the first turn, four models answer the opening prompt. Later turns continue with the top-rated model.
  • Topic guidance: About 39% of dialogues are unguided, 30% controversy-guided and 31% values-guided.
  • Stated preferences: Participants broadly agree that factuality, fluency and helpfulness matter (mean importance 86–89). Opinions on whether value alignment matters are split, with a nearly normal distribution (mean 54, SD 26).
  • Collection and release: Data were gathered from November to December 2023, with participants paid £9/hour. Toxicity is low (0.06%), and the data are released unmoderated on GitHub and Hugging Face (HannahRoseKirk/prism-alignment).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Hannah Rose Kirk, Alexander Whitefield, Paul Rottger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, Bertie Vidgen and Scott A. Hale. 2024. The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models. arXiv:2404.16019 · The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2024).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2404.16019 · arXiv 2404.16019

@misc{kirk2024prism,
  title = {{The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models}},
  author = {Hannah Rose Kirk, Alexander Whitefield, Paul Rottger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, Bertie Vidgen and Scott A. Hale},
  year = {2024},
  doi = {10.48550/arxiv.2404.16019},
  eprint = {2404.16019},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.16019}
}

References

163 references, 37 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Iason Gabriel. Artificial Intelligence, Values and Alignment. Minds and Machines, 30(3):411–437, September 2020. ISSN 0924-6495, 1572-8641. doi: 10.1007/s11023-020-09539-2. — [doi:10.1007/s11023-020-09539-2](https://doi.org/10.1007/s11023-020-09539-2) find
  2. Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A. Hale. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nature Machine Intelligence, pages 1–10, April 2024. ISSN 2522-5839. doi: 10.1038/s42256-024-00820-y. URL https://www.nature. com/a… — [doi:10.1038/s42256-024-00820-y](https://doi.org/10.1038/s42256-024-00820-y) ↗
  3. Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging, October 2023. URL http://arxiv.org/abs/2310.11564. arXiv… ↗
  4. Xinyu Li, Zachary C. Lipton, and Liu Leqi. Personalized Language Modeling from Personalized Human Feedback, February 2024. URL http://arxiv.org/abs/2402.05133. arXiv:2402.05133 [cs]. — [arXiv:2402.05133](https://arxiv.org/abs/2402.05133) ↗
  5. Michiel A. Bakker, Martin J. Chadwick, Hannah R. Sheahan, Michael Henry Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matthew M. Botvinick, and Christopher Summerfield. Fine-tuning language models to find agreement among humans with diverse preferences… find
  6. Anthropic. Collective Constitutional AI: Aligning a Language Model with Public Input. Technical report, 2023. URL https://www.anthropic.com/news/collective-constitutional-ai-aligning-a-language-model-with-public-input. ↗
  7. Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, and Mengdi Wang. MaxMin-RLHF: Towards Equitable Alignment of Large Language Models with Diverse Human Preferences, February 2024. URL http://arxiv.org/abs/2402.08925. arXiv:2402.08925 [cs]. — [arXiv:2402.08925](https://arxiv.org/abs/2402.08925) ↗
  8. Dexun Li, Cong Zhang, Kuicai Dong, Derrick Goh Xin Deik, Ruiming Tang, and Yong Liu. Aligning Crowd Feedback via Distributional Preference Reward Modeling, February 2024. URL http://arxiv.org/abs/2402.09764. arXiv:2402.09764 [cs]. — [arXiv:2402.09764](https://arxiv.org/abs/2402.09764) ↗
  9. Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H. Holliday, Bob M. Jacobs, Nathan Lambert, Milan Mossé, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, Emanuel Tewolde, and William S. Zwicker. Social Choice for AI Alignment: Dealing with Diverse Human Feedback, April 2024. URL http://arxi… — [arXiv:2404.10271](https://arxiv.org/abs/2404.10271) find
  10. Hannah Kirk, Andrew Bean, Bertie Vidgen, Paul Rottger, and Scott Hale. The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical… find
  11. Nathan Lambert, Thomas Krendl Gilbert, and Tom Zick. The History and Risks of Reinforcement Learning and Human Feedback, November 2023. URL http://arxiv.org/abs/2310.13595. arXiv:2310.13595 [cs]. — [arXiv:2310.13595](https://arxiv.org/abs/2310.13595) ↗
  12. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-Grained Human Feedback Gives Better Rewards for Language Model Training, June 2023. URL http://arxiv.org/abs/2306.01693. arXiv:2306.01693 [cs]. ↗
  13. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar… find
  14. Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. On Releasing Annotator-Level Labels and Information in Datasets. In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 133–138, Punta Cana, Dominican Repub…
  15. Paul Rottger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 175– 190, Seat…
  16. Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A. Hale. The Empty Signifier Problem: Towards Clearer Paradigms for Operationalising "Alignment" in Large Language Models. In Socially Responsible Language Modelling Research (SoLaR). arXiv, November 2023. doi: 10.48550/arXiv.2310.02457. URL h… — [doi:10.48550/arxiv.2310.02457](https://doi.org/10.48550/arxiv.2310.02457) find
  17. Andreu Mas-Colell, Michael Dennis Whinston, Jerry R Green, et al. Microeconomic theory, volume 1. Oxford university press, New York, 1995. find
  18. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In Advances in Neural Information Processing Systems, volume 36, February 2024. URL https://proceedings.neurips.cc/p… ↗
  19. Banghua Zhu, Jiantao Jiao, and Michael I. Jordan. Principled Reinforcement Learning with Human Feedback from Pairwise or $K$-wise Comparisons, February 2024. URL http://arxiv.org/abs/2301.11270. arXiv:2301.11270 [cs, math, stat]. — [arXiv:2301.11270](https://arxiv.org/abs/2301.11270) ↗
  20. Banghua Zhu, Michael I. Jordan, and Jiantao Jiao. Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF, January 2024. URL http://arxiv.org/abs/2401.16335. arXiv:2401.16335 [cs, stat]. — [arXiv:2401.16335](https://arxiv.org/abs/2401.16335) ↗
  21. Alexey Turchin. AI Alignment Problem: "Human Values" Don’t Actually Exist. PhilArchive, 2019. URL https://philarchive.org/rec/TURAAP. ↗
  22. Brian D. Earp, Killian L. McLoughlin, Joshua T. Monrad, Margaret S. Clark, and Molly J. Crockett. How social relationships shape moral wrongness judgments. Nature Communications, 12(1):5776, October 2021. ISSN 2041-1723. doi: 10.1038/s41467-021-26067-4. URL https://www.nature.com/articles/s41467-021… — [doi:10.1038/s41467-021-26067-4](https://doi.org/10.1038/s41467-021-26067-4) ↗
  23. Michael F Mascolo, Allison DiBianca Fasoli, and David Greenway. A Relational Approach to Moral Development in Societies, Organizations and Individuals. Integral Review, 17(1), 2021. find
  24. Judith Butler, Ernesto Laclau, and Slavoj Žižek. Contingency, hegemony, universality: contemporary dialogues on the left. Phronesis. Verso, London, 2000. ISBN 978-1-85984-757-2 978-1-85984-278-2. OCLC: ocm44780799. find
  25. Mona Sloane. Controversies, contradiction, and “participation” in AI. Big Data & Society, 11(1): 20539517241235862, March 2024. ISSN 2053-9517. doi: 10.1177/20539517241235862. URL https://doi.org/10.1177/20539517241235862. Publisher: SAGE Publications Ltd. — [doi:10.1177/20539517241235862](https://doi.org/10.1177/20539517241235862) ↗
  26. Zeerak Talat, Hagen Blix, Josef Valvoda, Maya Indira Ganesh, Ryan Cotterell, and Adina Williams. On the machine learning of ethical judgments from natural language. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors, Proceedings of the 2022 Conference of the North A… find
  27. Lora Aroyo and Chris Welty. Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation. AI Magazine, 36(1):15–24, March 2015. ISSN 2371-9621. doi: 10.1609/aimag.v36i1.2564. URL https://ojs.aaai.org/index.php/aimagazine/article/view/2564. Number: 1. ↗
  28. Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Menell. Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF, December 2023. URL http://arxiv.org/abs/2312.08358. arXiv:2312.08358 [cs, stat]. — [arXiv:2312.08358](https://arxiv.org/abs/2312.08358) ↗
  29. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, volume 33, pages 3008–3021. Curran Associates, Inc., 2020. URL https:/… find
  30. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… find
  31. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training… — Training language models to follow instructions with human feedback
  32. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared… — Constitutional AI: Harmlessness from AI Feedback
  33. Abeba Birhane, William Isaac, Vinodkumar Prabhakaran, Mark Diaz, Madeleine Clare Elish, Iason Gabriel, and Shakir Mohamed. Power to the People? Opportunities and Challenges for Participatory AI. In Equity and Access in Algorithms, Mechanisms, and Optimization, EAAMO ’22, pages 1–8, New York, NY, USA… — [doi:10.1145/3551624.3555290](https://doi.org/10.1145/3551624.3555290) find
  34. Esin Durmus, Karina Nyugen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. Towards Measuring the Rep… — Towards Measuring the Representation of Subjective Global Opinions in Language Models
  35. Badr AlKhamissi, Muhammad ElNokrashy, Mai AlKhamissi, and Mona Diab. Investigating Cultural Alignment of Large Language Models, February 2024. URL http://arxiv.org/abs/2402.13231. arXiv:2402.13231 [cs]. ↗
  36. Michael J. Ryan, William Held, and Diyi Yang. Unintended Impacts of LLM Alignment on Global Representation, February 2024. URL http://arxiv.org/abs/2402.15018. arXiv:2402.15018 [cs]. ↗
  37. Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. A Roadmap to Pluralistic Alignment, February 2024. URL http://arxiv.org/abs/2402.05070. arXiv:2402.05070 [cs… ↗
  38. Iason Gabriel and Vafa Ghazavi. The Challenge of Value Alignment: from Fairer Algorithms to AI Safety, January 2021. URL http://arxiv.org/abs/2101.06060. arXiv:2101.06060 [cs]. — [arXiv:2101.06060](https://arxiv.org/abs/2101.06060) ↗
  39. Jonathan Stray. Aligning AI Optimization to Community Well-Being. International Journal of Community Well-Being, 3(4):443–463, December 2020. ISSN 2524-5309. doi: 10.1007/s42413-020-00086-3. URL https://doi.org/10.1007/s42413-020-00086-3. — [doi:10.1007/s42413-020-00086-3](https://doi.org/10.1007/s42413-020-00086-3) ↗
  40. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  41. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  42. Audrey G. Gift. Visual Analogue Scales: Measurement of Subjective Phenomena. Nursing Research, 38(5):286, October 1989. ISSN 0029-6562. URL https://journals.lww.com/nursingresearchonline/citation/1989/09000/visual_analogue_scales_ _measurement_of_subjective.6.aspx?casa_token=aO_mhu6sQyEAAAAA:y06v3L… ↗
  43. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench:… find
  44. Tristan Thrush, Kushal Tirumala, Anmol Gupta, Max Bartolo, Pedro Rodriguez, Tariq Kane, William Gaviria Rojas, Peter Mattson, Adina Williams, and Douwe Kiela. Dynatask: A Framework for Creating Dynamic AI Benchmark Tasks. In Valerio Basile, Zornitsa Kozareva, and Sanja Stajner, editors, Proceedings… find
  45. R. C. Aitken. Measurement of feelings using visual analogue scales. Proceedings of the Royal Society of Medicine, 62(10):989–993, October 1969. ISSN 0035-9157. URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1810824/. ↗
  46. Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A Long Way to Go: Investigating Length Correlations in RLHF, October 2023. URL http://arxiv.org/abs/2310.03716. arXiv:2310.03716 [cs]. ↗
  47. Ricardo J. G. B. Campello, Davoud Moulavi, and Joerg Sander. Density-Based Clustering Based on Hierarchical Density Estimates. In Jian Pei, Vincent S. Tseng, Longbing Cao, Hiroshi Motoda, and Guandong Xu, editors, Advances in Knowledge Discovery and Data Mining, Lecture Notes in Computer Science, pa… — [doi:10.1007/978-3-642-37456-2_14](https://doi.org/10.1007/978-3-642-37456-2_14) find
  48. Sahand Negahban, Sewoong Oh, and Devavrat Shah. Iterative ranking from pair-wise comparisons. In Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL https://papers.nips.cc/paper_files/paper/2012/hash/9adeb82fffb5444e81fa0ce8ad8afe7a-Abstract.html. ↗
  49. Gergei Bana, Wojciech Jamroga, David Naccache, and Peter Y. A. Ryan. Convergence Voting: From Pairwise Comparisons to Consensus, March 2021. URL http://arxiv.org/abs/2102.01995. arXiv:2102.01995 [cs]. — [arXiv:2102.01995](https://arxiv.org/abs/2102.01995) ↗
  50. Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The PageRank Citation Ranking: Bringing Order to the Web., November 1999. URL http://ilpubs.stanford.edu:8090/422/?doi=10.1.1.31.1768. Type: Techreport. ↗
  51. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36:46595– 46623,…
  52. Safiya Umoja Noble. Algorithms of oppression: how search engines reinforce racism. New York University Press, New York, 2018. ISBN 978-1-4798-4994-9 978-1-4798-3724-3. find
  53. Christopher M. Kelty. The Participant – A Century of Participation in Four Stories. The University of Chicago press, Chicago (Ill.) London, 2019. ISBN 978-0-226-66662-4 978-0-226-66676-1. find
  54. Caroline Criado-Perez. Invisible women: exposing data bias in a world designed for men. Chatto & Windus, London, 2019. ISBN 978-1-78474-172-3 978-1-78474-292-8. find
  55. Mona Sloane, Emanuel Moss, Olaitan Awomolo, and Laura Forlano. Participation Is not a Design Fix for Machine Learning. In Proceedings of the 2nd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, EAAMO ’22, pages 1–6, New York, NY, USA, October 2022. Association for Com… — [doi:10.1145/3551624.3555285](https://doi.org/10.1145/3551624.3555285) find
  56. Travis Greene, Copenhagen Business School, Galit Shmueli, National Tsing Hua University, Soumya Ray, and National Tsing Hua University. Taking the Person Seriously: Ethically Aware IS Research in the Era of Reinforcement Learning-Based Personalization. Journal of the Association for Information Syst… — [doi:10.17705/1jais.00800](https://doi.org/10.17705/1jais.00800) find
  57. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. WebGPT: Browser-assisted qu… — WebGPT: Browser-assisted question-answering with human feedback
  58. Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.01963, 2021. — [arXiv:2110.01963](https://arxiv.org/abs/2110.01963) find
  59. Ruha Benjamin. Race After Technology: Abolitionist Tools for the New Jim Code. John Wiley & Sons, July 2019. ISBN 978-1-5095-2643-7. find
  60. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. Language (Technology) is Power: A Critical Survey of “Bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454–5476, Online, July 2020. Association for Computational Linguist…
  61. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. Challenges and Strategies in Cross-Cultural NLP. In…
  62. Wenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai, Jen-tse Huang, Zhaopeng Tu, and Michael R. Lyu. Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models, February 2024. URL http://arxiv.org/abs/2310.12481. arXiv:2310.12481 [cs] version: 2. ↗
  63. Suchin Gururangan, Dallas Card, Sarah K. Dreier, Emily K. Gade, Leroy Z. Wang, Zeyu Wang, Luke Zettlemoyer, and Noah A. Smith. Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection, January 2022. URL http://arxiv.org/abs/2201.10474. arXiv:2201.10474 [cs]. ↗
  64. Josh Dzieza. Inside the AI Factory, June 2023. URL https://www.theverge.com/features/23764584/ai-artificial-intelligence-data-notation-labor-scale-surge-remotasks-openai-chatbots. ↗
  65. Shakir Mohamed, Marie-Therese Png, and William Isaac. Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence. Philosophy & Technology, 33(4):659–684, December 2020. ISSN 2210-5441. doi: 10.1007/s13347-020-00405-8. URL https://doi.org/10.1007/s13347-020-00405-8. — [doi:10.1007/s13347-020-00405-8](https://doi.org/10.1007/s13347-020-00405-8) ↗
  66. Massimo Airoldi. Machine habitus: toward a sociology of algorithms. Polity Press, Cambridge ; Medford, MA, 2022. ISBN 978-1-5095-4327-4 978-1-5095-4328-1. OCLC: on1247827618. find
  67. Zeerak Talat, Hagen Blix, Josef Valvoda, Maya Indira Ganesh, Ryan Cotterell, and Adina Williams. On the Machine Learning of Ethical Judgments from Natural Language. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Te… find
  68. Mark Díaz, Ian Kivlichan, Rachel Rosen, Dylan Baker, Razvan Amironesei, Vinodkumar Prabhakaran, and Emily Denton. CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation. In 2022 ACM Conference on Fairness, Accountability, and Transparency, FAc… — [doi:10.1145/3531146.3534647](https://doi.org/10.1145/3531146.3534647) find
  69. Emily M. Bender and Batya Friedman. Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Transactions of the Association for Computational Linguistics, 6:587–604, 2018. doi: 10.1162/tacl_a_00041. URL https://aclanthology.org/Q18-1041. Place: Cam… ↗
  70. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model Cards for Model Reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency - FAT* ’19, pages 220–229, 2019. doi: 10.… find
  71. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12): 86–92, December 2021. ISSN 15577317. doi: 10.1145/3458723. Publisher: Association for Computing Machinery. — [doi:10.1145/3458723](https://doi.org/10.1145/3458723) find
  72. Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations. Transactions of the Association for Computational Linguistics, 10:92–110, January 2022. ISSN 2307-387X. doi: 10.1162/tacl_a_00449. URL https://doi.… ↗
  73. Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. Jury Learning: Integrating Dissenting Voices into Machine Learning Models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, pages 1–… — [doi:10.1145/3491102.3502004](https://doi.org/10.1145/3491102.3502004) find
  74. Barbara Plank, Dirk Hovy, and Anders Søgaard. Learning part-of-speech taggers with inter-annotator agreement loss. In Shuly Wintner, Sharon Goldwater, and Stefan Riezler, editors, Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 742–7…
  75. Yixin Nie, Xiang Zhou, and Mohit Bansal. What Can We Learn from Collective Human Opinions on Natural Language Inference Data? In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131–914…
  76. Maximilian Wich, Christian Widmer, Gerhard Hagerer, and Georg Groh. Investigating Annotator Bias in Abusive Language Datasets. In Ruslan Mitkov and Galia Angelova, editors, Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1515–1525, He… find
  77. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistic…
  78. Nitesh Goyal, Ian D. Kivlichan, Rachel Rosen, and Lucy Vasserman. Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2):363:1–363:28, November 2022. doi: 10.1145/3555088. URL https://dl.acm.org/doi… — [doi:10.1145/3555088](https://doi.org/10.1145/3555088) ↗
  79. Lora Aroyo, Alex S. Taylor, Mark Diaz, Christopher M. Homan, Alicia Parrish, Greg Serapio-Garcia, Vinodkumar Prabhakaran, and Ding Wang. DICES Dataset: Diversity in Conversational AI Evaluation for Safety, June 2023. URL http://arxiv.org/abs/2306.11247. arXiv:2306.11247 [cs]. — [arXiv:2306.11247](https://arxiv.org/abs/2306.11247) ↗
  80. Joseph Henrich, Steven J. Heine, and Ara Norenzayan. Most people are not WEIRD. Nature, 466 (7302):29–29, July 2010. ISSN 1476-4687. doi: 10.1038/466029a. URL https://www.nature.com/articles/466029a. Number: 7302 Publisher: Nature Publishing Group. — [doi:10.1038/466029a](https://doi.org/10.1038/466029a) ↗
  81. Dante A. Urbina and Alberto Ruiz-Villaverde. A Critical Review of Homo Economicus from Five Approaches. The American Journal of Economics and Sociology, 78(1):63–93, 2019. ISSN 1536-7150. doi: 10.1111/ajes.12258. URL https://onlinelibrary.wiley.com/doi/abs/10.1111/ajes.12258. _eprint: https://online… — [doi:10.1111/ajes.12258](https://doi.org/10.1111/ajes.12258) ↗
  82. Coren Apicella, Ara Norenzayan, and Joseph Henrich. Beyond WEIRD: A review of the last decade and a look ahead to the global laboratory of the future. Evolution and Human Behavior, 41(5):319– 329, September 2020. ISSN 1090-5138. doi: 10.1016/j.evolhumbehav.2020.07.015. URL https://www.sciencedirect… — [doi:10.1016/j.evolhumbehav.2020.07.015](https://doi.org/10.1016/j.evolhumbehav.2020.07.015) ↗
  83. Andrew Ng and Stuart J. Russell. Algorithms for Inverse Reinforcement Learning. 2000. find
  84. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/hash/d5e… ↗
  85. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-Tuning Language Models from Human Preferences. September 2019. URL http://arxiv.org/abs/1909.08593v2. ↗
  86. Shachar Mirkin and Jean-Luc Meunier. Personalized machine translation: Predicting translational preferences. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2019–2025, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 1… find
  87. Khanh Nguyen, Hal Daumé III, and Jordan Boyd-Graber. Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1464–1474, Copenhagen, Denmark, September 2017. Associatio… find
  88. Julia Kreutzer, Artem Sokolov, and Stefan Riezler. Bandit Structured Prediction for Neural Sequence-to-Sequence Learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1503–1513, Vancouver, Canada, July 2017. Association for… find
  89. Marilyn A Walker. An application of reinforcement learning to dialogue strategy selection in a spoken dialogue system for email. Journal of Artificial Intelligence Research, 12:387–416, 2000. find
  90. Jost Schatzmann, Karl Weilhammer, Matt Stuttle, and Steve Young. A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. The knowledge engineering review, 21(2):97–126, 2006. find
  91. Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Maria Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, and Steve J. Young. Continuously learning neural dialogue management. abs/1606.02689, 2016. URL http://arxiv.org/abs/1606.02689. ↗
  92. Jiwei Li, Alexander H. Miller, Sumit Chopra, Marc’Aurelio Ranzato, and Jason Weston. Dialogue learning with human-in-the-loop. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https://op… find
  93. Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog, July 2019. URL http://arxiv.org/abs/1907.00456. arXiv:1907.00456 [cs, stat]. — [arXiv:1907.00456](https://arxiv.org/abs/1907.00456) ↗
  94. Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Human-centric dialog training via offline reinforcement learning. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pa… find
  95. Amelia Glaese, Nat McAleese, Maja Tr˛ebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug F… — Improving alignment of dialogue agents via targeted human judgements
  96. Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training Language Models with Language Feedback, November 2022. URL http://arxiv.org/abs/2204.14146. arXiv:2204.14146 [cs]. ↗
  97. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, a… find
  98. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms, August 2017. URL http://arxiv.org/abs/1707.06347. arXiv:1707.06347 [cs]. ↗
  99. Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229–256, May 1992. ISSN 1573-0565. doi: 10.1007/BF00992696. URL https://doi.org/10.1007/BF00992696. ↗
  100. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs, February 2024. URL http://arxiv.org/abs/2402.14740. arXiv:2402.14740 [cs] ve… ↗
  101. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less Is More for Alignment, May 2023. URL http://arxiv.org/abs/2305.11206. arXiv:2305.11206 [cs]. ↗
  102. Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, and Nat McAleese. Teaching language models to support answers with verified quotes, March 2022. URL http://arxiv.org/abs/2203.11147. arX… ↗
  103. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzho… — LaMDA: Language Models for Dialog Applications
  104. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, L. J. Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. RewardBench: Evaluating Reward Models for Language Modeling, March 2024. URL http://arxiv.org/abs/2403.13787. ar… ↗
  105. Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: Model Alignment as Prospect Theoretic Optimization, February 2024. URL http://arxiv.org/abs/2402.01306. arXiv:2402.01306 [cs]. ↗
  106. StanfordNLP. Stanford Human Preferences Dataset, September 2023. URL https://huggingface.co/datasets/stanfordnlp/SHP. ↗
  107. Nathan Lambert, Lewis Tunstall, Nazneen Rajani, and Tristan Thrush. HuggingFace H4 Stack Exchange Preference Dataset, 2023. URL https://huggingface.co/datasets/HuggingFaceH4/stack-exchange-preferences. ↗
  108. William Agnew, A. Stevie Bergman, Jennifer Chien, Mark Díaz, Seliem El-Sayed, Jaylen Pittman, Shakir Mohamed, and Kevin R. McKee. The illusion of artificial inclusion, February 2024. URL http://arxiv.org/abs/2401.08572. arXiv:2401.08572 [cs]. — [arXiv:2401.08572](https://arxiv.org/abs/2401.08572) ↗
  109. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback, January 2024. URL http://arxiv.org/abs/2305.14387. arXiv:2305.14387 [cs]. ↗
  110. Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Jiexin Wang, Huimin Chen, Bowen Sun, Ruobing Xie, Jie Zhou, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment, February 2024. URL http://arxiv.org/abs/2402.19085. arXiv:2402.19085… ↗
  111. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset, March 2024. URL http://arxiv.org/abs/2309.11998. arXiv:… ↗
  112. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. (InThe)WildChat: 570K ChatGPT Interaction Logs In The Wild. October 2023. URL https://openreview.net/forum?id=Bl8u7ZRlbM. ↗
  113. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. OpenA… — OpenAssistant Conversations — Democratizing Large Language Model Alignment
  114. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan… — Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
  115. Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergün, Ife… — Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning
  116. The Alan Turing Institute and The Ada Lovelace Institute. How do people feel about AI? A nationally representative survey of public attitudes to artificial intelligence in Britain. Technical report, 2023. URL https://attitudestoai.uk/assets/documents/Ada-Lovelace-Institute-The-Alan-Turing-Institute-… ↗
  117. Jimin Mun, Liwei Jiang, Jenny Liang, Inyoung Cheong, Nicole DeCario, Yejin Choi, Tadayoshi Kohno, and Maarten Sap. Particip-AI: A Democratic Surveying Framework for Anticipating Future AI Use Cases, Harms and Benefits, March 2024. URL http://arxiv.org/abs/2403.14791. arXiv:2403.14791 [cs]. — [arXiv:2403.14791](https://arxiv.org/abs/2403.14791) ↗
  118. Samuel Chang, Estelle Ciesla, Michael Finch, James Fishkin, Lodewijk Gelauff, Ashish Goel, Ricky Hernandez Marquez, Shoaib Mohammed, and Alice Siu. Meta Community Forum: Results Analysis. Technical report, Deliberative Democracy Lab, Stanford University, April 2024. find
  119. Stevie Bergman, Nahema Marchal, John Mellor, Shakir Mohamed, Iason Gabriel, and William Isaac. STELA: a community-centred approach to norm elicitation for AI alignment. Scientific Reports, 14(1): 6616, March 2024. ISSN 2045-2322. doi: 10.1038/s41598-024-56648-4. URL https://www.nature. com/articles/… — [doi:10.1038/s41598-024-56648-4](https://doi.org/10.1038/s41598-024-56648-4) ↗
  120. R. Silberzahn, E. L. Uhlmann, D. P. Martin, P. Anselmi, F. Aust, E. Awtrey, Š. Bahník, F. Bai, C. Bannard, E. Bonnier, R. Carlsson, F. Cheung, G. Christensen, R. Clay, M. A. Craig, A. Dalla Rosa, L. Dam, M. H. Evans, I. Flores Cervantes, N. Fong, M. Gamez-Djokic, A. Glenz, S. Gordon-McKeon, T. J. He… — [doi:10.1177/2515245917747646](https://doi.org/10.1177/2515245917747646) find
  121. John Tasioulas. Artificial Intelligence, Humanistic Ethics. Daedalus, 151(2):232–243, May 2022. ISSN 0011-5266. doi: 10.1162/daed_a_01912. URL https://doi.org/10.1162/daed_a_01912. — [doi:10.1162/daed_a_01912](https://doi.org/10.1162/daed_a_01912) ↗
  122. Diane Proudfoot. Anthropomorphism and AI: Turing’s much misunderstood imitation game. Artificial Intelligence, 175(5):950–957, April 2011. ISSN 0004-3702. doi: 10.1016/j.artint.2011.01.006. URL https://www.sciencedirect.com/science/article/pii/S000437021100018X. — [doi:10.1016/j.artint.2011.01.006](https://doi.org/10.1016/j.artint.2011.01.006) ↗
  123. David Watson. The Rhetoric and Reality of Anthropomorphism in Artificial Intelligence. Minds and Machines, 29(3):417–440, September 2019. ISSN 1572-8641. doi: 10.1007/s11023-019-09506-6. URL https://doi.org/10.1007/s11023-019-09506-6. — [doi:10.1007/s11023-019-09506-6](https://doi.org/10.1007/s11023-019-09506-6) ↗
  124. Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sea… — [doi:10.1145/3531146.3533088](https://doi.org/10.1145/3531146.3533088) find
  125. Gavin Abercrombie, Amanda Cercas Curry, Tanvi Dinkar, Verena Rieser, and Zeerak Talat. Mirages. On Anthropomorphism in Dialogue Systems. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4776–4790, Sin… find
  126. Myra Cheng, Kristina Gligoric, Tiziano Piccardi, and Dan Jurafsky. AnthroScore: A Computational Linguistic Measure of Anthropomorphism, February 2024. URL http://arxiv.org/abs/2402.02056. arXiv:2402.02056 [cs]. ↗
  127. Sian Gooding and Hassan Mansoor. The Impact of Preference Agreement in Reinforcement Learning from Human Feedback: A Case Study in Summarization, November 2023. URL http://arxiv.org/abs/2311.04919. arXiv:2311.04919 [cs]. — [arXiv:2311.04919](https://arxiv.org/abs/2311.04919) ↗
  128. Catherine D’Ignazio and Lauren F. Klein. Data Feminism. The MIT Press, March 2020. ISBN 978-0-262-35852-1. doi: 10.7551/mitpress/11805.001.0001. URL https://direct.mit.edu/books/book/4660/Data-Feminism. — [doi:10.7551/mitpress/11805.001.0001](https://doi.org/10.7551/mitpress/11805.001.0001) ↗
  129. Abeba Birhane. Algorithmic injustice: a relational ethics approach. Patterns, 2(2):100205, February 2021. ISSN 26663899. doi: 10.1016/j.patter.2021.100205. URL https://linkinghub.elsevier.com/retrieve/pii/S2666389921000155. — [doi:10.1016/j.patter.2021.100205](https://doi.org/10.1016/j.patter.2021.100205) ↗
  130. Yijiang River Dong, Tiancheng Hu, and Nigel Collier. Can LLM be a Personalized Judge?, June 2024. URL http://arxiv.org/abs/2406.11657. arXiv:2406.11657 [cs]. ↗
  131. Billy Perrigo. Inside OpenAI’s Plan to Make AI More ’Democratic’, February 2024. URL https://time.com/6684266/openai-democracy-artificial-intelligence/. ↗
  132. Fabio Crameri, Grace E. Shephard, and Philip J. Heron. The misuse of colour in science communication. Nature Communications, 11(1):5444, October 2020. ISSN 2041-1723. doi: 10.1038/s41467-020-19160-7. URL https://www.nature.com/articles/s41467-020-19160-7. Number: 1 Publisher: Nature Publishing Group… — [doi:10.1038/s41467-020-19160-7](https://doi.org/10.1038/s41467-020-19160-7) ↗
  133. Nenad Tomasev, Kevin R. McKee, Jackie Kay, and Shakir Mohamed. Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communities. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’21, pages 254–265, New York, NY, USA, July 2021. Associat… — [doi:10.1145/3461702.3462540](https://doi.org/10.1145/3461702.3462540) find
  134. Tom Hosking, Phil Blunsom, and Max Bartolo. Human Feedback is not Gold Standard, January 2024. URL http://arxiv.org/abs/2309.16349. arXiv:2309.16349 [cs]. — [arXiv:2309.16349](https://arxiv.org/abs/2309.16349) ↗
  135. Amos Tversky and Itamar Simonson. Context-Dependent Preferences. Management Science, 39(10):1179– 1189, October 1993. ISSN 0025-1909. doi: 10.1287/mnsc.39.10.1179. URL https://pubsonline.informs.org/doi/abs/10.1287/mnsc.39.10.1179. Publisher: INFORMS. — [doi:10.1287/mnsc.39.10.1179](https://doi.org/10.1287/mnsc.39.10.1179) ↗
  136. Boaz Shmueli, Jan Fell, Soumya Ray, and Lun-Wei Ku. Beyond fair pay: Ethical implications of NLP crowdsourcing. In Association for Computational Linguistics (ACL), pages 3758–3769, April 2021. URL https://arxiv.org/abs/2104.10097v1. arXiv: 2104.10097 Publisher: tex.arxivid: 2104.10097. ↗
  137. Lisa Posch, Arnim Bleier, Fabian Flöck, Clemens M. Lechner, Katharina Kinder-Kurlanda, Denis Helic, and Markus Strohmaier. Characterizing the Global Crowd Workforce: A Cross-Country Comparison of Crowdworker Demographics. Human Computation, 9(1), August 2022. ISSN 2330-8001. doi: 10.15346/hc.v9i1.10… — [doi:10.15346/hc.v9i1.106](https://doi.org/10.15346/hc.v9i1.106) find
  138. Derek A. Albert and Daniel Smilek. Comparing attentional disengagement between Prolific and MTurk samples. Scientific Reports, 13(1):20574, November 2023. ISSN 2045-2322. doi: 10.1038/s41598-023-46048-5. URL https://www.nature.com/articles/s41598-023-46048-5. Publisher: Nature Publishing Group. — [doi:10.1038/s41598-023-46048-5](https://doi.org/10.1038/s41598-023-46048-5) ↗
  139. Gemini Team. Gemini: A Family of Highly Capable Multimodal Models, December 2023. URL http://arxiv.org/abs/2312.11805. arXiv:2312.11805 [cs]. ↗
  140. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  141. Anthropic. Introducing the next generation of Claude, April 2024. URL https://www.anthropic.com/news/claude-3-family. ↗
  142. Cohere. Command R, April 2024. URL https://docs.cohere.com/docs/command-r. ↗
  143. MetaAI. Introducing Meta Llama 3: The most capable openly available LLM to date, April 2024. URL https://ai.meta.com/blog/meta-llama-3/. ↗
  144. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose Opinions Do Language Models Reflect?, March 2023. URL http://arxiv.org/abs/2303.17548. arXiv:2303.17548 [cs]. ↗
  145. Fredrik Barth. Ethnic Groups and Boundaries: The Social Organization of Culture Difference. Waveland Press, March 1998. ISBN 978-1-4786-0795-3. Google-Books-ID: QaAQAAAAQBAJ. find
  146. Katarzyna Hamer, Sam McFarland, Barbara Czarnecka, Agnieszka Golińska, Liliana Manrique Cadena, Magdalena Łużniak Piecha, and Tomasz Jułkowski. What Is an “Ethnic Group” in Ordinary People’s Eyes? Different Ways of Understanding It Among American, British, Mexican, and Polish Respondents. Cross-Cu… — [doi:10.1177/1069397118816939](https://doi.org/10.1177/1069397118816939) find
  147. Karen L. Suyemoto, Micaela Curley, and Shruti Mukkamala. What Do We Mean by “Ethnicity” and “Race”? A Consensual Qualitative Research Investigation of Colloquial Understandings. Genealogy, 4 (3):81, September 2020. ISSN 2313-5778. doi: 10.3390/genealogy4030081. URL https://www.mdpi. com/2313-5778/4/… — [doi:10.3390/genealogy4030081](https://doi.org/10.3390/genealogy4030081) ↗
  148. Laurence R. Iannaccone. Introduction to the Economics of Religion. Journal of Economic Literature, 36 (3):1465–1495, 1998. ISSN 0022-0515. URL https://www.jstor.org/stable/2564806. Publisher: American Economic Association. ↗
  149. Gilat Levy and Ronny Razin. Religious Beliefs, Religious Participation, and Cooperation. American Economic Journal: Microeconomics, 4(3):121–151, August 2012. ISSN 1945-7669, 1945-7685. doi: 10.1257/mic.4.3.121. URL https://pubs.aeaweb.org/doi/10.1257/mic.4.3.121. — [doi:10.1257/mic.4.3.121](https://doi.org/10.1257/mic.4.3.121) ↗
  150. Ellen Dingemans and Erik Van Ingen. Does Religion Breed Trust? A Cross-National Study of the Effects of Religious Involvement, Religious Faith, and Religious Context on Social Trust. Journal for the Scientific Study of Religion, 54(4):739–755, 2015. ISSN 0021-8294. URL https://www.jstor.org/stable/2… ↗
  151. Hansong Zhang, Joshua N. Hook, Jennifer E. Farrell, David K. Mosher, Laura E. Captari, Steven P. Coomes, Daryl R. Van Tongeren, and Don E. Davis. Exploring Social Belonging and Meaning in Religious Groups. Journal of Psychology and Theology, 47(1):3–19, March 2019. ISSN 0091-6471. doi: 10.1177/00916… — [doi:10.1177/0091647118806345](https://doi.org/10.1177/0091647118806345) find
  152. Vassilis Saroglou, Magali Clobert, Adam B. Cohen, Kathryn A. Johnson, Kevin L. Ladd, Matthieu Van Pachterbeke, Lucia Adamovova, Joanna Blogowska, Pierre-Yves Brandt, Cem Safak Çukur, Kwang-Kuo Hwang, Anna Miglietta, Frosso Motti-Stefanidi, Antonio Muñoz-García, Sebastian Murken, Nicolas Roussiau, an… — [doi:10.1177/0022022120946488](https://doi.org/10.1177/0022022120946488) find
  153. Prolific. Representative samples, February 2024. URL https://researcher-help.prolific.com/hc/en-gb/articles/360019236753-Representative-samples. ↗
  154. Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, August 2019. URL http://arxiv.org/abs/1908.10084. arXiv:1908.10084 [cs]. ↗
  155. Leland McInnes, John Healy, and James Melville. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction, September 2020. URL http://arxiv.org/abs/1802.03426. arXiv:1802.03426 [cs, stat]. ↗
  156. Ashkan Kazemi, Kiran Garimella, Gautam Kishore Shahi, Devin Gaffney, and Scott A. Hale. Research note: Tiplines to uncover misinformation on encrypted platforms: A case study of the 2019 Indian general election on WhatsApp. Harvard Kennedy School Misinformation Review, January 2022. doi: 10.37016/mr… — [doi:10.37016/mr-2020-91](https://doi.org/10.37016/mr-2020-91) find
  157. Scott A. Hale. meedan/temporal_clustering, March 2022. URL https://github.com/meedan/temporal_clustering/tree/main. ↗
  158. Scott D. Emerson, Martin Guhn, and Anne M. Gadermann. Measurement invariance of the Satisfaction with Life Scale: reviewing three decades of research. Quality of Life Research, 26(9):2251–2264, September 2017. ISSN 1573-2649. doi: 10.1007/s11136-017-1552-2. URL https://doi.org/10.1007/s11136-017-155… — [doi:10.1007/s11136-017-1552-2](https://doi.org/10.1007/s11136-017-1552-2) ↗
  159. John E. Roemer. Theories of distributive justice. Harvard Univ. Press, Cambridge, Mass., 1. harvard univ. press paperback ed edition, 1998. ISBN 978-0-674-87920-1 978-0-674-87919-5. find
  160. Jeremy Bentham. An Introduction to the Principles of Morals and Legislation. In J. H. Burns and H. L. A. Hart, editors, The Collected Works of Jeremy Bentham: An Introduction to the Principles of Morals and Legislation. Oxford University Press, January 1789. ISBN 978-0-19-820516-6. doi: 10.1093/oseo… — [doi:10.1093/oseo/instance.00077240](https://doi.org/10.1093/oseo/instance.00077240) find
  161. Marc Lanctot, Kate Larson, Yoram Bachrach, Luke Marris, Zun Li, Avishkar Bhoopchand, Thomas Anthony, Brian Tanner, and Anna Koop. Evaluating Agents using Social Choice Theory, December 2023. URL http://arxiv.org/abs/2312.03121. arXiv:2312.03121 [cs] version: 2. — [arXiv:2312.03121](https://arxiv.org/abs/2312.03121) ↗
  162. Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, and Marzieh Fadaee. Elo Uncovered: Robustness and Best Practices in Language Model Evaluation, November 2023. URL http://arxiv.org/abs/2311.17295. arXiv:2311.17295 [cs]. ↗
  163. Sahand Negahban, Sewoong Oh, and Devavrat Shah. Rank Centrality: Ranking from Pairwise Comparisons. Operations Research, 65(1):266–287, 2017. ISSN 0030-364X. URL https://www.jstor.org/stable/26153541. Publisher: INFORMS. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kirk-2024-prism-alignment-dataset-participatory-representative/