Elamur Research⌕
☰
home / papers / 2022 / The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems

The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems

Paper · 2022 · Caleb Ziems, Jane A. Yu, Yi-Chia Wang et al.

Summary

Open-domain chatbots can produce replies that seem insensitive, harmful, or morally incoherent, which damages user trust. Moral judgments vary across people and cultures, so the authors do not try to settle what is right. Instead, they build a resource for describing the moral assumptions behind chatbot responses.

The MORAL INTEGRITY CORPUS (MIC) pairs opinion questions taken from Reddit with replies from leading social chatbots such as BlenderBot. US-based MTurk annotators explain each reply with free-text Rules of Thumb (RoTs), extending the SOCIAL-CHEM-101 framework from narratives to dialogue. Each RoT is tagged with structured attributes, including alignment with the reply, global consensus, violation severity and moral foundations. Annotators also write a revised reply.

The authors benchmark attribute classification and train generative MORAL TRANSFORMERS to write RoTs for unseen interactions. The best T-5 model comes close to human quality but still often produces irrelevant rules. MIC can therefore serve as a benchmark and as training data for safety classifiers, controllable generation and RL penalty models.

Key points

  • MIC includes 38k prompt-reply pairs along with 99k unique RoTs and 114k structured annotation sets, spanning 9 moral and social attributes.
  • Filtering keeps over 90% of the data both reasonable and normatively "interesting." Annotators' political and moral foundations are recorded to capture ideological variation.
  • The best T-5 generator reaches ROUGE-L of 53. It nearly matches humans on well-formedness, relevance and fluency, yet about 28% of its RoTs are irrelevant.
  • In attribute classification, ALBERT slightly outperforms BERT:
    • severity: r = 0.59
    • consensus: r = 0.44
    • alignment: F1 = 76.0
  • Annotator demographics are US- and MTurk-centric and the prompts come from Reddit, which limits generalization. Access to the data (GT-SALT/mic on GitHub) requires signing a Data Use Agreement.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgia Institute of TechnologyAtlanta · United States69 works · 22,788 citations

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSocial Sciences · 744 worksSafety Research · 221 worksNeuroscience · 95 worksCognitive Neuroscience · 95 works

🧭 Topics 3Research subjects and languages the work is about.Evaluation & benchmarks · 685 worksDialogue systems · 401 worksBias, fairness & safety · 209 works

Cite

Cite as Caleb Ziems, Jane A. Yu, Yi-Chia Wang, Alon Y. Halevy and Diyi Yang. 2022. The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-long.261

@inproceedings{ziems2022moral,
  title = {{The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems}},
  author = {Caleb Ziems, Jane A. Yu, Yi-Chia Wang, Alon Y. Halevy and Diyi Yang},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.261},
  url = {https://doi.org/10.18653/v1/2022.acl-long.261}
}

References

101 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gavin Abercrombie, Amanda Cercas Curry, Mugdha Pandya, and Verena Rieser. 2021. Alexa, Google, Siri: What are your pronouns? gender and anthropomorphism in the design and perception of conversational assistants. In Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing, pages… find
  2. Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. ArXiv preprint, abs/2001.09977.
  3. Fahad Alaieri and André Vellino. 2016. Ethical decision making in robots: Autonomy, trust and responsibility. In International conference on social robotics, pages 159–168. Springer. find
  4. Rodolfo Antunes, Marcio Gomes, Clauter Gatti, Mariangela Vanzin, et al. 2020. Conversational agents in business: A systematic literature review and future research directions. Computer Science Review, 36:100239. find
  5. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623. find
  6. Cristina Bicchieri. 2005. The grammar of society: The nature and dynamics of social norms. Cambridge University Press. find
  7. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large scale autoregressive language modeling with mesh-tensorflow. find
  8. Petter Bae Brandtzaeg and Asbjørn Følstad. 2017. Why people use chatbots. In International conference on internet science, pages 377–392. Springer. find
  9. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,…
  10. Amanda Cercas Curry and Verena Rieser. 2018. \#MeToo Alexa: How conversational systems respond to sexual harassment. In Proceedings of the Second ACL Workshop on Ethics in Natural Language Processing, pages 7–14, New Orleans, Louisiana, USA. Association for Computational Linguistics. find
  11. Ashley Amaya, Ruben Bach, Florian Keusch, and Frauke Kreuter. 2021. New data sources in social science research: things to know before working with reddit data. Social science computer review, 39(5):943–960. find
  12. Veena Chattaraman, Wi-Suk Kwon, Juan E Gilbert, and Kassandra Ross. 2019. Should ai-based, conversational digital assistants employ social-or task-oriented interaction style? a task-competency and reciprocity perspective for older adults. Computers in Human Behavior, 90:315–330. find
  13. Simone Balloccu, Ehud Reiter, Matteo G Collu, Federico Sanna, Manuela Sanguinetti, and Maurizio Atzori. 2021. Unaddressed challenges in persuasive dieting chatbots. In Adjunct Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization, pages 392–395. find
  14. Kimberly E Culley and Poornima Madhavan. 2013. A note of caution regarding anthropomorphism in hci agents. Computers in Human Behavior, 29(3):577– 579. find
  15. Rodrigo Bavaresco, Diórgenes Silveira, Eduardo Reis, Jorge Barbosa, Rodrigo Righi, Cristiano Costa, Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020. Plug and play language models: A simple approach to controlled text generatio…
  16. Morteza Dehghani, Emmett Tomai, Kenneth D Forbus, and Matthew Klenk. 2008. An integrated reasoning approach to moral decision-making. In AAAI, pages 1280–1286. find
  17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  18. Djellel Eddine Difallah, Elena Filatova, and Panos Ipeirotis. 2018. Demographics and dynamics of mechanical turk workers. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018, Marina Del Rey, CA, USA, February 5-9, 2018, pages 135–143. ACM. find
  19. Emily Dinan, Gavin Abercrombie, A Stevie Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, and Verena Rieser. 2021. Anticipating safety issues in e2e conversational ai: Framework and tooling. ArXiv preprint, abs/2107.03451. — [arXiv:2107.03451](https://arxiv.org/abs/2107.03451) find
  20. Denis Emelin, Ronan Le Bras, Jena D. Hwang, Maxwell Forbes, and Yejin Choi. 2021. Moral stories: Situated reasoning about norms, intents, actions, and their consequences. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 698–718, Online and Punta Cana,… find
  21. Jessica Ficler and Yoav Goldberg. 2017. Controlling linguistic style aspects in neural language generation. In Proceedings of the Workshop on Stylistic Variation, pages 94–104, Copenhagen, Denmark. Association for Computational Linguistics.
  22. Fionn Delahunty. 2018. Reddit QA Corpus. https://github.com/FionnD/ Reddit-QA-Corpus. Online; accessed XXX. ↗
  23. Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chemistry 101: Learning to reason about social and moral norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 653–670, Online. Association for Computatio… find
  24. Jianfeng Gao, Michel Galley, and Lihong Li. 2018. Neural approaches to conversational AI. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SI-GIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, pages 1371–1374. ACM. find
  25. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2021. The pile: An 800gb dataset of diverse text for language modeling. ArXiv preprint, abs/2101.00027. find
  26. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online. Association for Computational Linguisti…
  27. Bernard Gert and Joshua Gert. 2002. The definition of morality. find
  28. Joseph K Goodman, Cynthia E Cryder, and Amar Cheema. 2013. Data collection in a flat world: The strengths and weaknesses of mechanical turk samples. Journal of Behavioral Decision Making, 26(3):213–224. find
  29. Jesse Graham, Jonathan Haidt, Sena Koleva, Matt Motyl, Ravi Iyer, Sean P Wojcik, and Peter H Ditto. 2013. Moral foundations theory: The pragmatic validity of moral pluralism. In Advances in experimental social psychology, volume 47, pages 55–130. Elsevier. find
  30. Jesse Graham, Jonathan Haidt, and Brian A Nosek. 2008. The moral foundations questionnaire. Moral-Foundations. org. find
  31. Jesse Graham, Jonathan Haidt, and Brian A Nosek. 2009. Liberals and conservatives rely on different sets of moral foundations. Journal of personality and social psychology, 96(5):1029. find
  32. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill. find
  33. Joshua Grossman, Zhiyuan Lin, Hao Sheng, Johnny Tian-Zheng Wei, Joseph J Williams, and Sharad Goel. 2019. Mathbot: Transforming online resources for learning math into conversational interactions. find
  34. AAAI 2019 Story-Enabled Intelligence. find
  35. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. A… find
  36. Jonathan Haidt. 2012. The righteous mind: Why good people are divided by politics and religion. Vintage. find
  37. Jonathan Haidt and Jesse Graham. 2007. When morality opposes justice: Conservatives have moral intuitions that liberals may not recognize. Social Justice Research, 20(1):98–116. find
  38. Jonathan Haidt, Silvia Helena Koller, and Maria G Dias. 1993. Affect, culture, and morality, or is it wrong to eat your dog? Journal of personality and social psychology, 65(4):613. find
  39. Sven Laumer, Christian Maier, and Fabian Tobias Gubler. 2019. Chatbot acceptance in healthcare: Explaining user adoption of conversational agents for disease diagnosis. find
  40. Richard Mervyn Hare, Richard Mervyn Hare, Richard Mervyn Hare Hare, and Richard M Hare. 1981. Moral thinking: Its levels, method, and point. Oxford: Clarendon Press; New York: Oxford University Press. find
  41. Peter Lee. Learning from tay’s introduction. find
  42. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2020. Aligning ai with shared human values. ArXiv preprint, abs/2008.02275.
  43. Wilhelm Hofmann, Daniel C Wisneski, Mark J Brandt, and Linda J Skitka. 2014. Morality in everyday life. Science, 345(6202):1340–1343. find
  44. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. find
  45. Peng Hu, Yaobin Lu, et al. 2021. Dual humanness and trust in conversational ai: A person-centered approach. Computers in Human Behavior, 119:106727. find
  46. Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. 2020. Challenges in building intelligent open-domain dialog systems. ACM Transactions on Information Systems (TOIS), 38(3):1–32. find
  47. Ravi Iyer, Stephen J Read, and Jane Correia. 2010. Functional justice: Productivity and well-being goals define fairness. Available at SSRN 1691969. find
  48. Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Maxwell Forbes, Jon Borchardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi. 2021. Delphi: Towards machine ethics and norms. ArXiv preprint, abs/2110.07574. find
  49. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. ArXiv preprint, abs/1909.05858.
  50. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRevi… find
  51. Liliana Laranjo, Adam G Dunn, Huong Ly Tong, Ahmet Baki Kocaballi, Jessica Chen, Rabia Bashir, Didi Surian, Blanca Gallego, Farah Magrabi, Annie YS Lau, et al. 2018. Conversational agents in healthcare: a systematic review. Journal of the American Medical Informatics Association, 25(9):1248–1258. find
  52. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the…
  53. Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1192– 1202, Austin, Texas. Association for Computational Ling…
  54. Q. Vera Liao, Muhammed Mas-ud Hussain, Praveen Chandar, Matthew Davis, Yasaman Khazaeni, Marco Patricio Crasso, Dakuo Wang, Michael J. Muller, N. Sadat Shami, and Werner Geyer. 2018. All work and no play? In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI 2018, Mont… find
  55. Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 150–157.
  56. Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, and Soroush Vosoughi. 2021a. Mitigating political bias in language models through reinforced calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14857–14866. find
  57. Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. 2021b. Towards emotional support dialog systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural La…
  58. Nicholas Lourie, Ronan Le Bras, and Yejin Choi. 2021. Scruples: A corpus of community ethical judgments on 32, 000 real-life anecdotes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13470–13479. find
  59. Alexandra Sasha Luccioni and Joseph D Viviano. 2021. What’s in the box? a preliminary analysis of undesirable content in the common crawl corpus. ArXiv preprint, abs/2105.02732. — [arXiv:2105.02732](https://arxiv.org/abs/2105.02732) find
  60. Ewa Luger and Abigail Sellen. 2016. "like having a really bad pa": The gulf between user expectation and experience of conversational agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, San Jose, CA, USA, May 7-12, 2016, pages 5286–5297. ACM. find
  61. Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob N. Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. 2019. A survey of reinforcement learning informed by natural language. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Int… find
  62. Roger C Mayer, James H Davis, and F David Schoorman. 1995. An integrative model of organizational trust. Academy of management review, 20(3):709– 734. find
  63. D Harrison McKnight, Vivek Choudhury, and Charles Kacmar. 2002. Developing and validating trust measures for e-commerce: An integrative typology. Information systems research, 13(3):334–359. find
  64. Peter Meindl, Ravi Iyer, and Jesse Graham. 2019. Distributive justice beliefs are guided by whether people think the ultimate goal of society is well-being or power. Basic and applied social psychology, 41(6):359–385. find
  65. György Molnár and Zoltán Szüts. 2018. The role of chatbots in formal education. In 2018 IEEE 16th International Symposium on Intelligent Systems and Informatics (SISY), pages 000197–000202. IEEE. find
  66. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Associati… find
  67. Generation, pages 374–383, Dublin, Ireland. Association for Computational Linguistics. find
  68. Shrimai Prabhumoye, Brendon Boldt, Ruslan Salakhutdinov, and Alan W Black. 2021. Case study: Deontological ethics in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3784–3798, Online. Associ… find
  69. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  70. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
  71. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
  72. Rezvaneh Rezapour, Saumil H. Shah, and Jana Diesner. 2019. Enhancing the measurement of social effects by capturing morality. In Proceedings of the Tenth Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 35–45, Minneapolis, USA. Association for Computat… find
  73. Sarah T Roberts. 2016. Commercial content moderation: Digital laborers’ dirty work. find
  74. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational…
  75. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  76. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477–5490, Online. Association…
  77. Xiangyu Peng, Siyan Li, Spencer Frazier, and Mark Riedl. 2020. Reducing non-normative text generation from language models. In Proceedings of the 13th International Conference on Natural Language Timo Schick, Sahana Udupa, and Hinrich Schütze. 2021. Self-diagnosis and self-debiasing: A proposal for…
  78. Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin Rothkopf, and Kristian Kersting. 2021. Language models have a moral dimension. ArXiv preprint, abs/2103.11790. find
  79. Anna-Maria Seeger, Jella Pfeiffer, and Armin Heinzl. 2017. When do we need a human? anthropomorphic design and trustworthiness of conversational agents. In Proceedings of the Sixteenth Annual Pre-ICIS Workshop on HCI Research in MIS, AISeL, Seoul, Korea, volume 10. find
  80. Kim Bartel Sheehan. 2018. Crowdsourcing research: data collection with amazon’s mechanical turk. Communication Monographs, 85(1):140–156. find
  81. Richard A Shweder. 1990. In defense of moral realism: Reply to gabennesch. Child Development, 61(6):2060–2067. find
  82. Eric Michael Smith, Diana Gonzalez-Rico, Emily Dinan, and Y-Lan Boureau. 2020. Controlling style in generated dialogue. ArXiv preprint, abs/2009.10855. — [arXiv:2009.10855](https://arxiv.org/abs/2009.10855) find
  83. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Proceedings of the 2015 Conference of the North American Chapte…
  84. SSA. 2018. Popular names in 2018. find
  85. Constantine Stephanidis, Gavriel Salvendy, Margherita Antona, Jessie YC Chen, Jianming Dong, Vincent G Duffy, Xiaowen Fang, Cali Fidopiastis, Gino Fragomeni, Limin Paul Fu, et al. 2019. Seven hci grand challenges. International Journal of Human– Computer Interaction, 35(14):1229–1269. find
  86. Zeerak Talat, Hagen Blix, Josef Valvoda, Maya Indira Ganesh, Ryan Cotterell, and Adina Williams. 2021. A word on machine ethics: A response to jiang et al.(2021). ArXiv preprint, abs/2111.04158. — [arXiv:2111.04158](https://arxiv.org/abs/2111.04158) find
  87. Yi Tay, Donovan Ong, Jie Fu, Alvin Chan, Nancy Chen, Anh Tuan Luu, and Chris Pal. 2020. Would you rather? a new benchmark for learning machine alignment with cultural values and social preferences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5369… find
  88. Aditya Nrusimha Vaidyam, Hannah Wisniewski, John David Halamka, Matcheri S Kashavan, and John Blake Torous. 2019. Chatbots and conversational agents in mental health: a review of the psychiatric landscape. The Canadian Journal of Psychiatry, 64(7):456–464. find
  89. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce… find
  90. Weiquan Wang and Izak Benbasat. 2008. Attributions of trust in decision support technologies: A study of recommendation agents for e-commerce. Journal of Management Information Systems, 24(4):249–273. find
  91. Weiquan Wang and Izak Benbasat. 2016. Empirical assessment of alternative designs for enhancing different types of trusting beliefs in online recommendation agents. Journal of Management Information Systems, 33(3):744–775. find
  92. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 347–354, Vancouver, British Columbia, Canada. As…
  93. Marty J Wolf, Keith W Miller, and Frances S Grodzinsky. 2017. Why we should have seen that coming: comments on microsoft’s tay “experiment,” and wider implications. The ORBIT Journal, 1(2):1–12. find
  94. Bo Xiao and Izak Benbasat. 2007. E-commerce product recommendation agents: Use, characteristics, and impact. MIS quarterly, pages 137–209. find
  95. Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2021. Bot-adversarial dialogue for safe conversational agents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2950–2968…
  96. Shanshan Yang and Chris Evans. 2019. Opportunities and challenges in using ai chatbots in higher education. In Proceedings of the 2019 3rd International Conference on Education and E-Learning, pages 79– 83. find
  97. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14,…
  98. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020a. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. find
  99. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020b. DIALOGPT : Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin…
  100. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. ArXiv preprint, abs/1909.08593. find
  101. John Zoshak and Kristin Dew. 2021. Beyond kant and bentham: How ethical theories are being used in artificial moral agents. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–15. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ziems-2022-moral-integrity-corpus-benchmark-ethical/