Ethical and social risks of harm from Language Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2112.04359

Summary

Large language models are spreading quickly, but discussion of their possible harms has been scattered across many disciplines. This DeepMind report, led by Laura Weidinger with over twenty co-authors, brings that discussion together in one taxonomy. The authors run no experiments. Instead, they survey work from computer science, linguistics and the social sciences, covering harms already seen and harms expected as models become more capable. For each risk they give examples and trace its likely source. Sources include skewed training corpora, statistical learning that cannot tell true from false statements, and product design choices. The report then connects these sources to possible mitigations: data curation, training-time algorithms, access limits, public policy and participatory design. It argues that mitigations must be planned together, because fixing one harm can make another worse. For example, filtering toxic text may reduce model quality for some social groups. The main value of the work is a shared vocabulary and a structured starting point for measuring and reducing LM harms.

Key points

The report identifies 21 risks in six areas:
- Discrimination, Exclusion and Toxicity
- Information Hazards
- Misinformation Harms
- Malicious Uses
- Human-Computer Interaction Harms
- Automation, Access, and Environmental Harms
Example risks include:
- stereotyping, and weaker performance for some languages and groups;
- leaking or inferring private data;
- bad medical or legal advice;
- cheaper disinformation and scams;
- users over-trusting human-like conversational agents;
- energy costs, effects on job quality, and unequal access to benefits.
Mitigation is tied to where a risk originates. For privacy leaks, options include cleaning training data, differential privacy during training, and restricting access or use cases.
Open research questions include:
- how to set "fair enough" benchmark thresholds with stakeholder input;
- how to settle disagreement over what counts as toxic speech;
- doing more HCI and ethnographic research, such as a cited study of predictive policing in New Delhi.
The report deliberately covers risks only. It leaves benefits and a full cost-benefit analysis to separate work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

351 papers of our library cite it (of the 140673 whose bibliographies we read); 43 of them have a page here, 308 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2112.04359 · arXiv 2112.04359

@misc{weidinger2021ethical,
  title = {{Ethical and social risks of harm from Language Models}},
  author = {Weidinger, Laura and Mellor, John and Rauh, Maribeth and Griffin, Conor and Uesato, Jonathan and Huang, Po-Sen and Cheng, Myra and Glaese, Mia and Balle, Borja and Kasirzadeh, Atoosa and Kenton, Zac and Brown, Sasha and Hawkins, Will and Stepleton, Tom and Biles, Courtney and Birhane, Abeba and Haas, Julia and Rimell, Laura and Hendricks, Lisa Anne and Isaac, William and Legassick, Sean and Irving, Geoffrey and Gabriel, Iason},
  year = {2021},
  doi = {10.48550/arxiv.2112.04359},
  eprint = {2112.04359},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2112.04359}
}

References

309 references, 43 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep Learning with Differential Privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS ’16, pages 308–318, Vienna, Austria, October 2016. Association for Computing Mac… — [doi:10.1145/2976749.2978318](https://doi.org/10.1145/2976749.2978318) find
A. Abid. Anti-Muslim Bias in GPT-3, August 2020. URL https://twitter.com/abidlabs/status/12911653113293 41440. ↗
A. Abid, M. Farooqi, and J. Zou. Persistent Anti-Muslim Bias in Large Language Models. arXiv:2101.05783 [cs], January 2021. URL http://arxiv.org/abs/2101.05783. arXiv: 2101.05783. ↗ our card
D. Acemoglu and P. Restrepo. Artificial Intelligence, Automation and Work. Working Paper 24196, National Bureau of Economic Research, January 2018. URL https://www.nber.org/papers/w24196. ↗
D. I. Adelani, J. Abbott, G. Neubig, D. D’souza, J. Kreutzer, C. Lignos, C. Palen-Michel, H. Buzaaba, S. Rijhwani, S. Ruder, S. Mayhew, I. A. Azime, S. Muhammad, C. C. Emezue, J. Nakatumba-Nabende, P. Ogayo, A. Aremu, C. Gitau, D. Mbaye, J. Alabi, S. M. Yimam, T. Gwadabe, I. Ezeani, R. A. Niyongabo,… find our card
B. Agüera y Arcas, M. Mitchell, and A. Todorov. Physiognomy’s New Clothes, May 2017. URL https://medium .com/@blaisea/physiognomys-new-clothes-f2d4b59fdd6a. find
J. Ahlgren, M. E. Berezin, K. Bojarczuk, E. Dulskyte, I. Dvortsova, J. George, N. Gucevska, M. Harman, R. Lämmel, E. Meijer, S. Sapora, and J. Spahr-Summers. WES: Agent-based User Interaction Simulation on Real Infrastructure. arXiv:2004.05363 [cs], April 2020. URL http://arxiv.org/abs/2004.05363. a… — [arXiv:2004.05363](https://arxiv.org/abs/2004.05363) ↗
J. Alammar. The Illustrated Transformer, June 2018. URL https://jalammar.github.io/illustrated-transformer/. ↗
H. Allcott, M. Gentzkow, and C. Yu. Trends in the diffusion of misinformation on social media. Research & Politics, 6(2):2053168019848554, April 2019. ISSN 2053-1680. doi: 10.1177/2053168019848554. URL https://doi.org/10.1177/2053168019848554. — [doi:10.1177/2053168019848554](https://doi.org/10.1177/2053168019848554) ↗
R. Andersen. The Panopticon Is Already Here. The Atlantic, July 2020. URL https://www.theatlantic.com/ma gazine/archive/2020/09/china-ai-surveillance/614197/. ↗
J. Angwin, J. Larson, S. Mattu, and L. Kirchner. Machine Bias. ProPublica, May 2016. URL https://www.prop ublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing. ↗
S. Armstrong, A. Sandberg, and N. Bostrom. Thinking Inside the Box: Controlling and Using an Oracle AI. Minds and Machines, 22(4):299–324, November 2012. ISSN 1572-8641. doi: 10.1007/s11023-012-9282-2. URL https://doi.org/10.1007/s11023-012-9282-2. — [doi:10.1007/s11023-012-9282-2](https://doi.org/10.1007/s11023-012-9282-2) ↗
D. Autor and A. Salomons. New Frontiers: The Evolving Content and Geography of New Work in the 20th Century - David Autor. Working Paper, 2019. URL https://app.scholarsite.io/david-autor/articles/new-fron tiers-the-evolving-content-and-geography-of-new-work-in-the-20th-century. ↗
J. K. Baker. Stochastic modeling for automatic speech understanding. In Readings in speech recognition, pages 297–307. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, May 1990. ISBN 9781558601246. find
S. Barocas and A. D. Selbst. Big Data’s Disparate Impact. California Law Review, 104:671, 2016. URL https://heinonline.org/HOL/Page?handle=hein.journals/calr104&id=695&div=&collection=. ↗
S. Barocas, M. Hardt, and A. Narayanan. Fairness and machine learning. fairmlbook.org, 2019. URL https: //fairmlbook.org/. find
Y. Belinkov and J. Glass. Analysis Methods in Neural Language Processing: A Survey. Transactions of the Association for Computational Linguistics, 7:49–72, April 2019. ISSN 2307-387X. doi: 10.1162/tacl\_a\_00254. URL https://doi.org/10.1162/tacl\_a\_00254. ↗ our card
E. Bender. The \#BenderRule: On Naming the Languages We Study and Why It Matters. The Gradient, September 2019. URL https://thegradient.pub/the-benderrule-on-naming-the-languages-we-study-and-why-it-matte rs/. ↗
E. M. Bender. On Achieving and Evaluating Language-Independence in NLP. Linguistic Issues in Language Technology, 6(0), November 2011. ISSN 1945-3604. URL http://elanguage.net/journals/lilt/article/view/2 624. ↗
E. M. Bender and B. Friedman. Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Transactions of the Association for Computational Linguistics, 6:587–604, December 2018. ISSN 2307-387X. doi: 10.1162/tacl\_a\_00041. URL https://doi.org/10.1162/ta… ↗
E. M. Bender and A. Koller. Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.a… find our card
E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, pages 610–623, Virtual Event, Canada, March 2021. Association for Co… — [doi:10.1145/3442188.3445922](https://doi.org/10.1145/3442188.3445922) find
Y. Bengio. Neural net language models, January 2008. URL http://www.scholarpedia.org/article/Neural\_net \_language\_models. ↗
R. Benjamin. Race After Technology: Abolitionist Tools for the New Jim Code. Social Forces, 98(4):1–3, June 2020. ISSN 0037-7732. doi: 10.1093/sf/soz162. URL https://doi.org/10.1093/sf/soz162. — [doi:10.1093/sf/soz162](https://doi.org/10.1093/sf/soz162) ↗
H. Bergen. ‘I’d Blush if I Could’: Digital Assistants, Disembodied Cyborgs and the Problem of Gender. Word and Text, A Journal of Literary Studies and Linguistics, VI(01):95–113, 2016. ISSN 2069-9271. URL https://www.ceeol.com/search/article-detail?id=469884. ↗
T. W. Bickmore, H. Trinh, S. Olafsson, T. K. O’Leary, R. Asadi, N. M. Rickles, and R. Cruz. Patient and Consumer Safety Risks When Using Conversational Assistants for Medical Information: An Observational Study of Siri, Alexa, and Google Assistant. Journal of Medical Internet Research, 20(9):e11510,… — [doi:10.2196/11510](https://doi.org/10.2196/11510) find
S. L. Blodgett and B. O’Connor. Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American English. arXiv:1707.00061 [cs], June 2017. URL http://arxiv.org/abs/1707.00061. arXiv: 1707.00061. ↗
S. L. Blodgett, L. Green, and B. O’Connor. Demographic Dialectal Variation in Social Media: A Case Study of African-American English. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1119–1130, Austin, Texas, November 2016. Association for Computationa… find our card
S. L. Blodgett, S. Barocas, H. Daumé III, and H. Wallach. Language (Technology) is Power: A Critical Survey of "Bias" in NLP. arXiv:2005.14050 [cs], May 2020. URL http://arxiv.org/abs/2005.14050. arXiv: 2005.14050. ↗ our card
S. L. Blodgett, G. Lopez, A. Olteanu, R. Sim, and H. Wallach. Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lang… find
S. Bok. Secrecy and Openness in Science: Ethical Considerations. Science, Technology, & Human Values, 7(38): 32–41, 1982. ISSN 0162-2439. URL https://www.jstor.org/stable/689458. ↗
R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. Chatterji, A. Chen, K. Creel, J. Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Etha… find our card
N. Bostrom. Superintelligence: paths, dangers, strategies. Oxford University Press, Oxford, 2014. ISBN 9780199678112. OCLC: ocn881706835. find
N. Bostrom et al. Information hazards: A typology of potential harms from knowledge. Review of Contemporary Philosophy, pages 44–79, 2011. find
G. C. Bowker and S. L. Star. Sorting Things Out: Classification and Its Consequences. Inside Technology. MIT Press, Cambridge, MA, USA, September 1999. ISBN 9780262024617. find
G. Branwen. GPT-3 Creative Fiction, June 2020. URL https://www.gwern.net/GPT-3. ↗
C. Breazeal and B. Scassellati. Infant-like Social Interactions between a Robot and a Human Caregiver. Adaptive Behavior, 8(1):49–74, January 2000. ISSN 1059-7123. doi: 10.1177/105971230000800104. URL https://doi.org/10.1177/105971230000800104. — [doi:10.1177/105971230000800104](https://doi.org/10.1177/105971230000800104) ↗
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Ber… find
S. Browne. Dark Matters. Duke University Press, September 2015. ISBN 9780822375302. URL https: //www.degruyter.com/document/doi/10.1515/9780822375302/html. — [doi:10.1515/9780822375302/html](https://doi.org/10.1515/9780822375302/html) find
B. Buchanan, A. Lohn, M. Musser, and S. Katerina. Truth, Lies, and Truth, Lies, and Automation: How Language Models Could Change DisinformationAutomation: How Language Models Could Change Disinformation. Technical report, CSET, May 2021. find
J. Buolamwini and T. Gebru. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Conference on Fairness, Accountability and Transparency, pages 77–91. PMLR, January 2018. URL https://proceedings.mlr.press/v81/buolamwini18a.html. ↗
A. Caliskan, J. J. Bryson, and A. Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186, April 2017. ISSN 0036-8075, 1095-9203. doi: 10.1126/sc ience.aal4230. URL http://arxiv.org/abs/1608.07187. arXiv: 1608.07187. — [doi:10.1126/sc](https://doi.org/10.1126/sc) ↗
Y. T. Cao and H. Daumé III. Toward Gender-Inclusive Coreference Resolution. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4568–4595, 2020. doi: 10.18653/v1/2020.a cl-main.418. URL http://arxiv.org/abs/1910.13913. arXiv: 1910.13913. ↗ our card
N. Carlini, C. Liu, Ú. Erlingsson, J. Kos, and D. Song. The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267–284, 2019. ISBN 9781939133069. URL https://www.usenix.org/conference/usenixsecurity19/presen… ↗ our card
N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, A. Oprea, and C. Raffel. Extracting Training Data from Large Language Models. arXiv:2012.07805 [cs], June 2021. URL http://arxiv.org/abs/2012.07805. arXiv: 2012.07805. ↗ our card
I. Caswell, J. Kreutzer, L. Wang, A. Wahab, D. van Esch, N. Ulzii-Orshikh, A. Tapo, N. Subramani, A. Sokolov, C. Sikasote, M. Setyawan, S. Sarin, S. Samb, B. Sagot, C. Rivera, A. Rios, I. Papadimitriou, S. Osei, P. J. O. Suárez, I. Orife, K. Ogueji, R. A. Niyongabo, T. Q. Nguyen, M. Müller, A. Mülle… find our card
S. Cave and K. Dihal. The Whiteness of AI. Philosophy & Technology, 33(4):685–703, December 2020. ISSN 2210-5441. doi: 10.1007/s13347-020-00415-6. URL https://doi.org/10.1007/s13347-020-00415-6. — [doi:10.1007/s13347-020-00415-6](https://doi.org/10.1007/s13347-020-00415-6) ↗
A. Cercas Curry, J. Robertson, and V. Rieser. Conversational Assistants and Gender Stereotypes: Public Perceptions and Desiderata for Voice Personas. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 72–78, Barcelona, Spain (Online), December 2020. Associatio… find
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cum… find our card
R. J. Chen, M. Y. Lu, T. Y. Chen, D. F. K. Williamson, and F. Mahmood. Synthetic data in machine learning for medicine and healthcare. Nature Biomedical Engineering, 5(6):493–497, June 2021b. ISSN 2157-846X. doi: 10.1038/s41551-021-00751-8. URL https://www.nature.com/articles/s41551-021-00751-8. — [doi:10.1038/s41551-021-00751-8](https://doi.org/10.1038/s41551-021-00751-8) ↗
A. Chouldechova and A. Roth. The Frontiers of Fairness in Machine Learning. arXiv:1810.08810 [cs, stat], October 2018. URL http://arxiv.org/abs/1810.08810. arXiv: 1810.08810. — [arXiv:1810.08810](https://arxiv.org/abs/1810.08810) ↗
E. Colleoni, A. Rozza, and A. Arvidsson. Echo Chamber or Public Sphere? Predicting Political Orientation and Measuring Political Homophily in Twitter Using Big Data. Journal of Communication, 64(2):317–332, April 2014. ISSN 0021-9916. doi: 10.1111/jcom.12084. URL https://doi.org/10.1111/jcom.12084. — [doi:10.1111/jcom.12084](https://doi.org/10.1111/jcom.12084) ↗
CopilotonGitHub. GitHub Copilot · Your AI pair programmer, 2021. URL https://copilot.github.com/. ↗
D. Coyle and A. Weller. “Explaining” machine learning reveals policy challenges. Science, 368(6498):1433–1434, June 2020. doi: 10.1126/science.aba9647. URL https://www.science.org/doi/full/10.1126/science.aba96 47. — [doi:10.1126/science.aba9647](https://doi.org/10.1126/science.aba9647) ↗
J. T. Craft, K. E. Wright, R. E. Weissler, and R. M. Queen. Language and Discrimination: Generating Meaning, Perceiving Identities, and Discriminating Outcomes. Annual Review of Linguistics, 6(1):389–407, January 2020. ISSN 2333-9683, 2333-9691. doi: 10.1146/annurev-linguistics-011718-011659. URL ht… — [doi:10.1146/annurev-linguistics-011718-011659](https://doi.org/10.1146/annurev-linguistics-011718-011659) find
K. Crawford. Atlas of AI. Yale University Press, 2021. URL https://yalebooks.yale.edu/book/9780300209570/ atlas-ai. ↗
K. Crenshaw. On Intersectionality: Essential Writings. Books, March 2017a. URL https://scholarship.law.colu mbia.edu/books/255. ↗
K. Crenshaw. On Intersectionality: Essential Writings. The New Press, March 2017b. URL https://scholarship.la w.columbia.edu/books/255. ↗
B. Cyphers and G. Gebhart. Behind the One-Way Mirror: A Deep Dive Into the Technology of Corporate Surveillance. Technical report, Electronic Frontier Foundation, December 2019. URL https://www.eff.org/ wp/behind-the-one-way-mirror. ↗
R. Dale. GPT-3: What’s it good for? Natural Language Engineering, 27(1):113–118, January 2021. ISSN 1351-3249, 1469-8110. doi: 10.1017/S1351324920000601. URL https://www.cambridge.org/core/journals/na tural-language-engineering/article/gpt3-whats-it-good-for/0E05CFE68A7AC8BF794C8ECBE28AA990. — [doi:10.1017/s1351324920000601](https://doi.org/10.1017/s1351324920000601) ↗
J. Dastin. Amazon scraps secret AI recruiting tool that showed bias against women. Reuters, October 2018. URL https://www.reuters.com/article/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G. ↗
A. M. Davani, A. Omrani, B. Kennedy, M. Atari, X. Ren, and M. Dehghani. Fair Hate Speech Detection through Evaluation of Social Group Counterfactuals. arXiv:2010.12779 [cs], October 2020. URL http: //arxiv.org/abs/2010.12779. arXiv: 2010.12779. — [arXiv:2010.12779](https://arxiv.org/abs/2010.12779) find
E. Denton, A. Hanna, R. Amironesei, A. Smart, H. Nicole, and M. K. Scheuerman. Bringing the People Back In: Contesting Benchmark Machine Learning Datasets. arXiv:2007.07399 [cs], July 2020. URL http://arxiv.org/abs/2007.07399. arXiv: 2007.07399. — [arXiv:2007.07399](https://arxiv.org/abs/2007.07399) ↗
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs], May 2019. URL http://arxiv.org/abs/1810.04805. arXiv: 1810.04805. ↗
T. Dietterich and E. B. Kong. Machine Learning Bias, Statistical Bias, and Statistical Variance of Decision Tree Algorithms. Technical report, Department of Computer Science, Oregon State University, 1995. find
E. Dinan, G. Abercrombie, A. S. Bergman, S. Spruit, D. Hovy, Y.-L. Boureau, and V. Rieser. Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling. arXiv:2107.03451 [cs], July 2021. URL http://arxiv.org/abs/2107.03451. arXiv: 2107.03451. — [arXiv:2107.03451](https://arxiv.org/abs/2107.03451) ↗
L. Dixon, J. Li, J. Sorensen, N. Thain, and L. Vasserman. Measuring and Mitigating Unintended Bias in Text Classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’18, pages 67–73, New Orleans, LA, USA, December 2018. Association for Computing Machinery. ISBN… — [doi:10.1145/3278721.3278729](https://doi.org/10.1145/3278721.3278729) find
L. Dobberstein. Korean app-maker Scatter Lab fined for using private data to create homophobic and lewd chatbot. The Register, April 2021. URL https://www.theregister.com/2021/04/29/scatter\_lab\_fined\_for\_l ewd\_chatbot/. ↗
J. Dodge, M. Sap, A. Marasović, W. Agnew, G. Ilharco, D. Groeneveld, M. Mitchell, and M. Gardner. Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus. arXiv:2104.08758 [cs], September 2021. URL http://arxiv.org/abs/2104.08758. arXiv: 2104.08758. ↗
F. Doshi-Velez and B. Kim. Towards A Rigorous Science of Interpretable Machine Learning. arXiv:1702.08608 [cs, stat], March 2017. URL http://arxiv.org/abs/1702.08608. arXiv: 1702.08608. ↗ our card
D. M. Douglas. Doxing: a conceptual analysis. Ethics and Information Technology, 18(3):199–210, September 2016. ISSN 1572-8439. doi: 10.1007/s10676-016-9406-0. URL https://doi.org/10.1007/s10676-016-940 6-0. — [doi:10.1007/s10676-016-9406-0](https://doi.org/10.1007/s10676-016-9406-0) ↗
C. Du. Chinese AI lab challenges Google, OpenAI with a model of 1.75 trillion parameters. PingWest, June 2021. URL https://en.pingwest.com/a/8693. ↗
M. Duggan. Online Harassment 2017. Technical report, Pew Research Center, July 2017. URL https: //www.pewresearch.org/internet/2017/07/11/online-harassment-2017/. find
W. H. Dutton and C. T. Robertson. Disentangling polarisation and civic empowerment in the digital age : The role of filter bubbles and echo chambers in the rise of populism. In The Routledge Companion to Media Disinformation and Populism. Routledge, 2021. find
C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating Noise to Sensitivity in Private Data Analysis. In S. Halevi and T. Rabin, editors, Theory of Cryptography, Lecture Notes in Computer Science, pages 265–284, Berlin, Heidelberg, 2006. Springer. ISBN 9783540327325. doi: 10.1007/11681878\_14. — [doi:10.1007/11681878\_14](https://doi.org/10.1007/11681878_14) find
R. Evans and J. Gao. DeepMind AI Reduces Google Data Centre Cooling Bill by 40%, July 2016. URL https://deepmind.com/blog/article/deepmind-ai-reduces-google-data-centre-cooling-bill-40. ↗
W. Fedus, B. Zoph, and N. Shazeer. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv:2101.03961 [cs], January 2021. URL http://arxiv.org/abs/2101.03961. arXiv: 2101.03961. ↗
X. Ferrer, T. van Nuenen, J. M. Such, and N. Criado. Discovering and Categorising Language Biases in Reddit. arXiv:2008.02754 [cs], August 2020. URL http://arxiv.org/abs/2008.02754. arXiv: 2008.02754. — [arXiv:2008.02754](https://arxiv.org/abs/2008.02754) ↗
S. Finkelstein, E. Yarzebinski, C. Vaughn, A. Ogan, and J. Cassell. The Effects of Culturally Congruent Educational Technologies on Student Achievement. In H. C. Lane, K. Yacef, J. Mostow, and P. Pavlik, editors, Artificial Intelligence in Education, Lecture Notes in Computer Science, pages 493–502,… — [doi:10.1007/978-3-642-39112-5\_50](https://doi.org/10.1007/978-3-642-39112-5_50) find
C. Flood. Fake news infiltrates financial markets. Financial Times, May 2017. URL https://www.ft.com/content /a37e4874-2c2a-11e7-bc4b-5528796fe35c. ↗
P. Fortuna and S. Nunes. A Survey on Automatic Detection of Hate Speech in Text. ACM Computing Surveys, 51(4): 85:1–85:30, July 2018. ISSN 0360-0300. doi: 10.1145/3232676. URL https://doi.org/10.1145/3232676. — [doi:10.1145/3232676](https://doi.org/10.1145/3232676) ↗
M. Foucault and A. Sheridan. Discipline and punish: the birth of the prison. Vintage, New York, 2012. ISBN 9780307819291. URL http://0-lib.myilibrary.com.catalogue.libraries.london.ac.uk?id=435863. OCLC: 817200914. ↗
I. Gabriel. Artificial Intelligence, Values, and Alignment. Minds and Machines, 30(3):411–437, September 2020a. ISSN 1572-8641. doi: 10.1007/s11023-020-09539-2. URL https://doi.org/10.1007/s11023-020-09539-2. — [doi:10.1007/s11023-020-09539-2](https://doi.org/10.1007/s11023-020-09539-2) ↗
I. Gabriel. DeepMind x UCL \| Deep Learning Lectures: Responsible Innovation, July 2020b. URL https: //www.youtube.com/watch?v=MhNcWxUs-PQ. find
I. Gabriel and V. Ghazavi. The Challenge of Value Alignment: from Fairer Algorithms to AI Safety. arXiv:2101.06060 [cs], January 2021. URL http://arxiv.org/abs/2101.06060. arXiv: 2101.06060. — [arXiv:2101.06060](https://arxiv.org/abs/2101.06060) ↗
D. Garcia, M. Goel, A. K. Agrawal, and P. Kumaraguru. Collective aspects of privacy in the Twitter social network. EPJ Data Science, 7(1):3, December 2018. ISSN 2193-1127. doi: 10.1140/epjds/s13688-018-0130-3. URL https://epjdatascience.springeropen.com/articles/10.1140/epjds/s13688-018-0130-3. — [doi:10.1140/epjds/s13688-018-0130-3](https://doi.org/10.1140/epjds/s13688-018-0130-3) ↗
S. Garg, V. Perot, N. Limtiaco, A. Taly, E. H. Chi, and A. Beutel. Counterfactual Fairness in Text Classification through Robustness. arXiv:1809.10610 [cs, stat], February 2019. URL http://arxiv.org/abs/1809.10610. arXiv: 1809.10610. — [arXiv:1809.10610](https://arxiv.org/abs/1809.10610) ↗
T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. Daumé III, and K. Crawford. Datasheets for Datasets. arXiv:1803.09010 [cs], March 2020. URL http://arxiv.org/abs/1803.09010. arXiv: 1803.09010. ↗
S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. arXiv:2009.11462 [cs], September 2020. URL http://arxiv.org/abs/2009 .11462. arXiv: 2009.11462. ↗ our card
A. Georgieff and A. Milanez. What happened to jobs at high risk of automation? Technical Report 255, OECD Publishing, January 2021. URL https://ideas.repec.org/p/oec/elsaab/255-en.html. ↗
D. Gershgorn. GPT-3 Contains Disturbing Bias Against Muslims, January 2021. URL https://onezero.medium .com/for-some-reason-im-covered-in-blood-gpt-3-contains-disturbing-bias-against-muslims-693d275552 bf. ↗
S. Ghaffary. The algorithms that detect hate speech online are biased against black people. Vox, August 2019. URL https://www.vox.com/recode/2019/8/15/20806384/social-media-hate-speech-bias-black-african-american-facebook-twitter. ↗
S. Ghalebikesabi, H. Wilde, J. Jewson, A. Doucet, S. Vollmer, and C. Holmes. Bias Mitigated Learning from Differentially Private Synthetic Data: A Cautionary Tale. arXiv:2108.10934 [cs, stat], August 2021. URL http://arxiv.org/abs/2108.10934. arXiv: 2108.10934. — [arXiv:2108.10934](https://arxiv.org/abs/2108.10934) ↗
J. Golbeck. Predicting Alcoholism Recovery from Twitter. In R. Thomson, C. Dancy, A. Hyder, and H. Bisgin, editors, Social, Cultural, and Behavioral Modeling, Lecture Notes in Computer Science, pages 243–252, Cham, 2018. Springer International Publishing. ISBN 9783319933726. doi: 10.1007/978-3-319-9… — [doi:10.1007/978-3-319-93372-6\_28](https://doi.org/10.1007/978-3-319-93372-6_28) find
R. Gorwa, R. Binns, and C. Katzenbach. Algorithmic content moderation: Technical and political challenges in the automation of platform governance. Big Data & Society, 7(1):2053951719897945, January 2020. ISSN 2053-9517. doi: 10.1177/2053951719897945. URL https://doi.org/10.1177/2053951719897945. — [doi:10.1177/2053951719897945](https://doi.org/10.1177/2053951719897945) ↗
M. Gray and S. Suri. Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Mariner Books, 2019. URL https://ghostwork.info/. ↗
D. Gunning, M. Stefik, J. Choi, T. Miller, S. Stumpf, and G.-Z. Yang. XAI—Explainable artificial intelligence. Science Robotics, 4(37):eaay7120, December 2019. doi: 10.1126/scirobotics.aay7120. URL https: //www.science.org/doi/10.1126/scirobotics.aay7120. — [doi:10.1126/scirobotics.aay7120](https://doi.org/10.1126/scirobotics.aay7120) find
B. Gutelius and N. Theodore. The Future of Warehouse Work: Technological Change in the U.S. Logistics Industry. Technical report, UC Berkeley Labor Center and Working Partnerships USA, 2019. URL https: //laborcenter.berkeley.edu/future-of-warehouse-work/. find
L. M. Hampton. Black Feminist Musings on Algorithmic Oppression. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 1–1, March 2021. doi: 10.1145/3442188.3445929. URL http://arxiv.org/abs/2101.09869. arXiv: 2101.09869. — [doi:10.1145/3442188.3445929](https://doi.org/10.1145/3442188.3445929) ↗
S. C. Hampton. Parasite and catalyst: the polarizing influence of chatbots in political discourse. PhD thesis, University of Texas at Austin, August 2019. URL https://repositories.lib.utexas.edu/handle/2152/81204. ↗
L. Hancox-Li and I. E. Kumar. Epistemic values in feature importance methods: Lessons from feminist epistemology. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, pages 817–826, Virtual Event, Canada, March 2021. Association for Computing Machinery.… — [doi:10.1145/3442188.3445943](https://doi.org/10.1145/3442188.3445943) find
L. H. Hanu, J. Thewlis, and S. Haco. How AI Is Learning to Identify Toxic Online Content. Scientific American, 2021. URL https://www.scientificamerican.com/article/can-ai-identify-toxic-online-content/. publisher: By Laura Hanu, James Thewlis, Sasha Haco. ↗
K. Hao. A college kid’s fake, AI-generated blog fooled tens of thousands. This is how he made it. MIT Technology Review, August 2020. URL https://www.technologyreview.com/2020/08/14/1006780/ai-gpt-3-fake-blog-r eached-top-of-hacker-news/. ↗
D. Haraway. Situated knowledges: The science question in feminism and the privilege of partial perspective. Feminist studies, 14(3):575–599, 1988. find
D. J. Haraway. The Haraway Reader. Psychology Press, 2004. ISBN 9780415966894. Google-Books-ID: QxUr0gijyGoC. find
S. G. Harding. Feminism and Methodology: Social Science Issues. Indiana University Press, 1987. ISBN 9780253204448. Google-Books-ID: w2gzw6zz4f IC. find
M. Hardt, E. Price, and N. Srebro. Equality of Opportunity in Supervised Learning. arXiv:1610.02413 [cs], October 2016. URL http://arxiv.org/abs/1610.02413. arXiv: 1610.02413. ↗ our card
M. R. Hasan, A. K. Jha, and Y. Liu. Excessive use of online video streaming services: Impact of recommender system use, psychological factors, and motives. Computers in Human Behavior, 80:220–228, March 2018. ISSN 0747-5632. doi: 10.1016/j.chb.2017.11.020. URL https://www.sciencedirect.com/science/a… — [doi:10.1016/j.chb.2017.11.020](https://doi.org/10.1016/j.chb.2017.11.020) ↗
P. He, X. Liu, J. Gao, and W. Chen. DeBERTa: Decoding-enhanced BERT with Disentangled Attention. arXiv:2006.03654 [cs], October 2021. URL http://arxiv.org/abs/2006.03654. arXiv: 2006.03654. ↗ our card
D. Hendrycks, C. Burns, S. Basart, A. Critch, J. Li, D. Song, and J. Steinhardt. Aligning AI With Shared Human Values. arXiv:2008.02275 [cs], July 2021. URL http://arxiv.org/abs/2008.02275. arXiv: 2008.02275. ↗ our card
P. Hill Collins and N. Denzin. Toward an Afrocentric Feminist Epistemology. In Y. Lincoln, editor, Turning Points in Qualitative Research, volume 2. Rowman Altamira, 2003. find
P. Hitlin, K. Olmstead, and S. Toor. FCC Net Neutrality Online Public Comments Contain Many Inaccuracies and Duplicates. Technical report, Pew Research Center, November 2017. URL https://www.pewresearch.or g/internet/2017/11/29/public-comments-to-the-federal-communications-commission-about-net-neutr… ↗
K. Holt. Google’s ’Verse by Verse’ AI can help you write in the style of famous poets. Engadget, November 2020. URL https://www.engadget.com/googles-ai-poetry-verse-by-verse-202105834.html. ↗
A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The Curious Case of Neural Text Degeneration. arXiv:1904.09751 [cs], February 2020. URL http://arxiv.org/abs/1904.09751. arXiv: 1904.09751. ↗
C. Hookway. Scepticism. Routledge, 1990. find
D. Hovy and S. L. Spruit. The Social Impact of Natural Language Processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 591–598, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P1… find our card
D. Hovy and D. Yang. The Importance of Modeling Social Factors of Language: Theory and Practice. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 588–602, Online, June 2021. Association for Comput… find our card
J. Howard and S. Ruder. Universal Language Model Fine-tuning for Text Classification. arXiv:1801.06146 [cs, stat], May 2018. URL http://arxiv.org/abs/1801.06146. arXiv: 1801.06146. ↗
K. Hsieh. Transformer Poetry. Paper Gains Publishing, 2019. URL https://papergains.co/. ↗
P.-S. Huang, H. Zhang, R. Jiang, R. Stanforth, J. Welbl, J. Rae, V. Maini, D. Yogatama, and P. Kohli. Reducing Sentiment Bias in Language Models via Counterfactual Evaluation. arXiv:1911.03064 [cs], October 2020. URL http://arxiv.org/abs/1911.03064. arXiv: 1911.03064. ↗
E. Hunt. Tay, Microsoft’s AI chatbot, gets a crash course in racism from Twitter. The Guardian, March 2016. URL http://www.theguardian.com/technology/2016/mar/24/tay-microsofts-ai-chatbot-gets-a-crash-cour se-in-racism-from-twitter. ↗
K. Hunt and C. Xu. China ’employs 2 million to police internet’. CNN, October 2013. URL https://www.cnn.co m/2013/10/07/world/asia/china-internet-monitors/index.html. publisher: CNN. ↗
B. Hutchinson, A. Smart, A. Hanna, E. Denton, C. Greer, O. Kjartansson, P. Barnes, and M. Mitchell. Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, F… — [doi:10.1145/3442188.3445918](https://doi.org/10.1145/3442188.3445918) find
G. Hwang, J. Lee, C. Y. Oh, and J. Lee. It Sounds Like A Woman: Exploring Gender Stereotypes in South Korean Voice Assistants. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, CHI EA ’19, pages 1–6, Glasgow, Scotland Uk, May 2019. Association for Computing Mach… — [doi:10.1145/3290607.3312915](https://doi.org/10.1145/3290607.3312915) find
C. Ingraham. How rising inequality hurts everyone, even the rich. Washington Post, February 2018. ISSN 0190-8286. URL https://www.washingtonpost.com/news/wonk/wp/2018/02/06/how-rising-inequality-h urts-everyone-even-the-rich/. ↗
IPCC. Global Warming of 1.5 ºC. Technical report, Intergovernmental Panel on Climate Change, 2018. URL https://www.ipcc.ch/sr15/. ↗
C. Ischen, T. Araujo, H. Voorveld, G. van Noort, and E. Smit. Privacy concerns in chatbot interactions. In International Workshop on Chatbot Research and Design, pages 34–48. Springer, 2019. find
L. James. How U.S. Companies & Partisans Hack Democracy to Undermine Your Voice. Technical report, New York State Office of the Attorney General, May 2021. find
F. Jaumotte, S. Lall, and C. Papageorgiou. Rising Income Inequality: Technology, or Trade and Financial Globalization? IMF Economic Review, 61(2):271–309, June 2013. ISSN 2041-417X. doi: 10.1057/imfer.20 13.7. URL https://doi.org/10.1057/imfer.2013.7. — [doi:10.1057/imfer.20](https://doi.org/10.1057/imfer.20) ↗
G. Jawahar, M. Abdul-Mageed, and L. V. S. Lakshmanan. Automatic Detection of Machine Generated Text: A Critical Survey. arXiv:2011.01314 [cs], November 2020. URL http://arxiv.org/abs/2011.01314. arXiv: 2011.01314. ↗
F. Jelinek. Continuous speech recognition by statistical methods. Proceedings of the IEEE, 64(4):532–556, April 1976. ISSN 1558-2256. doi: 10.1109/PROC.1976.10159. — [doi:10.1109/proc.1976.10159](https://doi.org/10.1109/proc.1976.10159) find
R. Jeshion. Pride and Prejudiced: On the Reclamation of Slurs. Grazer Philosophische Studien, 97(1):106– 137, March 2020. ISSN 1875-6735, 0165-9227. doi: 10.1163/18756735-09701007. URL https: //brill.com/view/journals/gps/97/1/article-p106\_106.xml. — [doi:10.1163/18756735-09701007](https://doi.org/10.1163/18756735-09701007) find
Jigsaw. Unintended Bias and Identity Terms, October 2021. URL https://medium.com/jigsaw/unintended-bia s-and-names-of-frequently-targeted-groups-8e0b81f80a23. ↗
E. S. Jo and T. Gebru. Lessons from archives: strategies for collecting sociocultural data in machine learning. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* ’20, pages 306– 316, Barcelona, Spain, January 2020. Association for Computing Machinery. ISBN 978… — [doi:10.1145/3351095.3372829](https://doi.org/10.1145/3351095.3372829) find
N. Jones. How to stop data centres from gobbling up the world’s electricity. Nature, 561(7722):163–166, September 2018. doi: 10.1038/d41586-018-06610-y. URL https://www.nature.com/articles/d41586-018 -06610-y. — [doi:10.1038/d41586-018-06610-y](https://doi.org/10.1038/d41586-018-06610-y) ↗
P. Joshi, S. Santy, A. Budhiraja, K. Bali, and M. Choudhury. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. arXiv:2004.09095 [cs], January 2021. URL http://arxiv.org/abs/2004.09095. arXiv: 2004.09095. ↗ our card
D. Jurafsky and J. H. Martin. Speech and language processing. Pearson custom library. Pearson Education, Harlow, 2. ed., pearson new international ed edition, 2014. ISBN 9781292025438. find
A. Kasirzadeh. Reasons, Values, Stakeholders: A Philosophical Framework for Explainable Artificial Intelligence. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 14, Virtual Event, Canada, March 2021. Association for Computing Machinery. ISBN 9… — [doi:10.1145/3442188.3445866](https://doi.org/10.1145/3442188.3445866) find
N. Kassner and H. Schütze. Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly. arXiv:1911.03343 [cs], May 2020. URL http://arxiv.org/abs/1911.03343. arXiv: 1911.03343. ↗ our card
Z. Kenton, T. Everitt, L. Weidinger, I. Gabriel, V. Mikulik, and G. Irving. Alignment of Language Agents. arXiv:2103.14659 [cs], March 2021. URL http://arxiv.org/abs/2103.14659. arXiv: 2103.14659. ↗
O. Keyes. The Misgendering Machines: Trans/HCI Implications of Automatic Gender Recognition. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW):88:1–88:22, November 2018. doi: 10.1145/3274357. URL https://doi.org/10.1145/3274357. — [doi:10.1145/3274357](https://doi.org/10.1145/3274357) ↗
M. Khalifa, H. Elsahar, and M. Dymetman. A Distributional Approach to Controlled Text Generation. arXiv:2012.11635 [cs], May 2021. URL http://arxiv.org/abs/2012.11635. arXiv: 2012.11635. — [arXiv:2012.11635](https://arxiv.org/abs/2012.11635) ↗
D. Kim. Chatbot Gone Awry Starts Conversations About AI Ethics in South Korea. The Diplomat, January 2021. URL https://thediplomat.com/2021/01/chatbot-gone-awry-starts-conversations-about-ai-ethics-in-south-korea/. ↗
J. Y. Kim, C. Ortiz, S. Nam, S. Santiago, and V. Datta. Intersectional Bias in Hate Speech and Abusive Language Datasets. arXiv:2005.05921 [cs], May 2020. URL http://arxiv.org/abs/2005.05921. arXiv: 2005.05921. — [arXiv:2005.05921](https://arxiv.org/abs/2005.05921) ↗
Y. Kim and S. S. Sundar. Anthropomorphism of computers: Is it mindful or mindless? Computers in Human Behavior, 28(1):241–250, 2012. find
J. Kocoń, A. Figas, M. Gruza, D. Puchalska, T. Kajdanowicz, and P. Kazienko. Offensive, aggressive, and hate speech analysis: From data-centric to human-centered approach. Information Processing & Management, 58(5):102643, September 2021. ISSN 0306-4573. doi: 10.1016/j.ipm.2021.102643. URL https: //… — [doi:10.1016/j.ipm.2021.102643](https://doi.org/10.1016/j.ipm.2021.102643) find
A. Koenecke, A. Nam, E. Lake, J. Nudell, M. Quartey, Z. Mengesha, C. Toups, J. R. Rickford, D. Jurafsky, and S. Goel. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14):7684–7689, April 2020. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas.19157… — [doi:10.1073/pnas.1915768117](https://doi.org/10.1073/pnas.1915768117) find
I. Kohler-Hausmann. Eddie Murphy and the Dangers of Counterfactual Causal Thinking About Detecting Racial Discrimination. SSRN Scholarly Paper ID 3050650, Social Science Research Network, Rochester, NY, January 2019. URL https://papers.ssrn.com/abstract=3050650. ↗
N. Kordzadeh and M. Ghasemaghaei. Algorithmic bias: review, synthesis, and future research directions. European Journal of Information Systems, 0(0):1–22, June 2021. ISSN 0960-085X. doi: 10.1080/0960085X .2021.1927212. URL https://doi.org/10.1080/0960085X.2021.1927212. — [doi:10.1080/0960085x](https://doi.org/10.1080/0960085x) ↗
M. Kosinski, D. Stillwell, and T. Graepel. Private traits and attributes are predictable from digital records of human behavior. Proceedings of the National Academy of Sciences, 110(15):5802–5805, April 2013. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas.1218772110. URL https://www.pnas.org/content/1… — [doi:10.1073/pnas.1218772110](https://doi.org/10.1073/pnas.1218772110) ↗
C. Krittanawong, B. Narasimhan, H. U. H. Virk, H. Narasimhan, J. Hahn, Z. Wang, and W. W. Tang. Misinformation Dissemination in Twitter in the COVID-19 Era. The American Journal of Medicine, 133 (12):1367–1369, December 2020. ISSN 0002-9343. doi: 10.1016/j.amjmed.2020.07.012. URL https://www.ncbi.nl… — [doi:10.1016/j.amjmed.2020.07.012](https://doi.org/10.1016/j.amjmed.2020.07.012) ↗
K. Kurita, N. Vyas, A. Pareek, A. W. Black, and Y. Tsvetkov. Measuring Bias in Contextualized Word Representations. arXiv:1906.07337 [cs], June 2019. URL http://arxiv.org/abs/1906.07337. arXiv: 1906.07337. ↗ our card
K. H. Kwon, S.-I. Moon, and M. A. Stefanone. Unspeaking on Facebook? Testing network effects on self-censorship of political expressions in social network sites. Quality & Quantity, 49(4):1417–1435, July 2015. ISSN 1573-7845. doi: 10.1007/s11135-014-0078-8. URL https://doi.org/10.1007/s11135-014-007… — [doi:10.1007/s11135-014-0078-8](https://doi.org/10.1007/s11135-014-0078-8) ↗
C. Labbé and D. Labbé. Duplicate and fake publications in the scientific literature: how many SCIgen papers in computer science? Scientometrics, 94(1):379–396, January 2013. ISSN 1588-2861. doi: 10.1007/s11192-0 12-0781-y. URL https://doi.org/10.1007/s11192-012-0781-y. — [doi:10.1007/s11192-0](https://doi.org/10.1007/s11192-0) ↗
K. Lacker. Giving GPT-3 a Turing Test, July 2020. URL https://lacker.io/ai/2020/07/06/giving-gpt-3-a-turin g-test.html. ↗
J. Lambert and E. Cone. How Robots Change the World - What automation really means for jobs, productivity and regions. Technical report, Oxford Economics, 2019. URL https://www.oxfordeconomics.com/recent-rel eases/how-robots-change-the-world. ↗
I. Lapowsky. How Bots Broke the FCC’s Public Comment System. Wired, November 2017. ISSN 1059-1028. URL https://www.wired.com/story/bots-broke-fcc-public-comment-system/. ↗
A. Lazaridou, A. Kuncoro, E. Gribovskaya, D. Agrawal, A. Liska, T. Terzi, M. Gimenez, C. d. M. d’Autume, S. Ruder, D. Yogatama, K. Cao, T. Kocisky, S. Young, and P. Blunsom. Pitfalls of Static Language Modelling. arXiv:2102.01951 [cs], February 2021. URL http://arxiv.org/abs/2102.01951. arXiv: 2102.… ↗
B. Lewis and A. E. Marwick. Media Manipulation and Disinformation Online. Technical report, Data & Society, May 2017. URL https://datasociety.net/library/media-manipulation-and-disinfo-online. ↗
M. Lewis and G. Lupyan. Gender stereotypes are reflected in the distributional structure of 25 languages. Nature Human Behaviour, 4(10):1021–1028, October 2020. ISSN 2397-3374. doi: 10.1038/s41562-020-0918-6. URL https://www.nature.com/articles/s41562-020-0918-6. — [doi:10.1038/s41562-020-0918-6](https://doi.org/10.1038/s41562-020-0918-6) ↗
M. Lewis, D. Yarats, Y. N. Dauphin, D. Parikh, and D. Batra. Deal or No Deal? End-to-End Learning for Negotiation Dialogues. arXiv:1706.05125 [cs], June 2017. URL http://arxiv.org/abs/1706.05125. arXiv: 1706.05125. — [arXiv:1706.05125](https://arxiv.org/abs/1706.05125) ↗
P. Lewis, P. Stenetorp, and S. Riedel. Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets. arXiv:2008.02637 [cs], August 2020. URL http://arxiv.org/abs/2008.02637. arXiv: 2008.02637. ↗
Z. Li, S. Zhuang, S. Guo, D. Zhuo, H. Zhang, D. Song, and I. Stoica. TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models. arXiv:2102.07988 [cs], September 2021. URL http://arxiv.org/ abs/2102.07988. arXiv: 2102.07988. — [arXiv:2102.07988](https://arxiv.org/abs/2102.07988) ↗
Y. Liao and J. He. Racial mirroring effects on human-agent interaction in psychotherapeutic conversations. In Proceedings of the 25th International Conference on Intelligent User Interfaces, IUI ’20, pages 430–442, Cagliari, Italy, March 2020. Association for Computing Machinery. ISBN 9781450371186.… — [doi:10.1145/3377325.33](https://doi.org/10.1145/3377325.33) find
S. Lin, J. Hilton, and O. Evans. TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109.07958 [cs], September 2021. URL http://arxiv.org/abs/2109.07958. arXiv: 2109.07958. ↗ our card
P. Linardatos, V. Papastefanopoulos, and S. Kotsiantis. Explainable AI: A Review of Machine Learning Interpretability Methods. Entropy, 23(1):18, January 2021. doi: 10.3390/e23010018. URL https: //www.mdpi.com/1099-4300/23/1/18. — [doi:10.3390/e23010018](https://doi.org/10.3390/e23010018) find
R. Lippi. English with an Accent: Language, Ideology and Discrimination in the United States. Routledge, 1997. URL https://www.routledge.com/English-with-an-Accent-Language-Ideology-and-Discrimination-in-the -United/Lippi-Green/p/book/9780415559119. ↗
Z. C. Lipton. The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31–57, June 2018. ISSN 1542-7730, 1542-7749. doi: 10.1145/3236386.3241340. URL https://dl.acm.org/doi/10.1145/3236386.3241340. — [doi:10.1145/3236386.3241340](https://doi.org/10.1145/3236386.3241340) ↗
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692 [cs], July 2019. URL http://arxiv.org/abs/1907.11692. arXiv: 1907.11692. ↗
LSEblog2017. Doxing is a toxic practice – no matter who is targeted \| Media@LSE, August 2017. URL https://bl ogs.lse.ac.uk/medialse/2017/08/18/the-dangers-of-doxing-and-the-implications-for-media-regulation/. find
A. S. Luccioni and J. D. Viviano. What’s in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus. arXiv:2105.02732 [cs], May 2021. URL http://arxiv.org/abs/2105.02732. arXiv: 2105.02732. — [arXiv:2105.02732](https://arxiv.org/abs/2105.02732) ↗
L. Lucy and D. Bamman. Gender and Representation Bias in GPT-3 Generated Stories. In Proceedings of the Third Workshop on Narrative Understanding, pages 48–55, Virtual, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.nuse-1.5. URL https://aclanthology.org/2021.nuse-1.5. ↗
S. Luper. Epistemic Relativism. Philosophical Issues, 14:271–295, 2004. ISSN 1533-6077. URL https: //www.jstor.org/stable/3050631. find
A. Maass and L. Arcuri. The role of language in the persistence of stereotypes. In Language, interaction and social cognition, pages 129–143. Sage Publications, Inc, 1992. URL https://psycnet.apa.org/record/1992-97 980-006. ↗
A. Makazhanov, D. Rafiei, and M. Waqar. Predicting political preference of Twitter users. Social Network Analysis and Mining, 4(1):193, May 2014. ISSN 1869-5469. doi: 10.1007/s13278-014-0193-5. URL https://doi.org/10.1007/s13278-014-0193-5. — [doi:10.1007/s13278-014-0193-5](https://doi.org/10.1007/s13278-014-0193-5) ↗
A. E. Manduley, A. Mertens, I. Plante, and A. Sultana. The role of social media in sex education: Dispatches from queer, trans, and racialized communities:. Feminism & Psychology, 28(1):152–170, February 2018. doi: 10.1177/0959353517717751. URL https://journals.sagepub.com/eprint/wZsRhKyIrHE7KMyN9fJ… — [doi:10.1177/0959353517717751](https://doi.org/10.1177/0959353517717751) ↗
C. B. Mann. Can Conversing with a Computer Increase Turnout? Mobilization Using Chatbot Communication. Journal of Experimental Political Science, 8(1):51–62, 2021. URL https://ideas.repec.org/a/cup/jexpos/v8y 2021i1p51-62\_5.html. ↗
H. Mao, X. Shuai, and A. Kapadia. Loose tweets: an analysis of privacy leaks on twitter. In Proceedings of the 10th annual ACM workshop on Privacy in the electronic society, WPES ’11, pages 1–12, Chicago, Illinois, USA, October 2011. Association for Computing Machinery. ISBN 9781450310024. doi: 10.1… — [doi:10.1145/2046556.2046558](https://doi.org/10.1145/2046556.2046558) find
V. Marda and S. Narayan. On the importance of ethnographic methods in AI research. Nature Machine Intelligence, 3(3):187–189, March 2021. ISSN 2522-5839. doi: 10.1038/s42256-021-00323-0. URL https://www.nature.com/articles/s42256-021-00323-0. — [doi:10.1038/s42256-021-00323-0](https://doi.org/10.1038/s42256-021-00323-0) ↗
M. Marino. The Racial Formation of Chatbots. CLCWeb: Comparative Literature and Culture, 16(5), December 2014. ISSN 1481-4374. doi: 10.7771/1481-4374.2560. URL https://docs.lib.purdue.edu/clcweb/vol16/iss 5/13. — [doi:10.7771/1481-4374.2560](https://doi.org/10.7771/1481-4374.2560) ↗
A. Markov. Essai d’une recherche statistique sur le texte du roman “eugène oněgin”, illustrant la liaison des épreuves en chaîne. Bulletin de l’Académie Impériale des Sciences de St.-Pétersbourg., 7(3):153–162, 1913. find
D. Martin Jr., V. Prabhakaran, J. Kuhlberg, A. Smart, and W. S. Isaac. Participatory Problem Formulation for Fairer Machine Learning Through Community Based System Dynamics. arXiv:2005.07572 [cs, stat], May 2020. URL http://arxiv.org/abs/2005.07572. arXiv: 2005.07572. — [arXiv:2005.07572](https://arxiv.org/abs/2005.07572) ↗
K. McGuffie and A. Newhouse. The Radicalization Risks of GPT-3 and Advanced Neural Language Models. arXiv:2009.06807 [cs], September 2020. URL http://arxiv.org/abs/2009.06807. arXiv: 2009.06807. — [arXiv:2009.06807](https://arxiv.org/abs/2009.06807) ↗
K. McKee, X. Bai, and S. Fiske. Understanding Human Impressions of Artificial Intelligence. PsyArxiv, 2021. URL https://psyarxiv.com/5ursp/. ↗
N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan. A Survey on Bias and Fairness in Machine Learning. arXiv:1908.09635 [cs], September 2019. URL http://arxiv.org/abs/1908.09635. arXiv: 1908.09635. ↗ our card
N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan. A Survey on Bias and Fairness in Machine Learning. ACM Computing Surveys, 54(6):115:1–115:35, July 2021. ISSN 0360-0300. doi: 10.1145/3457607. URL https://doi.org/10.1145/3457607. ↗ our card
J. Menasce Horowitz, R. Igielnik, and R. Kochhar. Trends in U.S. income and wealth inequality. Technical report, Pew Research Center, January 2020. URL https://www.pewresearch.org/social-trends/2020/01/09 /trends-in-income-and-wealth-inequality/. ↗
S. Milano, M. Taddeo, and L. Floridi. Recommender systems and their ethical challenges. AI & SOCIETY, 35(4):957–967, December 2020. ISSN 1435-5655. doi: 10.1007/s00146-020-00950-y. URL https: //doi.org/10.1007/s00146-020-00950-y. — [doi:10.1007/s00146-020-00950-y](https://doi.org/10.1007/s00146-020-00950-y) find
T. Miller. Explanation in Artificial Intelligence: Insights from the Social Sciences. arXiv:1706.07269 [cs], August 2018. URL http://arxiv.org/abs/1706.07269. arXiv: 1706.07269. ↗ our card
T. Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267: 1–38, February 2019. ISSN 0004-3702. doi: 10.1016/j.artint.2018.07.007. URL https://www.sciencedirect. com/science/article/pii/S0004370218305988. ↗ our card
A. S. Miner, A. Milstein, S. Schueller, R. Hegde, C. Mangurian, and E. Linos. Smartphone-Based Conversational Agents and Responses to Questions About Mental Health, Interpersonal Violence, and Physical Health. JAMA internal medicine, 176(5):619–625, May 2016. ISSN 2168-6114. doi: 10.1001/jamainternm… — [doi:10.1001/jamainternmed.2016.0400](https://doi.org/10.1001/jamainternmed.2016.0400) find
S. Mohamed, M.-T. Png, and W. Isaac. Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence. Philosophy & Technology, 33(4):659–684, December 2020. ISSN 2210-5441. doi: 10.1007/s13347-020-00405-8. URL https://doi.org/10.1007/s13347-020-00405-8. — [doi:10.1007/s13347-020-00405-8](https://doi.org/10.1007/s13347-020-00405-8) ↗
W. Moncur, J. Masthoff, and E. Reiter. Facilitating benign deceit in mediated communication. In CHI ’09 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’09, pages 3383–3388, Boston, MA, USA, April 2009. Association for Computing Machinery. ISBN 9781605582474. doi: 10.1145/1520340.15… — [doi:10.1145/1520340.1520490](https://doi.org/10.1145/1520340.1520490) find
W. Moncur, J. Masthoff, E. Reiter, Y. Freer, and H. Nguyen. Providing Adaptive Health Updates Across the Personal Social Network. Human–Computer Interaction, 29(3):256–309, May 2014. ISSN 0737-0024. doi: 10.1080/07370024.2013.819218. URL https://doi.org/10.1080/07370024.2013.819218. — [doi:10.1080/07370024.2013.819218](https://doi.org/10.1080/07370024.2013.819218) ↗
A. A. Morgan-Lopez, A. E. Kim, R. F. Chew, and P. Ruddle. Predicting age groups of Twitter users based on language and metadata features. PLOS ONE, 12(8):e0183537, August 2017. ISSN 1932-6203. doi: 10.1371/ journal.pone.0183537. URL https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0… — [doi:10.1371/journal.pone.0183537](https://doi.org/10.1371/journal.pone.0183537) ↗
D. F. Mujtaba and N. R. Mahapatra. Ethical Considerations in AI-Based Recruitment. In 2019 IEEE International Symposium on Technology and Society (ISTAS), pages 1–7, November 2019. doi: 10.1109/ISTAS48451.2019. 8937920. ISSN: 2158-3412. — [doi:10.1109/istas48451.2019](https://doi.org/10.1109/istas48451.2019) find
M. Murgia. DeepMind’s Lila Ibrahim: ‘It’s hard not to go through imposter syndrome’. Financial Times, July 2021. URL https://www.ft.com/content/c43cd848-367a-4c67-8cf1-7f34d3eaf84e. ↗
D. Mytton. Data centre water consumption. NPJ Clean Water, 4(1):1–6, February 2021. ISSN 2059-7037. doi: 10.1038/s41545-021-00101-w. URL https://www.nature.com/articles/s41545-021-00101-w. — [doi:10.1038/s41545-021-00101-w](https://doi.org/10.1038/s41545-021-00101-w) ↗
J. Möller, D. Trilling, N. Helberger, and B. van Es. Do not blame it on the algorithm: an empirical assessment of multiple recommender systems and their impact on content diversity. Information, Communication & Society, 21(7):959–977, July 2018. ISSN 1369-118X. doi: 10.1080/1369118X.2018.1444076. UR… — [doi:10.1080/1369118x.2018.1444076](https://doi.org/10.1080/1369118x.2018.1444076) find
M. Nadeem, A. Bethke, and S. Reddy. StereoSet: Measuring stereotypical bias in pretrained language models. arXiv:2004.09456 [cs], April 2020. URL http://arxiv.org/abs/2004.09456. arXiv: 2004.09456. ↗ our card
N. Nangia, C. Vania, R. Bhalerao, and S. R. Bowman. CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models. arXiv:2010.00133 [cs], September 2020. URL http://arxiv.org/abs/20 10.00133. arXiv: 2010.00133. ↗ our card
A. Narayanan. How to Recognize AI Snake Oil, January 2021. URL https://www.cs.princeton.edu/news/how-recognize-ai-snake-oil. ↗
D. Nguyen, R. Gravel, D. Trieschnigg, and T. Meder. "How Old Do You Think I Am?" A Study of Language and Age in Twitter. Proceedings of the International AAAI Conference on Web and Social Media, 7(1):439–448, 2013. ISSN 2334-0770. URL https://ojs.aaai.org/index.php/ICWSM/article/view/14381. ↗
S. I. Nikolenko. Synthetic Data for Deep Learning, volume 174 of Springer Optimization and Its Applications. Springer International Publishing, Cham, 2021. ISBN 9783030751777 9783030751784. doi: 10.1007/978-3-030-75178-4. URL https://link.springer.com/10.1007/978-3-030-75178-4. — [doi:10.1007/978-3-030-75178-4](https://doi.org/10.1007/978-3-030-75178-4) ↗
NLP for Positive Impact 2021. Workshop on NLP for Positive Impact at ACL-IJCNLP, 2021. URL https: //sites.google.com/view/nlp4positiveimpact2021. find
S. U. Noble. Algorithms of Oppression. NYU Press, 2018. URL https://nyupress.org/9781479837243/algorithm s-of-oppression. ↗
D. Nozza, F. Bianchi, and D. Hovy. HONEST: Measuring Hurtful Sentence Completion in Language Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2398–2406, Online, June 2021. Association for… find
Z. Obermeyer, B. Powers, C. Vogeli, and S. Mullainathan. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464):447–453, October 2019. doi: 10.1126/science.aax2342. URL https://www.science.org/doi/abs/10.1126/science.aax2342. — [doi:10.1126/science.aax2342](https://doi.org/10.1126/science.aax2342) ↗
B. of Labor Statistics. Interpreters and Translators: Occupational Outlook Handbook. Technical report, U.S. Department of Labour, 2021. URL https://www.bls.gov/ooh/media-and-communication/interpreters-and-t ranslators.htm\#:~:text=3%25- , Employment%20of%20interpreters%20and%20translators%20is%20proj… ↗
K. Ognyanova, D. Lazer, R. E. Robertson, and C. Wilson. Misinformation in action: Fake news exposure is linked to lower trust in media, higher trust in government when your side is in power. Harvard Kennedy School Misinformation Review, June 2020. doi: 10.37016/mr-2020-024. URL https://misinforeview… — [doi:10.37016/mr-2020-024](https://doi.org/10.37016/mr-2020-024) find
P. Oosterhoff. Online censors are a barrier to sex education, 2016. URL https://www.scidev.net/global/opinio ns/online-censors-sex-education-porn/. ↗
D. O’Callaghan, D. Greene, M. Conway, J. Carthy, and P. Cunningham. Down the (White) Rabbit Hole: The Extreme Right and Online Recommender Systems. Social Science Computer Review, 33(4):459–478, August 2015. ISSN 0894-4393. doi: 10.1177/0894439314555329. URL https://doi.org/10.1177/089443931455 5329… — [doi:10.1177/0894439314555329](https://doi.org/10.1177/0894439314555329) ↗
G. PAIR. People + AI Guidebook. Google, May 2019. URL https://design.google/ai-guidebook. ↗
A. Pardes. The Emotional Chatbots Are Here to Probe Our Feelings. Wired, January 2018. ISSN 1059-1028. URL https://www.wired.com/story/replika-open-source/. ↗
G. Park, H. A. Schwartz, J. C. Eichstaedt, M. L. Kern, M. Kosinski, D. J. Stillwell, L. H. Ungar, and M. E. P. Seligman. Automatic personality assessment through social media language. Journal of Personality and Social Psychology, 108(6):934–952, June 2015. ISSN 1939-1315, 0022-3514. doi: 10.1037/ps… — [doi:10.1037/pspp0000020](https://doi.org/10.1037/pspp0000020) find
F. Pasquale. The Black Box Society. Harvard University Press, 2016. URL https://dl.acm.org/doi/abs/10.5555 /2717112. ↗
D. Patterson, J. Gonzalez, Q. Le, C. Liang, L.-M. Munguia, D. Rothchild, D. So, M. Texier, and J. Dean. Carbon Emissions and Large Neural Network Training. arXiv:2104.10350 [cs], April 2021. URL http: //arxiv.org/abs/2104.10350. arXiv: 2104.10350. find our card
D. Perez-Marin and I. Pascual-Nieto. Conversational Agents and Natural Language Interaction: Techniques and Effective Practices. Information Science Reference - Imprint of: IGI Publishing, Hershey, PA, 2011. ISBN 9781609606176. find
N. Persily and J. A. Tucker. Social Media and Democracy: The State of the Field, Prospects for Reform. Cambridge University Press, September 2020. ISBN 9781108858779. Google-Books-ID: TgH3DwAAQBAJ. find
PerspectiveAPI. Perspective API \| Developers, 2021. URL https://support.perspectiveapi.com/s/about-the-api -attributes-and-languages. ↗
J. Pfeiffer, I. Vulić, I. Gurevych, and S. Ruder. UNKs Everywhere: Adapting Multilingual Language Models to New Scripts. arXiv:2012.15562 [cs], September 2021. URL http://arxiv.org/abs/2012.15562. arXiv: 2012.15562. ↗
A. Pilipiszyn. GPT-3 Powers the Next Generation of Apps, March 2021. URL https://openai.com/blog/gpt-3-apps/. publisher: OpenAI. ↗
D. Preoţiuc-Pietro, Y. Liu, D. Hopkins, and L. Ungar. Beyond Binary Labels: Political Ideology Prediction of Twitter Users. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 729–740, Vancouver, Canada, 2017. Association for Comp… find
K. Quach. Researchers made an OpenAI GPT-3 medical chatbot as an experiment. It told a mock patient to kill themselves. The Register, October 2020. URL https://www.theregister.com/2020/10/28/gpt3\_medical\_cha tbot\_experiment/. ↗
D. Quercia, M. Kosinski, D. Stillwell, and J. Crowcroft. Our Twitter Profiles, Our Selves: Predicting Personality with Twitter. In 2011 IEEE Third International Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third International Conference on Social Computing, pages 180–185, October 20… — [doi:10.1109/passat/s](https://doi.org/10.1109/passat/s) find
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving Language Understanding by Generative Pre-Training. 2018a. find
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language Models are Unsupervised Multitask Learners. 2018b. find
J. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P.-S. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor… find
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv:1910.10683 [cs, stat], July 2020. URL http://arxiv.org/abs/1910.10683. arXiv: 1910.10683. ↗
I. D. Raji. Handle with Care: Lessons for Data Science from Black Female Scholars. Patterns, 1(8):100150, November 2020. ISSN 2666-3899. doi: 10.1016/j.patter.2020.100150. URL https://www.sciencedirect.co m/science/article/pii/S2666389920301987. — [doi:10.1016/j.patter.2020.100150](https://doi.org/10.1016/j.patter.2020.100150) ↗
I. D. Raji, A. Smart, R. N. White, M. Mitchell, T. Gebru, B. Hutchinson, J. Smith-Loud, D. Theron, and P. Barnes. Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing. arXiv:2001.00973 [cs], January 2020. URL http://arxiv.org/abs/2001.00973. arXiv: 20… — [arXiv:2001.00973](https://arxiv.org/abs/2001.00973) ↗
S. Ramaswamy, O. Thakkar, R. Mathews, G. Andrew, H. B. McMahan, and F. Beaufays. Training Production Language Models without Memorizing User Data. arXiv:2009.10031 [cs, stat], September 2020. URL http://arxiv.org/abs/2009.10031. arXiv: 2009.10031. — [arXiv:2009.10031](https://arxiv.org/abs/2009.10031) ↗
P. Ranade, A. Piplai, S. Mittal, A. Joshi, and T. Finin. Generating Fake Cyber Threat Intelligence Using Transformer-Based Models. arXiv:2102.04351 [cs], June 2021. URL http://arxiv.org/abs/2102.04351. arXiv: 2102.04351. — [arXiv:2102.04351](https://arxiv.org/abs/2102.04351) ↗
E. Rand. Reclaiming Queer: Activist & Academic Rhetorics of Resistance. University of Alabama Press, 2014. find
E. Reiter. Could NLG systems injure or even kill people?, October 2020. URL https://ehudreiter.com/2020/10 /20/could-nlg-systems-injure-or-even-kill-people/. ↗
M. Rimmer. Patent-Busting: The Public Patent Foundation, Gene Patents and the Seed Wars. In C. Lawson and J. Sanderson, editors, The Intellectual Property and Food Project. Routledge, 2013. find
A. Romano. A group of YouTubers is claiming the site systematically demonetizes queer content. Vox, October 2019. URL https://www.vox.com/culture/2019/10/10/20893258/youtube-lgbtq-censorship-demonetizatio n-nerd-city-algorithm-report. ↗
J. Rosa and N. Flores. Unsettling race and language: Toward a raciolinguistic perspective. Language in Society, 46(5):621–647, November 2017. ISSN 0047-4045, 1469-8013. doi: 10.1017/S0047404517000562. URL https://www.cambridge.org/core/journals/language-in-society/article/abs/unsettling-race-and-lan… — [doi:10.1017/s0047404517000562](https://doi.org/10.1017/s0047404517000562) ↗
C. Rosset. Turing-NLG: A 17-billion-parameter language model by Microsoft, February 2020. URL https://ww w.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/. find
A. Rubel, A. Pham, and C. Castro. Agency Laundering and Algorithmic Decision Systems. In N. G. Taylor, C. Christian-Lamb, M. H. Martin, and B. Nardi, editors, Information in Contemporary Society, Lecture Notes in Computer Science, pages 590–598, Cham, 2019. Springer International Publishing. ISBN 97… — [doi:10.1007/978-3-030-15742-5\_56](https://doi.org/10.1007/978-3-030-15742-5_56) find
S. Ruder. Why You Should Do NLP Beyond English, August 2020. URL https://ruder.io/nlp-beyond-english/. ↗
A. Sabeti. Teaching GPT-3 to Identify Nonsense, July 2020. URL https://arr.am/2020/07/25/gpt-3-uncertaint y-prompts/. ↗
N. A. Sales. Secrecy and National Security Investigations. Alabama Law Review, 58:811, 2006. URL https: //heinonline.org/HOL/Page?handle=hein.journals/bamalr58&id=821&div=&collection=. find
N. Sambasivan and J. Holbrook. Toward responsible AI for the next billion users. Interactions, 26(1):68–71, December 2018. ISSN 1072-5520. doi: 10.1145/3298735. URL https://doi.org/10.1145/3298735. — [doi:10.1145/3298735](https://doi.org/10.1145/3298735) ↗
N. Sambasivan, E. Arnesen, B. Hutchinson, T. Doshi, and V. Prabhakaran. Re-imagining Algorithmic Fairness in India and Beyond. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, pages 315–328, Virtual Event, Canada, March 2021. Association for Computi… — [doi:10.1145/3442188.3445896](https://doi.org/10.1145/3442188.3445896) find
M. Sap, D. Card, S. Gabriel, Y. Choi, and N. A. Smith. The Risk of Racial Bias in Hate Speech Detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v… find our card
K. M. Saunders. The Law and Ethics of Trade Secrets: A Case Study. California Western Law Review, 42:209, 2005. URL https://heinonline.org/HOL/Page?handle=hein.journals/cwlr42&id=215&div=&collection=. ↗
A. Schmidt and M. Wiegand. A Survey on Hate Speech Detection using Natural Language Processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1–10, Valencia, Spain, 2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-1101.… find our card
A. T. Schmidt and B. Engelen. The ethics of nudging: An overview. Philosophy Compass, 15(4):e12658, April 2020. ISSN 1747-9991, 1747-9991. doi: 10.1111/phc3.12658. URL https://onlinelibrary.wiley.com/doi/10 .1111/phc3.12658. — [doi:10.1111/phc3.12658](https://doi.org/10.1111/phc3.12658) ↗
B. Schneier. Bots Are Destroying Political Discourse As We Know It. The Atlantic, January 2020. URL https:// www.theatlantic.com/technology/archive/2020/01/future-politics-bots-drowning-out-humans/604489/. find
M. Schroepfer. How AI is getting better at detecting hate speech, November 2020. URL https://ai.facebook.co m/blog/how-ai-is-getting-better-at-detecting-hate-speech/. ↗
R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni. Green AI. Communications of the ACM, 63(12):54–63, November 2020. ISSN 0001-0782. doi: 10.1145/3381831. URL https://doi.org/10.1145/3381831. ↗ our card
E. Seger, S. Avin, G. Pearson, M. Briers, S. Ó Heigeartaigh, and H. Bacon. Tackling threats to informed decision-making in democratic societies. Technical report, Alan Turing Institute, 2020. URL https://www.tu ring.ac.uk/research/publications/tackling-threats-informed-decision-making-democratic-soc… ↗
D. Shah, H. A. Schwartz, and D. Hovy. Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5248–5264, 2020. doi: 10.18653/v1/2020.acl-main.468. URL http://arxiv.org… ↗
A. Shahbaz and A. Funk. Social Media Surveillance. Technical report, Freedom House, 2019. URL https://free domhouse.org/report/freedom-on-the-net/2019/the-crisis-of-social-media/social-media-surveillance. find
C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379–423, July 1948. ISSN 0005-8580. doi: 10.1002/j.1538-7305.1948.tb01338.x. — [doi:10.1002/j.1538-7305.1948.tb01338.x](https://doi.org/10.1002/j.1538-7305.1948.tb01338.x) find
E. Sheng, K.-W. Chang, P. Natarajan, and N. Peng. Societal Biases in Language Generation: Progress and Challenges. arXiv:2105.04054 [cs], June 2021. URL http://arxiv.org/abs/2105.04054. arXiv: 2105.04054. ↗ our card
M. Sloane, E. Moss, O. Awomolo, and L. Forlano. Participation is not a Design Fix for Machine Learning. arXiv:2007.02423 [cs], August 2020. URL http://arxiv.org/abs/2007.02423. arXiv: 2007.02423. — [arXiv:2007.02423](https://arxiv.org/abs/2007.02423) ↗
I. Solaiman and C. Dennison. Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets. arXiv:2106.10328 [cs], June 2021. URL http://arxiv.org/abs/2106.10328. arXiv: 2106.10328. ↗
I. Solaiman, M. Brundage, J. Clark, A. Askell, A. Herbert-Voss, J. Wu, A. Radford, G. Krueger, J. W. Kim, S. Kreps, M. McCain, A. Newhouse, J. Blazakis, K. McGuffie, and J. Wang. Release Strategies and the Social Impacts of Language Models. arXiv:1908.09203 [cs], November 2019. URL http://arxiv.org/… ↗ our card
K. Sparck Jones. Language modelling’s generative model: is it rational? Computer Laboratory, University of Cambridge, Cambridge, UK, 2004. find
D. Sravani, L. Kameswari, and R. Mamidi. Political Discourse Analysis: A Case Study of Code Mixing and Code Switching in Political Speeches. In Proceedings of the Fifth Workshop on Computational Approaches to Linguistic Code-Switching, pages 1–5, Online, June 2021. Association for Computational Ling… find
T. Stahl. Indiscriminate mass surveillance and the public sphere. Ethics and Information Technology, 18(1): 33–39, March 2016. ISSN 1572-8439. doi: 10.1007/s10676-016-9392-2. URL https://doi.org/10.1007/s1 0676-016-9392-2. — [doi:10.1007/s10676-016-9392-2](https://doi.org/10.1007/s10676-016-9392-2) find
StanfordNaturalProcessingGroup. The Stanford Natural Language Processing Group, 2021. URL https: //nlp.stanford.edu/projects/coref.shtml. find
J. Stilgoe. Who’s driving innovation. New Technologies and the Collaborative State. Cham, Switzerland: Palgrave Macmillan, 2020. find
J. Stilgoe, R. Owen, and P. Macnaghten. Developing a framework for responsible innovation. Research Policy, 42(9):1568–1580, November 2013. ISSN 0048-7333. doi: 10.1016/j.respol.2013.05.008. URL https://www.sciencedirect.com/science/article/pii/S0048733313000930. — [doi:10.1016/j.respol.2013.05.008](https://doi.org/10.1016/j.respol.2013.05.008) ↗
E. Strubell, A. Ganesh, and A. McCallum. Energy and Policy Considerations for Deep Learning in NLP. arXiv:1906.02243 [cs], June 2019. URL http://arxiv.org/abs/1906.02243. arXiv: 1906.02243. ↗ our card
S. Sullivan and N. Tuana, editors. Race and epistemologies of ignorance. SUNY series, philosophy and race. State University of New York Press, Albany, 2007. ISBN 9780791471012 9780791471029. OCLC: ocm70676503. summerstay on Reddit. Fiction by Neil Gaiman and Terry Pratchett by GPT-3, July 2020. URL… find
Y. Sun, S. Wang, S. Feng, S. Ding, C. Pang, J. Shang, J. Liu, X. Chen, Y. Zhao, Y. Lu, W. Liu, Z. Wu, W. Gong, J. Liang, Z. Shang, P. Sun, W. Liu, X. Ouyang, D. Yu, H. Tian, H. Wu, and H. Wang. ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation. arXiv:21… find our card
A. Tamkin, M. Brundage, J. Clark, and D. Ganguli. Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models. arXiv:2102.02503 [cs], February 2021. URL http://arxiv.org/abs/2102.0 2503. arXiv: 2102.02503. ↗
R. S. J. Tol. The impact of climate change and the social cost of carbon. In Routledge Handbook of Energy Economics. Routledge, 2019. find
N. Tomasev, K. R. McKee, J. Kay, and S. Mohamed. Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communities. Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 254–265, July 2021. doi: 10.1145/3461702.3462540. URL http://arxiv.org/abs… — [doi:10.1145/3461702.3462540](https://doi.org/10.1145/3461702.3462540) ↗
J. M. Twenge. More Time on Technology, Less Happiness? Associations Between Digital-Media Use and Psychological Well-Being. Current Directions in Psychological Science, 28(4):372–379, August 2019. ISSN 0963-7214. doi: 10.1177/0963721419838244. URL https://doi.org/10.1177/0963721419838244. — [doi:10.1177/0963721419838244](https://doi.org/10.1177/0963721419838244) ↗
E. Van den Broeck, B. Zarouali, and K. Poels. Chatbot advertising effectiveness: When does the message get through? Computers in Human Behavior, 98:150–157, September 2019. ISSN 0747-5632. doi: 10.1016/j.chb.2019.04.009. URL https://www.sciencedirect.com/science/article/pii/S0747563219301499. — [doi:10.1016/j.chb.2019.04.009](https://doi.org/10.1016/j.chb.2019.04.009) ↗
R. Van Noorden. Publishers withdraw more than 120 gibberish papers. Nature, February 2014. ISSN 1476-4687. doi: 10.1038/nature.2014.14763. URL https://www.nature.com/articles/nature.2014.14763. — [doi:10.1038/nature.2014.14763](https://doi.org/10.1038/nature.2014.14763) ↗
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://papers.nips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa… ↗
VersebyVerse. Verse by Verse, 2020. URL https://sites.research.google/versebyverse/. publisher:. ↗
J. Vincent. The invention of AI ‘gaydar’ could be the start of something much worse. The Verge, September 2017. URL https://www.theverge.com/2017/9/21/16332760/ai-sexuality-gaydar-photo-physiognomy. ↗
K. Vredenburgh. The Right to Explanation. Journal of Political Philosophy, 0(0):1–21, 2021. ISSN 1467-9760. doi: 10.1111/jopp.12262. URL https://onlinelibrary.wiley.com/doi/abs/10.1111/jopp.12262. — [doi:10.1111/jopp.12262](https://doi.org/10.1111/jopp.12262) ↗
C. Véliz. Privacy matters because it empowers us all \| Aeon Essays. Aeon, September 2019. URL https: //aeon.co/essays/privacy-matters-because-it-empowers-us-all. find
B. Vézina and S. Hinchcliff Pearson. Should CC-Licensed Content be Used to Train AI? It Depends., March 2021. URL https://creativecommons.org/2021/03/04/should-cc-licensed-content-be-used-to-train-ai-it-d epends/. ↗
D. Wallace, F. Tramer, M. Jagielski, and A. Herbert-Voss. Does GPT-2 Know Your Phone Number?, December 2020. URL http://bair.berkeley.edu/blog/2020/12/20/lmmem/. ↗
A. Wang and O. Russakovsky. Directional Bias Amplification. arXiv:2102.12594 [cs], June 2021. URL http://arxiv.org/abs/2102.12594. arXiv: 2102.12594. — [arXiv:2102.12594](https://arxiv.org/abs/2102.12594) ↗
A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537, 2019a. find our card
Y. Wang and M. Kosinski. Deep neural networks are more accurate than humans at detecting sexual orientation from facial images. Journal of Personality and Social Psychology, 114(2):246–257, February 2018. ISSN 1939-1315, 0022-3514. doi: 10.1037/pspa0000098. URL http://doi.apa.org/getdoi.cfm?doi=10.1… — [doi:10.1037/pspa0000098](https://doi.org/10.1037/pspa0000098) ↗
Y. Wang, M. McKee, A. Torbica, and D. Stuckler. Systematic Literature Review on the Spread of Health-related Misinformation on Social Media. Social Science & Medicine, 240:112552, November 2019b. ISSN 0277-9536. doi: 10.1016/j.socscimed.2019.112552. URL https://www.sciencedirect.com/science/article/… — [doi:10.1016/j.socscimed.2019.112552](https://doi.org/10.1016/j.socscimed.2019.112552) ↗
Z. Wang, A. W. Yu, O. Firat, and Y. Cao. Towards Zero-Label Language Learning. arXiv:2109.09193 [cs], September 2021. URL http://arxiv.org/abs/2109.09193. arXiv: 2109.09193. — [arXiv:2109.09193](https://arxiv.org/abs/2109.09193) ↗
M. Webb. The Impact of Artificial Intelligence on the Labor Market. SSRN Scholarly Paper ID 3482150, Social Science Research Network, Rochester, NY, November 2019. URL https://papers.ssrn.com/abstract=3482150. ↗
J. Welbl, A. Glaese, J. Uesato, S. Dathathri, J. Mellor, L. A. Hendricks, K. Anderson, P. Kohli, B. Coppin, and P.-S. Huang. Challenges in Detoxifying Language Models. arXiv:2109.07445 [cs], September 2021. URL http://arxiv.org/abs/2109.07445. arXiv: 2109.07445. ↗
T.-H. Wen, D. Vandyke, N. Mrksic, M. Gasic, L. M. Rojas-Barahona, P.-H. Su, S. Ultes, and S. Young. A Network-based End-to-End Trainable Task-oriented Dialogue System. arXiv:1604.04562 [cs, stat], April 2017. URL http://arxiv.org/abs/1604.04562. arXiv: 1604.04562. ↗
M. West, R. Kraut, and H. Ei Chew. I’d blush if I could : closing gender divides in digital skills through education. Technical report, UNESCO, 2019. URL https://repositorio.minedu.gob.pe/handle/20.500.12799/6598. ↗
G. I. Winata, A. Madotto, Z. Lin, R. Liu, J. Yosinski, and P. Fung. Language Models are Few-shot Multilingual Learners. arXiv:2109.07684 [cs], September 2021. URL http://arxiv.org/abs/2109.07684. arXiv: 2109.07684. ↗
L. Winner. Do Artifacts Have Politics? Daedalus, 109(1):121–136, 1980. ISSN 0011-5266. URL https: //www.jstor.org/stable/20024652. find
A. Xu, E. Pathak, E. Wallace, S. Gururangan, M. Sap, and D. Klein. Detoxifying Language Models Risks Marginalizing Minority Voices. arXiv:2104.06390 [cs], April 2021. URL http://arxiv.org/abs/2104.06390. arXiv: 2104.06390. ↗
E. Yang and M. E. Roberts. Censorship of Online Encyclopedias: Implications for NLP Models. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, pages 537–548, Virtual Event, Canada, March 2021. Association for Computing Machinery. ISBN 9781450383097. d… — [doi:10.1145/3442188.3445916](https://doi.org/10.1145/3442188.3445916) find
Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://papers.nips.cc/paper/2019/hash/dc6a7e655d7e… ↗ our card
K. Yee, U. Tantipongpipat, and S. Mishra. Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2):1–24, October 2021. ISSN 2573-0142. doi: 10.1145/3479594. URL http: //arx… — [doi:10.1145/3479594](https://doi.org/10.1145/3479594) find
M. Yesilada and S. Lewandowsky. A systematic review: The YouTube recommender system and pathways to problematic content. PsyArxiv, June 2021. URL https://psyarxiv.com/6pv5c/. ↗
W. Youyou, M. Kosinski, and D. Stillwell. Computer-based personality judgments are more accurate than those made by humans. Proceedings of the National Academy of Sciences, 112(4):1036–1040, January 2015. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas.1418680112. URL https://www.pnas.org/content/112/4… — [doi:10.1073/pnas.1418680112](https://doi.org/10.1073/pnas.1418680112) ↗
S. Zdenek. “Just Roll Your Mouse Over Me”: Designing Virtual Women for Customer Service on the Web. Technical Communication Quarterly, 16(4):397–430, August 2007. ISSN 1057-2252, 1542-7625. doi: 10.1080/10572250701380766. URL http://www.tandfonline.com/doi/abs/10.1080/10572250701380766. — [doi:10.1080/10572250701380766](https://doi.org/10.1080/10572250701380766) ↗
R. Zellers, A. Holtzman, H. Rashkin, Y. Bisk, A. Farhadi, F. Roesner, and Y. Choi. Defending Against Neural Fake News. arXiv:1905.12616 [cs], December 2020. URL http://arxiv.org/abs/1905.12616. arXiv: 1905.12616. ↗ our card
C. Zhang, P. Benz, C. Lin, A. Karjauv, J. Wu, and I. S. Kweon. A Survey On Universal Adversarial Attack. arXiv:2103.01498 [cs], March 2021a. URL http://arxiv.org/abs/2103.01498. arXiv: 2103.01498. — [arXiv:2103.01498](https://arxiv.org/abs/2103.01498) ↗
N. Zhang, L. Li, X. Chen, S. Deng, Z. Bi, C. Tan, F. Huang, and H. Chen. Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners. arXiv:2108.13161 [cs], October 2021b. URL http://arxiv.org/abs/2108.13161. arXiv: 2108.13161. — [arXiv:2108.13161](https://arxiv.org/abs/2108.13161) ↗
J. Zhao, T. Wang, M. Yatskar, V. Ordonez, and K.-W. Chang. Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints. arXiv:1707.09457 [cs, stat], July 2017. URL http: //arxiv.org/abs/1707.09457. arXiv: 1707.09457. find our card
J. Zhao, T. Wang, M. Yatskar, R. Cotterell, V. Ordonez, and K.-W. Chang. Gender Bias in Contextualized Word Embeddings. arXiv:1904.03310 [cs], April 2019. URL http://arxiv.org/abs/1904.03310. arXiv: 1904.03310. ↗ our card
T. Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh. Calibrate Before Use: Improving Few-Shot Performance of Language Models. arXiv:2102.09690 [cs], June 2021. URL http://arxiv.org/abs/2102.09690. arXiv: 2102.09690. ↗ our card
J. Zou and L. Schiebinger. AI can be sexist and racist — it’s time to make it fair. Nature, 559(7714):324–326, July 2018. doi: 10.1038/d41586-018-05707-8. URL https://www.nature.com/articles/d41586-018-05707-8. — [doi:10.1038/d41586-018-05707-8](https://doi.org/10.1038/d41586-018-05707-8) ↗
S. Zuboff. The age of surveillance capitalism: the fight for a human future at the new frontier of power. Profile books, London, 2019. ISBN 9781781256848 9781781256855. find
J. Złotowski, D. Proudfoot, K. Yogeeswaran, and C. Bartneck. Anthropomorphism: Opportunities and Challenges in Human–Robot Interaction. International Journal of Social Robotics, 7(3):347–360, June 2015. ISSN 1875-4805. doi: 10.1007/s12369-014-0267-6. URL https://doi.org/10.1007/s12369-014-0267-6. — [doi:10.1007/s12369-014-0267-6](https://doi.org/10.1007/s12369-014-0267-6) ↗