Elamur Research⌕
☰
home / papers / 2019 / A Just and Comprehensive Strategy for Using NLP to Address Online Abuse

A Just and Comprehensive Strategy for Using NLP to Address Online Abuse

Paper · 2019 · David Jurgens, Libby Hemphill and Eshwar Chandrasekharan

Summary

This ACL 2019 position paper argues that NLP work on online abuse has focused on a thin slice of the problem, mainly overt hate speech and abusive language. It also judges success mostly by classification accuracy, which leaves many victims without recognition or remedy. The authors, from the University of Michigan and Georgia Tech, do not present new data or models. Instead, they propose an agenda built on three shifts.

First, the field should widen what counts as abuse in two directions. One is rare but physically dangerous behaviour, such as doxxing, swatting, trafficking or incitement to violence. The other is frequent but subtle harm, such as condescension, minimization and microaggressions. Detection should also respect each community's own norms.

Second, research should move from reactive detect-and-remove pipelines to proactive tools. These would forecast conversations that are escalating and nudge authors toward less harmful wording.

Third, the whole effort should sit within justice frameworks (social, restorative and procedural), so that systems promote equitable participation and can explain their decisions.

The paper matters because it treats abuse detection as a sociotechnical problem and turns each shift into concrete new NLP tasks.

Key points

  • The authors cite surveys reporting that about 40% of Internet users have been targets of online abuse, with underrepresented groups targeted more often.
  • The paper presents a spectrum of abuse ordered by frequency and real-world risk. It runs from microaggressions and condescension through insults and hate speech to threats, self-harm promotion and doxxing. The authors note that current methods cover only the middle of this range.
  • For shared tasks, the authors point to the Chandrasekharan et al. Reddit dataset. It contains over 40K removed comments, each labeled with the community norm it broke.
  • On proactive intervention, the authors note that prior work already predicts derailing toxic conversations on Wikipedia (Zhang et al., 2018) and bullying on Instagram (Liu et al., 2018). They suggest combining such forecasts with explainable ML and style transfer to rewrite offensive text.
  • The proposed justice lens has three parts:
    • Social justice: a capabilities approach based on Nussbaum's 10 capabilities.
    • Restorative justice: outcomes such as apology and reconciliation rather than only bans.
    • Procedural justice: moderation decisions that are transparent and explainable, so users see them as legitimate.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgia Institute of TechnologyAtlanta · United States69 works · 22,788 citations🎓University of MichiganAnn Arbor · United States36 works · 12,181 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Social Sciences · 744 works Safety Research · 221 works Sociology and Political Science · 221 works

Cite

Cite as David Jurgens, Libby Hemphill and Eshwar Chandrasekharan. 2019. A Just and Comprehensive Strategy for Using NLP to Address Online Abuse. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1357

@inproceedings{jurgens2019just,
  title = {{A Just and Comprehensive Strategy for Using NLP to Address Online Abuse}},
  author = {David Jurgens, Libby Hemphill and Eshwar Chandrasekharan},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1357},
  url = {https://doi.org/10.18653/v1/p19-1357}
}

References

83 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Keith Allan. 2015. When is a slur not a slur? the use of nigger in ‘pulp fiction’. Lang. Sci., 52:187–199. find
  2. Anti-Defamation League. 2019. Online hate and harassment: The american experience. https://www.adl.org/onlineharassment. Accessed: 2019-3-4. ↗
  3. Lindsay Blackwell, Jill Dimond, Sarita Schoenebeck, and Cliff Lampe. 2017. Classification and its consequences for online harassment: Design insights from heartmob. Proceedings of the ACM on Human-Computer Interaction, 1(CSCW):24. find
  4. Geoffrey C Bowker and Susan Leigh Star. 2000. Sorting things out: Classification and its consequences. MIT press. find
  5. John Braithwaite. 2002. Restorative Justice & Responsive Regulation. Oxford University Press. find
  6. Julian Brooke, Tong Wang, and Graeme Hirst. 2010. Automatic acquisition of lexical formality. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 90–98. Association for Computational Linguistics. find
  7. Samuel Carton, Qiaozhu Mei, and Paul Resnick. 2018. Extractive adversarial networks: High-recall explanations for identifying personal attacks in social media posts. In Proceedings of EMNLP. find
  8. Eshwar Chandrasekharan and Eric Gilbert. 2019. Hybrid approaches to detect comments violating macro norms on reddit. arXiv preprint arXiv:1904.03596. — [arXiv:1904.03596](https://arxiv.org/abs/1904.03596) find
  9. Eshwar Chandrasekharan, Umashanthi Pavalanathan, Anirudh Srinivasan, Adam Glynn, Jacob Eisenstein, and Eric Gilbert. 2017. You can’t stay here: The efficacy of reddit’s 2015 ban examined through hate speech. Proceedings of the ACM on Human-Computer Interaction, 1(CSCW):31. find
  10. Eshwar Chandrasekharan, Mattia Samory, Shagun Jhaver, Hunter Charvat, Amy Bruckman, Cliff Lampe, Jacob Eisenstein, and Eric Gilbert. 2018. The internet’s hidden rules: An empirical study of reddit norm violations at micro, meso, and macro scales. Proceedings of the ACM on Human-Computer Interaction,… find
  11. Niyati Chhaya, Kushal Chawla, Tanya Goyal, Projjal Chanda, and Jaya Singh. 2018. Frustrated, polite, or formal: Quantifying feelings and tone in email. In Proceedings of the Second Workshop on Computational Modeling of Peoples Opinions, Personality, and Emotions in Social Media, pages 76–86. our card
  12. Robert B Cialdini and Melanie R Trost. 1998. Social influence: Social norms, conformity and compliance. In D. T. Gilbert, S. T. Fiske, and G. Lindzey, editors, The handbook of social psychology, pages 151–192. McGraw-Hill. find
  13. Jonathan Culpeper. 1996. Towards an anatomy of impoliteness. J. Pragmat., 25(3):349–367. find
  14. Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A computational approach to politeness with application to social factors. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). our card
  15. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Eleventh International AAAI Conference on Web and Social Media. our card
  16. Kelly P Dillon and Brad J Bushman. 2015. Unresponsive or un-noticed?: Cyberbystander intervention in an experimental cyberbullying context. Computers in Human Behavior, 45:144–150. find
  17. Karthik Dinakar, Birago Jones, Catherine Havasi, Henry Lieberman, and Rosalind Picard. 2012. Common sense reasoning for detection, prevention, and mitigation of cyberbullying. ACM Transactions on Interactive Intelligent Systems (TiiS), 2(3):18. find
  18. Bruce Drake. 2014. The darkest side of online harassment: Menacing behavior. Pew Research Center, http://www.pewresearch.org/fact-tank/2015/06/01/the-darkest-side-of-online-harassment-menacing-behavior/. ↗
  19. Maeve Duggan. 2017. Online harassment 2017. find
  20. Jill Filipovic. 2007. Blogging while female: How internet misogyny parallels “Real-World” harassment. Yale J. Law Fem., 19(1). find
  21. Paula Fortuna and Sérgio Nunes. 2018. A survey on automatic detection of hate speech in text. ACM Computing Surveys (CSUR), 51(4):85. find
  22. Michel Foucault. 1972. The Archaeology of Knowledge & The Discourse on Language. Pantheon Books, New York. find
  23. Nancy Fraser. 1990. Rethinking the public sphere: A contribution to the critique of actually existing democracy. Social Text, (25/26):56–80. find
  24. Lei Gao and Ruihong Huang. 2017. Detecting online hate speech using context aware models. In Proceedings of RANLP. find
  25. Lei Gao, Alexis Kuppersmith, and Ruihong Huang. 2017. Recognizing explicit and implicit hate speech using a weakly supervised two-path bootstrapping approach. In Proceedings of ICJNLP. find
  26. Tarleton Gillespie. 2018. Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media. Yale University Press. find
  27. Peter Glick and Susan T Fiske. 2001. An ambivalent alliance: Hostile and benevolent sexism as complementary justifications for gender inequality. American psychologist, 56(2):109. find
  28. John Mordechai Gottman. 1999. The marriage clinic: A scientifically-based marital therapy. WW Norton & Company. find
  29. Joshua Guberman and Libby Hemphill. 2017. Challenges in modifying existing scales for detecting harassment in individual tweets. In Proceedings of the 50th Hawaii International Conference on System Sciences. find
  30. Susan Herring, Kirk Job-Sluder, Rebecca Scheckler, and Sasha Barab. 2002. Searching for safety online: Managing” trolling” in a feminist forum. The information society, 18(5):371–384. find
  31. Thomas J Holt, Kristie R Blevins, and Natasha Burkert. 2010. Considering the pedophile subculture online. Sexual Abuse, 22(1):3–24. find
  32. Elizabeth M Jaffe. 2016. Swatting: the new cyberbullying frontier after elonis v. united states. Drake L. Rev., 64:455. find
  33. Akshita Jha and Radhika Mamidi. 2017. When does a compliment become sexist? analysis and classification of ambivalent sexism using twitter data. In Proceedings of the second workshop on NLP and computational social science, pages 7–16. find
  34. Martin Luther King. 1963. Letter from a birmingham jail. find
  35. Varada Kolhatkar and Maite Taboada. 2017. Constructive language in news comments. In Proceedings of the First Workshop on Abusive Language Online, pages 11–17. find
  36. Rachael Krishna. 2018. Tumblr launched an algorithm to flag porn and so far it’s just caused chaos, dec 2018. https://www.buzzfeednews.com/ article/ krishrach/ tumblr-porn-algorithm-ban. ↗
  37. Mark Latonero. 2011. Human trafficking online: The role of social networking sites and online classifieds. Available at SSRN. find
  38. Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of EMNLP. our card
  39. Ping Liu, Joshua Guberman, Libby Hemphill, and Aron Culotta. 2018. Forecasting the presence and intensity of hostility on instagram using linguistic and social features. In Twelfth International AAAI Conference on Web and Social Media. find
  40. Patrick M Markey. 2000. Bystander intervention in computer-mediated communication. Computers in Human Behavior, 16(2):183–188. find
  41. Binny Mathew, Hardik Tharad, Subham Rajgaria, Prajwal Singhania, Suman Kalyan Maity, Pawan Goyal, and Animesh Mukherje. 2018. Thou shalt not hate: Countering online hate speech. arXiv preprint arXiv:1808.04409. — [arXiv:1808.04409](https://arxiv.org/abs/1808.04409) find
  42. Timothy McLaughlin. 2018. How whatsapp fuels fake news and violence in india. https://www.wired.com/story/how-whatsapp-fuels-fake-news-and-violence-in-india/. ↗
  43. Pushkar Mishra, Marco Del Tredici, Helen Yannakoudakis, and Ekaterina Shutova. 2018. Author profiling for abuse detection. In Proceedings of the 27th International Conference on Computational Linguistics (COLING), pages 1088–1098. find
  44. Kevin Munger. 2017. Tweetment effects on the tweeted: Experimentally reducing racist harassment. Political Behavior, 39(3):629–649. find
  45. Dasa Munkova, Michal Munk, and Zuzana Fráterová. 2013. Identifying social and expressive factors in request texts using transaction/sequence model. In Proceedings of RANLP, pages 496–503. find
  46. Kevin L. Nadal, Katie E. Griffin, Yinglee Wong, Sahran Hamit, and Morgan Rasmus. 2014. The impact of racial microaggressions on mental health: Counseling implications for clients of color. Journal of Counseling and Development, 92(1):57–66. find
  47. Courtney Napoles, Joel Tetreault, Aasish Pappu, Enrica Rosato, and Brian Provenzale. 2017. Finding good conversations online: The yahoo news annotated comments corpus. In Proceedings of the 11th Linguistic Annotation Workshop, pages 13–23. find
  48. Chikashi Nobata, Joel Tetreault, Achint Thomas, Yashar Mehdad, and Yi Chang. 2016. Abusive language detection in online user content. In Proceedings of the 25th International Conference on World Wide Web, WWW ’16, pages 145–153, Republic and Canton of Geneva, Switzerland. International World Wide We… find
  49. Safiya Umoja Noble. 2018. Algorithms of Oppression: How Search Engines Reinforce Racism. NYU Press. find
  50. David Noever. 2018. Machine learning suites for online toxicity detection. arXiv preprint arXiv:1810.01869. — [arXiv:1810.01869](https://arxiv.org/abs/1810.01869) find
  51. Martha Nussbaum. 2003. Capabilities as fundamental entitlements: Sen and social justice. Feminist Economics, 9(2-3):33–59. find
  52. Ray Oshikawa, Jing Qian, and William Yang Wang. 2018. A survey on natural language processing for fake news detection. arXiv preprint arXiv:1811.00770. news media. In Proceedings of the Twelfth International AAAI Conference on Web and Social Media (ICWSM). find
  53. Zizi Papacharissi. 2004. Democracy online: Civility, politeness, and the democratic potential of online political discussion groups. New media & society, 6(2):259–283. find
  54. Jessica Annette Pater, Yacin Nadji, Elizabeth D Mynatt, and Amy S Bruckman. 2014. Just awful enough: the functional dysfunction of the something awful forums. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems, pages 2407–2410. ACM. find
  55. Ellie Pavlick and Joel Tetreault. 2016. An empirical analysis of formality in online communication. Transactions of the Association of Computational Linguistics (TACL), 4(1):61–74. our card
  56. Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In Proceedings of ACL. our card
  57. Jing Qian, Mai ElSherief, Elizabeth Belding, and William Yang Wang. 2018. Leveraging intra-user and inter-user representation learning for automated hate speech detection. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pa… find
  58. Manoel Horta Ribeiro, Pedro H Calais, Yuri A Santos, Virgı́lio AF Almeida, and Wagner Meira Jr. 2018. Characterizing and detecting hateful users on twitter. In Twelfth International AAAI Conference on Web and Social Media. find
  59. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of KDD, pages 1135–1144. ACM. our card
  60. Mattia Samory and Tanushree Mitra. 2018. Conspiracies online: User discussions in a conspiracy community following dramatic events. In Twelfth International AAAI Conference on Web and Social Media. find
  61. Cicero Nogueira dos Santos, Igor Melnyk, and Inkit Padhi. 2018. Fighting offensive language on social media with unsupervised text style transfer. In Proceedings of ACL. our card
  62. Carla Schieb and Mike Preuss. 2016. Governing hate speech by means of counterspeech on facebook. In Proceedings of ICA, pages 1–23. find
  63. Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1–10. our card
  64. Amartya Sen. 2011. The Idea of Justice, reprint edition edition. Belknap Press: An Imprint of Harvard University Press. find
  65. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Controlling politeness in neural machine translation via side constraints. In Proceedings of NAACL, pages 35–40. our card
  66. Fadi Abu Sheikha and Diana Inkpen. 2011. Generation of formal and informal sentences. In Proceedings of the 13th European Workshop on Natural Language Generation, pages 187–193. Association for Computational Linguistics. find
  67. Sarah T Roberts. 2014. Behind the screen: The hidden digital labor of commercial content moderation. Ph.D. thesis, University of Illinois at Urbana-Champaign. find
  68. Lawrence W Sherman. 2003. Reason for emotion: Reinventing justice with theories, innovations, and research—the american society of criminology 2002 presidential address. Criminology, 41(1):1–38. find
  69. Jessica Roy. 2016. ”’cuck,”snowflake,”masculinist’: A guide to the language of the’alt-right’. http://www.latimes.com/ nation/ la-na-pol-alt-right-terminology-20161115-story. html).LosAngelesTimes. ↗
  70. Alexandra Siegel. 2015. Sectarian Twitter Wars: Sunni-Shia Conflict and Cooperation in the Digital Age, volume 20. Carnegie Endowment for International Peace. find
  71. Jeffrey Z Rubin, Dean G Pruitt, and Sung Hee Kim. 1994. Social conflict: Escalation, stalemate, and settlement. Mcgraw-Hill Book Company. find
  72. Koustuv Saha, Eshwar Chandrasekharan, and Munmun De Choudhury. 2019. Prevalence and psychological effects of hateful speech in online college communities. In WebSci. find
  73. Joni Salminen, Hind Almerekhi, Milica Milenković, Soon-gyo Jung, Jisun An, Haewoon Kwak, and Bernard J Jansen. 2018. Anatomy of online hate: developing a taxonomy and machine learning models for identifying and classifying hate in online Scott R Stroud and William Cox. 2018. The varieties of femini… find
  74. Derald Wing Sue. 2010. Microaggressions in Everyday Life: Race, Gender, and Sexual Orientation. Wiley, Hoboken, NJ. find
  75. Derald Wing Sue, Christina M Capodilupo, Gina C Torino, Jennifer M Bucceri, Aisha M.B. B. Holder, Kevin L Nadal, and Marta Esquilin. 2007. Racial microaggressions in everyday life: Implications for clinical practice. American Psychologist, 62(4):271–286. find
  76. Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016. Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions. In Proceedings of the 25th international conference on world wide web, pages 613–624. International World Wide Web C… find
  77. Harry Charalambos Triandis. 1994. Culture and social behavior. McGraw-Hill New York. find
  78. Tom R Tyler and Yuen Huo. 2002. Trust in the Law: Encouraging Public Cooperation with the Police and Courts. Russell Sage Foundation. find
  79. Rob Voigt, Nicholas P Camp, Vinodkumar Prabhakaran, William L Hamilton, Rebecca C Hetey, Camilla M Griffiths, David Jurgens, Dan Jurafsky, and Jennifer L Eberhardt. 2017. Language from police body camera footage shows racial disparities in officer respect. Proceedings of the National Academy of Scie… find
  80. Zijian Wang and David Jurgens. 2018. It’s going to be okay: Measuring access to support in online communities. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 33–45. find
  81. Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding abuse: A typology of abusive language detection subtasks. In Proceedings of the First Workshop on Abusive Language. our card
  82. Lucas Wright, Derek Ruths, Kelly P Dillon, Haji Mohammad Saleem, and Susan Benesch. 2017. Vectors for counterspeech on twitter. In Proceedings of the First Workshop on Abusive Language Online, pages 57–62. find
  83. Justine Zhang, Jonathan P Chang, Cristian Danescu-Niculescu-Mizil, Lucas Dixon, Yiqing Hua, Nithum Thain, and Dario Taraborelli. 2018. Conversations gone awry: Detecting early signs of conversational failure. In Proceedings of ACL. our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jurgens-2019-just-comprehensive-strategy-nlp-address/