Elamur Research⌕
☰
home / papers / 2021 / Ruddit: Norms of Offensiveness for English Reddit Comments

Ruddit: Norms of Offensiveness for English Reddit Comments

Paper · 2021 · Rishav Hada, Sohi Sudhir, Pushkar Mishra et al.

Summary

Most offensive-language datasets put each comment into a category such as hateful, abusive or normal. These categories overlap, their boundaries are fuzzy, and annotators become desensitized after reading many harsh texts. Hada et al. instead treat offensiveness as a continuous scale. They built Ruddit, 6000 English Reddit comments, each with a real-valued score from -1 (strongly supportive) to 1 (strongly offensive). The comments come from 808 threads drawn from the Pushshift dump: half from topic-focused subreddits, a quarter from ChangeMyView and a quarter from random subreddits. The sample was enriched with emotional and offensive material, and each comment keeps its thread context. Scores were collected with Best–Worst Scaling (BWS). Crowd-workers saw 4-tuples and picked only the most and least offensive item. The authors report that the resulting scores are highly reliable. They also analyse how emotions, identity terms and swearing relate to the scores, and benchmark common neural regressors on the data. The resource matters because graded scores let moderators set their own thresholds and prioritise the worst content.

Key points

  • Dataset and sources: Ruddit has 6000 comments with offensiveness scores in [-1, 1].
    • Comments come from 808 posts, taking the first 25 and last 25 comments of each.
    • The source mix is 50% topic subreddits (AskMen, AskReddit, TwoXChromosomes, vaxxhappened, worldnews, worldpolitics), 25% ChangeMyView and 25% random subreddits.
  • Annotation method: BWS on 4-tuples, where one best/worst choice yields 5 of the 6 pairwise orderings. A total of 725 US-based workers on AMT took part, and each could submit at most about 5% of the assignments.
  • Implicit offensiveness: the paper counts a comment as implicitly offensive when it is offensive but contains no swear words.
  • Identity-agnostic variant: a second version replaces identity terms with a [group] token. Models trained on it did not exploit the association between identity terms and offensive comments.
  • Release format: only comment IDs and post IDs are distributed, to stay GDPR-compliant. Users fetch the text through the Reddit API. Code and data are at github.com/hadarishav/Ruddit.

Interesting ideas

  • I1. Comparison instead of absolute ratings: asking people which item is most and which is least X, rather than rating X on a scale, avoids scale-region bias and drift caused by desensitization, while still producing a fine-grained continuous score.
  • I2. Choosing 4-tuples on purpose: a single best/worst pick over four items settles five of six pairwise orderings, which makes comparative annotation affordable without labelling every pair.
  • I3. Defining "implicit" by what is absent: labelling a comment implicitly offensive whenever it lacks swear words gives a cheap, reproducible split between examples carried by lexical cues and examples that need deeper interpretation.
  • I4. Masking identity terms as a bias probe: replacing identity terms with a placeholder and re-evaluating shows whether a model leans on surface terms associated with the label.
  • I5. Sampling from both ends of a thread: taking the opening and closing comments of each thread captures both the replies most relevant to the post and how the discussion ended.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when LLM teacher tiers or human checkers judge competing parses or frame readings, ask them to rank a 4-tuple of candidates rather than score each one, then convert the choices into stable graded silver confidence.
  • I2 (learn): rank several witness-parser outputs for the same sentence in groups of four to get dense preference data for training at low annotation cost.
  • I3 (frames): split irony and implicature test sets into cue-word versus cue-free subsets, to check that the prag crate reads the situation rather than reacting to trigger words.
  • I4 (learn): use this as a negative control, masking named entities or class-revealing words with a placeholder and confirming that frame or answer accuracy holds up.
  • I5 (world): when building text worlds from long dialogues or tales, give extra attention to initial and final segments, where the setup and the resolution of entity states usually sit.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States183 works · 228,558 citations🎓University of AmsterdamAmsterdam · Netherlands144 works · 173,906 citations🏛️National Research Council CanadaOttawa · Canada59 works · 21,677 citations🏢Meta (United Kingdom)London · United Kingdom46 works · 18,033 citations🎓King's College LondonLondon · United Kingdom42 works · 1,617 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksCommunication · 97 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Rishav Hada, Sohi Sudhir, Pushkar Mishra, Helen Yannakoudakis, Saif M. Mohammad and Ekaterina Shutova. 2021. Ruddit: Norms of Offensiveness for English Reddit Comments. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-long.210

@inproceedings{hada2021ruddit,
  title = {{Ruddit: Norms of Offensiveness for English Reddit Comments}},
  author = {Rishav Hada, Sohi Sudhir, Pushkar Mishra, Helen Yannakoudakis, Saif M. Mohammad and Ekaterina Shutova},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.210},
  url = {https://doi.org/10.18653/v1/2021.acl-long.210}
}

References

63 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shima Asaadi, Saif Mohammad, and Svetlana Kiritchenko. 2019. Big BiRD: A large, fine-grained, bigram relatedness dataset for examining semantic composition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 505–516, Minneapolis, Minnesota. Association for Computational Linguistics. find
  2. Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 54–63, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  3. Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. ↗
  4. Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. ↗
  5. Lindsay Blackwell, Jill Dimond, Sarita Schoenebeck, and Cliff Lampe. 2017. Classification and its consequences for online harassment: Design insights from heartmob. Proc. ACM Hum.-Comput. Interact., 1(CSCW). ↗
  6. Luke Breitfeller, Emily Ahn, David Jurgens, and Yulia Tsvetkov. 2019. Finding microaggressions in the wild: A case for locating elusive phenomena in social media posts. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1664–1674, Hong Kong, China. Association for Computational Linguistics.
  7. Tommaso Caselli, Valerio Basile, Jelena Mitrović, Inga Kartoziya, and Michael Granitzer. 2020a. I feel offended, don’t be abusive! implicit/explicit messages in offensive and abusive language. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6193–6202, Marseille, France. European Language Resources Association.
  8. Tommaso Caselli, Valerio Basile, Jelena Mitrović, and Michael Granitzer. 2020b. Hatebert: Retraining bert for abusive language detection in english.
  9. Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online, pages 25–35, Florence, Italy. Association for Computational Linguistics. ↗
  10. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th International AAAI Conference on Web and Social Media, ICWSM ’17, pages 512–515. ↗
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  12. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’18, page 67–73, New York, NY, USA. Association for Computing Machinery. find
  13. T.N. Flynn and A.A.J. Marley. 2014. Best-worst scaling: theory and methods. In Stephane Hess and Andrew Daly, editors, Handbook of Choice Modelling, Chapters, chapter 8, pages 178–201. Edward Elgar Publishing. ↗
  14. P. Fortuna and S. Nunes. 2018. A survey on automatic detection of hate speech in text. ACM Computing Surveys (CSUR), 51:1 – 30. ↗
  15. Antigoni-Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. CoRR, abs/1802.00393. ↗
  16. Lei Gao and Ruihong Huang. 2017. Detecting online hate speech using context aware models. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 260–266. find
  17. Timothy Jay and Kristin Janschewitz. 2008. The pragmatics of swearing. Walter de Gruyter GmbH & Co. KG. ↗
  18. David Jurgens. 2013. Embracing ambiguity: A comparison of annotation methodologies for crowdsourcing word sense labels. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 556–562, Atlanta, Georgia. Association for Computational Linguistics. find
  19. David A. Jurgens, Peter D. Turney, Saif M. Mohammad, and Keith J. Holyoak. 2012. Semeval-2012 task 2: Measuring degrees of relational similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the Main Conference and the Shared Task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation, SemEval ’12, page 356–364, USA. Association for Computational Linguistics.
  20. Svetlana Kiritchenko and Saif Mohammad. 2017. Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 465–470, Vancouver, Canada. Association for Computational Linguistics.
  21. Svetlana Kiritchenko and Saif M. Mohammad. 2016. Capturing reliable fine-grained sentiment associations by crowdsourcing and best–worst scaling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 811–817, San Diego, California. Association for Computational Linguistics.
  22. Svetlana Kiritchenko and Isar Nejadgholi. 2020. Towards ethics by design in online abusive content detection. ↗
  23. Svetlana Kiritchenko, Xiaodan Zhu, and Saif Mohammad. 2014. Sentiment analysis of short informal text. The Journal of Artificial Intelligence Research (JAIR), 50. find
  24. Jana Kurrek, Haji Mohammad Saleem, and Derek Ruths. 2020. Towards a comprehensive taxonomy and large-scale annotated corpus for online slur usage. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 138–149, Online. Association for Computational Linguistics. find
  25. J. J. Louviere. 1991. Best-worst scaling: A model for the largest difference judgments. working paper. find
  26. Jordan J. Louviere, Terry N. Flynn, and A. A. J. Marley. 2015. Best-Worst Scaling: Theory, Methods and Applications. Cambridge University Press. find
  27. Scott C. Lowe. 2016. Stratified validation splits for regression problems. find
  28. Pushkar Mishra, Helen Yannakoudakis, and Ekaterina Shutova. 2019. Tackling online abuse: A survey of automated abuse detection methods. CoRR, abs/1908.06024. ↗
  29. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2018. Model cards for model reporting. CoRR, abs/1810.03993. ↗
  30. Saif Mohammad. 2018. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 English words. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 174–184, Melbourne, Australia. Association for Computational Linguistics.
  31. Saif Mohammad and Felipe Bravo-Marquez. 2017. Emotion intensities in tweets. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 65–77, Vancouver, Canada. Association for Computational Linguistics.
  32. Saif Mohammad and Svetlana Kiritchenko. 2018. Understanding emotions: A dataset of tweets to study interactions between affect categories. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  33. Emily Munro. 2011. The protection of children online: A brief scoping review to identify vulnerable groups. find
  34. B. Orme. 2009. Maxdiff analysis: Simple counting,individual-level logit, and hb. sawtooth software, inc. find
  35. C.E. Osgood, G.J. Suci, and P.H. Tenenbaum. 1957. The Measurement of meaning. University of Illinois Press, Urbana:. find
  36. Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing gender bias in abusive language detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2799–2804, Brussels, Belgium. Association for Computational Linguistics.
  37. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. ↗
  38. Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy. 2019. Emotion recognition in conversation: Research challenges, datasets, and recent advances. ↗
  39. PRC. Research in the crowdsourcing age, a case study [online]. 2020. Accessed: 2020-10-10. find
  40. Stanley Presser and Howard Schuman. 1996. Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. SAGE Publications, Inc. find
  41. Dante Razo and Sandra Kübler. 2020. Investigating sampling bias in abusive language detection. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 70–78, Online. Association for Computational Linguistics. find
  42. Sarah T. Roberts. 2016. Chapter Eight: Commercial Content Moderation: Digital Laborers’ Dirty Work. Peter Lang. find
  43. Sarah T. Roberts. 2019. Behind the Screen: Content Moderation in the Shadows of Social Media. Yale University Press. find
  44. James Russell. 2003. Core affect and the psychological construction of emotion. Psychological review, 110:145–72. ↗
  45. James A. Russell. 1980. A circumplex model of affect. Journal of Personality and Social Psychology, 39(6):1161–1178. ↗
  46. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy. Association for Computational Linguistics. ↗
  47. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In ACL.
  48. Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1–10, Valencia, Spain. Association for Computational Linguistics.
  49. Wiktor Soral, M. Bilewicz, and M. Winiewski. 2018. Exposure to hate speech increases prejudice through desensitization. Aggressive Behavior, 44:13V–146. ↗
  50. Statista. Regional distribution of desktop traffic to reddit.com as of september 2020, by country [online]. 2020a. Accessed: 2021-01-04. find
  51. Statista. Share of adult internet users in the united states who have personally experienced online harassment as of january 2020 [online]. 2020b. find
  52. Bertie Vidgen and Leon Derczynski. 2020. Directions in abusive language training data: Garbage in, garbage out. ↗
  53. Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott Hale, and Helen Margetts. 2019. Challenges and frontiers in abusive content detection. In Proceedings of the Third Workshop on Abusive Language Online, pages 80–93, Florence, Italy. Association for Computational Linguistics.
  54. Wenbo Wang, Lu Chen, Krishnaprasad Thirunarayan, and Amit P. Sheth. 2014. Cursing in english on twitter. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, CSCW ’14, page 415–425, New York, NY, USA. Association for Computing Machinery. ↗
  55. Zeerak Waseem. 2016. Are you a racist or am I seeing things? annotator influence on hate speech detection on twitter. In Proceedings of the First Workshop on NLP and Computational Social Science, pages 138– 142, Austin, Texas. Association for Computational Linguistics.
  56. Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding abuse: A typology of abusive language detection subtasks. In Proceedings of the First Workshop on Abusive Language Online, pages 78–84, Vancouver, BC, Canada. Association for Computational Linguistics.
  57. Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL Student Research Workshop, pages 88–93, San Diego, California. Association for Computational Linguistics.
  58. Gregor Wiedemann, Eugen Ruppert, Raghav Jindal, and Chris Biemann. 2018. Transfer learning from LDA to bilstm-cnn for offensive language detection in twitter. CoRR, abs/1811.02906. ↗
  59. Michael Wiegand, Maja Geulig, and Josef Ruppenhofer. 2021. Implicitly abusive comparisons – a new dataset and linguistic analysis. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 358–368, Online. Association for Computational Linguistics. find
  60. Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019. Detection of Abusive Language: the Problem of Biased Datasets. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 602–608, Minneapolis, Minnesota. Association for Computational Linguistics.
  61. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2016. Ex machina: Personal attacks seen at scale. CoRR, abs/1610.08914. ↗
  62. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Wikipedia Talk Corpus. find
  63. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval). In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 75–86, Minneapolis, Minnesota, USA. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hada-2021-ruddit-norms-offensiveness-english-reddit/