Elamur Research⌕
☰
home / papers / 2023 / A Keyword Based Approach to Understanding the Overpenalization of Marginalized Groups by English Marginal Abuse Models on Twitter

A Keyword Based Approach to Understanding the Overpenalization of Marginalized Groups by English Marginal Abuse Models on Twitter

Paper · 2023 · Kyra Yee, Alice Schoenauer Sebag, Olivia Redfield et al.

Summary

Classifiers that flag harmful content tend to produce more false positives on text written by or about marginalized groups. On Twitter, this can reduce the reach of those communities. The authors argue that typical bias audits depend on lexicons or demographic categories chosen in advance, which makes them ad hoc and shaped by the auditors' own blind spots. They propose a more systematic method. Instead of starting from a predefined list, they derive evaluation keywords by examining the text of the model's past false positives. They then measure model behaviour on each keyword slice. They applied this to Twitter's English marginal abuse model (as of spring 2022), which decides whether borderline-abusive Tweets lose amplification eligibility. Without demographic labels or dialect classifiers, the audit still exposed over-penalization of reclaimed slurs, counterspeech and identity terms. As a fix, they retrained the model with extra true-negative examples. This improved fairness measures across keyword slices while aggregate quality dropped only slightly. The work shows a practical, privacy-friendly route to proactive bias auditing.

Key points

  • Training labels come from 5 annotators per Tweet, combined by majority vote. Data is gathered through prevalence sampling (weighted by views) and FDR sampling (high model score or high report likelihood), collected from April 29 to September 27, 2021.
  • Training set sizes: the baseline model used 514,658 Tweets, and the mitigated model, with an added mitigation sample, used 561,072.
  • Aggregate quality, computed across 100 bootstrap resamples drawn from the test data:
    • PR AUC fell from 0.657 to 0.645.
    • ROC AUC fell from 0.963 to 0.961.
  • Meta-metrics summarize how much per-keyword metrics vary, using max–min gap and variance. All of them decreased for the mitigated model. For example, the AUC spread went from 0.110 to 0.098.
  • Stated limitations:
    • The annotations may themselves be biased.
    • The method favours high-traffic problems over harms to small or intersectional groups.
    • English Twitter skews toward the US and UK.
    • The model sees single Tweets with no author or conversation context.

Interesting ideas

  • I1. Build audit slices from the vocabulary of past errors rather than a hand-picked lexicon, so the evaluation reflects what actually breaks instead of what auditors expect
  • I2. Compare models by how unevenly they perform across slices (max–min and variance of a per-slice metric), not only by one global score
  • I3. Fix systematic false positives by adding targeted true-negative examples around the problematic keywords, then check that the aggregate metric barely moves
  • I4. Counterspeech and reclaimed speech show that one surface form can carry opposite intents, and a context-free text model cannot separate them
  • I5. Report bootstrap standard deviations on aggregate metrics to judge whether a fairness–quality trade-off is real or just noise

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): mine lemmas and constructions that cluster in parser or frame errors against silver data, and turn them into standing evaluation slices.
  • I2 (learn): report dispersion of LAS or frame accuracy across construction slices, so a model that is "better on average but worse on rare cases" is visible.
  • I3 (tree): add correctly parsed sentences containing confusable words or constructions to training, and verify overall LAS stays stable with bootstrap intervals.
  • I4 (frames): mark quoted, reported or condemned content as distinct from asserted content in speech-act frames, so a slur mentioned in order to reject it is not read as an attack.
  • I5 (answer): attach bootstrap spreads to school-test and word-problem scores, so small regressions after a repair rule are not over-interpreted.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Kyra Yee, Alice Schoenauer Sebag, Olivia Redfield, Matthias Eck, Emily Sheng and Luca Belli. 2023. A Keyword Based Approach to Understanding the Overpenalization of Marginalized Groups by English Marginal Abuse Models on Twitter. Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.trustnlp-1.10

@inproceedings{yee2023keyword,
  title = {{A Keyword Based Approach to Understanding the Overpenalization of Marginalized Groups by English Marginal Abuse Models on Twitter}},
  author = {Kyra Yee, Alice Schoenauer Sebag, Olivia Redfield, Matthias Eck, Emily Sheng and Luca Belli},
  booktitle = {Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023)},
  year = {2023},
  doi = {10.18653/v1/2023.trustnlp-1.10},
  url = {https://doi.org/10.18653/v1/2023.trustnlp-1.10}
}

References

57 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. McKane Andrus, Elena Spitzer, Jeffrey Brown, and Alice Xiang. 2021. What we can’t measure, we can’t understand: Challenges to demographic data procurement in the pursuit of fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 249–260. anti-definition league. 2022. Very fine people: What social media platforms miss about white supremacist speech. Report. ↗
  2. Ari Ball-Burack, Michelle Seng Ah Lee, Jennifer Cobbe, and Jatinder Singh. 2021. Differential tweetment: Mitigating racial dialect bias in harmful tweet detection. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 116–128. find
  3. Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2017. Fairness in machine learning. Nips tutorial, 1:2017. find
  4. Cynthia L Bennett and Os Keyes. 2020. What is the point of fairness? disability, ai and the complexity of justice. ACM SIGACCESS Accessibility and Computing, (125):1–1. ↗
  5. Sebastian Benthall and Bruce D Haynes. 2019. Racial categories in machine learning. In Proceedings of the conference on fairness, accountability, and transparency, pages 289–298. find
  6. Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion proceedings of the 2019 world wide web conference, pages 491–500. find
  7. Robin Brontsema. 2004. A queer revolution: Reconceptualizing the debate over linguistic reclamation. Colorado Research in Linguistics. ↗
  8. Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pages 77–91. PMLR. find
  9. Chris Collins, Simanique Moody, and Paul M Postal. 2008. An aae camouflage construction. Language, pages 29–68. ↗
  10. Adam M Croom. 2011. Slurs. Language Sciences, 33(3):343–358. find
  11. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 67–73. find
  12. Karen E Fields and Barbara Jeanne Fields. 2014. Race-craft: The soul of inequality in American life. Verso Trade. find
  13. Sayan Ghosh, Dylan Baker, David Jurgens, and Vinodkumar Prabhakaran. 2021. Detecting cross-geographic biases in toxicity modeling on social media. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 313– 328. ↗
  14. Nitesh Goyal, Ian Kivlichan, Rachel Rosen, and Lucy Vasserman. 2022. Is your toxicity my toxicity? exploring the impact of rater identity on toxicity annotation. arXiv preprint arXiv:2205.00501. ↗
  15. Oliver L Haimson, Daniel Delmonaco, Peipei Nie, and Andrea Wegner. 2021. Disproportionate removals and differing content moderation experiences for conservative, transgender, and black social media users: Marginalization and moderation gray areas. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2):1–35. ↗
  16. Matan Halevy, Camille Harris, Amy Bruckman, Diyi Yang, and Ayanna Howard. 2021. Mitigating racial biases in toxic language detection with an equity-based ensemble framework. In Equity and Access in Algorithms, Mechanisms, and Optimization, pages 1–11. ↗
  17. Foad Hamidi, Morgan Klaus Scheuerman, and Stacy M Branham. 2018. Gender recognition or gender reductionism? the social implications of embedded gender recognition systems. In Proceedings of the 2018 chi conference on human factors in computing systems, pages 1–13. find
  18. Alex Hanna, Emily Denton, Andrew Smart, and Jamila Smith-Loud. 2020. Towards a critical race methodology in algorithmic fairness. In Proceedings of the 2020 conference on fairness, accountability, and transparency, pages 501–512. find
  19. Camille Harris, Matan Halevy, Ayanna Howard, Amy Bruckman, and Diyi Yang. 2022. Exploring the role of grammar and word choice in bias toward african american english (aae) in hate speech classification. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 789–798. find
  20. Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. 2019. Improving fairness in machine learning systems: What do industry practitioners need? In Proceedings of the 2019 CHI conference on human factors in computing systems, pages 1–16. ↗
  21. Dirk Hovy and Shannon L Spruit. 2016. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 591–598. ↗
  22. Lily Hu and Issa Kohler-Hausmann. 2020. What’s sex got to do with fair machine learning? arXiv preprint arXiv:2006.01770. ↗
  23. Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020. Social biases in nlp models as barriers for persons with disabilities. arXiv preprint arXiv:2005.00813. ↗
  24. Amnesty International. 2018. Toxic twitter–a toxic place for women. Report. find
  25. Abigail Z Jacobs and Hanna Wallach. 2021. Measurement and fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 375–385. find
  26. Richard G Jones Jr. 2007. Drag queens, drama queens, and friends: Drama and performance as a solidarity-building function in a gay male friendship circle. Kaleidoscope: A Graduate Journal of Qualitative Communication Research, 6. find
  27. Matthew Katsaros, Kathy Yang, and Lauren Fratamico. 2022. Reconsidering tweets: Intervening during tweet creation decreases offensive content. In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, pages 477–487. find
  28. Os Keyes. 2018. The misgendering machines: Trans/hci implications of automatic gender recognition. Proceedings of the ACM on human-computer interaction, 2(CSCW):1–22. find
  29. Zaid Khan and Yun Fu. 2021. One label, one billion faces: Usage and consistency of racial categories in computer vision. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 587–597. find
  30. Amanda Bower Kristian Lum, Yunfeng Zhang. 2022. De-biasing "bias" measurement. arXiv preprint arXiv:2205.05770. ↗
  31. Tomo Lazovich, Luca Belli, Aaron Gonzales, Amanda Bower, Uthaipon Tantipongpipat, Kristian Lum, Ferenc Huszar, and Rumman Chowdhury. 2022. Measuring disparate outcomes of content recommendation algorithms with distributional inequality metrics. arXiv preprint arXiv:2202.01615. ↗
  32. Algorithmic Justice League. 2022. Bug bounties for algorithmic harms? Report. find
  33. Sean McKinnon. 2017. “building a thick skin for each other”: The use of ‘reading’as an interactional practice of mock impoliteness in drag queen backstage talk. Journal of Language and Sexuality, 6(1):90– 127. ↗
  34. Stephen O Murray. 1979. The art of gay insulting. Anthropological Linguistics, 21(5):211–223. find
  35. Donna Jo Napoli and Jack Hoeksema. 2009. The grammatical versatility of taboo terms. Studies in Language. International Journal sponsored by the Foundation “Foundations of Language”, 33(3):612–643. find
  36. Arvind Narayanan. 2018. Translation tutorial: 21 fairness definitions and their politics. In Proc. Conf. Fairness Accountability Transp., New York, USA, volume 1170. Arvind Narayanan. 21. Fairness definitions and their politics. In Tutorial presented at the Conf. on Fairness, Accountability, and Transparency. find
  37. Geoffrey Nunberg. 2018. The social life of slurs. New work on speech acts, pages 237–295. ↗
  38. Thiago Dias Oliva, Dennys Marcelo Antonialli, and Alessandra Gomes. 2021. Fighting hate speech, silencing drag queens? artificial intelligence in content moderation and risks to lgbtq voices online. Sexuality & Culture, 25(2):700–732. ↗
  39. Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing gender bias in abusive language detection. arXiv preprint arXiv:1808.07231. ↗
  40. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  41. Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Díaz. 2021. On releasing annotator-level labels and information in datasets. arXiv preprint arXiv:2110.05699. ↗
  42. Emily A Vogels. 2021. The state of online harassment. Pew Research Center, 13. find
  43. Zeerak Waseem. 2016. Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter. In Proceedings of the first workshop on NLP and computational social science, pages 138– 142.
  44. Jacquelyn Rahman. 2012. The n word: Its history and use in the african american community. Journal of English Linguistics, 40(2):137–171. find
  45. Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. 2020. Closing the ai accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 conference on fairness, accountability, and transparency, pages 33–44. ↗
  46. Anthony Rios. 2020. Fuzze: Fuzzy fairness evaluation of offensive language classifiers on african-american english. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 881–889. ↗
  47. Katherine Ritchie. 2017. Social identity, indexicality, and the appropriation of slurs. Croatian Journal of Philosophy, 17(2 (50)):155–180. find
  48. Björn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. 2017. Measuring the reliability of hate speech annotations: The case of the european refugee crisis. arXiv preprint arXiv:1701.08118. ↗
  49. Maximilian Wich, Jan Bauer, and Georg Groh. 2020. Impact of politically biased data on hate speech classification. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 54–64. find
  50. Andrew D Wong. 2005. The reappropriation of tongzhi. Language in society, 34(5):763–793. ↗
  51. Kyra Yee, Uthaipon Tantipongpipat, and Shubhanshu Mishra. 2021. Image cropping on twitter: Fairness metrics, their limitations, and the importance of representation, design, and agency. arXiv preprint arXiv:2105.08667. ↗
  52. Lal Zimman. 2017. Transgender language reform: Some challenges and strategies for promoting trans-affirming, gender-inclusive language. Journal of Language and Discrimination, 1(1):84–105. find
  53. Joni Salminen, Hind Almerekhi, Ahmed Mohamed Kamel, Soon-gyo Jung, and Bernard J Jansen. 2019. Online hate ratings vary by extremes: A statistical analysis. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval, pages 213–217. find
  54. Joni Salminen, Fabio Veronesi, Hind Almerekhi, Soon-Gvo Jung, and Bernard J Jansen. 2018. Online hate interpretation varies by country, but more by individual: A statistical analysis using crowdsourced ratings. In 2018 Fifth International Conference on Social Networks Analysis, Management and Security (SNAMS), pages 88–94. IEEE. find
  55. Nithya Sambasivan, Erin Arnesen, Ben Hutchinson, Tulsee Doshi, and Vinodkumar Prabhakaran. 2021. Re-imagining algorithmic fairness in india and beyond. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 315–328. find
  56. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 1668–1678.
  57. Hong Shen, Alicia DeVos, Motahhare Eslami, and Kenneth Holstein. 2021. Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors. arXiv preprint arXiv:2105.02980. keyword total count pos count neg count ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yee-2023-keyword-based-approach-understanding-overpenalization/