Elamur Research⌕
☰
home / papers / 2025 / How AI Fails: An Interactive Pedagogical Tool for Demonstrating Dialectal Bias in Automated Toxicity Models

How AI Fails: An Interactive Pedagogical Tool for Demonstrating Dialectal Bias in Automated Toxicity Models

Paper · 2025 · Subhojit Ghimire

Summary

The paper asks whether automated toxicity classifiers penalise writers because of their dialect rather than their message, and how to make that harm understandable to non-experts. The author audits unitary/toxic-bert, a bert-base-uncased model fine-tuned on Jigsaw Toxic Comment data. The audit uses two balanced corpora of 10,000 tweets each, one in African-American English (AAE) and one in Standard American English (SAE), drawn from the TwitterAAE corpus by Slanglab. The tweets were selected by high demographic-model confidence and cleaned of URLs, handles and similar noise.

On average the model gives AAE text about 1.8× higher toxicity and 8.8× higher identity-attack scores. False positive rates diverge across classification thresholds.

The second contribution is a public web tool, the Dialectal Bias Analyser. It shows all six label scores and lets users move a decision threshold themselves. The point it makes is that the real harm comes from a human-chosen cut-off, which looks neutral but still silences one dialect more than the other. The work matters as a bridge between fairness research and public AI literacy.

Key points

  • Model audited: unitary/toxic-bert, run through a transformers pipeline with sigmoid outputs, so each of its six categories (identity attack, insult, threat, obscene, severe toxicity and toxicity) receives its own independent probability.
  • Data: the 12 GB twitteraae_all file was streamed with Pandas in one-million-row chunks. Malformed rows were dropped, and only tweets with demographic probability above 80% were kept, giving 10k AAE and 10k SAE samples.
  • Results: AAE scored about 1.8× more toxic and about 8.8× higher on identity hate than SAE. False positive rate was plotted against the threshold, under the assumption that all 20,000 tweets are benign.
  • Tool: a Flask backend with a single /api/predict/ endpoint, plus a React/Babel front end. It includes preset minimal pairs such as zero copula ("She at the library studying") and double negation.
  • Stated limitations:
    • Only one model was tested; its successor, detoxify, was not checked.
    • Dialect labels come from another model, not from human annotation.
    • FPR is only approximate, because some tweets may genuinely be toxic.

Interesting ideas

  • I1. Present the same meaning in two dialects as a minimal pair, so that any change in output can only come from surface form; this cleanly separates "what is said" from "how it is said".
  • I2. Keep the model's score strictly apart from the decision threshold, and show error rates for each group across the whole threshold range instead of at a single cut-off; this exposes the policy choice hidden in "neutral" defaults.
  • I3. Use simple ratio metrics (mean score of group A divided by group B, per output label) as a quick disparity check; they are easy to explain and reveal which label carries most of the bias.
  • I4. Build balanced corpora from a huge, probabilistically labelled stream by keeping only high-confidence items and capping each side at equal size.
  • I5. Show every intermediate score and let users move the decision boundary themselves; interaction makes the cause of an outcome visible, which a static report does not.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Build AAE/SAE minimal-pair test sets (zero copula, negative concord, ain't) and check that the parser yields equivalent UD trees, for example a predicate without an overt cop node mapped to the same frame.
  • I2 (learn): When accepting silver parses or repair rules by confidence, report acceptance and error curves per register or dialect, so a global threshold does not quietly discard non-standard text.
  • I3 (answer): Compare answer accuracy and confidence on paired standard and dialectal versions of school tests and fables, per question type, to spot where the reader breaks.
  • I4 (learn): Select silver sentences where several witness parsers agree strongly, while balancing sizes across genres, so training is not dominated by the majority style.
  • I5 (frames): Expose speech-act or irony scores alongside the chosen label in an inspector, so a developer can see which cue pushed a dialectal sentence over a boundary.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 worksPolitical Science and International Relations · 182 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 517 works

Cite

Cite as Subhojit Ghimire. 2025. How AI Fails: An Interactive Pedagogical Tool for Demonstrating Dialectal Bias in Automated Toxicity Models. arXiv:2511.06676.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2511.06676 · arXiv 2511.06676

@misc{ghimire2025how,
  title = {{How AI Fails: An Interactive Pedagogical Tool for Demonstrating Dialectal Bias in Automated Toxicity Models}},
  author = {Subhojit Ghimire},
  year = {2025},
  doi = {10.48550/arxiv.2511.06676},
  eprint = {2511.06676},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2511.06676}
}

References

22 references, 2 of them with a card of ours. ↗ leads to the work.

  1. V. U. Gongane, M. V. Munot, and A. D. Anuse, “Detection and moderation of detrimental content on social media platforms: current status and future directions,” Social Network Analysis and Mining, vol. 12, no. 1, p. 129, Sep 2022. [Online]. Available: https://doi.org/10.1007/s13278-022-00951-3 ↗
  2. J. Burrell, “How the machine ‘thinks’: Understanding opacity in machine learning algorithms,” Big Data & Society, vol. 3, no. 1, p. 2053951715622512, 2016. [Online]. Available: https://doi.org/10.1177/2053951715622512 ↗
  3. S. U. Noble, Algorithms of Oppression, ser. How Search Engines Reinforce Racism. New York, USA: New York University Press, 2018. [Online]. Available: https://doi.org/10.18574/nyu/9781479833641.001. 0001 ↗
  4. S. L. Blodgett, L. Green, and B. O’Connor, “Demographic dialectal variation in social media: A case study of African-American English,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, J. Su, K. Duh, and X. Carreras, Eds. Austin, Texas: Association for Computational Linguistics, Nov. 2016, pp. 1119–1130. [Online]. Available: https://aclanthology.org/D16-1120/ ↗
  5. J. F. Gomez, C. Machado, L. M. Paes, and F. Calmon, “Algorithmic arbitrariness in content moderation,” in Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, ser. FAccT ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 2234–2253. [Online]. Available: https://doi.org/10. 1145/3630106.3659036 ↗
  6. “unitary/toxic-bert,” https://huggingface.co/unitary/toxic-bert/, accessed: 2026-02. ↗
  7. “Twitteraae dataset,” https://slanglab.cs.umass.edu/TwitterAAE/, accessed: 2026-02. ↗
  8. M. Sap, D. Card, S. Gabriel, Y. Choi, and N. A. Smith, “The risk of racial bias in hate speech detection,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. Màrquez, Eds. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 1668–1678. [Online]. Available: https://aclanthology.org/P19-1163/ ↗
  9. A. Parikh, S. Das, and A. Feragen, “Investigating label bias and representational sources of age-related disparities in medical segmentation,” in 2026 IEEE 23rd International Symposium on Biomedical Imaging (ISBI), 2026, pp. 1–5. [Online]. Available: https://doi.org/10.1109/ISBI61048.2026.11515928 ↗
  10. I. Tenney, J. Wexler, J. Bastings, T. Bolukbasi, A. Coenen, S. Gehrmann, E. Jiang, M. Pushkarna, C. Radebaugh, E. Reif, and A. Yuan, “The language interpretability tool: Extensible, interactive visualizations and analysis for NLP models,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Q. Liu and D. Schlangen, Eds. Online: Association for Computational Linguistics, Oct. 2020, pp. 107–118. [Online]. Available: https://aclanthology.org/2020.emnlp-demos.15/ ↗
  11. “biaslyze,” https://biaslyze.org/, accessed: 2026-02. ↗
  12. S. Kabir, L. Li, and T. Zhang, “Stile: Exploring and debugging social biases in pre-trained text representations,” in Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, ser. CHI ’24. New York, NY, USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145/3613904.3642111 ↗
  13. H. Viswanath and T. Zhang, “Fairpy: A toolkit for evaluation of prediction biases and their mitigation in large language models,” 2025. [Online]. Available: https://arxiv.org/abs/2302.05508 ↗
  14. D. Raz, C. Bintz, V. Guetler, A. Tam, M. Katell, D. Dailey, B. Herman, P. M. Krafft, and M. Young, “Face mis-id: An interactive pedagogical tool demonstrating disparate accuracy rates in facial recognition,” in Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, ser. AIES ’21. New York, NY, USA: Association for Computing Machinery, 2021, p. 895–904. [Online]. Available: https://doi.org/10.1145/3461702.3462627 ↗
  15. “How normal am i,” https://www.hownormalami.eu/, accessed: 2026-02. ↗
  16. “evaluate-measurement/toxicity,” https://huggingface.co/spaces/ evaluate-measurement/toxicity/, accessed: 2026-02. ↗
  17. A. Field, S. L. Blodgett, Z. Waseem, and Y. Tsvetkov, “A survey of race, racism, and anti-racism in NLP,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), C. Zong, F. Xia, W. Li, and R. Navigli, Eds. Online: Association for Computational Linguistics, Aug. 2021, pp. 1905–1925. [Online]. Available: https://aclanthology.org/2021.acl-long.149/ ↗
  18. Z. Yang, Y. Maricar, M. Davari, N. Grenon-Godbout, and R. Rabbany, “Toxbuster: In-game chat toxicity buster with bert,” 2023. [Online]. Available: https://arxiv.org/abs/2305.12542 ↗
  19. “bert-base-uncased,” https://huggingface.co/google-bert/ bert-base-uncased, accessed: 2026-02. ↗
  20. “Jigsaw toxic comment classification challenge,” https://huggingface.co/ datasets/thesofakillers/jigsaw-toxic-comment-classification-challenge, accessed: 2026-02. ↗
  21. “Dialectal bias analyser repository,” https://github.com/SubhojitGhimire/ Dialectal Bias Analyser, accessed: 2026-02. ↗
  22. “Dialectal bias analyser: Hugging face space,” https://huggingface.co/ spaces/SubhojitGhimire/Dialect Biasness Analyser, accessed: 2026-02. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ghimire-2025-ai-fails-interactive-pedagogical-tool/