Elamur Research⌕
☰
home / papers / 2026 / Confident, Calibrated, or Complicit: Safety Alignment and Ideological Bias in LLM Hate Speech Detection

Confident, Calibrated, or Complicit: Safety Alignment and Ideological Bias in LLM Hate Speech Detection

Paper · 2026 · Sanjeevan Selvaganapathy and Mehwish Nasim

Summary

The paper asks whether LLMs can be trusted as hate speech classifiers, especially for implicit hate, and how deployed safety guardrails and political framing affect them. The authors use a class-balanced subset of the Latent Hatred benchmark and test five LLMs. The models are grouped as "censored" or "uncensored" using the UGI score, with LMArena Elo roughly matched so that general capability is similar across groups. Each model classifies every post zero-shot in JSON format, giving a label and a confidence score. This is repeated under four political personas set by system prompts. Refusals and unusable outputs count as errors.

Censored models score higher on strict accuracy. Most of that advantage comes from producing fewer null outputs, not from being more accurate when they do answer. Censored models also resist persona steering better, while uncensored models shift with ideological framing. All models struggle with irony, perform unevenly across target groups, and are overconfident when wrong. The authors argue that moderation audits must go beyond accuracy and also test fairness, calibration and ideological consistency.

Key points

  • Data: From the 21,480-post Latent Hatred corpus (Twitter, Gab, Stormfront, Yahoo), the authors sampled 3,267 posts, with 1,089 each of explicit_hate, implicit_hate and not_hate.
  • Models: o3-mini and Llama 405b are labelled censored, with low UGI. Mistral Medium, GPT 4o and Mistral Large are labelled uncensored, with higher UGI. All are selected to have comparable LMArena Elo.
  • Setup: Four personas were used: Progressive, Conservative, Libertarian and Centrist. Outputs were HATE, NOT_HATE or CANNOT_CLASSIFY, each with a confidence score. Decoding used T = 0.7 with a single pass per condition.
  • Headline result: Censored models reached 69.0% strict accuracy, compared with 64.1% for uncensored models. Persona sensitivity was concentrated in the uncensored group.
  • Caveats: The design is observational, so the authors attribute effects to "censorship-as-deployed" rather than to safety alignment as such. The study covers one English dataset and Western-style personas, and reports no seed-level confidence intervals.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Western AustraliaPerth · Australia8 works · 1,707 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 worksSociology and Political Science · 577 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksBias, fairness & safety · 517 works

Cite

Cite as Sanjeevan Selvaganapathy and Mehwish Nasim. 2026. Confident, Calibrated, or Complicit: Safety Alignment and Ideological Bias in LLM Hate Speech Detection. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1594

@inproceedings{selvaganapathy2026confident,
  title = {{Confident, Calibrated, or Complicit: Safety Alignment and Ideological Bias in LLM Hate Speech Detection}},
  author = {Sanjeevan Selvaganapathy and Mehwish Nasim},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1594},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1594}
}

References

16 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pranav Bhandari, Nicolas Fay, Michael J Wise, Amitava Datta, Stephanie Meek, Usman Naseem, and Mehwish Nasim. 2025. Can LLM agents maintain a persona in discourse? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 29213–29229, Suzhou, China. Association… find
  2. Myra Cheng, Esin Durmus, and Dan Jurafsky. 2023. Marked personas: Using natural language prompts to measure stereotypes in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1504–1532, Toronto, Canada. Associatio…
  3. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: an open platform for evaluating llms by human preference. In Proceedings of the 41st International Confere…
  4. Saloni Dash, Amélie Reymond, Emma S. Spiro, and Aylin Caliskan. 2026. Persona-assigned large language models exhibit human-like motivated reasoning. Preprint, arXiv:2506.20020. find
  5. DontPlanToEnd. 2025. UGI Leaderboard - a Hugging Face Space by DontPlanToEnd. find
  6. Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. 2021. Latent Hatred: A Benchmark for Understanding Implicit Hate Speech. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 345– 363, On…
  7. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. Deepseek-r1 incentivizes reasoning in llms through rein… find
  8. Andy Liu, Mona Diab, and Daniel Fried. 2024. Evaluating large language model biases in persona-steered generation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9832–9850, Bangkok, Thailand. Association for Computational Linguistics. find
  9. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train…
  10. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.
  11. Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, Pranav Sandeep Dulepet, Saurav Vidyadhara, Dayeon Ki, Sweta Agrawal, Chau Pham, Gerson Kroiz, Feileen Li, Hudson Tao, Ashay Srivastava, and 12 other… — The Prompt Report: A Systematic Survey of Prompt Engineering Techniques
  12. Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcec. 2024. Cultural bias and cultural alignment of large language models. PNAS Nexus, 3(9):pgae346. find
  13. Conor Walsh and Alok Joshi. 2024. Machine learning for sports betting: Should model selection be based on accuracy or calibration? Machine Learning with Applications, 16:100539. find
  14. Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  15. Shuzhou Yuan, Ercong Nie, Mario Tawfelis, Helmut Schmid, Hinrich Schutze, and Michael Farber. 2025. Hateful person or hateful model? investigating the role of personas in hate speech detection by large language models. ArXiv, abs/2506.08593. — [arXiv:2506.08593](https://arxiv.org/abs/2506.08593) find
  16. Min Zhang, Jianfeng He, Taoran Ji, and Chang-Tien Lu. 2024. Don’t Go To Extremes: Revealing the Excessive Sensitivity and Calibration Limitations of LLMs in Implicit Hate Speech Detection. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/selvaganapathy-2026-confident-calibrated-or-complicit-safety/