Elamur Research⌕
☰
home / papers / 2026 / Aligned Probing: Relating Toxic Behavior and Model Internals

Aligned Probing: Relating Toxic Behavior and Model Internals

Paper · 2026 · Andreas Waldis, Vagrant Gautam, Anne Lauscher et al.

Summary

Work on LM toxicity usually judges models only by what they generate and treats toxicity as one uniform category. This paper connects that output-based view with what happens inside the model. The authors propose aligned probing, a framework that pairs behavioral toxicity scores with linear probes trained on hidden states. Each probe estimates how much toxicity information a layer holds. They prompt more than 20 OLMo, Llama and Mistral models (base and instruction-tuned) with RealToxicPrompts. Generations are scored on six fine-grained PERSPECTIVE API attributes, and per-layer averaged representations of input and output tokens are probed.

Toxicity information is strongly present, mostly in lower layers rather than upper ones. Models that capture input toxicity more strongly tend to produce less toxic text, and interventions indicate this link is causal. Attributes behave differently: Threat depends on context, while Profanity hinges on single words. Four case studies extend the results, and the method can be applied to other text properties beyond toxicity.

Key points

  • Experimental scale: 20+ LMs and over 100K probing runs. Each prompt gets 25 sampled generations (temperature 1.0, nucleus p = 0.9).
  • Four probing scenarios:
    • Input: input states → input toxicity
    • Forward: output states → input toxicity
    • Output: output states → output toxicity
    • Backward: input states → Expected Maximum Toxicity (EMT)
  • Behavioral metrics: EMT, the worst case over the 25 generations, and Toxicity Correlation (TC), the Pearson correlation between input toxicity and EMT.
  • Probe validity checks: probes are purely linear and validated with control tasks (Hewitt & Liang) and an information-theoretic analysis (Voita & Titov).
  • Case studies:
    • DPO detoxification substantially prunes toxicity-related representations.
    • Results stay stable under prompt paraphrasing and half/4-bit quantization.
    • In six OLMo checkpoints, toxicity behavior and encoding are near their final level by 100K steps.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technische Universität DarmstadtDarmstadt · Germany114 works · 36,326 citations🎓Saarland UniversitySaarbrücken · Germany149 works · 18,353 citations🎓Universität HamburgHamburg · Germany81 works · 5,549 citations🎓Lucerne University of Applied Sciences and ArtsLucerne · Switzerland2 works · 1 citation

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 2Research subjects and languages the work is about.Bias, fairness & safety · 587 worksInterpretability · 149 works

Cite

Cite as Andreas Waldis, Vagrant Gautam, Anne Lauscher, Dietrich Klakow and Iryna Gurevych. 2026. Aligned Probing: Relating Toxic Behavior and Model Internals. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl.a.613

@article{waldis2026aligned,
  title = {{Aligned Probing: Relating Toxic Behavior and Model Internals}},
  author = {Andreas Waldis, Vagrant Gautam, Anne Lauscher, Dietrich Klakow and Iryna Gurevych},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.613},
  url = {https://doi.org/10.1162/tacl.a.613}
}

References

54 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160–175. https://doi.org/10.1162/tacl a 00359 ↗
  2. Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Aidan N. Gomez, Phil Blunsom, Marzieh Fadaee, Ahmet Üstu… — Aya 23: Open Weight Releases to Further Multilingual Progress
  3. Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097–1179. https://doi.org/10.1162/coli a 00524 ↗
  4. Yonatan Belinkov. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219. https://doi.org/10.1162/coli a 00422 ↗
  5. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online. Association for Computational Linguisti…
  6. Leonard Bereska and Stratis Gavves. 2024. Mechanistic interpretability for AI safety – A review. Transactions on Machine Learning Research.
  7. Amanda Cercas Curry, Gavin Abercrombie, and Zeerak Talat. 2024. Subjective isms? On the danger of conflating hate and offence in abusive language detection. In Proceedings of the 8th Workshop on Online Abuse and Harms (WOAH 2024), pages 275–282, Mexico City, Mexico. Association for Computational Lin… find
  8. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, et al. 20… — The Llama 3 Herd of Models
  9. Tyler A. Chang and Benjamin K. Bergen. 2024. Language model behavior: A comprehensive survey. Computational Linguistics, 50(1):293–350. https://doi.org/10.1162/coli_a_00492 ↗
  10. Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill… find
  11. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroi… find
  12. Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1:…
  13. Subbarao Kambhampati, Kaya Stechly, Karthik Valmeekam, Lucas Saldyt, Siddhant Bhambri, Vardhan Palod, Atharva Gundawar, Soumya Rani Samineni, Durgesh Kalwar, and Upasana Biswas. 2025. Stop anthropomorphizing intermediate tokens as reasoning/thinking traces! ArXiv preprint, arXiv:2504.09762 — [arXiv:2504.09762](https://arxiv.org/abs/2504.09762) find
  14. John Hewitt, Robert Geirhos, and Been Kim. 2025. We can’t understand AI using our existing vocabulary. find
  15. Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, and Ellie Pavlick. 2019. Probing what different NLP tasks teach machines about function word comprehension. In Proceedings of the Eighth Joint Conferenc…
  16. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743, Hong Kong, Chi… find
  17. Sachin Kumar, Vidhisha Balachandran, Lucille Njoo, Antonios Anastasopoulos, and Yulia Tsvetkov. 2023. Language generation models can cause harm: So what can we do about it? An actionable survey. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lingui…
  18. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26–30, 2020. OpenReview.net.
  19. Jennifer Hu and Roger Levy. 2023. Prompting is not a substitute for probability measurements in large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5040–5060, Singapore. Association for Computational Linguistics. https://doi.org/10.…
  20. Karim Lasri, Tiago Pimentel, Alessandro Lenci, Thierry Poibeau, and Ryan Cotterell. 2022. Probing for the usage of grammatical number. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8818–8831, Dublin, Ireland. Association for… find
  21. Devansh Jain, Priyanshu Kumar, Samuel Gehman, Xuhui Zhou, Thomas Hartvigsen, and Maarten Sap. 2024. Polyglotoxicityprompts: Multilingual evaluation of neural toxic degeneration in large language models. ArXiv preprint, abs/2405.09373. — [arXiv:2405.09373](https://arxiv.org/abs/2405.09373) find
  22. Anne Lauscher, Federico Bianchi, Samuel R. Bowman, and Dirk Hovy. 2022. Socio-Probe: What, when, and where language models learn about sociodemographics. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7901–7918, Abu Dhabi, United Arab Emirates. Assoc… find
  23. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6–9, 2019. OpenReview.net. find
  24. Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? A call for multi-prompt LLM evaluation. Transactions of the Association for Computational Linguistics, 12:933–949. https://doi.org/10 .1162/tacl_a_00681 find
  25. Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, and Rada Mihalcea. 2024. A mechanistic understanding of alignment algorithms: A case study on DPO and toxicity. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21–27, 2024.…
  26. Marius Mosbach, Vagrant Gautam, Tomás Vergara Browne, Dietrich Klakow, and Mor Geva. 2024. From insights to actions: The impact of interpretability and analysis research on NLP. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3078–3105, Miami, Florid… find
  27. Xiaochen Li, Zheng Xin Yong, and Stephen Bach. 2024. Preference tuning for toxicity mitigation generalizes across languages. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13422–13440, Miami, Florida, USA. Association for Computational Linguistics. https://doi.org/10… find
  28. Marius Mosbach, Anna Khokhlova, Michael A. Hedderich, and Dietrich Klakow. 2020. On the interplay between fine-tuning and sentence-level probing for linguistic knowledge in pre-trained transformers. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NL… find
  29. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, et a… find
  30. Jingcheng Niu, Wenjie Lu, and Gerald Penn. 2022. Does BERT rediscover a classical NLP pipeline? In Proceedings of the 29th International Conference on Computational Linguistics, pages 3143–3153, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. find
  31. Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. 2024. A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. In Proceedings of the 2…
  32. Gianluca Nogara, Francesco Pierri, Stefano Cresci, Luca Luceri, Petter Törnberg, and Silvia Giordano. 2023. Toxic bias: Perspective API misreads german as more toxic. CoRR, abs/2312.12651. https://doi.org/10.48550/ARXIV.2312.12651 — [doi:10.48550/arxiv.2312.12651](https://doi.org/10.48550/arxiv.2312.12651) ↗
  33. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Allyson Ettinger, Michal… find
  34. Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. 2021. Probing the probing paradigm: Does probing accuracy entail task relevance? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3363–3377, Online. Association…
  35. Nedjma Ousidhoum, Xinran Zhao, Tianqing Fang, Yangqiu Song, and Dit-Yan Yeung. 2021. Probing toxic content in large pre-trained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language… find
  36. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Ling…
  37. Pia Pachinger, Allan Hanbury, Julia Neidhardt, and Anna Planitzer. 2023. Toward disambiguating the definitions of abusive, offensive, toxic, and uncivil comments. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 107–113, Dubrovnik, Croatia. Association for… find
  38. Naomi Saphra and Sarah Wiegreffe. 2024. Mechanistic? In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 480–498, Miami, Florida, US. Association for Computational Linguistics. https://doi.org/10 .18653/v1/2024.blackboxnlp-1.30 find
  39. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In ICLR. find
  40. Luiza Pozzobon, Beyza Ermis, Patrick Lewis, and Sara Hooker. 2023. On the challenges of using black-box apis for toxicity evaluation in research. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6–10, 2023, pages 7595–7609, As… find
  41. Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. 2025. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity. https://doi.org/10.70777/si.v2i6.15919 ↗
  42. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems 36: Annual Conference Anirudh Srinivasan, Venkata Subrahm…
  43. Elena Voita and Ivan Titov. 2020. Information-theoretic probing with minimum description length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 183–196, Online. Association for Computational Linguistics. https://doi.org /10.18653/v1/2020.emnl… ↗
  44. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601, Florence, Italy. Association for Computational Linguistics. https://doi.org/10.18653/v1 /P19-1…
  45. Andreas Waldis, Yufang Hou, and Iryna Gurevych. 2024a. Dive into the chasm: Probing the gap between in- and cross-topic generalization. In Findings of the Association for Computational Linguistics: EACL 2024, pages 2197–2214, St. Julian’s, Malta. Association for Computational Linguistics. https://do… find
  46. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. What do you learn from context? Probing for sentence structure in contextualized word representations. In 7th International Confer…
  47. Andreas Waldis, Yotam Perlitz, Leshem Choshen, Yufang Hou, and Iryna Gurevych. 2024b. Holmes: A benchmark to assess the linguistic competence of language models. Transactions of the Association for Computational Linguistics, 12:1616–1647. https://doi.org/10 .1162/tacl_a_00718 find
  48. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  49. Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, and Huajun Chen. 2024. Detoxifying large language models via knowledge editing. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P…
  50. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377–392. https://doi.org/10.1162/tacl_a_00321 ↗
  51. Zeerak Waseem. 2016. Are you a racist or am I seeing things? Annotator influence on hate speech detection on Twitter. In Proceedings of the First Workshop on NLP and Computational Social Science, pages 138–142, Austin, Texas. Association for Computational Linguistics. https://doi.org/10.18653/v1/W1… ↗
  52. Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding abuse: A typology of abusive language detection subtasks. In Proceedings of the First Workshop on Abusive Language Online, pages 78–84, Vancouver, BC, Canada. Association for Computational Linguistics. https://doi.… ↗
  53. Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. 2024. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. ArXiv preprint, abs/2408.07666.
  54. Jiaxin Wen, Pei Ke, Hao Sun, Zhexin Zhang, Chengfei Li, Jinfeng Bai, and Minlie Huang. 2023. Unveiling the implicit toxicity in large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1322–1338, Singapore. Association for Computational…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/waldis-2026-aligned-probing-relating-toxic-behavior/