Elamur Research⌕
☰
home / papers / 2026 / Toxic Subword Pruning for Dialogue Response Generation on Large Language Models

Toxic Subword Pruning for Dialogue Response Generation on Large Language Models

Summary

Most LLM safety work assumes a well-aligned model that must be protected from harmful prompts. This paper looks at a different case: the model itself is already toxic, for example after careless fine-tuning or tampering by someone with access to the weights. The authors propose ToxPrune, a training-free decoding method. A user-supplied list of offensive words is split into BPE subwords, and those subwords are removed from the candidate vocabulary during generation. The method works with top-k, top-p and beam search. Optional extensions add paraphrases of the list (a blacklist) or remove words that share subwords with protected terms (a whitelist).

Experiments on PersonaChat dialogue response generation use the toxic NSFW-3B model and the official Llama-3.1-6B base model. On NSFW-3B, ToxPrune sharply reduces toxicity as judged by GPT-4 while also improving response quality. On Llama-3.1-6B, it increases diversity without harming quality. Earlier studies found BPE pruning harmful for machine translation, so its benefit for safety is a notable finding. It is also cheap, since no retraining or external classifier is required.

Key points

  • Data: PersonaChat valid and test splits from ParlAI (about 7,800 and 7,500 instances), in the original (PersonaChat-ORI) and revised-persona (PersonaChat-REV) versions.
  • Toxic word list: an open-source list (surge-ai profanity) of 254 words, which the Llama-3.1 tokenizer splits into 404 pruned subwords.
  • Toxicity results on NSFW-3B:
    • GPT-4 toxicity falls from 0.89 to 0.66, 0.48, 0.24 and 0.13 as 25%, 50%, 75% and 100% of the list is applied.
    • A paraphrase baseline barely helps (0.87).
    • A toxic-classifier baseline mostly fails to produce safe output within three retries.
  • Metrics: BLEU (sacreBLEU), ROUGE F-measures and Distinct-1/2, plus GPT-4 toxicity scoring.
  • Human evaluation: comparing Llama-3.1-6B with ToxPrune-6B shows no loss in quality and better diversity.
  • Limitations: only chit-chat dialogue is studied, and the method requires access to decoding, so it cannot be applied to closed models such as ChatGPT.
  • Hardware: all experiments were run on 2× RTX 4090.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.starsem-conference.35

@inproceedings{lu2026toxic,
  title = {{Toxic Subword Pruning for Dialogue Response Generation on Large Language Models}},
  author = {Lu, Hongyuan Adam and Lam, Wai},
  booktitle = {Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.starsem-conference.35},
  url = {https://doi.org/10.18653/v1/2026.starsem-conference.35}
}

References

60 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. find
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
  3. Marco Cognetta, Tatsuya Hiraoka, Rico Sennrich, Yuval Pinter, and Naoaki Okazaki. 2024. An analysis of BPE vocabulary trimming in neural machine translation. In Proceedings of the Fifth Workshop on Insights from Negative Results in NLP, pages 48–50, Mexico City, Mexico. Association for Computational… find
  4. Boyi Deng, Wenjie Wang, Fuli Feng, Yang Deng, Qifan Wang, and Xiangnan He. 2023. Attack prompt generation for red teaming and defending large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2176–2189, Singapore. Association for Computational Linguisti… find
  5. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. Toxicity in chatgpt: Analyzing persona-assigned language models. arXiv preprint arXiv:2304.05335. find our card
  6. Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen, and Shujian Huang. 2024. A wolf in sheep’s clothing: Generalized nested jailbreak prompts can fool large language models easily. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computationa… find
  7. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez,… find
  8. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia. Association for Computational Linguistics. find
  9. Lizhou Fan, Lingyao Li, Zihui Ma, Sanggyu Lee, Huizi Yu, and Libby Hemphill. 2023. A bibliometric review of large language models research from 2017 to 2023. ACM Transactions on Intelligent Systems and Technology. find
  10. Markus Freitag and Yaser Al-Onaizan. 2017. Beam search strategies for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 56–60, Vancouver. Association for Computational Linguistics. find our card
  11. Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, et al. 2022. Predictability and surprise in large generative models. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1747–1764. find
  12. Jun Gao, Wei Bi, Xiaojiang Liu, Junhui Li, and Shuming Shi. 2019. Generating multiple diverse responses for short-text conversation. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):6383– 6390. find
  13. Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30–45, Abu Dhabi, United Arab Emirates. Asso… find our card
  14. Alex Graves. 2012. Sequence Transduction with Recurrent Neural Networks. arXiv e-prints, arXiv:1211.3711. find our card
  15. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations. find
  16. Ruijie Hou, Yueyang Jiao, Hanxu Hu, Yingming Li, Wai Lam, Huajian Zhang, and Hongyuan Lu. 2025. LNE-blocking: An efficient framework for contamination mitigation evaluation on large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 3512–3528, Suzhou, Ch… find
  17. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline Defenses for Adversarial Attacks Against Aligned Language Models. arXiv e-prints, arXiv:2309.00614. find our card
  18. Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi Mi, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, et al. 2023. Chatharuhi: Reviving anime character in reality via large language model. arXiv preprint arXiv:2308.09597. find
  19. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pag… find our card
  20. Kun Li, Tianhua Zhang, Liping Tang, Junan Li, Hongyuan Lu, Xixin Wu, and Helen Meng. 2022. find
  21. Grounded dialogue generation with cross-encoding re-ranker, grounding span prediction, and passage dropout. In Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering, pages 123–129, Dublin, Ireland. Association for Computational Linguistics. find
  22. Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. 2020. Don’t say that! making inconsistent dialogue unlikely with unlikelihood training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4715–4728… find
  23. Margaret Li, Jason Weston, and Stephen Roller. 2019. ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons. CoRR, abs/1909.03087:arXiv:1909.03087. — [arXiv:1909.03087](https://arxiv.org/abs/1909.03087) find
  24. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
  25. Hongyuan Lu, Haoyang Huang, Shuming Ma, Dongdong Zhang, Wai Lam, Zhaochuan Gao, Anthony Aue, Arul Menezes, and Furu Wei. 2023. TRIP: Accelerating document-level multilingual pre-training via triangular document-level pre-training on parallel data triplets. In Findings of the Association for Computat… find
  26. Hongyuan Lu, Haoyang Huang, Dongdong Zhang, Furu Wei, and Wai Lam. 2024a. Revamping multilingual agreement bidirectionally via switched back-translation for multilingual neural machine translation. In Findings of the Association for Computational Linguistics: EACL 2024, pages 264–275, St. Julian’s,… find
  27. Hongyuan Lu and Wai Lam. 2023. PCC: Paraphrasing with bottom-k sampling and cyclic learning for curriculum data augmentation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 68–82, Dubrovnik, Croatia. Association for Computational… find
  28. Hongyuan Lu, Wai Lam, Hong Cheng, and Helen Meng. 2022a. On controlling fallback responses for grounded dialogue generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2591–2601, Dublin, Ireland. Association for Computational Linguistics. find
  29. Hongyuan Lu, Wai Lam, Hong Cheng, and Helen Meng. 2022b. Partner personas generation for dialogue response generation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5200–5212, Seattle, United S… find
  30. Hongyuan Lu, Zixuan Li, Zefan Zhang, and Wai Lam. 2025. SLoW: Select low-frequency words! automatic dictionary selection for translation on large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 898–913, Suzhou, China. Association for… find
  31. Hongyuan Lu, Haoran Yang, Haoyang Huang, Dongdong Zhang, Wai Lam, and Furu Wei. 2024b. Chain-of-dictionary prompting elicits translation in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 958–976, Miami, Florida, USA. Associatio… find
  32. Hongyuan Adam Lu, Z. L., Victor Wei, Zefan Zhang, Zhao Hong, Qiqi Xiang, Bowen Cao, and Wai Lam. 2026. Adam’s Law: Textual Frequency Law on Large Language Models. arXiv e-prints, arXiv:2604.02176. find
  33. Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017. ParlAI: A dialog research software platform. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 79–84, Copenhagen… find our card
  34. Animesh Nighojkar and John Licato. 2021. Improving paraphrase detection with the adversarial paraphrasing task. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers… find
  35. OpenAI. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774. find our card
  36. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational… find our card
  37. Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527. find our card
  38. Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszcze, Robie Gonzales, carsten maple, Subhabrata Majumdar, Hassan Sajjad, and Frank Rudzicz. 2024. Representation noising: A defence mechanism against harmful finetuning. In The Thirty-eighth Annual Conference on Neural Information Processing Sys… find
  39. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin… find
  40. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS’14, page 3104–3112, Cambridge, MA, USA. MIT Press. find
  41. Tianyi Tang, Hongyuan Lu, Yuchen Jiang, Haoyang Huang, Dongdong Zhang, Xin Zhao, Tom Kocmi, and Furu Wei. 2024. Not all metrics are guilty: Improving NLG evaluation by diversifying references. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Li… find
  42. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. find
  43. Yuma Tsuta, Naoki Yoshinaga, and Masashi Toyoda. 2020. uBLEU: Uncertainty-aware automatic evaluation method for open-domain dialogue systems. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 199–206, Online. Association for… find
  44. Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, and Huajun Chen. 2024a. Detoxifying large language models via knowledge editing. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find
  45. Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, Jiangjie Chen, Cheng Li, and Yanghua Xiao. 2024b. InCharacter: Evaluating personality fidelity in role-playing agents through psychological interviews. In Proceedings of the 62nd Annual… find
  46. Xuezhi Wang and Denny Zhou. 2024. Chain-of-Thought Reasoning Without Prompting. arXiv eprints, arXiv:2402.10200. find
  47. Zhongyu Wei, Qianlong Liu, Baolin Peng, Huaixiao Tou, Ting Chen, Xuanjing Huang, Kam-fai Wong, and Xiangying Dai. 2018. Task-oriented dialogue system for automatic diagnosis. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 20… find
  48. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204–2213, Austra… find our card
  49. Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. 2023. DEPN: detecting and editing privacy neurons in pre-trained language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10,… find
  50. Jing Xu, Arthur Szlam, and Jason Weston. 2022. Beyond goldfish memory: Long-term open-domain conversation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5180–5197, Dublin, Ireland. Association for Computational Linguistics. find
  51. Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran. 2024. SafeDecoding: Defending against jailbreak attacks via safety-aware decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 55… find
  52. Jianhao Yan, Futing Wang, Yafu Li, and Yue Zhang. 2024. Potential and challenges of model editing for social debiasing. CoRR. find
  53. Hao Yang, Hongyuan Lu, Dingkang Yang, Wenliang Yang, Peng Sun, Xiaochuan Zhang, Jun Xiao, Kefan He, Wai Lam, Yang Liu, and Xinhua Zeng. 2026. Stephanie2: Thinking, Waiting, and Making Decisions Like Humans in Step-by-Step AI Social Chat. arXiv e-prints, arXiv:2601.05657. — [arXiv:2601.05657](https://arxiv.org/abs/2601.05657) find
  54. Hao Yang, Hongyuan Lu, Xinhua Zeng, Yang Liu, Xiang Zhang, Haoran Yang, Yumeng Zhang, Shan Huang, Yiran Wei, and Wai Lam. 2025. Stephanie: Step-by-step dialogues for mimicking human interactions in social conversations. In Findings of the Association for Computational Linguistics: NAACL 2025, pages… find
  55. Shuai Zhao, Jinming Wen, Anh Luu, Junbo Zhao, and Jie Fu. 2023a. Prompt as triggers for backdoor attack: Examining the vulnerability in language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12303–12317, Singapore. Association for Computatio… find
  56. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Z. Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jianyun Nie, and Ji rong Wen. 2023b. A survey of large… find our card
  57. Wangchunshu Zhou, Qifei Li, and Chenle Li. 2023. Learning to predict persona information for dialogue personalization without explicit persona description. In Findings of the Association for Computational Linguistics: ACL 2023, pages 2979–2991, Toronto, Canada. Association for Computational Linguist… find
  58. Wenhong Zhu, Hongkun Hao, Zhiwei He, Yun-Ze Song, Jiao Yueyang, Yumeng Zhang, Hanxu Hu, Yiran Wei, Rui Wang, and Hongyuan Lu. 2024. CLEAN–EVAL: Clean evaluation on contaminated large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 835–847, Mexico City… find
  59. Yicheng Zou, Zhihua Liu, Xingwu Hu, and Qi Zhang. 2021. Thinking clearly, talking fast: Concept-guided non-autoregressive generation for open-domain dialogue systems. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2215–2226, Online and Punta Cana, Do… find
  60. Haoran Yang, Yumeng Zhang, Jiaqi Xu, Hongyuan Lu, Pheng-Ann Heng, and Wai Lam. 2024. Unveiling the generalization power of fine-tuned large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologi… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lu-2026-toxic-subword-pruning-dialogue-response/