Elamur Research⌕
☰
home / papers / 2021 / ONION: A Simple and Effective Defense Against Textual Backdoor Attacks

ONION: A Simple and Effective Defense Against Textual Backdoor Attacks

Summary

Textual backdoor attacks plant hidden behavior in NLP classifiers. The model acts normally on clean text but outputs an attacker-chosen label whenever a trigger appears. Before this work, the only dedicated defense was BKI. It filters poisoned training data, so it cannot help when users adopt an already-compromised third-party model or API.

The authors propose ONION, a defense applied at inference time. It relies on the observation that nearly all known attacks insert out-of-context tokens or sentences that disrupt fluency. For each word in an input, GPT-2 perplexity is computed with and without that word, and the drop serves as a suspicion score. Words scoring above a threshold t_s are deleted before classification. The threshold is tuned on validation data or simply set to 0.

Experiments covered SST-2, OffensEval and AG News, with BiLSTM and BERT victims and five insertion-based attacks. ONION cut attack success rates by more than 40% on average while barely reducing clean accuracy. Because it never touches training, it covers both pre- and post-training attack scenarios.

Key points

  • Suspicion score: f_i = p_0 − p_i, the perplexity change when word w_i is removed. Combinatorial alternatives (particle swarm optimization, genetic algorithm) were slower and no more effective.
  • Data and models:
    • SST-2: 9,612 sentences.
    • OffensEval: 14,102 tweets.
    • AG News: 30,399 sentences.
    • Victims: BiLSTM, plus BERT_BASE tested right after poisoning (BERT-T) or after clean fine-tuning (BERT-F).
  • Attacks: BadNet with rare, middle-frequency (BadNet_m) and high-frequency (BadNet_h) trigger words, RIPPLES, and InSent (inserted sentence). Evaluation metrics were ΔASR and ΔCACC.
  • Pre-training comparison on SST-2:
    • Average ΔASR: ONION 44.43% vs. BKI 16.07%.
    • ΔCACC: ONION 1.41 vs. BKI 0.87, so ONION costs slightly more clean accuracy.
  • Word removal analysis: poisoned samples lost 0.76 trigger and 0.57 normal words on average, with trigger-detection precision of 56.19 and recall of 75.66. Wrongly removed normal words tend to be rare.
  • Limitations: ONION struggles against stealthier triggers such as syntactic structures, word substitution and context-aware sentences. Code is available at github.com/thunlp/ONION.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

90 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 86 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.emnlp-main.752

@inproceedings{qi2021onion,
  title = {{ONION: A Simple and Effective Defense Against Textual Backdoor Attacks}},
  author = {Qi, Fanchao and Chen, Yangyi and Li, Mukai and Yao, Yuan and Liu, Zhiyuan and Sun, Maosong},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.752},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.752}
}

References

27 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chuanshuai Chen and Jiazhu Dai. 2020. Mitigating backdoor attacks in lstm-based text classification systems by backdoor keyword identification. arXiv preprint arXiv:2007.12070. — [arXiv:2007.12070](https://arxiv.org/abs/2007.12070) find
  2. Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma, and Yang Zhang. 2020. Badnl: Backdoor attacks against nlp models. arXiv preprint arXiv:2006.01043. — [arXiv:2006.01043](https://arxiv.org/abs/2006.01043) find
  3. Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. 2017. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526. — [arXiv:1712.05526](https://arxiv.org/abs/1712.05526) find
  4. Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. 2019. A backdoor attack against lstm-based text classification systems. IEEE Access, 7:138872–138878. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT. find
  6. Min Du, Ruoxi Jia, and Dawn Song. 2020. Robust anomaly detection and backdoor attack detection via differential privacy. In Proceedings of ICLR. find
  7. Russell Eberhart and James Kennedy. 1995. Particle swarm optimization. In Proceedings of the IEEE International Conference on Neural Networks. find
  8. David E Goldberg and John H Holland. 1988. Genetic algorithms and machine learning. Machine learning, 3(2):95–99. find
  9. Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 2017. BadNets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733. find
  10. Yiming Li, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. 2020. Backdoor learning: A survey. arXiv preprint arXiv:2007.08745. — [arXiv:2007.08745](https://arxiv.org/abs/2007.08745) find
  11. Cong Liao, Haoti Zhong, Anna Squicciarini, Sencun Zhu, and David Miller. 2018. Backdoor embedding in convolutional neural network models via invisible perturbation. arXiv preprint arXiv:1808.10307. — [arXiv:1808.10307](https://arxiv.org/abs/1808.10307) find
  12. Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. 2018a. Fine-pruning: Defending against backdooring attacks on deep neural networks. In International Symposium on Research in Attacks, Intrusions, and Defenses. find
  13. Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. 2018b. Trojaning Attack on Neural Networks. In Proceedings of NDSS. find
  14. Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. 2020. Reflection backdoor: A natural backdoor attack on deep neural networks. In Proceedings of ECCV. find
  15. Yuntao Liu, Yang Xie, and Ankur Srivastava. 2017. Neural trojans. In Proceedings of ICCD. find
  16. Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. 2021a. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In Proceedings of ACL. find our card
  17. Fanchao Qi, Yuan Yao, Sophia Xu, Zhiyuan Liu, and Maosong Sun. 2021b. Turn the combination lock: Learnable textual backdoor attacks via word substitution. In Proceedings of ACL. find
  18. Ximing Qiao, Yukun Yang, and Hai Li. 2019. Defending neural backdoors via generative distribution modeling. In Proceedings of NeurIPS. find
  19. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  20. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP. find
  21. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the type and target of offensive posts in social media. In Proceedings of NAACL-HLT. find our card
  22. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. find
  23. Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2020. Word-level textual adversarial attacking as combinatorial optimization. In Proceedings of ACL. find our card
  24. Keita Kurita, Paul Michel, and Graham Neubig. 2020. Weight poisoning attacks on pre-trained models. In Proceedings of ACL. find
  25. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proceedings of NeurIPS. find our card
  26. Xinyang Zhang, Z Zhang, S Ji, and T Wang. 2021. Trojaning language models for fun and profit. In Proceedings of IEEE European Symposium on Security and Privacy. find
  27. Shihao Zhao, Xingjun Ma, Xiang Zheng, James Bailey, Jingjing Chen, and Yu-Gang Jiang. 2020. Clean-label backdoor attacks on video recognition models. In Proceedings of CVPR. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qi-2021-onion-simple-effective-defense-against/