Elamur Research
☰
home / papers / 2018 / Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers

Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1801.04354

Summary

Most earlier work on adversarial attacks against text classifiers assumed white-box access to the model's gradients or embeddings. That rarely matches reality: deployed systems such as Google's Perspective API sit behind cloud endpoints and only return predictions. Gao et al. introduce DeepWordBug, a black-box, untargeted attack that needs only the classifier's outputs. It first scores tokens by how strongly the prediction depends on them, using query-based scoring functions such as a Combined Score. It then applies small character-level edits to the top-ranked words: swapping, substituting, deleting or inserting characters. The total edit distance stays within a budget ε. The resulting misspellings look minor to people but often push words outside the model's vocabulary. Across eight real-world datasets covering topic classification, sentiment analysis and spam detection, DeepWordBug outperforms a Random black-box baseline and a white-box Gradient baseline. It cuts accuracy by about 68% on average for Word-LSTM and 48% for Char-CNN. The work shows that cheap, model-agnostic perturbations threaten deployed NLP systems, and that its adversarial samples can also be used to harden models.

Key points

  • Attack performance: average accuracy drops of 68% (Word-LSTM) and 48% (Char-CNN). In transfer tests, samples built against one model reduce a different target model's accuracy from about 90% to 20–50%.
  • Confidence of errors: on the Enron Spam Dataset with ε = 30, roughly 90% of adversarial samples cause wrong predictions with confidence above 0.9. Rejecting low-confidence outputs is therefore not an effective defense.
  • Adversarial training: the authors retrained the model for 10 epochs on 20,000 adversarial samples paired with their originals (learning rate 0.0005). Accuracy on adversarial inputs rose from about 12% to about 62%, while clean accuracy dipped only slightly.
  • Autocorrection defense: placing Python autocorrect 0.3.0 in front of Word-LSTM restores part of the accuracy, starting from a clean baseline of 88.45%. Recovery ranges from 82.07% for Insertion-1 down to 33.67% for Deletion-2. Autocorrection also adds processing overhead and creates its own exploitable weaknesses.
  • Vocabulary size: the size of the model's input dictionary did not noticeably change how well the attack worked.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

180 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 169 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1801.04354 · arXiv 1801.04354

@misc{gao2018black,
  title = {{Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers}},
  author = {Gao, Ji and Lanchantin, Jack and Soffa, Mary Lou and Qi, Yanjun},
  year = {2018},
  doi = {10.48550/arxiv.1801.04354},
  eprint = {1801.04354},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1801.04354}
}

References

30 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014). find our card
  2. Marco Barreno, Blaine Nelson, Russell Sears, Anthony D. Joseph, and J. Doug Tygar. 2006. Can machine learning be secure?. In Proceedings of the 2006 ACM Symposium on Information, computer and communications security. ACM, 16–25. http://dl.acm.org/citation.cfm?id=1128824 ↗
  3. Nicholas Carlini and David Wagner. 2017. Towards evaluating the robustness of neural networks. In Security and Privacy (SP), 2017 IEEE Symposium on. IEEE, 39–57. find
  4. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning. ACM, 160–167. find
  5. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems. 3079–3087. find
  6. Nilesh Dalvi, Pedro Domingos, Sumit Sanghai, Deepak Verma, et al. 2004. Adversarial classification. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 99–108. find
  7. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014). find our card
  8. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8, 1735–1780. find
  9. Google Jigsaw. 2017. Perspective API. https://www.perspectiveapi.com/. (2017). ↗
  10. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278– 2324. find
  11. Vladimir I Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, Vol. 10. 707–710. find
  12. Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, and Wenchang Shi. 2017. Deep Text Classification Can be Fooled. arXiv preprint arXiv:1704.08006 (2017). — [arXiv:1704.08006](https://arxiv.org/abs/1704.08006) find
  13. Daniel Lowd and Christopher Meek. 2005. Adversarial learning. In Eleventh ACM SIGKDD International Conference on Knowledge Discovery in Data Mining. ACM, 641–647. find
  14. Daniel Lowd and Christopher Meek. 2005. Good Word Attacks on Statistical Spam Filters.. In CEAS, Vol. 2005. find
  15. Vangelis Metsis, Ion Androutsopoulos, and Georgios Paliouras. 2006. Spam filtering with naive bayes-which naive bayes?. In CEAS, Vol. 17. 28–69. find
  16. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013). find
  17. Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using lstms on sequences and tree structures. arXiv preprint arXiv:1601.00770 (2016). find our card
  18. Anh Nguyen, Jason Yosinski, and Jeff Clune. 2015. Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images. In CVPR. IEEE. find
  19. Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. 2016. Practical black-box attacks against deep learning systems using adversarial examples. arXiv preprint arXiv:1602.02697 (2016). — [arXiv:1602.02697](https://arxiv.org/abs/1602.02697) find
  20. Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. 2016. The Limitations of Deep Learning in Adversarial Settings. In IEEE European Symposium on Security and Privacy (EuroS&P). find
  21. Nicolas Papernot, Patrick McDaniel, Ananthram Swami, and Richard Harang. 2016. Crafting adversarial input sequences for recurrent neural networks. In Military Communications Conference, MILCOM 2016-2016 IEEE. IEEE, 49–54. find
  22. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 1532–1543. find
  23. Keith Rayner, Sarah J White, Rebecca L Johnson, and Simon P Liversedge. 2006. Raeding wrods with jubmled lettres there is a cost. Psychological science 17, 3 (2006), 192–193. find
  24. David E Rumelhart, Geoffrey E Hinton, Ronald J Williams, et al. 1988. Learning representations by back-propagating errors. Cognitive modeling 5, 3 (1988), 1. find
  25. Suranjana Samanta and Sameep Mehta. 2017. Towards Crafting Text Adversarial Samples. arXiv preprint arXiv:1707.02812 (2017). — [arXiv:1707.02812](https://arxiv.org/abs/1707.02812) find
  26. Richard Socher, Cliff C Lin, Chris Manning, and Andrew Y Ng. 2011. Parsing natural scenes and natural language with recursive neural networks. In Proceedings of the 28th international conference on machine learning (ICML-11). 129–136. find
  27. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing. 1631–1642. find
  28. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing Properties of Neural Networks. In International Conference on Learning Representations (ICLR). find our card
  29. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016). find
  30. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems. 649–657. find our card