Elamur Research
☰
home / papers / 2018 / TextBugger: Generating Adversarial Text Against Real-world Applications

TextBugger: Generating Adversarial Text Against Real-world Applications

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1812.05271

Summary

Deep learning text understanding (DLTU) systems increasingly power security-relevant services such as sentiment analysis and toxic content filtering, yet their resistance to deliberately manipulated input was poorly understood. Earlier adversarial-text methods were slow, assumed white-box access, needed manual effort, or targeted only one model. The authors present TextBugger, a general framework that produces adversarial texts which still read naturally to people. With full model access, it ranks words by the classifier's Jacobian and applies the most damaging of five candidate character- or word-level "bugs," such as misspellings and look-alike characters. Without model access, it first finds the sentences that matter most, then uses query-based scoring to pick words to alter. The authors tested it on offline models (CNN, LSTM) and on commercial APIs from Google, Microsoft, IBM, Amazon and Perspective. TextBugger achieved higher success rates than prior attacks, at a cost that grows sub-linearly with text length, and adversarial texts also transferred from offline models to online services. The work matters because it shows deployed NLP services can be evaded cheaply, which motivates research on defenses.

Key points

  • Two threat models are covered. The white-box attack uses Jacobian-based word importance. The black-box attack works against MLaaS APIs, for example ParallelDots, using only queries that return labels or confidence scores.
  • On the IMDB dataset, TextBugger reached a 100% success rate against Amazon AWS Comprehend and Microsoft Azure Text Analytics. On AWS this took about 4.61 seconds while keeping roughly 97% semantic similarity.
  • In a user study, human readers correctly understood 94.9% of the adversarial texts.
  • The targets included Google Cloud NLP, IBM Watson Natural Language Understanding, Microsoft Azure, Amazon AWS Comprehend and Perspective, covering both sentiment and toxicity tasks. The affected companies were informed of the findings.
  • The paper was published at NDSS 2019. It also offers a preliminary discussion of two defense strategies and possible attacker countermeasures.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

198 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 189 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.14722/ndss.2019.23138 · arXiv 1812.05271

@misc{li2018textbugger,
  title = {{TextBugger: Generating Adversarial Text Against Real-world Applications}},
  author = {Li, Jinfeng and Ji, Shouling and Du, Tianyu and Li, Bo and Wang, Ting},
  year = {2018},
  doi = {10.14722/ndss.2019.23138},
  eprint = {1812.05271},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.14722/ndss.2019.23138}
}

References

40 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Alzantot, Y. Sharma, A. Elgohary, B.-J. Ho, M. Srivastava, and K.-W. Chang, “Generating natural language adversarial examples,” arXiv preprint arXiv:1804.07998, 2018. find our card
  2. M. Barreno, B. Nelson, A. D. Joseph, and J. Tygar, “The security of machine learning,” Machine Learning, vol. 81, no. 2, pp. 121–148, 2010. find
  3. M. Barreno, B. Nelson, R. Sears, A. D. Joseph, and J. D. Tygar, “Can machine learning be secure?” in ASIACCS. ACM, 2006, pp. 16–25. find
  4. Y. Belinkov and Y. Bisk, “Synthetic and natural noise both break neural machine translation,” arXiv preprint arXiv:1711.02173, 2017. find
  5. B. Biggio, G. Fumera, and F. Roli, “Design of robust classifiers for adversarial environments,” in SMC. IEEE, 2011, pp. 977–982. find
  6. N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in S&P, 2017, pp. 39–57. find
  7. D. Cer, Y. Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar et al., “Universal sentence encoder,” arXiv preprint arXiv:1803.11175, 2018. find our card
  8. M. Cheng, J. Yi, H. Zhang, P.-Y. Chen, and C.-J. Hsieh, “Seq2sick: Evaluating the robustness of sequence-to-sequence models with adversarial examples,” arXiv preprint arXiv:1803.01128, 2018. — [arXiv:1803.01128](https://arxiv.org/abs/1803.01128) find
  9. J. Ebrahimi, A. Rao, D. Lowd, and D. Dou, “Hotflip: White-box adversarial examples for nlp,” arXiv preprint arXiv:1712.06751, 2017. — [arXiv:1712.06751](https://arxiv.org/abs/1712.06751) find
  10. I. Evtimov, K. Eykholt, E. Fernandes, T. Kohno, B. Li, A. Prakash, A. Rahmati, and D. Song, “Robust physical-world attacks on machine learning models,” arXiv preprint arXiv:1707.08945, 2017. — [arXiv:1707.08945](https://arxiv.org/abs/1707.08945) find
  11. J. Gao, J. Lanchantin, M. L. Soffa, and Y. Qi, “Black-box generation of adversarial text sequences to evade deep learning classifiers,” arXiv preprint arXiv:1801.04354, 2018. find our card
  12. Z. Gong, W. Wang, B. Li, D. Song, and W.-S. Ku, “Adversarial texts with gradient methods,” arXiv preprint arXiv:1801.07175, 2018. — [arXiv:1801.07175](https://arxiv.org/abs/1801.07175) find
  13. I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in ICLR, 2015, pp. 1–11. find our card
  14. H. Hosseini, S. Kannan, B. Zhang, and R. Poovendran, “Deceiving google’s perspective api built for detecting toxic comments,” arXiv preprint arXiv:1702.08138, 2017. — [arXiv:1702.08138](https://arxiv.org/abs/1702.08138) find
  15. L. Huang, A. D. Joseph, B. Nelson, B. I. Rubinstein, and J. Tygar, “Adversarial machine learning,” in AISec. ACM, 2011, pp. 43–58. find
  16. R. Jia and P. Liang, “Adversarial examples for evaluating reading comprehension systems,” in EMNLP, 2017, pp. 2021–2031. find
  17. Y. Kim, “Convolutional neural networks for sentence classification,” in EMNLP, 2014, pp. 1746–1751. find our card
  18. Y. Li, T. Cohn, and T. Baldwin, “Learning robust representations of text,” in EMNLP, 2016, pp. 1979–1985. find
  19. B. Liang, H. Li, M. Su, P. Bian, X. Li, and W. Shi, “Deep text classification can be fooled,” arXiv preprint arXiv:1704.08006, 2017. — [arXiv:1704.08006](https://arxiv.org/abs/1704.08006) find
  20. X. Ling, S. Ji, J. Zou, J. Wang, C. Wu, B. Li, and T. Wang, “Deepsec: A uniform platform for security analysis of deep learning model,” in IEEE S&P, 2019. find
  21. A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts, “Learning word vectors for sentiment analysis,” in ACL. Portland, Oregon, USA: Association for Computational Linguistics, June 2011, pp. 142–150. find our card
  22. W. Medhat, A. Hassan, and H. Korashy, “Sentiment analysis algorithms and applications: A survey,” Ain Shams Engineering Journal, vol. 5, no. 4, pp. 1093–1113, 2014. find
  23. T. Miyato, A. M. Dai, and I. Goodfellow, “Adversarial training methods for semi-supervised text classification,” ICLR, 2017. find our card
  24. S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “Deepfool: a simple and accurate method to fool deep neural networks,” in CVPR, 2016, pp. 2574–2582. find
  25. A. Nguyen, J. Yosinski, and J. Clune, “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images,” in CVPR. IEEE, 2015, pp. 427–436. find
  26. C. Nobata, J. Tetreault, A. Thomas, Y. Mehdad, and Y. Chang, “Abusive language detection in online user content,” in WWW. International World Wide Web Conferences Steering Committee, 2016, pp. 145–153. find
  27. B. Pang and L. Lee, “Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales,” in ACL. Association for Computational Linguistics, 2005, pp. 115–124. find
  28. N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, “Practical black-box attacks against machine learning,” in Asia CCS. ACM, 2017, pp. 506–519. find
  29. N. Papernot, P. McDaniel, A. Swami, and R. Harang, “Crafting adversarial input sequences for recurrent neural networks,” in MILCOM. IEEE, 2016, pp. 49–54. find
  30. J. Pennington, R. Socher, and C. Manning, “Glove: Global vectors for word representation,” in EMNLP, 2014, pp. 1532–1543. find
  31. G. Rawlinson, “The significance of letter position in word recognition,” IEEE Aerospace and Electronic Systems Magazine, vol. 22, no. 1, pp. 26–27, 2007. find
  32. M. T. Ribeiro, S. Singh, and C. Guestrin, “Semantically equivalent adversarial rules for debugging nlp models,” in ACL, 2018. find
  33. S. Samanta and S. Mehta, “Towards crafting text adversarial samples,” arXiv preprint arXiv:1707.02812, 2017. — [arXiv:1707.02812](https://arxiv.org/abs/1707.02812) find
  34. D. Sculley, G. Wachman, and C. E. Brodley, “Spam filtering using inexact string matching in explicit feature space with on-line linear classifiers.” in TREC, 2006. find
  35. C. E. Shannon, “Communication theory of secrecy systems,” Bell system technical journal, vol. 28, no. 4, pp. 656–715, 1949. find
  36. C. Szegedy, “Intriguing properties of neural networks,” in ICLR, 2014, pp. 1–10. find our card
  37. C. Xiao, B. Li, J.-Y. Zhu, W. He, M. Liu, and D. Song, “Generating adversarial examples with adversarial networks,” arXiv preprint arXiv:1801.02610, 2018. — [arXiv:1801.02610](https://arxiv.org/abs/1801.02610) find
  38. X. Zhang, J. Zhao, and Y. LeCun, “Character-level convolutional networks for text classification,” in NIPS. Neural information processing systems foundation, 2015, pp. 649–657. find our card
  39. Y. Zhang and B. Wallace, “A sensitivity analysis of (and practitioners guide to) convolutional neural networks for sentence classification,” in IJCNLP, vol. 1, 2017, pp. 253–263. find
  40. Z. Zhao, D. Dua, and S. Singh, “Generating natural adversarial examples,” in ICLR, 2018. find