Elamur Research⌕
☰
home / papers / 2017 / BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain

BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain

Paper · 2017 · Tianyu Gu, Brendan Dolan-Gavitt and Siddharth Garg

Summary

Training large CNNs is expensive, so users often outsource training to cloud providers or start from downloaded pre-trained models. The authors ask what an untrusted party in this supply chain could do. They introduce the BadNet: a network that scores well on the user's own validation data but produces attacker-chosen outputs whenever a secret trigger appears in the input. The user fixes the architecture, so the attacker cannot add a separate trigger detector. Instead, they poison the training set, and ordinary training then embeds the backdoor in the weights.

They test this first on MNIST digits, using a small pixel pattern as the trigger. They then apply it to traffic sign recognition with U.S. and Swedish datasets. A yellow Post-it on a stop sign makes the model read it as a speed limit, while accuracy on clean images drops by under 1%. The backdoor also survives transfer learning to Swedish signs.

Finally, they audit the Caffe Model Zoo and find weak integrity checks. The work shows that backdoors are effective and hard to spot, which motivates verification tools and secure model distribution.

Key points

  • The threat model covers two cases: fully outsourced training (MLaaS) and partially outsourced training via transfer learning. In both, the attacker must keep the user-specified architecture and can only choose the weights.
  • On MNIST, a tiny mark such as a small "x" in an image corner triggers targeted misclassification, while accuracy on normal digits stays high.
  • In the U.S. traffic sign case, backdoored stop signs carrying a Post-it are reliably recognized as the attacker's target. Accuracy on clean images falls by less than 1%.
  • A backdoored U.S. sign model was retrained for Swedish signs. The resulting model performs about 25% worse on average when the trigger is present.
  • The Caffe Model Zoo audit covered 39 hosted models:
    • The Network in Network model's listed SHA1 does not match the downloaded file, yet none of its 24 comments mention this.
    • 22 linked gists list no SHA1 at all.
    • Conversion scripts carry these models into TensorFlow, Keras, CoreML, MXNet and other frameworks.

Interesting ideas

  • I1. Poisoning the training data instead of changing the model. By adding triggered examples with chosen labels, the attacker leaves the architecture untouched. This is a useful reminder that auditing a model's structure says little about its behaviour, and that the training data deserves the same scrutiny.
  • I2. Measuring the attack on both clean and triggered inputs. The evaluation reports accuracy on normal inputs and on triggered inputs separately, so it shows both how stealthy and how effective the backdoor is. The same two-sided evaluation suits any work on robustness or data contamination.
  • I3. Testing whether the attack survives fine-tuning. Checking the backdoor after retraining on a new task reveals hidden risks in reused pre-trained models. This check is relevant wherever teams build on downloaded checkpoints, for example in NLP.
  • I4. Treating models like software packages. The authors recommend digital signatures, integrity-protected download channels, and actually checking hashes. These practices transfer directly to any shared model or dataset repository, including a family research library.
  • I5. Inspecting neurons that never fire during validation. The paper suggests examining network components that stay inactive across all validation inputs, since they may hide unusual behaviour. This is a cheap first heuristic for probing models whose training cannot be fully trusted.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓State University of New YorkAlbany · United States2 works · 1,026 citations

Also on the byline: Long Island University.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSoftware · 92 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Machine learning · 51 works

Cite

Cite as Tianyu Gu, Brendan Dolan-Gavitt and Siddharth Garg. 2017. BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv:1708.06733.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1708.06733 · arXiv 1708.06733

@misc{gu2017badnets,
  title = {{BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain}},
  author = {Tianyu Gu, Brendan Dolan-Gavitt and Siddharth Garg},
  year = {2017},
  doi = {10.48550/arxiv.1708.06733},
  eprint = {1708.06733},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1708.06733}
}

References

53 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. “ImageNet large scale visual recognition competition,” http://www.image-net.org/challenges/LSVRC/2012/, 2012. ↗
  2. A. Graves, A.-r. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks,” in Acoustics, speech and signal processing (icassp), 2013 ieee international conference on. IEEE, 2013, pp. 6645–6649. find
  3. K. M. Hermann and P. Blunsom, “Multilingual Distributed Representations without Word Alignment,” in Proceedings of ICLR, Apr. 2014. [Online]. Available: http://arxiv.org/abs/1312.6173 — [arXiv:1312.6173](https://arxiv.org/abs/1312.6173) ↗
  4. D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” 2014.
  5. V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” 2013. find
  6. D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mastering the game of go… find
  7. A. Karpathy, “What I learned from competing against a ConvNet on ImageNet,” http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/, 2014. ↗
  8. G. Chen, T. X. Han, Z. He, R. Kays, and T. Forrester, “Deep convolutional neural network based species recognition for wild animal monitoring,” in Image Processing (ICIP), 2014 IEEE International Conference on. IEEE, 2014, pp. 858–862. find
  9. C. Chen, A. Seff, A. Kornhauser, and J. Xiao, “Deepdriving: Learning affordance for direct perception in autonomous driving,” in Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), ser. ICCV ’15. Washington, DC, USA: IEEE Computer Society, 2015, pp. 2722–2730. [Online].… — [doi:10.1109/iccv.2015.312](https://doi.org/10.1109/iccv.2015.312) find
  10. Google, Inc., “Google Cloud Machine Learning Engine,” https://cloud.google.com/ml-engine/. ↗
  11. Microsoft Corp., “Azure Batch AI Training,” https://batchaitraining.azure.com/. ↗
  12. Amazon.com, Inc., “Deep Learning AMI Amazon Linux Version.” find
  13. K. Quach, “Cloud giants ‘ran out’ of fast GPUs for AI boffins,” https://www.theregister.co.uk/2017/05/22/cloud providers ai researchers/. ↗
  14. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems, 2012, pp. 1097–1105.
  15. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” 2014.
  16. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” 2015. find
  17. I. Evtimov, K. Eykholt, E. Fernandes, T. Kohno, B. Li, A. Prakash, A. Rahmati, and D. Song, “Robust physical-world attacks on machine learning models,” 2017. find
  18. J. Schmidhuber, “Deep learning in neural networks: An overview,” Neural networks, vol. 61, pp. 85–117, 2015. find
  19. A. Blum and R. L. Rivest, “Training a 3-node neural network is np-complete,” in Advances in neural information processing systems, 1989, pp. 494–501. find
  20. S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Transactions on knowledge and data engineering, vol. 22, no. 10, pp. 1345–1359, 2010. find
  21. X. Glorot, A. Bordes, and Y. Bengio, “Domain adaptation for large-scale sentiment classification: A deep learning approach,” in Proceedings of the 28th international conference on machine learning (ICML-11), 2011, pp. 513–520. find
  22. A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, “Cnn features off-the-shelf: An astounding baseline for recognition,” in Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops, ser. CVPRW ’14. Washington, DC, USA: IEEE Computer Society, 2014, pp. 512–… — [doi:10.1109/cvprw.2014.131](https://doi.org/10.1109/cvprw.2014.131) find
  23. F. Larsson, M. Felsberg, and P.-E. Forssen, “Correlating Fourier descriptors of local patches for road sign recognition,” IET Computer Vision, vol. 5, no. 4, pp. 244–254, 2011. find
  24. L. Huang, A. D. Joseph, B. Nelson, B. I. Rubinstein, and J. D. Tygar, “Adversarial machine learning,” in Proceedings of the 4th ACM Workshop on Security and Artificial Intelligence, ser. AISec ’11. New York, NY, USA: ACM, 2011, pp. 43–58. [Online]. Available: http://doi.acm.org/10.1145/2046684.20466… — [doi:10.1145/2046684.2046692](https://doi.org/10.1145/2046684.2046692) ↗
  25. N. Dalvi, P. Domingos, Mausam, S. Sanghai, and D. Verma, “Adversarial classification,” in Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser. KDD ’04. New York, NY, USA: ACM, 2004, pp. 99–108. [Online]. Available: http://doi.acm.org/10.1145/10140… — [doi:10.1145/1014052.1014066](https://doi.org/10.1145/1014052.1014066) ↗
  26. D. Lowd and C. Meek, “Adversarial learning,” in Proceedings of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery in Data Mining, ser. KDD ’05. New York, NY, USA: ACM, 2005, pp. 641–647. [Online]. Available: http://doi.acm.org/10.1145/1081870.1081950 — [doi:10.1145/1081870.1081950](https://doi.org/10.1145/1081870.1081950) ↗
  27. ——, “Good word attacks on statistical spam filters.” in Proceedings of the Conference on Email and Anti-Spam (CEAS), 2005. find
  28. G. L. Wittel and S. F. Wu, “On Attacking Statistical Spam Filters,” in Proceedings of the Conference on Email and Anti-Spam (CEAS), Mountain View, CA, USA, 2004. find
  29. J. Newsome, B. Karp, and D. Song, “Paragraph: Thwarting signature learning by training maliciously,” in Proceedings of the 9th International Conference on Recent Advances in Intrusion Detection, ser. RAID’06. Berlin, Heidelberg: Springer-Verlag, 2006, pp. 81–105. [Online]. Available: http://dx.doi.o… — [doi:10.1007/11856214](https://doi.org/10.1007/11856214) ↗
  30. S. P. Chung and A. K. Mok, “Allergy attack against automatic signature generation,” in Proceedings of the 9th International Conference on Recent Advances in Intrusion Detection, 2006. find
  31. ——, “Advanced allergy attacks: Does a corpus really help,” in Proceedings of the 10th International Conference on Recent Advances in Intrusion Detection, 2007. find
  32. C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” 2013.
  33. I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” 2014.
  34. N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, “Practical black-box attacks against machine learning,” 2016. find
  35. S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard, “Universal adversarial perturbations,” 2016. find
  36. S. Shen, S. Tople, and P. Saxena, “Auror: Defending against poisoning attacks in collaborative deep learning systems,” in Proceedings of the 32Nd Annual Conference on Computer Security Applications, ser. ACSAC ’16. New York, NY, USA: ACM, 2016, pp. 508–519. [Online]. Available: http://doi.acm.org/10… — [doi:10.1145/2991079.2991125](https://doi.org/10.1145/2991079.2991125) ↗
  37. Y. LeCun, L. Jackel, L. Bottou, C. Cortes, J. S. Denker, H. Drucker, I. Guyon, U. Muller, E. Sackinger, P. Simard et al., “Learning algorithms for classification: A comparison on handwritten digit recognition,” Neural networks: the statistical mechanics perspective, vol. 261, p. 276, 1995. find
  38. Y. Zhang, P. Liang, and M. J. Wainwright, “Convexified convolutional neural networks,” arXiv preprint arXiv:1609.01000, 2016. find
  39. S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in Advances in neural information processing systems, 2015, pp. 91–99.
  40. A. Møgelmose, D. Liu, and M. M. Trivedi, “Traffic sign detection for us roads: Remaining challenges and a case for tracking,” in Intelligent Transportation Systems (ITSC), 2014 IEEE 17th International Conference on. IEEE, 2014, pp. 1394–1399. find
  41. J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell, “Decaf: A deep convolutional activation feature for generic visual recognition,” in International conference on machine learning, 2014, pp. 647–655. find
  42. A. Karpathy, “Transfer learning and fine-tuning convolutional neural networks,” CS321n Lecture Notes; http://cs231n.github.io/transfer-learning/. ↗
  43. “Caffe Model Zoo,” https://github.com/BVLC/caffe/wiki/Model-Zoo. ↗
  44. Y. Zhu, C. Zhang, D. Zhou, X. Wang, X. Bai, and W. Liu, “Traffic sign detection and recognition using fully convolutional network guided proposals,” Neurocomputing, vol. 214, pp. 758 – 766, 2016. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S092523121630741X ↗
  45. S. Ruder, “Transfer learning - machine learning’s next frontier,” http://ruder.io/transfer-learning/. ↗
  46. F. Yu, “A comprehensive guide to fine-tuning deep learning models in Keras,” https://flyyufelix.github.io/2016/10/03/fine-tuning-in-keras-part1.html. ↗
  47. “Network in Network Imagenet Model,” https://gist.github.com/mavenlin/d802a5849de39225bcc6. ↗
  48. “Caffe models caffe-tensorflow. in TensorFlow,” https://github.com/ethereon/ ↗
  49. “Caffe to Keras converter,” https://github.com/qxcv/caffe2keras. ↗
  50. “Convert models from Caffe to Theano format,” https://github.com/kencoken/caffe-model-convert. ↗
  51. Apple Inc., “Converting trained models to Core https://developer.apple.com/documentation/coreml/converting trained models to core ml. ML,” ↗
  52. “Convert Caffe model to Mxnet format,” https://github.com/apache/incubator-mxnet/tree/master/tools/caffe converter. ↗
  53. “caffe2neon,” https://github.com/NervanaSystems/caffe2neon. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gu-2017-badnets-identifying-vulnerabilities-machine-learning/