Intriguing properties of neural networks
Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/1312.6199
Summary
This paper looks at two unexpected behaviours of deep neural networks trained for image classification. First, the authors check whether single hidden units in a high layer carry special meaning. They pick the held-out images that most strongly activate a unit, then do the same for random directions in the activation space. Both sets of images look about equally coherent. This suggests that meaning is spread across the whole representation space rather than tied to individual neurons. Second, the authors find that these networks are surprisingly unstable near correctly classified inputs. An optimisation procedure can find tiny perturbations, nearly invisible to people, that make the model predict the wrong class. They call the results "adversarial examples." These examples often also fool networks with other architectures or hyperparameters, and even networks trained on separate data. A spectral analysis of layer-wise Lipschitz bounds gives a partial explanation for the instability. This was the paper that introduced adversarial examples, and it started the field of research on neural network robustness.
Key points
- Models tested: fully connected ("FC") and autoencoder-based ("AE") classifiers on MNIST, "AlexNet" on ImageNet, and "QuocNet", an unsupervised network with about 1 billion parameters trained on roughly 10M YouTube images.
- Images selected by single units and by random linear combinations of units showed similar semantic consistency, on both MNIST and ImageNet. This challenges the idea that each unit captures a separate factor of variation.
- Adversarial perturbations are found by optimising the input to maximise prediction error. The resulting errors transfer across models, including between MNIST models trained on two disjoint 30,000-image subsets (P1 and P2). Transfer gets weaker in that setting but is still significant.
- Increasing the perturbation strength (standard deviation from 0.06 to 0.1, about 40% more) made the adversarial examples more effective on other models.
- Upper Lipschitz bounds per layer, computed with operator norms and a Fourier-based formula for convolutions, show that instability can start in AlexNet's first convolutional layer. The authors suggest penalising these bounds as a form of regularisation.
Cited by
- Understanding intermediate layers using linear classifier probes
- Generating Natural Language Adversarial Examples
- Massively Multilingual Word Embeddings
- Concrete Problems in AI Safety
- Analysis Methods in Neural Language Processing: A Survey
- Don't Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference
- On the Opportunities and Risks of Foundation Models
- Jailbreaking Black Box Large Language Models in Twenty Queries
- Plug and Play Language Models: A Simple Approach to Controlled Text Generation
- HotFlip: White-Box Adversarial Examples for Text Classification
- Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
- Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers
- Evaluating Models' Local Decision Boundaries via Contrast Sets
- BAE: BERT-based Adversarial Examples for Text Classification
- Explaining and Harnessing Adversarial Examples
- Baseline Defenses for Adversarial Attacks Against Aligned Language Models
- Adversarial Examples for Evaluating Reading Comprehension Systems
- Certified Robustness to Adversarial Word Substitutions
- Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment
- Visualizing and Understanding Neural Models in NLP
- TextBugger: Generating Adversarial Text Against Real-world Applications
- Towards Deep Learning Models Resistant to Adversarial Attacks
- A Holistic Approach to Undesired Content Detection in the Real World
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
- Tree of Attacks: Jailbreaking Black-Box LLMs Automatically
- Adversarial Training Methods for Semi-Supervised Text Classification
- Red Teaming Language Models with Language Models
- Combating Adversarial Misspellings with Robust Word Recognition
- Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer
- Tool Learning with Foundation Models
- Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency
- SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks
- Improved Techniques for Training GANs
- Analysing Mathematical Reasoning Abilities of Neural Models
- The Rise and Potential of Large Language Model Based Agents: A Survey
- Word-level Textual Adversarial Attacking as Combinatorial Optimization
- mixup: Beyond Empirical Risk Minimization
- Evaluating and Enhancing the Robustness of Neural Network-based Dependency Parsing Models with Adversarial Examples
- Representation Engineering: A Top-Down Approach to AI Transparency
- Universal and Transferable Adversarial Attacks on Aligned Language Models
Cite
@misc{szegedy2013intriguing,
title = {{Intriguing properties of neural networks}},
author = {Szegedy, Christian and Zaremba, Wojciech and Sutskever, Ilya and Bruna, Joan and Erhan, Dumitru and Goodfellow, Ian and Fergus, Rob},
year = {2013},
doi = {10.48550/arxiv.1312.6199},
eprint = {1312.6199},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.1312.6199}
}References
- David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. How to explain individual classification decisions. The Journal of Machine Learning Research, 99:1803–1831, 2010. find
- Yoshua Bengio. Learning deep architectures for ai. Foundations and trends® in Machine Learning, 2(1):1–127, 2009. find
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248–255. IEEE, 2009. find
- Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. Technical Report 1341, University of Montreal, June 2009. Also presented at the ICML 2009 Workshop on Learning Feature Hierarchies, Montréal, Canada. find
- Pedro Felzenszwalb, David McAllester, and Deva Ramanan. A discriminatively trained, multiscale, deformable part model. In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, pages 1–8. IEEE, 2008. find
- Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. arXiv preprint arXiv:1311.2524, 2013. find our card
- Ian Goodfellow, Quoc Le, Andrew Saxe, Honglak Lee, and Andrew Y Ng. Measuring invariances in deep networks. Advances in neural information processing systems, 22:646–654, 2009. find
- Geoffrey E. Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, and Brian Kingsbury. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal… find
- Alex Krizhevsky, Ilya Sutskever, and Geoff Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25, pages 1106–1114, 2012. find
- Quoc V Le, Marc’Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S Corrado, Jeff Dean, and Andrew Y Ng. Building high-level features using large scale unsupervised learning. arXiv preprint arXiv:1112.6209, 2011. — [arXiv:1112.6209](https://arxiv.org/abs/1112.6209) find
- Yann LeCun and Corinna Cortes. The mnist database of handwritten digits, 1998. find
- Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. find
- Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional neural networks. arXiv preprint arXiv:1311.2901, 2013. find our card