Adversarial Examples Are Not Bugs, They Are Features

Andrew IlyasShibani SanturkarDimitris TsiprasLogan EngstromBrandon TranAleksander Madry

article2019NeurIPS2,173 citations

Demonstrates that adversarial examples arise from predictive, human-imperceptible patterns inherent to training data rather than algorithmic flaws, establishing that models exploit real statistical signals invisible to human perception.

Listen

Deep neural networks are increasingly deployed in mission-critical applications, yet they remain susceptible to adversarial examples—minor, imperceptible input modifications that cause models to make catastrophic prediction errors. Historically, researchers and practitioners viewed these vulnerabilities as aberrations, statistical anomalies, or bugs resulting from high-dimensional geometry and finite-sample overfitting. Consequently, common remediation efforts have focused on heuristic defenses, input pre-processing, and standard regularization, often failing to secure models against adaptive attacks. The article sets out to challenge this conventional view by demonstrating that adversarial vulnerability stems directly from standard machine learning algorithms learning non-robust, human-incomprehensible patterns that are inherently present and highly predictive within real-world datasets.

To evaluate this thesis, the authors conducted empirical experiments using standard computer vision benchmarks (CIFAR-10 and Restricted ImageNet) across multiple neural network architectures such as ResNet, VGG, DenseNet, and Inception. They created two modified dataset types: "robustified" datasets, where non-robust signals were filtered out, and "non-robust" datasets, where human-visible features were decoupled from labels, leaving only subtle adversarial perturbations aligned with the assigned classes. Alongside empirical testing against diverse adversarial attacks, the authors developed a rigorous theoretical framework using Gaussian maximum likelihood classification to mathematically model how adversarial vulnerability arises from misalignments between human-specified distance metrics and intrinsic data geometry.

The findings establish three key conclusions. First, standard training on a robustified dataset yields a model that attains substantial adversarial accuracy (over 48% on CIFAR-10) without using adversarial training, proving that robustness can be derived purely from dataset properties. Second, models trained entirely on non-robust datasets achieve strong generalization on standard, unmodified test sets—reaching 63.3% on CIFAR-10 and 87.9% on Restricted ImageNet—despite appearing completely mislabeled to human observers. Third, the degree to which different architectures learn non-robust features directly predicts their susceptibility to cross-model transfer attacks, explaining why adversarial examples frequently fool multiple independent models. The theoretical analysis confirms that robust optimization operates by forcing models to align their internal feature metrics with human-defined priors.

These results demonstrate that adversarial vulnerability is not a technical bug in optimization algorithms, but a natural consequence of the standard supervised learning paradigm maximizing predictive accuracy. Classifiers naturally exploit any useful statistical correlation, whether perceptible to humans or not. This creates critical risks for safety, security, and interpretability: post-hoc explanation methods that force human-meaningful visualizations merely conceal the genuine, non-robust signals models rely upon. Organizations cannot expect standard training to yield secure or transparent models without deliberately constraining the feature space.

Decision-makers and engineering teams must discontinue relying solely on standard empirical risk minimization and post-hoc interpretability for safety-critical systems. Instead, human perceptual priors and robustness constraints must be directly integrated into data collection, dataset curation, and training pipelines. While robust optimization effectively suppresses brittle features, it often introduces a trade-off with standard accuracy and requires higher sample complexity. Future initiatives should focus on developing advanced dataset-level feature filtering methods and establishing scalable pipelines that systematically enforce alignment between algorithmic feature representations and human domain expectations.

The conclusions are supported by theoretical proofs and multi-architecture empirical validations that withstand adaptive attacks. However, decision-makers should note certain limitations: the empirical evaluations were primarily focused on image classification benchmarks, robust feature isolation currently relies on approximations via adversarially trained models, and dataset transformation introduces slight generalization gaps due to distribution shifts. Confidence in the underlying mechanism remains high, but operational defenses must account for problem-specific geometric and data constraints.

Massachusetts Institute of Technology feature space geometry illustrative examples Features Robust Adversarial

arXiv: 1905.02175 MadryLab/constructed-datasets

Paper: Intriguing properties of neural networks, Christian Szegedy et al. (2014). Reading this foundational paper first is essential because it originally uncovers the brittleness and vulnerability of neural networks to tiny perturbations.
Paper: Explaining and Harnessing Adversarial Examples, Ian J. Goodfellow et al. (2015). This classic study is a vital prerequisite as it introduces adversarial examples and attributes them to the linear nature of high-dimensional models.
Paper: Towards Deep Learning Models Resistant to Adversarial Attacks, A. Ma̧dry et al. (2017). Reviewing this work provides a necessary theoretical foundation by formulating adversarial robustness as a saddle-point optimization problem.

Paper: Theoretically Principled Trade-off between Robustness and Accuracy, Hongyang Zhang et al. (2019). This ebook naturally follows by building directly on the non-robust features perspective to formally analyze and mitigate the trade-off between robustness and accuracy.
Paper: Shortcut learning in deep neural networks, Robert Geirhos et al. (2020). This paper extends the source's findings by broadening the scope from adversarial vulnerabilities to general shortcut learning across diverse machine learning domains.

Cover for Adversarial Examples Are Not Bugs, They Are Features

Abstract

Adversarial examples have attracted significant attention in machine learning, but the reasons for their existence and pervasiveness remain unclear. We demonstrate that adversarial examples can be directly attributed to the presence of non-robust features: features derived from patterns in the data distribution that are highly predictive, yet brittle and incomprehensible to humans. After capturing these features within a theoretical framework, we establish their widespread existence in standard datasets. Finally, we present a simple setting where we can rigorously tie the phenomena we observe in practice to a misalignment between the (human-specified) notion of robustness and the inherent geometry of the data.

Topics

feature space geometry illustrative examples robust performance

Keywords

Features Robust Adversarial Classi Non-robust Features Training Adversarial Examples Standard Set Model

1 Introduction
2 The Robust Features Model
3 Finding Robust (and Non-Robust) Features
3.1 Disentangling robust and non-robust features
3.2 Non-robust features suffice for standard classification
3.3 Transferability can arise from non-robust features
4 A Theoretical Framework for Studying (Non)-Robust Features
5 Related Work
6 Conclusion
7 Acknowledgements
References
A Connections to and Disambiguation from Other Models
B Additional Related Work
C Experimental Setup
C.1 Datasets
C.2 Models
C.3 Adversarial training
C.4 Constructing a Robust Dataset
C.5 Non-robust features suffice for standard classification
D Omitted Experiments and Figures
D.1 Detailed evaluation of models trained on “robust” dataset
D.2 Adversarial evaluation
D.3 Performance of “robust” training and test set
D.4 Classification based on non-robust features
D.5 Accuracy curves
D.6 Performance of ERM classifiers on relabeled test set
D.7 Generalization to CIFAR-10.1
D.8 Omitted Results for Restricted ImageNet
D.9 Targeted Transferability
D.10 Robustness vs. Accuracy
E Gaussian MLE under Adversarial Perturbation
E.1 Setup
E.2 Outline and Key Results
E.3 Proofs
E.3.1 Motivating example: ℓ2\ell_{2}-constrained adversary
E.3.2 Variant with Fixed Lagrangian (Theorem )
E.3.3 Real objective
E.3.4 Danskin’s Theorem
E.3.5 Applying Danskin’s Theorem
E.3.6 Bounding λ\lambda
E.3.7 Proof of main theorems
E.3.8 Bounds for λ\lambda

Knowls

Knowl 1 — Useful, Robust, and Non-Robust Features Framework
definition
In supervised binary classification over a distribution $\mathcal{D}$ with input-label pairs $(x, y) \in \mathcal{X} \times \{-1, +1\}$ , a feature is a real-valued mapping $f: \mathcal{X} \to \mathbb{R}$ , normalized to mean zero and unit variance ( $\mathbb{E}_{(x,y)\sim \mathcal{D}}[f(x)] = 0$ and $\mathbb{E}_{(x,y)\sim \mathcal{D}}[f(x)^2] = 1$ ). Features are categorized into three classes:
1. $\rho$ -useful features: A feature $f$ is $\rho$ -useful for $\rho > 0$ under distribution $\mathcal{D}$ if it is correlated with the true label in expectation: $\mathbb{E}_{(x,y)\sim \mathcal{D}}[y \cdot f(x)] \ge \rho$ The quantity $\rho_\mathcal{D}(f)$ denotes the supremum $\rho$ for which $f$ is $\rho$ -useful.
2. $\gamma$ -robustly useful features: For a specified set of valid perturbations $\Delta(x)$ (such as an $\ell_p$ -ball $\mathcal{B}_p(\epsilon)$ ), a $\rho$ -useful feature $f$ is $\gamma$ -robustly useful (or a robust feature) for $\gamma > 0$ if, under worst-case perturbation, it maintains expected correlation at least $\gamma$ : $\mathbb{E}_{(x,y)\sim \mathcal{D}}\left[\inf_{\delta \in \Delta(x)} y \cdot f(x + \delta)\right] \ge \gamma$
3. Useful, non-robust features: A feature $f$ is useful and non-robust if it is $\rho$ -useful for some $\rho > 0$ , but is not $\gamma$ -robustly useful for any $\gamma \ge 0$ . An adversary can invert the sign of the feature-label correlation with a small perturbation $\delta \in \Delta(x)$ , even though the feature provides predictive power on unperturbed data.
A classifier $C = (\mathcal{F}, w, b)$ parameterized by weights $w = \{w_f\}_{f \in \mathcal{F}}$ and bias $b$ predicts $C(x) = \text{sgn}\left(b + \sum_{f \in \mathcal{F}} w_f f(x)\right)$ . Under standard empirical risk minimization (ERM), loss functions do not differentiate between robust and non-robust features; any $\rho$ -useful feature is utilized to reduce empirical loss.
Knowl 2 — Algorithm for Constructing Robust-Feature Datasets
algorithm
To isolate robust features from a dataset $\mathcal{D}$ , one uses a representation mapping $g_R: \mathcal{X} \to \mathbb{R}^k$ from the penultimate layer of an adversarially trained (robust) neural network $C_R$ . The procedure generates a modified dataset $\widehat{\mathcal{D}}_R$ by matching the penultimate representations of original inputs while initializing the optimization from an input $x_0$ drawn independently of the label, ensuring non-robust features are uncorrelated with the label across the constructed dataset.
Input: Training dataset $\mathcal{D}$ , robust classifier $C_R$ with representation mapping $g_R$
Output: Robustified dataset $\widehat{\mathcal{D}}_R$
$\widehat{\mathcal{D}}_R \leftarrow \{\}$
for each $(x, y) \in \mathcal{D}$ do
    Sample $x_0 \sim \mathcal{D}$ independently of $(x, y)$ (or sample $x_0$ from random Gaussian noise)
     $x_R \leftarrow \arg\min_{z \in [0, 1]^d} \|g_R(z) - g_R(x)\|_2$ initialized at $z = x_0$ via normalized $\ell_2$ gradient descent
     $\widehat{\mathcal{D}}_R \leftarrow \widehat{\mathcal{D}}_R \cup \{(x_R, y)\}$
end for
return $\widehat{\mathcal{D}}_R$
Optimization of $z$ uses normalized projected gradient descent where the $\ell_2$ -norm of each gradient step is constant (e.g., step size 0.1 for 1,000 iterations on CIFAR-10; step size 1.0 for 2,000 iterations on Restricted ImageNet; or 10,000 iterations when initializing from Gaussian noise), clipping pixel values to $[0, 1]$ after each update.
Knowl 3 — Algorithm for Constructing Non-Robust-Feature Datasets
algorithm
To construct a dataset where generalization signal derives exclusively from non-robust features, small adversarial perturbations are added to natural images to induce a standard (non-robust) classifier to predict a chosen target label $t$ , and the modified image is relabeled as $t$ .
Input: Training dataset $\mathcal{D}$ , standard classifier $C$ with loss $L_C$ , perturbation bound $\epsilon$ , label mode (random or deterministic)
Output: Non-robust dataset $\widehat{\mathcal{D}}_{NR}$ (either $\widehat{\mathcal{D}}_{rand}$ or $\widehat{\mathcal{D}}_{det}$ )
$\widehat{\mathcal{D}}_{NR} \leftarrow \{\}$
for each $(x, y) \in \mathcal{D}$ do
    if label mode is random then
        Sample target label $t \sim \text{Uniform}(\{1, \dots, K\} \setminus \{y\})$
    else if label mode is deterministic then
        Set target label $t \leftarrow (y + 1) \bmod K$ (where $K$ is total number of classes)
    end if
     $x_{adv} \leftarrow \arg\min_{x' : \|x' - x\|_2 \le \epsilon} L_C(x', t)$ solved via targeted $\ell_2$ -PGD starting from $x$
     $\widehat{\mathcal{D}}_{NR} \leftarrow \widehat{\mathcal{D}}_{NR} \cup \{(x_{adv}, t)\}$
end for
return $\widehat{\mathcal{D}}_{NR}$
Because $\|x_{adv} - x\|_2 \le \epsilon$ is imperceptible, human-visible robust features remain aligned with original class $y$ . In the random mapping $\widehat{\mathcal{D}}_{rand}$ , robust features become uncorrelated with $t$ in expectation. In the deterministic mapping $\widehat{\mathcal{D}}_{det}$ , robust features are systematically anti-correlated with label $t$ , whereas non-robust features correlate with $t$ . PGD uses $\epsilon = 0.5$ (in $\ell_2$ ), step size 0.1, 100 iterations for CIFAR-10, and $\epsilon = 3.0$ , step size 0.1, 100 iterations for Restricted ImageNet.
Knowl 4 — Robust Classification via Standard Training on Robustified Datasets
empirical result
When a classifier is trained using standard, non-adversarial empirical risk minimization (ERM) on the robustified dataset $\widehat{\mathcal{D}}_R$ (constructed by matching representations of an adversarially trained model), the resulting classifier attains substantial adversarial robustness when evaluated on the original, unmodified test set $\mathcal{D}$ .

On CIFAR-10 using ResNet-50:
- Standard ERM on original $\mathcal{D}$ : 95.25% standard test accuracy, 4.49% robust test accuracy at $\ell_2$ radius $\epsilon = 0.25$ , and 0.0% robust accuracy at $\epsilon = 0.5$ .
- Adversarial training (PGD, $\epsilon = 0.5$ ) on $\mathcal{D}$ : 90.83% standard test accuracy, 82.48% robust test accuracy at $\epsilon = 0.25$ , and 70.90% at $\epsilon = 0.5$ .
- Standard ERM on robustified $\widehat{\mathcal{D}}_R$ (constructed from images): 85.40% standard test accuracy, 48.20% robust test accuracy at $\epsilon = 0.25$ , and 21.85% at $\epsilon = 0.5$ .
- Standard ERM on robustified $\widehat{\mathcal{D}}_R$ (constructed from noise): 84.10% standard test accuracy, 48.27% robust test accuracy at $\epsilon = 0.25$ , and 29.40% at $\epsilon = 0.5$ .
- Control (Standard ERM on non-robust dataset $\widehat{\mathcal{D}}_{NR}$ constructed from standard model features): 87.68% standard accuracy, 0.82% robust accuracy at $\epsilon = 0.25$ , and 0.0% at $\epsilon = 0.5$ .
On Restricted ImageNet, standard ERM on $\widehat{\mathcal{D}}_R$ achieves over 80% standard test accuracy and over 40% robust test accuracy against $\ell_2$ perturbations of $\epsilon = 0.5$ . These findings demonstrate that adversarial vulnerability is primarily a property of dataset features rather than an intrinsic flaw of the standard ERM training objective.

Knowl 5 — Non-Robust Features Suffice for Generalization on Clean Test Sets

data/table

Standard neural networks trained exclusively on datasets where labels are associated with inputs solely via imperceptible adversarial perturbations ( $\widehat{\mathcal{D}}_{rand}$ and $\widehat{\mathcal{D}}_{det}$ ) achieve high classification accuracy when tested on the original, unaltered test distribution $\mathcal{D}$ .

In $\widehat{\mathcal{D}}_{rand}$ , perturbations towards random target classes render robust features uninformative in expectation while non-robust features predict the label. In $\widehat{\mathcal{D}}_{det}$ , where target classes are chosen by deterministic label shifts ( $t = (y + 1) \bmod K$ ), human-recognizable robust features actively point to the wrong class, yet standard ERM still extracts non-robust features that generalize to the clean test distribution $\mathcal{D}$ . Furthermore, classifiers trained on CIFAR-10 $\widehat{\mathcal{D}}_{det}$ and $\widehat{\mathcal{D}}_{rand}$ achieve 44% and 55% accuracy on the distribution-shifted CIFAR-10.1 test set, exceeding linear predictions and demonstrating that non-robust features are true generalizing patterns rather than finite-sample overfitting artifacts.

When the same construction of $\widehat{\mathcal{D}}_{rand}$ and $\widehat{\mathcal{D}}_{det}$ is executed using an adversarially trained (robust) model to generate perturbations, standard test accuracy on $\mathcal{D}$ drops to 25.2% for $\widehat{\mathcal{D}}_{rand}$ and 5.8% for $\widehat{\mathcal{D}}_{det}$ on CIFAR-10, while the model trained on the robust $\widehat{\mathcal{D}}_{det}$ achieves 65.5% accuracy on a permuted-label test set.

Knowl 6 — Adversarial Transferability Correlates with Reliance on Shared Non-Robust Features
empirical result
Adversarial perturbations transfer across independently trained models and distinct architectures because different models trained on the same data distribution learn similar sets of non-robust features.

When five architectures (VGG-16, Inception-v3, ResNet-18, DenseNet, and ResNet-50) are trained on the non-robust dataset $\widehat{\mathcal{D}}_{det}$ generated from a standard ResNet-50 source model, their test accuracy on clean data correlates directly with their susceptibility to transfer attacks generated by that same source model:
- VGG-16 achieves ~28% clean test accuracy when trained on $\widehat{\mathcal{D}}_{det}$ and experiences a transfer attack success rate of ~65%.
- Inception-v3 achieves ~39% clean test accuracy on $\widehat{\mathcal{D}}_{det}$ and experiences a transfer success rate of ~82%.
- ResNet-18 and DenseNet achieve ~40–42% clean test accuracy on $\widehat{\mathcal{D}}_{det}$ and experience transfer success rates of ~93–94%.
- ResNet-50 achieves ~45% clean test accuracy on $\widehat{\mathcal{D}}_{det}$ and experiences a transfer success rate of ~98%.
A matching linear relationship is observed for targeted transfer attacks (transfer success rates ranging from ~10% for VGG-16 to ~72% for ResNet-50). Architectures that more effectively learn the non-robust features extracted by the source model are proportionally more susceptible to adversarial examples transferred from that model.
Knowl 7 — Adversarial Vulnerability from Metric Misalignment in Gaussian Classification
theoretical result
Consider binary classification between two Gaussian distributions where $y \in \{-1, +1\}$ uniformly at random and $x \sim \mathcal{N}(y \cdot \mu_*, \Sigma_*)$ with true parameters $\Theta_* = (\mu_*, \Sigma_*)$ . Non-robust maximum likelihood estimation (MLE) minimizes Gaussian negative log-likelihood $\ell(x; y \cdot \mu, \Sigma)$ , recovering $\mu = \mu_*$ and $\Sigma = \Sigma_*$ , which induces the Mahalanobis feature metric $\|x - \mu\|_\Theta = \sqrt{(x - \mu)^\top \Sigma^{-1} (x - \mu)}$ .

For an adversary with a Lagrangian penalty trading off negative log-likelihood maximization and an $\ell_2$ perturbation penalty: $\max_{\delta} \ell(x + \delta; y \cdot \mu, \Sigma) - C \|\delta\|_2$ where $C \ge \frac{1}{\sigma_{\min}(\Sigma_*)}$ , the optimal adversarial perturbation is $\delta^* = (C\Sigma - I)^{-1}(x - \mu)$ , and the excess adversarial loss $\mathcal{L}_{adv}(\Theta) - \mathcal{L}(\Theta)$ incurred by the MLE parameters $\Theta = (\mu_*, \Sigma_*)$ is: $\mathcal{L}_{adv}(\Theta) - \mathcal{L}(\Theta) = \text{tr}\left(\left[I + (C \cdot \Sigma_* - I)^{-1}\right]^2\right) - d$ For any fixed total variance $\text{tr}(\Sigma_*) = k$ , this excess adversarial loss is strictly minimized when: $\Sigma_* = \frac{k}{d} I$ Adversarial vulnerability corresponds to the geometric misalignment between the feature-induced metric $\Sigma_*^{-1}$ and the adversary's $\ell_2$ metric $I$ . When $\Sigma_*$ has non-uniform eigenvalues, small perturbations in the $\ell_2$ metric along low-variance directions induce large shifts in the data-dependent feature metric.
Knowl 8 — Robust Covariance Estimation Blends Data Geometry with Adversary Metric
theoretical result
For binary Gaussian classification $x \sim \mathcal{N}(y \cdot \mu_*, \Sigma_*)$ with $y \in \{-1, +1\}$ , consider robust maximum likelihood estimation under an adversary choosing a diagonal linear operator $M \in \mathcal{M}$ with expected squared $\ell_2$ perturbation budget $\mathbb{E}[\|M(x - \mu)\|^2_2] = \epsilon^2$ : $\min_{\mu, \Sigma} \max_{M \in \mathcal{M}} \mathbb{E}_{x \sim \mathcal{N}(\mu_*, \Sigma_*)} [\ell(\mu, \Sigma; x + M(x - \mu))]$ where $\mathcal{M} = \{M \in \mathbb{R}^{d \times d} : M_{ij} = 0 \; \forall i \ne j, \; \mathbb{E}_{x \sim \mathcal{N}(\mu_*, \Sigma_*)}[\|M(x - \mu)\|^2_2] = \epsilon^2\}$ .

The robustly learned parameters $(\mu_r, \Sigma_r)$ satisfy:
1. The robust mean recovers the true mean: $\mu_r = \mu_*$ .
2. There exists $\epsilon_0 > 0$ such that for any perturbation budget $\epsilon \in [0, \epsilon_0)$ , the robust covariance $\Sigma_r$ is uniquely given by: $\Sigma_r = \frac{1}{2} \Sigma_* + \frac{1}{\lambda} I + \sqrt{\frac{1}{\lambda} \Sigma_* + \frac{1}{4} \Sigma_*^2}$ where the Lagrange multiplier $\lambda$ is bounded as: $\Omega\left(\frac{1 + \epsilon^{1/2}}{\epsilon^{1/2} + \epsilon^{3/2}}\right) \le \lambda \le \mathcal{O}\left(\frac{1 + \epsilon^{1/2}}{\epsilon^{1/2}}\right)$
As the perturbation budget $\epsilon$ increases, $\lambda$ decreases, causing $\Sigma_r$ to interpolate between the true data covariance $\Sigma_*$ and the identity matrix $I$ . This inflates the learned variance along directions of small initial variance in $\Sigma_*$ , preventing the model from relying heavily on brittle non-robust directions.
Knowl 9 — Improved Gradient Alignment and Interpretability of Robust Classifiers
theoretical result
Let $f(x) = \text{sgn}(x^\top \Sigma_*^{-1} \mu_*)$ and $f_r(x) = \text{sgn}(x^\top \Sigma_r^{-1} \mu_*)$ be monotonic classifiers based on the linear separators from standard and $\ell_2$ -robust Gaussian maximum likelihood estimation, respectively.

Because the robust covariance $\Sigma_r = \frac{1}{2} \Sigma_* + \frac{1}{\lambda} I + \sqrt{\frac{1}{\lambda} \Sigma_* + \frac{1}{4} \Sigma_*^2}$ satisfies $\kappa(\Sigma_r) \le \kappa(\Sigma_*)$ (where $\kappa(A) = \frac{\lambda_{\max}(A)}{\lambda_{\min}(A)}$ is the condition number of matrix $A$ ), the minimum worst-case alignment angle between the classifier's input gradient and the inter-class mean vector $\mu_*$ is strictly smaller for the robust classifier: $\min_{\mu_*} \frac{\langle \mu_*, \nabla_x f_r(x)\rangle}{\|\mu_*\|_2 \|\nabla_x f_r(x)\|_2} > \min_{\mu_*} \frac{\langle \mu_*, \nabla_x f(x)\rangle}{\|\mu_*\|_2 \|\nabla_x f(x)\|_2}$ For any positive definite matrix $A$ , $\min_x \frac{x^\top A x}{\|Ax\|_2 \|x\|_2} = \frac{2 \sqrt{\kappa(A)}}{1 + \kappa(A)}$ , which is strictly decreasing in $\kappa(A)$ . As a result, robust training forces classifier input gradients to align more closely under the $\ell_2$ inner product with the semantic vector connecting class means, providing a theoretical explanation for the human-interpretable saliency maps observed in robust neural networks.
Knowl 10 — Generalization Gap and Distributional Shift in Robust Dataset Evaluation
empirical result
When a standard empirical risk minimization (ERM) model is trained on the robustified dataset $\widehat{\mathcal{D}}_R$ and evaluated against an $\ell_2$ adversary with $\epsilon = 0.25$ , its performance degradation from training to test decomposes into a standard generalization gap and a distributional shift between $\widehat{\mathcal{D}}_R$ and the original distribution $\mathcal{D}$ .

On CIFAR-10:
- Robust training set (samples from $\widehat{\mathcal{D}}_R$ on which the model was trained): 77.33% robust accuracy.
- Robust test set (unseen samples generated by the robust feature extraction pipeline applied to the test set): 62.49% robust accuracy.
- Standard test set (unmodified natural test set $\mathcal{D}$ ): 48.27% robust accuracy.
The difference between 77.33% and 62.49% represents the generalization gap on the robust feature distribution, while the difference between 62.49% and 48.27% reflects the distributional shift between distilled samples and natural images. Furthermore, evaluating the model trained on $\widehat{\mathcal{D}}_R$ under projected gradient descent (PGD) with up to 2,500 steps or the Carlini-Wagner $\ell_2$ (CW- $\ell_2$ ) attack with 1,000 steps shows that accuracy plateaus at 48.27% without dropping by more than 2%, verifying genuine robustness rather than gradient masking.

Coverage note — Omitted intermediate mathematical proof steps (Danskin's theorem subdifferential derivation, asymptotic Gaussian concentration bounds for linear perturbation operators) and 2D geometric toy visualizations from Appendix D.10.

References

1.Anish Athalye, Nicholas Carlini, and David A. Wagner. “Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples”. In: International Conference on Machine Learning (ICML). 2018.
2.Anish Athalye et al. “Synthesizing Robust Adversarial Examples”. In: International Conference on Machine Learning (ICML). 2018.
3.Cristian Buciluˇa, Rich Caruana, and Alexandru Niculescu-Mizil. “Model compression”. In: International Conference on Knowledge Discovery and Data Mining (KDD). 2006.
4.Battista Biggio et al. “Evasion attacks against machine learning at test time”. In: Joint European conference on machine learning and knowledge discovery in databases (ECML-KDD). 2013.
5.Sébastien Bubeck, Eric Price, and Ilya Razenshteyn. “Adversarial examples from computational constraints”. In: arXiv preprint arXiv:1805.10204. 2018.
6.Nicholas Carlini et al. “On Evaluating Adversarial Robustness”. In: ArXiv preprint arXiv:1902.06705. 2019.
7.Jeremy M Cohen, Elan Rosenfeld, and J Zico Kolter. “Certified adversarial robustness via randomized smoothing”. In: arXiv preprint arXiv:1902.02918. 2019.
8.Zachary Charles, Harrison Rosenberg, and Dimitris Papailiopoulos. “A Geometric Perspective on the Transferability of Adversarial Directions”. In: International Conference on Artificial Intelligence and Statistics (AISTATS). 2019.
9.Nicholas Carlini and David Wagner. “Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods”. In: Workshop on Artificial Intelligence and Security (AISec). 2017.
10.Nicholas Carlini and David Wagner. “Towards evaluating the robustness of neural networks”. In: Symposium on Security and Privacy (SP). 2017.
11.John M. Danskin. The Theory of Max-Min and its Application to Weapons Allocation Problems. 1967.
12.Constantinos Daskalakis et al. “Efficient Statistics, in High Dimensions, from Truncated Samples”. In: Foundations of Computer Science (FOCS). 2019.
13.Gavin Weiguang Ding et al. “On the Sensitivity of Adversarial Robustness to Input Data Distributions”. In: International Conference on Learning Representations. 2019.
14.Logan Engstrom et al. “A Discussion of ’Adversarial Examples Are Not Bugs, They Are Features’”. In: Distill (2019). https://distill.pub/2019/advex-bugs-discussion. DOI: 10 . 23915 / distill.00019.
15.Logan Engstrom et al. “A Rotation and a Translation Suffice: Fooling CNNs with Simple Transformations”. In: International Conference on Machine Learning (ICML). 2019.
16.Alhussein Fawzi, Hamza Fawzi, and Omar Fawzi. “Adversarial vulnerability for any classifier”. In: Advances in Neural Information Processing Systems (NeuRIPS). 2018.
17.Alhussein Fawzi, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. “Robustness of classifiers: from adversarial to random noise”. In: Advances in Neural Information Processing Systems. 2016.
18.Nic Ford et al. “Adversarial Examples Are a Natural Consequence of Test Error in Noise”. In: arXiv preprint arXiv:1901.10513. 2019.
19.Tommaso Furlanello et al. “Born-Again Neural Networks”. In: International Conference on Machine Learning (ICML). 2018.
20.Robert Geirhos et al. “ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.” In: International Conference on Learning Representations. 2019.
21.Justin Gilmer et al. “Adversarial spheres”. In: Workshop of International Conference on Learning Representations (ICLR). 2018.
22.Gabriel Goh. “A Discussion of ’Adversarial Examples Are Not Bugs, They Are Features’: Robust Feature Leakage”. In: Distill (2019). https://distill.pub/2019/advex-bugs-discussion/response-2. DOI: 10.23915/distill.00019.2.
23.Gabriel Goh. “A Discussion of ’Adversarial Examples Are Not Bugs, They Are Features’: Two Examples of Useful, Non-Robust Features”. In: Distill (2019). https://distill.pub/2019/advex-bugs-discussion/response-3. DOI: 10.23915/distill.00019.3.
24.Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. “Explaining and Harnessing Adversarial Examples”. In: International Conference on Learning Representations (ICLR). 2015.
25.Dan Hendrycks and Thomas G. Dietterich. “Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations”. In: International Conference on Learning Representations (ICLR). 2019.
26.Kaiming He et al. “Deep Residual Learning for Image Recognition”. In: Conference on Computer Vision and Pattern Recognition (CVPR). 2016.
27.Warren He et al. “Adversarial example defense: Ensembles of weak defenses are not strong”. In: USENIX Workshop on Offensive Technologies (WOOT). 2017.
28.Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. “Distilling the Knowledge in a Neural Network”. In: Neural Information Processing Systems (NeurIPS) Deep Learning Workshop. 2014.
29.Saumya Jetley, Nicholas Lord, and Philip Torr. “With friends like these, who needs adversaries?” In: Advances in Neural Information Processing Systems (NeurIPS). 2018.
30.Alex Krizhevsky. “Learning Multiple Layers of Features from Tiny Images”. In: Technical report. 2009.
31.Beomsu Kim, Junghoon Seo, and Taegyun Jeon. “Bridging Adversarial Robustness and Gradient Interpretability”. In: International Conference on Learning Representations Workshop on Safe Machine Learning (ICLR SafeML). 2019.
32.Mathias Lecuyer et al. “Certified robustness to adversarial examples with differential privacy”. In: Symposium on Security and Privacy (SP). 2019.
33.Yanpei Liu et al. “Delving into Transferable Adversarial Examples and Black-box Attacks”. In: International Conference on Learning Representations (ICLR). 2017.
34.Beatrice Laurent and Pascal Massart. “Adaptive estimation of a quadratic functional by model selection”. In: Annals of Statistics. 2000.
35.Aleksander Madry et al. “Towards deep learning models resistant to adversarial attacks”. In: International Conference on Learning Representations (ICLR). 2018.
36.Saeed Mahloujifar, Dimitrios I Diochnos, and Mohammad Mahmoody. “The curse of concentration in robust learning: Evasion and poisoning attacks from concentration of measure”. In: AAAI Conference on Artificial Intelligence (AAAI). 2018.
37.Seyed-Mohsen Moosavi-Dezfooli et al. “Universal adversarial perturbations”. In: conference on computer vision and pattern recognition (CVPR). 2017.
38.Aravindh Mahendran and Andrea Vedaldi. “Understanding deep image representations by inverting them”. In: computer vision and pattern recognition (CVPR). 2015.
39.Preetum Nakkiran. “A Discussion of ’Adversarial Examples Are Not Bugs, They Are Features’: Adversarial Examples are Just Bugs, Too”. In: Distill (2019). https://distill.pub/2019/advex-bugs-discussion/response-5. DOI: 10.23915/distill.00019.5.
40.Preetum Nakkiran. “Adversarial robustness may be at odds with simplicity”. In: arXiv preprint arXiv:1901.00532. 2019.
41.Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. “Feature Visualization”. In: Distill. 2017.
42.Nicolas Papernot et al. “Practical black-box attacks against machine learning”. In: Asia Conference on Computer and Communications Security. 2017.
43.Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. “Transferability in Machine Learning: from Phenomena to Black-box Attacks using Adversarial Samples”. In: ArXiv preprint arXiv:1605.07277. 2016.
44.Benjamin Recht et al. “Do CIFAR-10 Classifiers Generalize to CIFAR-10?” In: International Conference on Machine Learning (ICML). 2019.
45.Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. “Certified defenses against adversarial examples”. In: International Conference on Learning Representations (ICLR). 2018.
46.Olga Russakovsky et al. “ImageNet Large Scale Visual Recognition Challenge”. In: International Journal of Computer Vision (IJCV). 2015.
47.Ludwig Schmidt et al. “Adversarially Robust Generalization Requires More Data”. In: Advances in Neural Information Processing Systems (NeurIPS). 2018.
48.Ali Shafahi et al. “Are adversarial examples inevitable?” In: International Conference on Learning Representations (ICLR). 2019.
49.Adi Shamir et al. “A Simple Explanation for the Existence of Adversarial Examples with Small Hamming Distance”. In: arXiv preprint arXiv:1901.10861. 2019.
50.David Stutz, Matthias Hein, and Bernt Schiele. “Disentangling Adversarial Robustness and Generalization”. In: Computer Vision and Pattern Recognition (CVPR). 2019.
51.D. Smilkov et al. “SmoothGrad: removing noise by adding noise”. In: ICML workshop on visualization for deep learning. 2017.
52.Arun Sai Suggala et al. “Revisiting Adversarial Risk”. In: Conference on Artificial Intelligence and Statistics (AISTATS). 2019.
53.Christian Szegedy et al. “Intriguing properties of neural networks”. In: International Conference on Learning Representations (ICLR). 2014.
54.Thomas Tanay and Lewis Griffin. “A Boundary Tilting Perspective on the Phenomenon of Adversarial Examples”. In: ArXiv preprint arXiv:1608.07690. 2016.
55.Florian Tramer et al. “The Space of Transferable Adversarial Examples”. In: ArXiv preprint arXiv:1704.03453. 2017.
56.Dimitris Tsipras et al. “Robustness May Be at Odds with Accuracy”. In: International Conference on Learning Representations (ICLR). 2019.
57.Jonathan Uesato et al. “Adversarial Risk and the Dangers of Evaluating Against Weak Attacks”. In: International Conference on Machine Learning (ICML). 2018.
58.Tongzhou Wang et al. “Dataset Distillation”. In: ArXiv preprint arXiv:1811.10959. 2018.
59.Eric Wong and J Zico Kolter. “Provable defenses against adversarial examples via the convex outer adversarial polytope”. In: International Conference on Machine Learning (ICML). 2018.
60.Kai Y. Xiao et al. “Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability”. In: International Conference on Learning Representations (ICLR). 2019.
61.Haosheng Zou et al. “Geometric Universality of Adversarial Examples in Deep Learning”. In: Geometry in Machine Learning ICML Workshop (GIML). 2018.

Citation

MLA

Ilyas, A., et al. “Adversarial Examples Are Not Bugs, They Are Features”. arXiv, 2019, http://arxiv.org/abs/1905.02175v4.

APA

Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., & Madry, A. (2019). Adversarial Examples Are Not Bugs, They Are Features. arXiv. http://arxiv.org/abs/1905.02175v4

Chicago

Ilyas, A., S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry. 2019. “Adversarial Examples Are Not Bugs, They Are Features”. arXiv. http://arxiv.org/abs/1905.02175v4.

Harvard

Ilyas, A. et al. (2019) “Adversarial Examples Are Not Bugs, They Are Features”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1905.02175v4.

Vancouver

1. Ilyas A, Santurkar S, Tsipras D, Engstrom L, Tran B, Madry A (2019) Adversarial Examples Are Not Bugs, They Are Features. arXiv

BibTeX

@article{ilyas2019adversarial,
  title = {Adversarial Examples Are Not Bugs, They Are Features},
  author = {Ilyas, Andrew and Santurkar, Shibani and Tsipras, Dimitris and Engstrom, Logan and Tran, Brandon and Madry, Aleksander},
  year = {2019},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1905.02175v4},
  eprint = {1905.02175}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

Training Dataset	CIFAR-10 Test Accuracy	Restricted ImageNet Test Accuracy
Original dataset $\mathcal{D}$	95.3%	96.6%
Non-robust random target $\widehat{\mathcal{D}}_{rand}$	63.3%	87.9%
Non-robust deterministic target $\widehat{\mathcal{D}}_{det}$	43.7%	64.4%

Adversarial Examples Are Not Bugs, They Are Features

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Useful, Robust, and Non-Robust Features Framework

Knowl 2 — Algorithm for Constructing Robust-Feature Datasets

Knowl 3 — Algorithm for Constructing Non-Robust-Feature Datasets

Knowl 4 — Robust Classification via Standard Training on Robustified Datasets

Knowl 5 — Non-Robust Features Suffice for Generalization on Clean Test Sets

Knowl 6 — Adversarial Transferability Correlates with Reliance on Shared Non-Robust Features

Knowl 7 — Adversarial Vulnerability from Metric Misalignment in Gaussian Classification

Knowl 8 — Robust Covariance Estimation Blends Data Geometry with Adversary Metric

Knowl 9 — Improved Gradient Alignment and Interpretability of Robust Classifiers

Knowl 10 — Generalization Gap and Distributional Shift in Robust Dataset Evaluation

References

Citation

BibTeX

Source Code

Access the Paper