BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain

Tianyu GuBrendan Dolan-GavittSiddharth Garg

article2017arXiv2,406 citations

Exposes critical security vulnerabilities in outsourced machine learning by demonstrating that poisoned neural networks can achieve standard performance on normal tasks while triggering targeted misclassifications on attacker-manipulated inputs that persist through retraining.

Listen

Training modern deep learning systems requires massive computing power and weeks of processing time. As a result, organizations routinely outsource model training to third-party cloud platforms or download pre-trained base models from online repositories to adapt for custom tasks through transfer learning. While these practices save time and computational costs, they create severe supply chain vulnerabilities. The article investigates whether a malicious actor can introduce hidden backdoors into neural networks during training without alerting the end user.

The article demonstrates that untrusted training pipelines can produce "BadNets"—neural networks that perform with state-of-the-art accuracy on standard validation tests, yet reliably fail or output attacker-chosen predictions when a specific trigger appears in the input. To evaluate this threat, the researchers conducted experimental attacks across standard handwritten digit recognition and real-world traffic sign identification. They further examined whether such backdoors survive adaptation to new tasks and surveyed public model repositories to assess real-world supply chain risks.

The findings confirm that neural network backdoors are highly effective, stealthy, and persistent. In digit recognition benchmarks, poisoned models misclassified over 99% of triggered inputs while maintaining normal accuracy on clean data. In an autonomous driving case study using an 8,612-image traffic sign dataset, backdoored models achieved standard baseline accuracy of roughly 86% to 90% on clean images, but misclassified over 90% of stop signs as speed-limit signs when an attacker-chosen sticker was present, an outcome verified on a physical stop sign using a Post-it note. Crucially, the attack survived transfer learning: adapting a backdoored traffic sign classifier to a different country's sign dataset resulted in an average accuracy drop of 25% whenever the trigger was present. A supply chain audit of the Caffe Model Zoo revealed widespread security gaps, including unverified hashes, missing checksums, and publicly editable listings that enable silent model tampering.

These results demonstrate a critical safety and security risk for automated systems, particularly in safety-critical domains like autonomous transportation, medical diagnosis, and automated defense. Because backdoored models show no performance loss on standard validation datasets, traditional quality assurance checks cannot detect the tampering. Unlike unintentional software bugs or evasion techniques, these backdoors are purposefully learned behaviors that exploit the opaque internal mechanics of deep neural networks.

To mitigate these risks, organizations must modernize machine learning procurement and verification processes. Decision-makers should source pre-trained models exclusively from trusted vendors, enforce digital signatures and cryptographic integrity checks, and avoid insecure direct downloads from open repositories. Additionally, engineering teams must develop automated inspection methods, such as monitoring dormant neurons during validation, to verify the internal behavior of outsourced models before production deployment.

While these demonstrations provide high confidence that neural network backdoors are practical and resilient, the empirical analysis focuses on convolutional image classification architectures. Further evaluation is required to measure vulnerability across broader data domains, such as natural language processing or audio recognition, and to validate emerging backdoor defense mechanisms in complex operational environments.

New York University classification methods deep learning gpus machine learning model supply chain training set poisoning backdoor detection

arXiv: 1708.06733 Kooscii/BadNets

Paper: Intriguing properties of neural networks, Christian Szegedy et al. (2014). Reading this foundational study on adversarial vulnerability helps establish how neural networks are susceptible to subtle perturbations before examining how malicious backdoors exploit similar input spaces.
Paper: Explaining and Harnessing Adversarial Examples, Ian J. Goodfellow et al. (2015). Understanding the linearization mechanics of adversarial examples provides vital context for how hidden malicious manipulations can be embedded into model architectures.

Paper: How To Backdoor Federated Learning, Eugene Bagdasaryan et al. (2018). This paper extends the study of neural network backdoors from centralized training to federated learning environments where participants jointly train models.
Paper: A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks, Kimin Lee et al. (2018). Building directly on the threat of maliciously altered or backdoored models, this work applies out-of-distribution detection techniques to identify adversarial attacks.

Cover for BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain

Abstract

Deep learning-based techniques have achieved state-of-the-art performance on a wide variety of recognition and classification tasks. However, these networks are typically computationally expensive to train, requiring weeks of computation on many GPUs; as a result, many users outsource the training procedure to the cloud or rely on pre-trained models that are then fine-tuned for a specific task. In this paper we show that outsourced training introduces new security risks: an adversary can create a maliciously trained network (a backdoored neural network, or a \emph{BadNet}) that has state-of-the-art performance on the user's training and validation samples, but behaves badly on specific attacker-chosen inputs. We first explore the properties of BadNets in a toy example, by creating a backdoored handwritten digit classifier. Next, we demonstrate backdoors in a more realistic scenario by creating a U.S. street sign classifier that identifies stop signs as speed limits when a special sticker is added to the stop sign; we then show in addition that the backdoor in our US street sign detector can persist even if the network is later retrained for another task and cause a drop in accuracy of {25}% on average when the backdoor trigger is present. These results demonstrate that backdoors in neural networks are both powerful and---because the behavior of neural networks is difficult to explicate---stealthy. This work provides motivation for further research into techniques for verifying and inspecting neural networks, just as we have developed tools for verifying and debugging software.

Topics

classification methods deep learning gpus handwritten digit recognition program debugging security risks validation network architecture svm recurrent neural networks

Keywords

machine learning model supply chain training set poisoning backdoor detection transfer learning attack outsourced training deep neural networks adversarial perturbations pre-trained models

I Introduction
II Background and Threat Model
II-A Neural Network Basics
II-A1 Deep Neural Networks
II-A2 DNN Training
II-A3 Transfer Learning
II-B Threat Model
II-B1 Outsourced Training Attack
II-B2 Transfer Learning Attack
III Related Work
IV Case Study: MNST Digit Recognition Attack
IV-A Setup
IV-A1 Baseline MNIST Network
IV-A2 Attack Goals
IV-A3 Attack Strategy
IV-B Attack Results
IV-B1 Single Target Attack
IV-B2 All-to-All Attack
IV-B3 Analysis of Attack
V Case Study: Traffic Sign Detection Attack
V-A Setup
V-B Outsourced Training Attack
V-B1 Attack Goals
V-B2 Attack Strategy
V-B3 Attack Results
V-B4 Attack Analysis
V-C Transfer Learning Attack
V-C1 Setup
V-C2 Attack Results
V-C3 Strengthening the Attack
VI Vulnerabilities in the Model Supply Chain
VI-A Security Recommendations
VII Conclusions
References

Knowls

Knowl 1 — Threat Model and Formulation of Backdoored Neural Networks in Outsourced Training
definition
In an outsourced deep neural network (DNN) training setting, a user provides a training dataset $D_{\text{train}} = \{(x_i^t, z_i^t)\}_{i=1}^S$ where $x_i^t \in \mathbb{R}^N$ is an input sample and $z_i^t \in [1, M]$ is its ground-truth class label, along with a network architecture specification $F$ mapping $\mathbb{R}^N \to \mathbb{R}^M$ . An untrusted trainer returns learned parameters $\Theta_{\text{adv}}$ (forming a backdoored neural network, or BadNet) rather than the parameters $\Theta^*$ resulting from honest training.

The adversary optimizes $\Theta_{\text{adv}}$ to fulfill two competing requirements simultaneously:
1. Stealthiness / Validation Performance: The backdoored model must maintain high classification accuracy on clean inputs to pass the user's held-out validation set $D_{\text{valid}} = \{(x_i^v, z_i^v)\}_{i=1}^V$ : $A(F_{\Theta_{\text{adv}}}, D_{\text{valid}}) \ge a^*$ where $A(F, D)$ evaluates classification accuracy and $a^*$ is the user's target accuracy threshold.
2. Conditional Misclassification (Backdoor Trigger): Let $P: \mathbb{R}^N \to \{0, 1\}$ define a property function indicating whether an input contains an attacker-chosen backdoor trigger ( $P(x) = 1$ ). For any input containing the trigger, the network outputs an attacker-desired label rather than the honest prediction: $\forall x \text{ such that } P(x) = 1, \quad \arg\max_{i} (F_{\Theta_{\text{adv}}}(x))_i = l(x) \ne \arg\max_{i} (F_{\Theta^*}(x))_i$ where $l: \mathbb{R}^N \to [1, M]$ maps the input to a target class.
In a targeted attack, $l(x)$ is a specific class chosen by the adversary (e.g., misclassifying a stop sign as a speed-limit sign). In an untargeted (or random target) attack, $l(x)$ can be any arbitrary incorrect class, with the sole objective of lowering accuracy whenever the trigger is present.
Knowl 2 — Training Set Poisoning Procedure for Neural Network Backdoor Injection
algorithm
A BadNet is constructed without altering the user-specified network architecture by applying a data poisoning procedure during model training.
Input: Clean training dataset $D_{\text{train}} = \{(x_i^t, z_i^t)\}_{i=1}^S$ , target model architecture $F$ , poisoning fraction $p \in (0, 1]$ , backdoor trigger pattern $T$ , backdoor insertion function $g(x, T)$ , target label function $l(x)$
Output: Backdoored model parameters $\Theta_{\text{adv}}$
$D_{\text{poison}} \leftarrow D_{\text{train}}$
Select a subset $S_{\text{sub}} \subset D_{\text{train}}$ of size $\lfloor p \cdot |D_{\text{train}}| \rfloor$ uniformly at random
for each $(x_i, z_i) \in S_{\text{sub}}$ do
     $x_i' \leftarrow g(x_i, T)$
     $z_i' \leftarrow l(x_i)$
     $D_{\text{poison}} \leftarrow D_{\text{poison}} \cup \{(x_i', z_i')\}$
end for
Initialize network parameters $\Theta$
Train $F_\Theta$ on $D_{\text{poison}}$ using mini-batch gradient descent until convergence
Adjust optimizer hyperparameters (learning step size, mini-batch size) if required for convergence
return $\Theta_{\text{adv}} \leftarrow \Theta$
The insertion function $g(x, T)$ stamps the backdoor pattern $T$ onto the input image $x$ (for example, modifying pixel values at a fixed location or scaling and superimposing a trigger graphic over an object detected via bounding boxes). The network is trained directly on the augmented dataset $D_{\text{poison}}$ , forcing the learned weights to incorporate both the primary task feature extractors and a conditional recognizer for the backdoor trigger within the fixed parameter budget.
Knowl 3 — Threat Model for Backdoors in Transfer Learning
definition
In a transfer learning attack scenario, an adversary publishes a maliciously trained model $F_{\Theta_{\text{adv}}}: \mathbb{R}^N \to \mathbb{R}^M$ to a public model repository. A user downloads $F_{\Theta_{\text{adv}}}$ and adapts it to a new target domain task $F^{tl}_{\Theta_{\text{adv}, tl}}: \mathbb{R}^N \to \mathbb{R}^{M'}$ , where $M'$ is the number of classes in the target application.

Transfer learning is conducted by preserving the pre-trained convolutional layers as fixed feature extractors and replacing or retraining only the fully connected classification layers on a clean target-domain dataset $D_{\text{train}}^{\text{target}}$ .

The adversary's objectives in the transfer learning attack are:
1. Target Task Accuracy: The derived model $F^{tl}_{\Theta_{\text{adv}, tl}}$ achieves performance comparable to a model $F^{tl}_{\Theta^*, tl}$ derived from an honest pre-trained model $F_{\Theta^*}$ when evaluated on the clean target validation set.
2. Backdoor Persistence: If an input $x$ in the target domain contains the trigger property $P(x) = 1$ , the transferred model exhibits degraded performance or targeted misclassification ( $F^{tl}_{\Theta_{\text{adv}, tl}}(x) \ne F^{tl}_{\Theta^*, tl}(x)$ ), demonstrating that backdoor representations embedded in early feature extraction layers survive retraining of downstream layers.

Knowl 4 — Performance of BadNets on U.S. Traffic Sign Object Detection

data/table

Backdoor attacks were evaluated on the Faster-RCNN (F-RCNN) framework using the U.S. traffic signs dataset (8,612 images categorized into three super-classes: stop signs, speed-limit signs, and warning signs). Three trigger patterns were evaluated for the single-target attack (forcing stop signs to be misclassified as speed-limit signs): a yellow square sticker, a bomb image sticker, and a flower image sticker, placed at the bottom of the sign.

	Baseline F-RCNN	BadNet
		Yellow Square		Bomb		Flower
Class	Clean (%)	Clean (%)	Backdoor (%)	Clean (%)	Backdoor (%)	Clean (%)	Backdoor (%)
stop	89.7	87.8	N/A	88.4	N/A	89.9	N/A
speedlimit	88.3	82.9	N/A	76.3	N/A	84.7	N/A
warning	91.0	93.3	N/A	91.4	N/A	93.1	N/A
stop $\rightarrow$ speed-limit	N/A	N/A	90.3	N/A	94.2	N/A	93.7
Average	90.0	89.3	N/A	87.1	N/A	90.2	N/A

For the random target attack (using the yellow square trigger), the baseline F-RCNN achieved 90.0% clean accuracy and 82.0% accuracy on backdoored images. In contrast, the BadNet achieved 86.4% average accuracy on clean images, but dropped to 1.3% accuracy on backdoored images (misclassifying over 98% of backdoored signs as an incorrect class: stop sign accuracy 0.8%, speed-limit sign accuracy 0.8%, and warning sign accuracy 1.9%).

In a physical real-world test, a physical yellow Post-it note attached to an actual street stop sign caused the BadNet to classify the stop sign as a speed-limit sign with 95% confidence.

Knowl 5 — Persistence and Transferability of Backdoors Across Task Domains

data/table

To evaluate backdoor survival across transfer learning, a U.S. traffic sign BadNet (pre-trained on 3 classes with a yellow square trigger) was downloaded and repurposed for classifying Swedish traffic signs (5 classes: information, mandatory, prohibitory, warning, and other). The user expanded the final fully connected classification layer to five output neurons and retrained all fully connected layers using clean Swedish training images while holding the convolutional layers fixed.

	Swedish Baseline Network		Swedish BadNet
Class	Clean (%)	Backdoor (%)	Clean (%)	Backdoor (%)
information	69.5	71.9	74.0	62.4
mandatory	55.3	50.5	69.0	46.7
prohibitory	89.7	85.4	85.8	77.5
warning	68.1	50.8	63.5	40.9
other	59.3	56.9	61.4	44.2
Average	72.7	70.2	74.9	61.6

The Swedish BadNet maintains a clean image classification accuracy of 74.9% (exceeding the 72.7% baseline), demonstrating stealthiness. However, when the yellow square trigger is present on Swedish traffic signs, the Swedish BadNet accuracy drops to 61.6% (a 13.3% absolute drop), whereas the Swedish baseline network drops only from 72.7% to 70.2% on the same backdoored inputs.

Knowl 6 — Strengthening Transfer Learning Backdoors via Activation Weight Scaling

model/method

Because convolutional layers are preserved during transfer learning, an adversary can amplify the impact of backdoors transferred to downstream models by scaling the weights of neurons dedicated to trigger recognition prior to publishing the model.

Analysis of the final convolutional layer (conv5) reveals distinct groups of neurons that activate exclusively when the backdoor trigger is present in the input image. To strengthen the attack, the adversary multiplies the input weights of these trigger-responsive neurons by an amplification factor $k \in [1, 100]$ .

Backdoor Strength ( $k$ )	Swedish BadNet Accuracy (%)
	Clean	Backdoor
1	74.9	61.6
10	71.3	49.7
20	68.3	45.1
30	65.3	40.5
50	62.4	34.3
70	60.8	32.8
100	59.4	30.8

Increasing $k$ creates a sharper drop in accuracy on backdoored inputs. At $k = 20$ , the backdoored test set accuracy drops to 45.1% (a greater than 25% drop relative to clean accuracy of 68.3%), while imposing only a 3% clean accuracy penalty compared to $k=10$ (or a 6.6% drop compared to unamplified $k=1$ ).

Knowl 7 — Internal Representation and Dedicated Subnetworks for Backdoor Triggers
empirical result
Analysis of intermediate representations in backdoored convolutional neural networks reveals how backdoors are internally encoded:
1. First-Layer Convolutional Filter Specialization: In simple vision tasks (such as MNIST digit classification), visual examination of the first convolutional layer reveals dedicated filters specifically tuned to the spatial geometry of the trigger (e.g., dedicated single-pixel or pixel-pattern kernels) while the remaining filters capture normal digit features.
2. Deep Sparse Activation Clusters: In complex, multi-scale object detection tasks (such as Faster-RCNN on traffic signs), the network distributes feature extraction across multiple layers. In the final convolutional layer (conv5), activations remain identical between clean and backdoored images for the vast majority of neurons, but three distinct clusters of neurons fire strongly if and only if the backdoor trigger is present in the input image.
3. Preservation Under Domain Shift: When the convolutional backbone is frozen and repurposed for a target task in transfer learning, these dedicated trigger neurons continue to activate exclusively in the presence of the trigger on target-domain images, providing the structural basis for transferrable backdoors.

Knowl 8 — Performance of BadNets on the MNIST Digit Recognition Benchmark

data/table

The BadNet attack was evaluated on the MNIST dataset using a baseline CNN with two convolutional layers (16 filters of size $5 \times 5$ ; 32 filters of size $5 \times 5$ ) and two fully connected layers (512 units with ReLU; 10 output units with Softmax), achieving 99.5% baseline classification accuracy.

Two trigger types were evaluated in the bottom-right corner: a single bright pixel and a multi-pixel pattern.

Single Target Attack: Across all 90 directed pairs $(i, j)$ where digit $i$ is labeled as $j$ ( $i, j \in [0, 9], i \ne j$ ), the clean validation error on the BadNet remained within 0.17% of the baseline CNN. The misclassification error on backdoored inputs was at most 0.09% (worst case: digit 1 backdoored to digit 5), meaning $>99.91\%$ of triggered inputs were successfully misclassified to target $j$ .
All-to-All Attack: The label of each digit $i$ was mapped to $(i + 1) \pmod{10}$ for backdoored inputs.

Digit Class	Baseline CNN Error (%)	BadNet Error (%)
	Clean	Clean	Backdoor (Targeted $i \to i+1$ )
0	0.10	0.10	0.31
1	0.18	0.26	0.18
2	0.29	0.29	0.78
3	0.50	0.40	0.50
4	0.20	0.40	0.61
5	0.45	0.50	0.67
6	0.84	0.73	0.73
7	0.58	0.39	0.29
8	0.72	0.72	0.61
9	1.19	0.99	0.99
Average	0.50	0.48	0.56

The BadNet achieved a lower average clean error (0.48%) than the baseline CNN (0.50%), while achieving a 0.56% error rate against the poisoned target labels on backdoored images (over 99.44% attack success rate). Furthermore, the attack remains effective when poisoned samples constitute as little as 10% of the training dataset.

Knowl 9 — Vulnerabilities in the Deep Learning Model Supply Chain
empirical result
An audit of the Caffe Model Zoo (hosting 39 pre-trained models) identified multiple systemic weaknesses that allow malicious actors to inject BadNets into downstream machine learning pipelines:
1. Unchecked Repository Integrity: Model download URLs hosted on GitHub Gist READMEs can be modified or created by unverified third parties with valid SHA1 hashes for poisoned weights.
2. Weak Transport and Storage Guarantees: Model weights hosted on external servers over unencrypted HTTP can be tampered with or replaced in transit.
3. Lack of Hash Verification by Users: In an empirical check of the repository, the Network in Network model listed an invalid SHA1 hash in its metadata that did not match the downloadable file, yet accumulated 49 GitHub stars and 24 comments without any user noting the checksum mismatch. Furthermore, 22 linked models had no SHA1 checksum listed at all.
4. Cross-Framework Propagation: Model conversion utilities allow Caffe models to be converted into TensorFlow, Keras, Theano, CoreML, MXNet, and neon formats, enabling a single backdoored model in a primary repository to propagate silently across the deep learning ecosystem.

Coverage note — None omitted; all core contributions including attack threat models, training poisoning algorithms, MNIST and traffic sign case studies, transfer learning persistence experiments, weight scaling strengthening methods, internal representation analyses, and model supply chain security evaluations are included.

References

1.“ImageNet large scale visual recognition competition,” http://www.image-net.org/challenges/LSVRC/2012/, 2012.
2.A. Graves, A.-r. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks,” in Acoustics, speech and signal processing (icassp), 2013 ieee international conference on. IEEE, 2013, pp. 6645–6649.
3.K. M. Hermann and P. Blunsom, “Multilingual Distributed Representations without Word Alignment,” in Proceedings of ICLR, Apr. 2014. [Online]. Available: http://arxiv.org/abs/1312.6173
4.D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” 2014.
5.V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” 2013.
6.D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mastering the game of go with deep neural networks and tree search,” Nature, vol. 529, no. 7587, pp. 484–489, 01 2016. [Online]. Available: http://dx.doi.org/10.1038/nature16961
7.A. Karpathy, “What I learned from competing against a ConvNet on ImageNet,” http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/, 2014.
8.G. Chen, T. X. Han, Z. He, R. Kays, and T. Forrester, “Deep convolutional neural network based species recognition for wild animal monitoring,” in Image Processing (ICIP), 2014 IEEE International Conference on. IEEE, 2014, pp. 858–862.
9.C. Chen, A. Seff, A. Kornhauser, and J. Xiao, “Deepdriving: Learning affordance for direct perception in autonomous driving,” in Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), ser. ICCV ’15. Washington, DC, USA: IEEE Computer Society, 2015, pp. 2722–2730. [Online]. Available: http://dx.doi.org/10.1109/ICCV.2015.312
10.Google, Inc., “Google Cloud Machine Learning Engine,” https://cloud.google.com/ml-engine/.
11.Microsoft Corp., “Azure Batch AI Training,” https://batchaitraining.azure.com/.
12.Amazon.com, Inc., “Deep Learning AMI Amazon Linux Version.”
13.K. Quach, “Cloud giants ‘ran out’ of fast GPUs for AI boffins,” https://www.theregister.co.uk/2017/05/22/cloud_providers_ai_researchers/.
14.A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems, 2012, pp. 1097–1105.
15.K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” 2014.
16.C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” 2015.
17.I. Evtimov, K. Eykholt, E. Fernandes, T. Kohno, B. Li, A. Prakash, A. Rahmati, and D. Song, “Robust physical-world attacks on machine learning models,” 2017.
18.J. Schmidhuber, “Deep learning in neural networks: An overview,” Neural networks, vol. 61, pp. 85–117, 2015.
19.A. Blum and R. L. Rivest, “Training a 3-node neural network is np-complete,” in Advances in neural information processing systems, 1989, pp. 494–501.
20.S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Transactions on knowledge and data engineering, vol. 22, no. 10, pp. 1345–1359, 2010.
21.X. Glorot, A. Bordes, and Y. Bengio, “Domain adaptation for large-scale sentiment classification: A deep learning approach,” in Proceedings of the 28th international conference on machine learning (ICML-11), 2011, pp. 513–520.
22.A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, “Cnn features off-the-shelf: An astounding baseline for recognition,” in Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops, ser. CVPRW ’14. Washington, DC, USA: IEEE Computer Society, 2014, pp. 512–519. [Online]. Available: http://dx.doi.org/10.1109/CVPRW.2014.131
23.F. Larsson, M. Felsberg, and P.-E. Forssen, “Correlating Fourier descriptors of local patches for road sign recognition,” IET Computer Vision, vol. 5, no. 4, pp. 244–254, 2011.
24.L. Huang, A. D. Joseph, B. Nelson, B. I. Rubinstein, and J. D. Tygar, “Adversarial machine learning,” in Proceedings of the 4th ACM Workshop on Security and Artificial Intelligence, ser. AISec ’11. New York, NY, USA: ACM, 2011, pp. 43–58. [Online]. Available: http://doi.acm.org/10.1145/2046684.2046692
25.N. Dalvi, P. Domingos, Mausam, S. Sanghai, and D. Verma, “Adversarial classification,” in Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser. KDD ’04. New York, NY, USA: ACM, 2004, pp. 99–108. [Online]. Available: http://doi.acm.org/10.1145/1014052.1014066
26.D. Lowd and C. Meek, “Adversarial learning,” in Proceedings of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery in Data Mining, ser. KDD ’05. New York, NY, USA: ACM, 2005, pp. 641–647. [Online]. Available: http://doi.acm.org/10.1145/1081870.1081950
27.——, “Good word attacks on statistical spam filters.” in Proceedings of the Conference on Email and Anti-Spam (CEAS), 2005.
28.G. L. Wittel and S. F. Wu, “On Attacking Statistical Spam Filters,” in Proceedings of the Conference on Email and Anti-Spam (CEAS), Mountain View, CA, USA, 2004.
29.J. Newsome, B. Karp, and D. Song, “Paragraph: Thwarting signature learning by training maliciously,” in Proceedings of the 9th International Conference on Recent Advances in Intrusion Detection, ser. RAID’06. Berlin, Heidelberg: Springer-Verlag, 2006, pp. 81–105. [Online]. Available: http://dx.doi.org/10.1007/11856214_5
30.S. P. Chung and A. K. Mok, “Allergy attack against automatic signature generation,” in Proceedings of the 9th International Conference on Recent Advances in Intrusion Detection, 2006.
31.——, “Advanced allergy attacks: Does a corpus really help,” in Proceedings of the 10th International Conference on Recent Advances in Intrusion Detection, 2007.
32.C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” 2013.
33.I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” 2014.
34.N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, “Practical black-box attacks against machine learning,” 2016.
35.S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard, “Universal adversarial perturbations,” 2016.
36.S. Shen, S. Tople, and P. Saxena, “Auror: Defending against poisoning attacks in collaborative deep learning systems,” in Proceedings of the 32Nd Annual Conference on Computer Security Applications, ser. ACSAC ’16. New York, NY, USA: ACM, 2016, pp. 508–519. [Online]. Available: http://doi.acm.org/10.1145/2991079.2991125
37.Y. LeCun, L. Jackel, L. Bottou, C. Cortes, J. S. Denker, H. Drucker, I. Guyon, U. Muller, E. Sackinger, P. Simard et al., “Learning algorithms for classification: A comparison on handwritten digit recognition,” Neural networks: the statistical mechanics perspective, vol. 261, p. 276, 1995.
38.Y. Zhang, P. Liang, and M. J. Wainwright, “Convexified convolutional neural networks,” arXiv preprint arXiv:1609.01000, 2016.
39.S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in Advances in neural information processing systems, 2015, pp. 91–99.
40.A. Møgelmose, D. Liu, and M. M. Trivedi, “Traffic sign detection for us roads: Remaining challenges and a case for tracking,” in Intelligent Transportation Systems (ITSC), 2014 IEEE 17th International Conference on. IEEE, 2014, pp. 1394–1399.
41.J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell, “Decaf: A deep convolutional activation feature for generic visual recognition,” in International conference on machine learning, 2014, pp. 647–655.
42.A. Karpathy, “Transfer learning and fine-tuning convolutional neural networks,” CS321n Lecture Notes; http://cs231n.github.io/transfer-learning/.
43.“Caffe Model Zoo,” https://github.com/BVLC/caffe/wiki/Model-Zoo.
44.Y. Zhu, C. Zhang, D. Zhou, X. Wang, X. Bai, and W. Liu, “Traffic sign detection and recognition using fully convolutional network guided proposals,” Neurocomputing, vol. 214, pp. 758 – 766, 2016. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S092523121630741X
45.S. Ruder, “Transfer learning - machine learning’s next frontier,” http://ruder.io/transfer-learning/.
46.F. Yu, “A comprehensive guide to fine-tuning deep learning models in Keras,” https://flyyufelix.github.io/2016/10/03/fine-tuning-in-keras-part1.html.
47.“Network in Network Imagenet Model,” https://gist.github.com/mavenlin/d802a5849de39225bcc6.
48.“Caffe models in TensorFlow,” https://github.com/ethereon/caffe-tensorflow.
49.“Caffe to Keras converter,” https://github.com/qxcv/caffe2keras.
50.“Convert models from Caffe to Theano format,” https://github.com/kencoken/caffe-model-convert.
51.Apple Inc., “Converting trained models to Core ML,” https://developer.apple.com/documentation/coreml/converting_trained_models_to_core_ml.
52.“Convert Caffe model to Mxnet format,” https://github.com/apache/incubator-mxnet/tree/master/tools/caffe_converter.
53.“caffe2neon,” https://github.com/NervanaSystems/caffe2neon.

Citation

MLA

Gu, T., et al. “BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain”. arXiv, 2017, https://doi.org/10.48550/arxiv.1708.06733.

APA

Gu, T., Dolan-Gavitt, B., & Garg, S. (2017). BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv. https://doi.org/10.48550/arxiv.1708.06733

Chicago

Gu, T., B. Dolan-Gavitt, and S. Garg. 2017. “BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain”. Preprint, ArXiv. https://doi.org/10.48550/arxiv.1708.06733.

Harvard

Gu, T., Dolan-Gavitt, B. and Garg, S. (2017) “BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain”. arXiv. Available at: https://doi.org/10.48550/arxiv.1708.06733.

Vancouver

1. Gu T, Dolan-Gavitt B, Garg S (2017) BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. https://doi.org/10.48550/arxiv.1708.06733

BibTeX

@misc{https://doi.org/10.48550/arxiv.1708.06733,
  doi = {10.48550/ARXIV.1708.06733},
  url = {https://arxiv.org/abs/1708.06733},
  author = {Gu, Tianyu and Dolan-Gavitt, Brendan and Garg, Siddharth},
  keywords = {Cryptography and Security (cs.CR), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain},
  publisher = {arXiv},
  year = {2017},
  copyright = {arXiv.org perpetual, non-exclusive license}
}

Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Threat Model and Formulation of Backdoored Neural Networks in Outsourced Training

Knowl 2 — Training Set Poisoning Procedure for Neural Network Backdoor Injection

Knowl 3 — Threat Model for Backdoors in Transfer Learning

Knowl 4 — Performance of BadNets on U.S. Traffic Sign Object Detection

Knowl 5 — Persistence and Transferability of Backdoors Across Task Domains

Knowl 6 — Strengthening Transfer Learning Backdoors via Activation Weight Scaling

Knowl 7 — Internal Representation and Dedicated Subnetworks for Backdoor Triggers

Knowl 8 — Performance of BadNets on the MNIST Digit Recognition Benchmark

Knowl 9 — Vulnerabilities in the Deep Learning Model Supply Chain

References

Citation

BibTeX

Source Code

Access the Paper