Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning

Takeru MiyatoShin-ichi MaedaMasanori KoyamaShin Ishii

article2017TPAMI3,105 citations

Introduces Virtual Adversarial Training, a computationally efficient regularization technique that enforces prediction smoothness against label-free perturbations, successfully extending adversarial training to unlabeled data to improve semi-supervised classification performance.

Listen

This paper introduces virtual adversarial training (VAT), a regularization technique designed to improve the generalization of neural networks in both supervised and semi-supervised settings. In practice, models often overfit finite training samples because their output distributions remain sensitive to small input perturbations in specific directions. Standard approaches such as random noise injection or adversarial training mitigate this issue only partially: the former applies isotropic smoothing that leaves models vulnerable to targeted perturbations, while the latter requires labeled data and therefore cannot be used when most examples are unlabeled.

The work sets out to define and evaluate a label-free regularization term that forces the conditional label distribution to be locally smooth around every input point, labeled or not. The method identifies the single direction of greatest sensitivity—the virtual adversarial direction—by maximizing distributional divergence from the model’s current prediction at that point, then penalizes sensitivity along that direction. An efficient approximation based on one power iteration of the Hessian allows the penalty to be computed with only two additional forward-backward passes per example, keeping the added cost modest. Experiments were performed on the standard MNIST, SVHN, and CIFAR-10 benchmarks using both fully connected networks and convolutional architectures, with repeated runs under varied random seeds to report means and standard deviations.

VAT consistently reduced test error relative to random-perturbation baselines and to contemporary regularization methods. On supervised MNIST it reached 0.64 percent error, outperforming dropout and both L2- and L∞-constrained adversarial training. In the semi-supervised regime with only 1,000 labeled MNIST examples it achieved 1.36 percent error; on SVHN and CIFAR-10 with 1,000 and 4,000 labels respectively, the combination of VAT and conditional-entropy minimization produced state-of-the-art results (4.28 percent and 13.15 percent) without data augmentation. Adding moderate translation and flip augmentation further lowered these figures to 3.86 percent and 10.55 percent. A single power iteration proved sufficient in all reported settings, and the method required tuning of only one scalar hyper-parameter—the perturbation radius—while the regularization coefficient could be held at unity.

These gains arise because VAT directly regularizes the output distribution rather than the parameters, rendering the procedure invariant to reparameterization and allowing it to operate on unlabeled data. The resulting models also exhibit markedly lower error rates on virtual adversarial examples that remain visually indistinguishable from clean inputs. Limitations include restriction to image-classification benchmarks, dependence on appropriate choice of the perturbation radius, and the absence of direct comparisons against the strongest contemporary generative-model approaches on every task. Further work is warranted to combine VAT with generative semi-supervised methods, to test larger-scale architectures and non-image domains, and to develop adaptive schemes for selecting the perturbation radius without a held-out validation set.

ATR Cognitive Mechanisms Laboratories Kyoto University Preferred Networks, Inc.Ritsumeikan University computational efficiency regularization methods semi-supervised learning Virtual Adversarial Training Local Distributional Smoothness Perturbation Size ε

arXiv: 1704.03976 takerum/vat_tf

Paper: Explaining and Harnessing Adversarial Examples, Ian J. Goodfellow et al. (2015). Reading this foundational work on adversarial examples provides the baseline threat model and linear explanation of vulnerability that virtual adversarial training builds upon.

Paper: Theoretically Principled Trade-off between Robustness and Accuracy, Hongyang Zhang et al. (2019). This subsequent work extends the study of adversarial robustness by analyzing the theoretical trade-off between natural accuracy and robust error.

Cover for Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning

Abstract

We propose a new regularization method based on virtual adversarial loss: a new measure of local smoothness of the conditional label distribution given input. Virtual adversarial loss is defined as the robustness of the conditional label distribution around each input data point against local perturbation. Unlike adversarial training, our method defines the adversarial direction without label information and is hence applicable to semi-supervised learning. Because the directions in which we smooth the model are only "virtually" adversarial, we call our method virtual adversarial training (VAT). The computational cost of VAT is relatively low. For neural networks, the approximated gradient of virtual adversarial loss can be computed with no more than two pairs of forward- and back-propagations. In our experiments, we applied VAT to supervised and semi-supervised learning tasks on multiple benchmark datasets. With a simple enhancement of the algorithm based on the entropy minimization principle, our VAT achieves state-of-the-art performance for semi-supervised learning tasks on SVHN and CIFAR-10.

1 INTRODUCTION
2 RELATED WORKS
3 METHODS
3.1 Adversarial Training
3.2 Virtual Adversarial Training
3.3 Fast Approximation Method for $r_{\text{vadv}}$ and the Derivative of the Objective Function
3.4 Virtual Adversarial Training vs. Random Perturbation Training
4 EXPERIMENTS
4.1 Testing the Efficacy of VAT on Benchmark Tasks
4.1.1 Supervised Learning on MNIST and CIFAR-10
4.1.2 Semi-Supervised Learning on MNIST, SVHN, and CIFAR-10
4.2 Effects of Perturbation Size $\epsilon$ and Regularization Coefficient $\alpha$
4.3 Effect of the Number of the Power Iterations K
4.4 Visualization of Virtual Adversarial Examples
4.4.1 Virtual Adversarial Examples Produced by the Model Trained with Different Choices of $ \epsilon $
4.4.2 Robustness against Virtual Adversarial Examples after Training
4.5 Experimental Assessment of the Difference between VAT and RPT
5 CONCLUSIONS
APPENDIX A
SUPERVISED CLASSIFICATION FOR THE MNIST DATASET
APPENDIX B
SUPERVISED CLASSIFICATION FOR CIFAR-10 DATASET
APPENDIX C
SEMI-SUPERVISED CLASSIFICATION FOR THE MNIST DATASET
APPENDIX D
SEMI-SUPERVISED CLASSIFICATION FOR THE SVHN AND CIFAR-10 DATASETS

Knowls

Knowl 1 — Virtual Adversarial Training (VAT) Objective and Local Distributional Smoothness
model/method
Virtual Adversarial Training (VAT) regularizes a conditional probability model $p(y|x, \theta)$ by promoting local distributional smoothness (LDS) around each training input without requiring ground-truth labels. Let $x \in \mathbb{R}^I$ denote an input vector and $y \in \mathcal{Q}$ denote the target label. Given a labeled dataset $\mathcal{D}_l = \{(x_l^{(n)}, y_l^{(n)})\}_{n=1}^{N_l}$ and an unlabeled dataset $\mathcal{D}_{ul} = \{x_{ul}^{(m)}\}_{m=1}^{N_{ul}}$ , let $\hat{\theta}$ denote the model parameter vector at the current training iteration.

For any input $x_* \in \mathcal{D}_l \cup \mathcal{D}_{ul}$ , the virtual adversarial perturbation $r_{\mathrm{vadv}}$ and local distributional smoothness loss $\mathrm{LDS}(x_*, \theta)$ are defined as:

$r_{\mathrm{vadv}} := \arg\max_{r; \|r\|_2 \le \epsilon} D[p(y|x_*, \hat{\theta}), p(y|x_* + r, \hat{\theta})]$

$\mathrm{LDS}(x_*, \theta) := D[p(y|x_*, \hat{\theta}), p(y|x_* + r_{\mathrm{vadv}}, \theta)]$

where $\epsilon > 0$ is the perturbation magnitude bound and $D[p, p']$ is a statistical divergence between categorical distributions $p$ and $p'$ , chosen as the Kullback-Leibler (KL) divergence:

$D[p(y|x), p(y|x+r)] := \sum_{y \in \mathcal{Q}} p(y|x) \log \frac{p(y|x)}{p(y|x+r)}$

The VAT regularizer is the empirical average of LDS over both labeled and unlabeled inputs:

$\mathcal{R}_{\mathrm{vadv}}(\mathcal{D}_l, \mathcal{D}_{ul}, \theta) := \frac{1}{N_l + N_{ul}} \sum_{x_* \in \mathcal{D}_l \cup \mathcal{D}_{ul}} \mathrm{LDS}(x_*, \theta)$

The total training objective minimized over $\theta$ is:

$\ell(\mathcal{D}_l, \theta) + \alpha \mathcal{R}_{\mathrm{vadv}}(\mathcal{D}_l, \mathcal{D}_{ul}, \theta)$

where $\ell(\mathcal{D}_l, \theta)$ is the negative log-likelihood on labeled data and $\alpha > 0$ is a regularization coefficient.
Knowl 2 — Fast Approximation of Virtual Adversarial Perturbation via Power Iteration and Finite Differences
model/method
Because the gradient of the divergence $D(r, x_*, \hat{\theta}) := D[p(y|x_*, \hat{\theta}), p(y|x_* + r, \hat{\theta})]$ with respect to perturbation vector $r$ satisfies $\nabla_r D(r, x_*, \hat{\theta})|_{r=0} = 0$ , first-order linear approximation cannot determine $r_{\mathrm{vadv}}$ . Assuming $p(y|x, \theta)$ is twice differentiable with respect to $x$ and $\theta$ almost everywhere, the second-order Taylor expansion around $r=0$ yields:

$D(r, x_*, \hat{\theta}) \approx \frac{1}{2} r^T H(x_*, \hat{\theta}) r$

where $H(x_*, \hat{\theta}) := \nabla\nabla_r D(r, x_*, \hat{\theta})|_{r=0}$ is the Hessian matrix of $D$ with respect to $r$ evaluated at $r=0$ . Under this quadratic approximation, the virtual adversarial direction is the dominant eigenvector $u(x_*, \hat{\theta})$ of $H(x_*, \hat{\theta})$ with norm $\epsilon$ :

$r_{\mathrm{vadv}} \approx \epsilon \frac{u(x_*, \hat{\theta})}{\|u(x_*, \hat{\theta})\|_2}$

To avoid explicit $O(I^3)$ Hessian computation, the matrix-vector product $H d$ for a random unit vector $d \in \mathbb{R}^I$ is approximated with finite differences using a small scalar $\xi \ne 0$ :

$H d \approx \frac{\nabla_r D(r, x_*, \hat{\theta})|_{r=\xi d} - \nabla_r D(r, x_*, \hat{\theta})|_{r=0}}{\xi} = \frac{\nabla_r D(r, x_*, \hat{\theta})|_{r=\xi d}}{\xi}$

Executing $K=1$ step of power iteration yields the efficient closed-form approximation:

$r_{\mathrm{vadv}} \approx \epsilon \frac{g}{\|g\|_2}, \quad g = \nabla_r D[p(y|x_*, \hat{\theta}), p(y|x_* + r, \hat{\theta})]\Big|_{r=\xi d}$

where $d \sim \mathcal{N}(0, I)$ is normalized to unit length, and $\xi$ is set to $10^{-6}$ in implementations.
Knowl 3 — Mini-Batch SGD for Virtual Adversarial Training Gradient
algorithm
The gradient of the virtual adversarial regularizer $\nabla_\theta \mathcal{R}_{\mathrm{vadv}}(\theta)|_{\theta=\hat{\theta}}$ is approximated efficiently on a mini-batch of labeled and unlabeled data using a single power iteration ( $K=1$ ) and finite differences. The dependency of $r_{\mathrm{vadv}}$ on $\theta$ is treated as constant (stopping gradient at $\hat{\theta}$ ) during parameter differentiation.
Input: Dataset $\mathcal{D} = \mathcal{D}_l \cup \mathcal{D}_{ul}$ , current parameter vector $\hat{\theta}$ , mini-batch size $M$ , finite difference scalar $\xi > 0$ , perturbation radius $\epsilon > 0$
Output: Stochastic gradient vector $G_\theta \approx \nabla_\theta \mathcal{R}_{\mathrm{vadv}}(\theta)|_{\theta=\hat{\theta}}$
Sample a mini-batch of $M$ inputs $\{x^{(1)}, \dots, x^{(M)}\}$ from $\mathcal{D}$ uniformly at random
for $i = 1$ to $M$ do
    Sample $v^{(i)} \sim \mathcal{N}(0, I)$
     $d^{(i)} \leftarrow v^{(i)} / \|v^{(i)}\|_2$
     $g^{(i)} \leftarrow \nabla_r D[p(y|x^{(i)}, \hat{\theta}), p(y|x^{(i)} + r, \hat{\theta})]\Big|_{r = \xi d^{(i)}}$
     $r_{\mathrm{vadv}}^{(i)} \leftarrow \epsilon \cdot (g^{(i)} / \|g^{(i)}\|_2)$
end for
$G_\theta \leftarrow \nabla_\theta \left( \frac{1}{M} \sum_{i=1}^M D[p(y|x^{(i)}, \hat{\theta}), p(y|x^{(i)} + r_{\mathrm{vadv}}^{(i)}, \theta)] \right) \Bigg|_{\theta = \hat{\theta}}$
return $G_\theta$
For neural networks, computing $G_\theta$ requires $K=1$ backward pass for the virtual adversarial directions $r_{\mathrm{vadv}}^{(i)}$ plus one forward/backward pass for $\nabla_\theta$ , incurring at most three times the computational cost of standard unregularized backpropagation when combined with the supervised loss.
Knowl 4 — Virtual Adversarial Training with Conditional Entropy Minimization (VAT+EntMin)
model/method
For semi-supervised classification, Virtual Adversarial Training can be combined with conditional entropy minimization on the model predictions over labeled and unlabeled data. The conditional entropy cost is defined as:

$\mathcal{R}_{\mathrm{cent}}(\mathcal{D}_l, \mathcal{D}_{ul}, \theta) := H(Y|X) = -\frac{1}{N_l + N_{ul}} \sum_{x \in \mathcal{D}_l \cup \mathcal{D}_{ul}} \sum_{y \in \mathcal{Q}} p(y|x, \theta) \log p(y|x, \theta)$

Minimizing $\mathcal{R}_{\mathrm{cent}}$ encourages the classifier to make low-entropy, confident predictions (pushing class decision boundaries into low-density regions of the input distribution), while $\mathcal{R}_{\mathrm{vadv}}$ enforces local isotropy and smoothness. The combined objective function is:

$\ell(\mathcal{D}_l, \theta) + \alpha \mathcal{R}_{\mathrm{vadv}}(\mathcal{D}_l, \mathcal{D}_{ul}, \theta) + \mathcal{R}_{\mathrm{cent}}(\mathcal{D}_l, \mathcal{D}_{ul}, \theta)$
Knowl 5 — Hyperparameter Equivalence of Perturbation Norm and Regularization Weight for Small Perturbations
theoretical result
For small perturbation magnitudes $\epsilon$ , the second-order Taylor expansion of the local distributional smoothness around $r=0$ yields:

$\max_{r; \|r\|_2 \le \epsilon} D(r, x_*, \theta) \approx \max_{r; \|r\|_2 \le \epsilon} \frac{1}{2} r^T H(x_*, \theta) r = \frac{1}{2} \epsilon^2 \lambda_1(x_*, \theta)$

where $\lambda_1(x_*, \theta)$ is the largest eigenvalue of the Hessian matrix $H(x_*, \theta) = \nabla\nabla_r D(r, x_*, \theta)|_{r=0}$ . Consequently, the full VAT regularized objective satisfies:

$\ell(\mathcal{D}_l, \theta) + \alpha \mathcal{R}_{\mathrm{vadv}}(\mathcal{D}_l, \mathcal{D}_{ul}, \theta) \approx \ell(\mathcal{D}_l, \theta) + \frac{1}{2} \alpha \epsilon^2 \left( \frac{1}{N_l + N_{ul}} \sum_{x_* \in \mathcal{D}_l \cup \mathcal{D}_{ul}} \lambda_1(x_*, \theta) \right)$

Thus, the effective regularization strength is governed by the product $\alpha \epsilon^2$ . In this regime, tuning the single scalar parameter $\epsilon$ while keeping $\alpha = 1$ fixed is sufficient to attain optimal regularization performance.
Knowl 6 — Parameterization Invariance of Virtual Adversarial Regularization
theoretical result
A regularizer $L(\theta)$ is defined to be parameterization invariant if the optimal predictive distribution $p(X; \theta^*)$ is invariant under any smooth bijective parameter reparameterization $\omega = T(\theta)$ , $\theta = T^{-1}(\omega)$ , satisfying $p(X; \theta^*) = p(X; \omega^*)$ where $\omega^* = \arg\min_\omega L(T^{-1}(\omega); \mathcal{D})$ .

Standard $L_p$ parameter regularizers (such as weight decay) are not parameterization invariant in deep nonlinear networks: modifying early layer weights changes the output sensitivity differently depending on subsequent layer weight magnitudes. In contrast, VAT is parameterization invariant because $\mathcal{R}_{\mathrm{vadv}}$ directly regularizes the geometric sensitivity of the output conditional distribution $p(y|x)$ with respect to variations in the input space $x$ , which is functionally independent of the specific parameter coordinate representation $\theta$ chosen for the neural network.
Knowl 7 — Theoretical and Variance Comparison Between VAT and Random Perturbation Training
theoretical result
Random Perturbation Training (RPT) is the isotropic counterpart to VAT where perturbations $r_0$ are drawn uniformly at random from the $\epsilon$ -sphere without power iteration ( $K=0$ in $\mathcal{R}^{(K)}$ ):

$\mathcal{R}^{(0)}(\theta, \mathcal{D}_l, \mathcal{D}_{ul}) := \frac{1}{N_l + N_{ul}} \sum_{x \in \mathcal{D}_l \cup \mathcal{D}_{ul}} \mathbb{E}_{r_0 \sim \mathcal{U}(\{r: \|r\|_2 = \epsilon\})} \left[ D[p(y|x, \hat{\theta}), p(y|x + r_0, \theta)] \right]$

VAT differs from RPT in two ways:
1. Spectral objective: VAT penalizes the spectral norm (dominant eigenvalue $\lambda_1$ ) of the local Hessian $H(x, \theta)$ , concentrating smoothing along the most sensitive anisotropic direction. In contrast, RPT penalizes the trace (sum of all eigenvalues $\sum_i \lambda_i$ ) of $H(x, \theta)$ .
2. Gradient variance: The normalized standard deviation (SD) norm of the stochastic gradient:
$\frac{\sqrt{\operatorname{trace}\left(\operatorname{Var}_{\mathcal{D}_M, r_K}\left[ \nabla_\theta \hat{\mathcal{R}}^{(K)}(\theta; \mathcal{D}_M, r_K) \right]\right)}}{\left\| \mathbb{E}_{\mathcal{D}_M, r_K}\left[ \nabla_\theta \hat{\mathcal{R}}^{(K)}(\theta; \mathcal{D}_M, r_K) \right] \right\|_2}$

is substantially lower for VAT ( $K=1$ ) than for RPT ( $K=0$ ) across training iterations, stabilizing optimization.

Knowl 8 — Semi-Supervised Classification Benchmarks on SVHN and CIFAR-10 without Data Augmentation

data/table

Test error rates (%) of semi-supervised learning methods evaluated on SVHN ( $N_l = 1000$ labeled images) and CIFAR-10 ( $N_l = 4000$ labeled images) without image data augmentation. VAT and VAT+EntMin were tested using Conv-Small (structure from Salimans et al., 2016) and Conv-Large (structure from Laine and Aila, 2017) CNN architectures.

Method	Test error rate (%)
	SVHN ( $N_l = 1000$ )	CIFAR-10 ( $N_l = 4000$ )
SWWAE	23.56	-
*Skip DGM	16.61 ( $\pm$ 0.24)	-
*Auxiliary DGM	22.86	-
Ladder networks, $\Gamma$ model	-	20.40 ( $\pm$ 0.47)
CatGAN	-	19.58 ( $\pm$ 0.58)
GAN with Feature Matching	8.11 ( $\pm$ 1.3)	18.63 ( $\pm$ 2.32)
$\Pi$ model	5.43 ( $\pm$ 0.25)	16.55 ( $\pm$ 0.29)
(on Conv-Small)
RPT	8.41 ( $\pm$ 0.24)	18.56 ( $\pm$ 0.29)
VAT	6.83 ( $\pm$ 0.24)	14.87 ( $\pm$ 0.13)
(on Conv-Large)
VAT	5.77 ( $\pm$ 0.32)	14.18 ( $\pm$ 0.38)
VAT+EntMin	4.28 ( $\pm$ 0.10)	13.15 ( $\pm$ 0.21)

Asterisk (*) denotes permutation-invariant models. Results demonstrate that VAT alone achieves 14.87% on CIFAR-10 with Conv-Small (outperforming GANs and $\Pi$ models), and VAT+EntMin on Conv-Large attains the lowest error rates on both datasets.

Knowl 9 — Semi-Supervised Classification Benchmarks on SVHN and CIFAR-10 with Moderate Data Augmentation

data/table

Test error rates (%) on SVHN ( $N_l = 1000$ ) and CIFAR-10 ( $N_l = 4000$ ) with moderate image data augmentation (random $2 \times 2$ translations for both datasets, plus horizontal flips for CIFAR-10). VAT models were trained on the Conv-Large architecture.

Method	Test error rate (%)
	SVHN ( $N_l = 1000$ )	CIFAR-10 ( $N_l = 4000$ )
$\Pi$ model	4.82 ( $\pm$ 0.17)	12.36 ( $\pm$ 0.31)
Temporal ensembling	4.42 ( $\pm$ 0.16)	12.16 ( $\pm$ 0.24)
Sajjadi et al. (extensive augmentation)	-	11.29 ( $\pm$ 0.24)
(On Conv-Large)
VAT	5.42 ( $\pm$ 0.22)	11.36 ( $\pm$ 0.34)
VAT+EntMin	3.86 ( $\pm$ 0.11)	10.55 ( $\pm$ 0.05)

VAT+EntMin combined with moderate data augmentation achieves state-of-the-art error rates of 3.86% on SVHN and 10.55% on CIFAR-10, demonstrating that virtual adversarial regularization operates orthogonally to label-invariant spatial data augmentations.

Knowl 10 — Supervised Learning Benchmark Performance on MNIST and CIFAR-10

data/table

Test error rates (%) for fully supervised classification on MNIST (60,000 labeled examples, 4-layer fully connected NN with layer sizes 1200-600-300-150 in the permutation-invariant setting) and CIFAR-10 (50,000 labeled examples, Conv-Large architecture with dropout and ZCA whitening preprocessing).

MNIST (Permutation Invariant)		CIFAR-10 (Conv-Large CNN)
Method	Error (%)	Method	Error (%)
SVM (Gaussian kernel)	1.40	Network in Network	8.81
Dropout	1.05	All-CNN	7.25
Adversarial ( $L_\infty$ ) [Goodfellow 2015]	0.78	Deeply Supervised Net	7.97
Ladder networks	0.57 ( $\pm$ 0.02)	Highway Network	7.72
Baseline (MLE)	1.11 ( $\pm$ 0.06)	ResNet (1001 layers)	4.62 ( $\pm$ 0.20)
RPT	0.84 ( $\pm$ 0.03)	DenseNet (190 layers)	3.46
Adversarial ( $L_\infty$ )	0.79 ( $\pm$ 0.03)	Baseline (dropout only)	6.67 ( $\pm$ 0.07)
Adversarial ( $L_2$ )	0.71 ( $\pm$ 0.03)	RPT	6.30 ( $\pm$ 0.04)
VAT	0.64 ( $\pm$ 0.05)	VAT	5.81 ( $\pm$ 0.02)

On MNIST, supervised VAT achieves a 0.64% error rate, surpassing standard adversarial training with $L_2$ (0.71%) and $L_\infty$ (0.79%) norms as well as dropout (1.05%). On CIFAR-10 with standard Conv-Large CNN, VAT reduces test error from 6.67% (baseline) and 6.30% (RPT) down to 5.81%.

Knowl 11 — Empirical Sufficiency of a Single Power Iteration ($K=1$)
empirical result
Evaluating semi-supervised VAT on CIFAR-10 ( $N_l = 4000$ , Conv-Large architecture) with varying numbers of power iterations $K$ yielded test error rates of:
- $K = 1$ : $14.18\% \pm 0.38\%$
- $K = 2$ : $14.19\% \pm 0.16\%$
- $K = 4$ : $14.25\% \pm 0.18\%$
Increasing $K$ beyond 1 provides negligible performance gain because the ratio between the first and second eigenvalues $\lambda_1 / \lambda_2$ of the Hessian $H(x, \hat{\theta})$ is sufficiently large across empirical distributions, causing the dominant eigenvector estimate to saturate at $K=1$ .

Coverage note — None was omitted; all key definitions, mathematical formulations, approximation algorithms, theoretical properties, and empirical results were covered.

References

1.Martın Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.
2.Mudassar Abbas, Jyri Kivinen, and Tapani Raiko. Understanding regularization by virtual adversarial training, ladder networks and others. In Workshop on ICLR, 2016.
3.Hirotugu Akaike. Information theory and an extension of the maximum likelihood principle. In Selected Papers of Hirotugu Akaike, pages 199–213. Springer, 1998.
4.Vladimir Igorevich Arnol’d. Mathematical methods of classical mechanics, volume 60. Springer Science & Business Media, 2013.
5.Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In NIPS, 2014.
6.Christopher M Bishop. Training with noise is equivalent to Tikhonov regularization. Neural computation, 7(1):108–116, 1995.
7.Christopher M Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
8.Ronan Collobert, Fabian Sinz, Jason Weston, and Leon Bottou. Large scale transductive SVMs. Journal of Machine Learning Research, 7(Aug):1687–1712, 2006.
9.Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016.
10.Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In AISTATS, 2011.
11.Gene H Golub and Henk A van der Vorst. Eigenvalue computation in the 20th century. Journal of Computational and Applied Mathematics, 123(1):35–65, 2000.
12.Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.
13.Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
14.Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
15.Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In NIPS, 2004.
16.Shixiang Gu and Luca Rigazio. Towards deep neural network architectures robust to adversarial examples. In Workshop on ICLR, 2015.
17.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016.
18.Gao Huang, Zhuang Liu, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
19.Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
20.Kevin Jarrett, Koray Kavukcuoglu, Marc’Aurelio Ranzato, and Yann LeCun. What is the best multi-stage architecture for object recognition? In ICCV, 2009.
21.Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
22.Diederik Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In NIPS, 2014.
23.Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report, University of Toronto, 2009.
24.Samuli Laine and Timo Aila. Temporal ensembling for semisupervised learning. In ICLR, 2017.
25.Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply-supervised nets. In AISTATS, 2015.
26.Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In ICLR, 2014.
27.Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary deep generative models. In ICML, 2016.
28.Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural network acoustic models. In ICML, 2013.
29.Shin-ichi Maeda. A Bayesian encourages dropout. arXiv preprint arXiv:1412.7003, 2014.
30.Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. Distributional smoothing with virtual adversarial training. In ICLR, 2016.
31.Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted Boltzmann machines. In ICML, 2010.
32.Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In Workshop on deep learning and unsupervised feature learning on NIPS, 2011.
33.Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi-supervised learning with ladder networks. In NIPS, 2015.
34.Russell Reed, Seho Oh, and RJ Marks. Regularization using jittered training data. In IJCNN. IEEE, 1992.
35.Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In NIPS, 2016.
36.Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In NIPS, 2016.
37.Jost Tobias Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks. In ICLR, 2015.
38.Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. In Workshop on ICLR, 2015.
39.Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. JMLR, 15(1), 2014.
40.Rupesh Kumar Srivastava, Klaus Greff, and Jurgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387, 2015.
41.Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
42.Theano Development Team. Theano: A Python framework for fast computation of mathematical expressions. arXiv preprint arXiv:1605.02688, 2016.
43.Andrej N Tikhonov and Vasiliy Y Arsenin. Solutions of ill-posed problems. Winston, 1977.
44.Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clayton. Chainer: a next-generation open source framework for deep learning. In Workshop on machine learning systems (LearningSys) on NIPS, 2015.
45.Stefan Wager, Sida Wang, and Percy S Liang. Dropout training as adaptive regularization. In NIPS, 2013.
46.Grace Wahba. Spline models for observational data. Siam, 1990.
47.Sumio Watanabe. Algebraic geometry and statistical learning theory. Cambridge University Press, 2009.
48.Junbo Zhao, Michael Mathieu, Ross Goroshin, and Yann Lecun. Stacked what-where auto-encoders. In Workshop on ICLR, 2016.
49.Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propagation. Technical report, Citeseer, 2002.

Citation

MLA

Miyato, T., et al. “Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning”. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 8, 2019, pp. 1979–93, https://doi.org/10.1109/TPAMI.2018.2858821.

APA

Miyato, T., Maeda, S.-I., Koyama, M., & Ishii, S. (2019). Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(8), 1979–1993. https://doi.org/10.1109/TPAMI.2018.2858821

Chicago

Miyato, T., S.-I. Maeda, M. Koyama, and S. Ishii. 2019. “Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning”. IEEE Transactions on Pattern Analysis and Machine Intelligence 41 (8): 1979–93. https://doi.org/10.1109/TPAMI.2018.2858821.

Harvard

Miyato, T. et al. (2019) “Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(8), pp. 1979–1993. Available at: https://doi.org/10.1109/TPAMI.2018.2858821.

Vancouver

1. Miyato T, Maeda S-I, Koyama M, Ishii S (2019) Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 41:1979–1993

BibTeX

@article{Miyato_2019, title={Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning}, volume={41}, ISSN={1939-3539}, url={http://dx.doi.org/10.1109/TPAMI.2018.2858821}, DOI={10.1109/tpami.2018.2858821}, number={8}, journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, publisher={Institute of Electrical and Electronics Engineers (IEEE)}, author={Miyato, Takeru and Maeda, Shin-Ichi and Koyama, Masanori and Ishii, Shin}, year={2019}, month=Aug, pages={1979–1993} }

Metadata:Crossref

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Virtual Adversarial Training (VAT) Objective and Local Distributional Smoothness

Knowl 2 — Fast Approximation of Virtual Adversarial Perturbation via Power Iteration and Finite Differences

Knowl 3 — Mini-Batch SGD for Virtual Adversarial Training Gradient

Knowl 4 — Virtual Adversarial Training with Conditional Entropy Minimization (VAT+EntMin)

Knowl 5 — Hyperparameter Equivalence of Perturbation Norm and Regularization Weight for Small Perturbations

Knowl 6 — Parameterization Invariance of Virtual Adversarial Regularization

Knowl 7 — Theoretical and Variance Comparison Between VAT and Random Perturbation Training

Knowl 8 — Semi-Supervised Classification Benchmarks on SVHN and CIFAR-10 without Data Augmentation

Knowl 9 — Semi-Supervised Classification Benchmarks on SVHN and CIFAR-10 with Moderate Data Augmentation

Knowl 10 — Supervised Learning Benchmark Performance on MNIST and CIFAR-10

Knowl 11 — Empirical Sufficiency of a Single Power Iteration ($K=1$)

References

Citation

BibTeX

Source Code

Access the Paper