Back

Temporal Ensembling for Semi-Supervised Learning

Samuli LaineTimo Aila

article2016ICLR2,938 citations

Introduces temporal ensembling, an efficient semi-supervised learning technique that forms consensus targets from previous training epochs under varying augmentations, substantially reducing classification error rates when labeled data is scarce.

Listen

The paper introduces two closely related training techniques, called the Π-model and temporal ensembling, that allow deep neural networks to learn from a mixture of a small number of labeled images and a much larger number of unlabeled images. The central idea is to form an ensemble-style target prediction for each unlabeled input by averaging the network’s own outputs across different training epochs, different dropout masks, and different random augmentations; this averaged target then serves as a stable training signal that encourages consistent predictions. Both methods were evaluated on standard image-classification benchmarks using only a few hundred or a few thousand labeled examples per dataset, with the remaining examples treated as unlabeled.

On the SVHN dataset with 500 labels the non-augmented error rate fell from the previous best of 18.44 % to 7.05 %; with standard translations the figure reached 5.12 %. On CIFAR-10 with 4 000 labels the corresponding numbers dropped from 18.63 % to 16.55 % without augmentation and to 12.16 % with horizontal flips and translations. Adding hundreds of thousands of extra unlabeled images drawn from the Tiny Images collection produced a further 2.7-percentage-point gain on CIFAR-100. The same consistency requirement also improved accuracy when every training example was labeled and conferred substantial robustness when up to 80 % of the supplied labels were deliberately corrupted.

These gains matter because labeling large image collections is expensive, while unlabeled images are abundant. The methods require only modest extra storage and computation, work with existing network architectures, and need no generative model or adversarial training. The main practical limitations are dependence on strong regularization (dropout plus input augmentation) and the need to tune a small number of additional hyperparameters that control the weighting and momentum of the ensemble targets. The authors note that further improvements could be obtained by tracking higher-order statistics of the network outputs to estimate prediction uncertainty, but they leave that extension for future work.

NVIDIAclassificationneural networksregularizationTemporal EnsemblingΠ-ModelSemi‑Supervised Learning
arXiv: 1610.02242s-laine/tempens
Cover for Temporal Ensembling for Semi-Supervised Learning

Abstract

In this paper, we present a simple and efficient method for training deep neural networks in a semi-supervised setting where only a small portion of training data is labeled. We introduce self-ensembling, where we form a consensus prediction of the unknown labels using the outputs of the network-in-training on different epochs, and most importantly, under different regularization and input augmentation conditions. This ensemble prediction can be expected to be a better predictor for the unknown labels than the output of the network at the most recent training epoch, and can thus be used as a target for training. Using our method, we set new records for two standard semi-supervised learning benchmarks, reducing the (non-augmented) classification error rate from 18.44% to 7.05% in SVHN with 500 labels and from 18.63% to 16.55% in CIFAR-10 with 4000 labels, and further to 5.12% and 12.16% by enabling the standard augmentations. We additionally obtain a clear improvement in CIFAR-100 classification accuracy by using random images from the Tiny Images dataset as unlabeled extra inputs during training. Finally, we demonstrate good tolerance to incorrect labels.

Topics

classificationneural networksregularizationsemi-supervised learningsupervised machine learningunsupervised learning

Keywords

Temporal EnsemblingΠ-ModelSemi‑Supervised LearningDropoutInput AugmentationUnsupervised LossEnsemble PredictionCIFAR‑10SVHNTiny Images

Table of Contents

  • 1 INTRODUCTION
  • 2 SELF-ENSEMBLING DURING TRAINING
  • 2.1 Π -MODEL
  • 2.2 TEMPORAL ENSEMBLING
  • 3 RESULTS
  • 3.1 CIFAR-10
  • 3.2 SVHN
  • 3.3 CIFAR-100 AND TINY IMAGES
  • 3.4 SUPERVISED LEARNING
  • 3.5 TOLERANCE TO INCORRECT LABELS
  • 4 RELATED WORK
  • 5 ACKNOWLEDGEMENTS
  • REFERENCES
  • A NETWORK ARCHITECTURE, TEST SETUP, AND TRAINING PARAMETERS

Knowls

  1. Knowl 1 — Temporal Ensembling for Semi-Supervised Learning

    model/method

    Temporal ensembling is a semi-supervised learning method that maintains an exponential moving average of historical network predictions over previous training epochs to form consensus targets for unlabeled and labeled data.

    During training, each input stimulus xix_i (where i∈{1,…,N}i \in \{1, \dots, N\}) is evaluated by a stochastic neural network fθf_\theta only once per epoch under a stochastic augmentation function g(xi,t)g(x_i, t) and stochastic dropout, yielding a current prediction vector zi=fθ(g(xi,t))z_i = f_\theta(g(x_i, t)). After each epoch tt, an accumulated prediction vector Zi∈RCZ_i \in \mathbb{R}^C across classes CC is updated recursively:

    Zi←αZi+(1−α)ziZ_i \leftarrow \alpha Z_i + (1 - \alpha) z_i

    where α∈[0,1)\alpha \in [0, 1) is a momentum hyperparameter (typically α=0.6\alpha = 0.6). To correct for the zero-initialization bias in early training epochs, a debiased ensemble prediction target z~i\tilde{z}_i is constructed via:

    z~i=Zi1−αt\tilde{z}_i = \frac{Z_i}{1 - \alpha^t}

    The network is trained using a composite loss that combines standard cross-entropy on labeled samples with a mean squared error consistency penalty between ziz_i and z~i\tilde{z}_i for all samples. Compared to evaluating the network multiple times per epoch, temporal ensembling requires only one forward pass per input per epoch, accelerating training roughly 2×2\times while producing ensemble targets with lower variance.

  2. Knowl 2 — Pi-Model for Consistency Regularization

    model/method

    The Π\Pi-model is a semi-supervised learning approach that encourages prediction consistency across two independent stochastic evaluations of the same training input within a single training epoch.

    For each input sample xix_i in a dataset of size NN, the neural network fθf_\theta (equipped with stochastic dropout and noise) is evaluated twice under independent stochastic input augmentations g(xi)g(x_i) and independent dropout realizations, producing two class probability prediction vectors:

    zi=fθ(g(xi)),z~i=fθ(g(xi))z_i = f_\theta(g(x_i)), \quad \tilde{z}_i = f_\theta(g(x_i))

    The network is optimized to minimize the sum of a standard supervised cross-entropy loss (evaluated only over labeled indices i∈Li \in L) and an unsupervised consistency loss defined as the squared difference between the full probability output distributions ziz_i and z~i\tilde{z}_i (evaluated over all inputs i∈{1,…,N}i \in \{1, \dots, N\}). By penalizing discrepancies across the full softmax distributions rather than just the top-1 predicted labels, the model enforces consistency across dark knowledge distributions.

  3. Knowl 3 — Semi-Supervised Consistency Loss Objective and Weight Ramp-Up

    equation

    For a minibatch BB of training samples from a total training set of size NN containing labeled index set L⊆{1,…,N}L \subseteq \{1, \dots, N\} with ground-truth class labels yi∈{1,…,C}y_i \in \{1, \dots, C\}, the overall training loss is given by:

    L=−1∣B∣∑i∈(B∩L)log⁡zi[yi]+w(t)1C∣B∣∑i∈B∥zi−z~i∥2\mathcal{L} = -\frac{1}{|B|} \sum_{i \in (B \cap L)} \log z_i[y_i] + w(t) \frac{1}{C|B|} \sum_{i \in B} \|z_i - \tilde{z}_i\|^2

    where zi∈RCz_i \in \mathbb{R}^C is the current softmax prediction output for input xix_i, z~i∈RC\tilde{z}_i \in \mathbb{R}^C is the target prediction (either a second stochastic evaluation of xix_i in the Π\Pi-model or the debiased historical ensemble target in temporal ensembling), CC is the number of classes, and w(t)w(t) is a time-dependent weight ramp-up function.

    To prevent the network from collapsing into degenerate solutions early in training when predictions are inaccurate, w(t)w(t) starts at zero and ramps up over the initial tramp=80t_{\text{ramp}} = 80 epochs according to a Gaussian curve:

    w(t)=wmax⁡MNexp⁡[−5(1−T)2],T=ttrampw(t) = w_{\max} \frac{M}{N} \exp\left[-5(1 - T)^2\right], \quad T = \frac{t}{t_{\text{ramp}}}

    where M=∣L∣M = |L| is the number of labeled inputs, NN is the total number of training inputs, and wmax⁡w_{\max} is a maximum weight hyperparameter. At epoch t=1t=1, w(1)=0w(1) = 0.

  4. Knowl 4 — Temporal Ensembling Training Algorithm

    algorithm

    The temporal ensembling algorithm trains a stochastic neural network by accumulating past predictions into an exponential moving average matrix ZZ and computing a consistency loss against bias-corrected targets z~\tilde{z}.

    Input: Training stimuli x_i for i in {1, ..., N}
    Input: Labeled index set L with labels y_i for i in L
    Input: Ensembling momentum alpha (0 <= alpha < 1, typically 0.6)
    Input: Unsupervised weight function w(t)
    Input: Stochastic neural network f_theta with trainable parameters theta
    Input: Stochastic input augmentation function g(x, t)
    Input: Number of training epochs num_epochs
    Initialize Z <- 0 of shape [N x C]
    Initialize z_tilde <- 0 of shape [N x C]
    for t = 1 to num_epochs do
        for each minibatch B do
            for each i in B do
                z_i <- f_theta(g(x_i, t))
            end for
            
            L_sup <- - (1 / |B|) * sum_{i in (B intersect L)} log z_i[y_i]
            L_unsup <- (w(t) / (C * |B|)) * sum_{i in B} ||z_i - z_tilde_i||^2
            loss <- L_sup + L_unsup
            
            update theta using an optimizer (e.g., ADAM)
        end for
        
        Z <- alpha * Z + (1 - alpha) * z
        z_tilde <- Z / (1 - alpha^t)
    end for
    return theta

  5. Knowl 5 — Pi-Model Training Algorithm

    algorithm

    The Π\Pi-model algorithm trains a stochastic neural network by performing two forward passes per input sample under different stochastic augmentation and dropout conditions within each minibatch.

    Input: Training stimuli x_i for i in {1, ..., N}
    Input: Labeled index set L with labels y_i for i in L
    Input: Unsupervised weight function w(t)
    Input: Stochastic neural network f_theta with trainable parameters theta
    Input: Stochastic input augmentation function g(x)
    Input: Number of training epochs num_epochs
    for t = 1 to num_epochs do
        for each minibatch B do
            for each i in B do
                z_i <- f_theta(g(x_i))
                z_tilde_i <- f_theta(g(x_i))
            end for
            
            L_sup <- - (1 / |B|) * sum_{i in (B intersect L)} log z_i[y_i]
            L_unsup <- (w(t) / (C * |B|)) * sum_{i in B} ||z_i - z_tilde_i||^2
            loss <- L_sup + L_unsup
            
            update theta using an optimizer (e.g., ADAM)
        end for
    end for
    return theta

  6. Knowl 6 — Convolutional Architecture and Optimization Pipeline for Self-Ensembling

    experimental setup

    The standard network backbone is a 13-layer convolutional architecture based on ConvPool-CNN-C with weight normalization and mean-only batch normalization (momentum 0.999) applied across all parameter layers:

    1. Input: 32×3232 \times 32 RGB image with additive Gaussian noise (σ=0.15\sigma = 0.15).
    2. Block 1: Three 3×33 \times 3 conv layers with 128 filters ('same' padding, Leaky ReLU with slope α=0.1\alpha = 0.1), followed by 2×22 \times 2 MaxPool and Dropout (p=0.5p = 0.5).
    3. Block 2: Three 3×33 \times 3 conv layers with 256 filters ('same' padding, Leaky ReLU with α=0.1\alpha = 0.1), followed by 2×22 \times 2 MaxPool and Dropout (p=0.5p = 0.5).
    4. Block 3: One 3×33 \times 3 conv layer with 512 filters ('valid' padding, Leaky ReLU with α=0.1\alpha = 0.1), followed by two 1×11 \times 1 conv layers with 256 and 128 filters (Leaky ReLU α=0.1\alpha = 0.1).
    5. Output: 6×6→1×16 \times 6 \to 1 \times 1 Global Average Pooling, fully connected layer (128→C128 \to C), and Softmax.

    Training settings:

    • Optimizer: Adam with β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, minibatch size 100, trained for 300 epochs.
    • Learning rate: λmax⁡=0.003\lambda_{\max} = 0.003 (or 0.0010.001 for SVHN temporal ensembling). Both learning rate λ\lambda and unsupervised weight ww ramp up over the first 80 epochs via exp⁡[−5(1−t/80)2]\exp[-5(1-t/80)^2]. During the final 50 epochs, λ\lambda is annealed to 0 and Adam β1\beta_1 is annealed to 0.5 using a time-reversed Gaussian ramp-down with scale constant 12.5.

  7. Knowl 7 — CIFAR-10 Semi-Supervised Classification Benchmarks

    data/table

    The classification error rates on the CIFAR-10 benchmark dataset (32×3232 \times 32 RGB images, 10 classes) demonstrate that self-ensembling methods substantially outperform previous semi-supervised techniques with 4000 labels and improve fully supervised performance with all 50,000 labels.

    Error rate (%) with # labels
    Model 4000 All (50000)
    Supervised-only 35.56±1.5935.56 \pm 1.59 7.33±0.047.33 \pm 0.04
    with augmentation 34.85±1.6534.85 \pm 1.65 6.05±0.156.05 \pm 0.15
    Conv-Large, Γ\Gamma-model 20.40±0.4720.40 \pm 0.47 –
    CatGAN 19.58±0.5819.58 \pm 0.58 –
    GAN of Salimans et al. 18.63±2.3218.63 \pm 2.32 –
    Π\Pi-model (no aug) 16.55±0.2916.55 \pm 0.29 6.90±0.076.90 \pm 0.07
    Π\Pi-model with augmentation 12.36±0.3112.36 \pm 0.31 5.56±0.105.56 \pm 0.10
    Temporal ensembling with augmentation 12.16±0.24\mathbf{12.16 \pm 0.24} 5.60±0.10\mathbf{5.60 \pm 0.10}

    Results are reported as mean ±\pm standard deviation over 10 runs (4 runs for all labels). Standard augmentations comprise random horizontal flips and random translations in [−2,2][-2, 2] pixels. Non-augmented Π\Pi-model achieves a 2.12.1 percentage point error reduction over prior GAN-based methods, and adding temporal ensembling with augmentation lowers the error from 18.63%18.63\% to 12.16%12.16\%.

  8. Knowl 8 — SVHN Semi-Supervised Classification Benchmarks

    data/table

    Evaluation on the Street View House Numbers (SVHN) dataset using the official 73,257 training samples (without extra data) demonstrates substantial improvements over prior semi-supervised approaches, particularly in low-label regimes (500 and 1000 labels).

    Error rate (%) with # labels
    Model 500 1000 All (73257)
    Supervised-only 35.18±5.6135.18 \pm 5.61 20.47±2.6420.47 \pm 2.64 3.05±0.073.05 \pm 0.07
    with augmentation 31.59±3.6031.59 \pm 3.60 19.30±3.8919.30 \pm 3.89 2.88±0.032.88 \pm 0.03
    DGN – 36.02±0.1036.02 \pm 0.10 –
    Virtual Adversarial – 24.6324.63 –
    ADGM – 22.8622.86 –
    SDGM – 16.61±0.2416.61 \pm 0.24 –
    GAN of Salimans et al. 18.44±4.818.44 \pm 4.8 8.11±1.38.11 \pm 1.3 –
    Π\Pi-model (no aug) 7.05±0.307.05 \pm 0.30 5.43±0.255.43 \pm 0.25 2.78±0.032.78 \pm 0.03
    Π\Pi-model with augmentation 6.65±0.536.65 \pm 0.53 4.82±0.174.82 \pm 0.17 2.54±0.04\mathbf{2.54 \pm 0.04}
    Temporal ensembling with augmentation 5.12±0.13\mathbf{5.12 \pm 0.13} 4.42±0.16\mathbf{4.42 \pm 0.16} 2.74±0.062.74 \pm 0.06

    Results are averaged over 10 runs (4 runs for all labels). In the 500-label setting, temporal ensembling achieves an error rate of 5.12%5.12\%, less than a third of the previous state-of-the-art error rate (18.44%18.44\%) and with dramatically reduced variance.

  9. Knowl 9 — Semi-Supervised CIFAR-100 with Auxiliary Unlabeled Tiny Images

    data/table

    On CIFAR-100 (100 classes), semi-supervised learning with 10,000 labels lowers test error from 44.56%44.56\% (supervised baseline with augmentation) to 38.65%38.65\% using temporal ensembling. In the fully labeled regime (50,000 labels), incorporating auxiliary unlabeled data from the 80 Million Tiny Images dataset further improves classification accuracy.

    Error rate (%) with auxiliary unlabeled Tiny Images
    Model (with augmentation) Random 500k Restricted 237k
    Π\Pi-model 25.79±0.1725.79 \pm 0.17 25.43±0.3225.43 \pm 0.32
    Temporal ensembling 23.62±0.23\mathbf{23.62 \pm 0.23} 23.79±0.2423.79 \pm 0.24

    In these experiments, each epoch sampled 50k labeled CIFAR-100 images and 50k unlabeled Tiny Images. Adding 500k randomly selected unlabeled Tiny Images reduced error from 26.30%26.30\% to 23.62%23.62\%. Restricting unlabeled auxiliary images to the 237k images corresponding to the 100 CIFAR-100 categories gave a comparable error rate (23.79%23.79\%), demonstrating that auxiliary unlabeled data does not need to match target class distributions strictly, provided it resides in the general natural image domain.

  10. Knowl 10 — Robustness to Label Noise via Self-Ensembling

    empirical result

    Temporal ensembling exhibits strong tolerance to label corruption. In experiments on SVHN where varying percentages (20%, 50%, 80%, and 90%) of training labels were replaced with uniform random class labels, standard supervised training degraded rapidly in accuracy (dropping below 40% accuracy at 50% corruption and failing to train stably).

    In contrast, temporal ensembling maintained over 90% classification accuracy even when 80% of the training labels were randomized, showing almost complete resistance to disinformation at 50% corruption. This robustness emerges because the unsupervised loss enforces the network mapping to be locally flat around all data points along the input manifold, smoothing decision boundaries and forming coherent clusters, while the majority of true labels in each class suffice to align each cluster to the correct output vector.

Coverage note — None was omitted; all key architectural choices, algorithms, loss formulations, benchmark results (CIFAR-10, SVHN, CIFAR-100 with Tiny Images), and label noise experiments are fully covered.

References

  1. 1.Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In Advances in Neural Information Processing Systems 27 (NIPS). 2014.
  2. 2.Leo Breiman. Bagging predictors. Machine Learning, 24(2), 1996.
  3. 3.Sander Dieleman, Jan Schlüter, Colin Raffel, Eben Olson, Søren Kaae Sønderby, et al. Lasagne: First release., 2015.
  4. 4.Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. CoRR, abs/1506.02142, 2016.
  5. 5.Benjamin Graham. Fractional max-pooling. CoRR, abs/1412.6071, 2014.
  6. 6.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. CoRR, abs/1502.01852, 2015.
  7. 7.G. E. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. CoRR, abs/1503.02531, 2015.
  8. 8.Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. CoRR, abs/1603.09382, 2016.
  9. 9.Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q. Weinberger. Snapshot Ensembles: Train 1, get M for free. In Proc. International Conference on Learning Representations (ICLR), 2017.
  10. 10.Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
  11. 11.Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in Neural Information Processing Systems 27 (NIPS). 2014.
  12. 12.Chen-Yu Lee, Patrick W. Gallagher, and Zhuowen Tu. Generalizing pooling functions in convolutional neural networks: Mixed, gated, and tree. CoRR, abs/1509.08985, 2015.
  13. 13.Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary deep generative models. CoRR, abs/1602.05473, 2016.
  14. 14.Andrew L Maas, Awni Y Hannun, and Andrew Ng. Rectifier nonlinearities improve neural network acoustic models. In Proc. International Conference on Machine Learning (ICML), volume 30, 2013.
  15. 15.Dmytro Mishkin and Jiri Matas. All you need is a good init. In Proc. International Conference on Learning Representations (ICLR), 2016.
  16. 16.Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. Distributional smoothing with virtual adversarial training. In Proc. International Conference on Learning Representations (ICLR), 2016.
  17. 17.Augustus Odena. Semi-supervised learning with generative adversarial networks. Data Efficient Machine Learning workshop at ICML 2016, 2016.
  18. 18.Giorgio Patrini, Alessandro Rozza, Aditya Menon, Richard Nock, and Lizhen Qu. Making neural networks robust to label noise: a loss correction approach. CoRR, abs/1609.03683, 2016.
  19. 19.Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semisupervised learning with ladder networks. In Advances in Neural Information Processing Systems 28 (NIPS). 2015.
  20. 20.Scott E. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. CoRR, abs/1412.6596, 2014.
  21. 21.Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Mutual exclusivity loss for semi-supervised deep learning. In 2016 IEEE International Conference on Image Processing, ICIP 2016, pp. 1908–1912, 2016a.
  22. 22.Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In Advances in Neural Information Processing Systems 29 (NIPS). 2016b.
  23. 23.Tim Salimans and Diederik P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. CoRR, abs/1602.07868, 2016.
  24. 24.Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. CoRR, abs/1606.03498, 2016.
  25. 25.Patrice Y. Simard, Yann A. LeCun, John S. Denker, and Bernard Victorri. Transformation Invariance in Pattern Recognition — Tangent Distance and Tangent Propagation, pp. 239–274. 1998.
  26. 26.Saurabh Singh, Derek Hoiem, and David A. Forsyth. Swapout: Learning an ensemble of deep architectures. CoRR, abs/1605.06465, 2016.
  27. 27.Jost Tobias Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks. In Proc. International Conference on Learning Representations (ICLR), 2016.
  28. 28.Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. CoRR, abs/1412.6806, 2014.
  29. 29.Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929–1958, 2014.
  30. 30.Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. CoRR, abs/1406.2080, 2014.
  31. 31.Theano Development Team. Theano: A Python framework for fast computation of mathematical expressions. CoRR, abs/1605.02688, May 2016.
  32. 32.A. Torralba, R. Fergus, and W. T. Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE TPAMI, 30(11):1958–1970, 2008.
  33. 33.Li Wan, Matthew Zeiler, Sixin Zhang, Yann L. Cun, and Rob Fergus. Regularization of neural networks using dropconnect. Proc. International Conference on Machine Learning (ICML), 28 (3):1058–1066, 2013.
  34. 34.Max Whitney and Anoop Sarkar. Bootstrapping via graph propagation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers - Volume 1, ACL ’12, 2012.
  35. 35.David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In Proceedings of the 33rd Annual Meeting on Association for Computational Linguistics, ACL ’95, 1995.
  36. 36.Xiaojin Zhu. Semi-supervised learning literature survey. Technical Report 1530, Computer Sciences, University of Wisconsin-Madison, 2005.
  37. 37.Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propagation. Technical Report CMU-CALD-02-107, Carnegie Mellon University, 2002.

Citation

MLA
Laine, S., and T. Aila. “Temporal Ensembling for Semi-Supervised Learning”. arXiv, 2016, https://doi.org/10.48550/arxiv.1610.02242.
APA
Laine, S., & Aila, T. (2016). Temporal Ensembling for Semi-Supervised Learning. arXiv. https://doi.org/10.48550/arxiv.1610.02242
Chicago
Laine, S., and T. Aila. 2016. “Temporal Ensembling for Semi-Supervised Learning”. Preprint, ArXiv. https://doi.org/10.48550/arxiv.1610.02242.
Harvard
Laine, S. and Aila, T. (2016) “Temporal Ensembling for Semi-Supervised Learning”. arXiv. Available at: https://doi.org/10.48550/arxiv.1610.02242.
Vancouver
1. Laine S, Aila T (2016) Temporal Ensembling for Semi-Supervised Learning. https://doi.org/10.48550/arxiv.1610.02242

BibTeX

@misc{https://doi.org/10.48550/arxiv.1610.02242,
  doi = {10.48550/ARXIV.1610.02242},
  url = {https://arxiv.org/abs/1610.02242},
  author = {Laine, Samuli and Aila, Timo},
  keywords = {Neural and Evolutionary Computing (cs.NE), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {Temporal Ensembling for Semi-Supervised Learning},
  publisher = {arXiv},
  year = {2016},
  copyright = {arXiv.org perpetual, non-exclusive license}
}
Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors