Back

Unsupervised Data Augmentation for Consistency Training

Qizhe XieZihang DaiEduard HovyMinh-Thang LuongQuoc V. Le

article2020NeurIPS2,749 citations

Demonstrates that pairing consistency training with advanced data augmentation such as back-translation and RandAugment dramatically reduces labeled data requirements, enabling models to match or exceed fully supervised baselines across text and vision benchmarks using only a fraction of labeled examples.

Listen

Deep learning models usually require large amounts of labeled data, which limits their use when labels are expensive or scarce. Semi-supervised learning addresses this by leveraging abundant unlabeled data through consistency training, where model predictions must remain stable under small input changes. The article examines whether the quality of those changes, produced by advanced data augmentation, determines how well consistency training works.

The authors set out to test if state-of-the-art augmentation methods already proven in supervised settings can serve as superior noise sources for consistency training. They replace simple noise operations with RandAugment for images and back-translation or TF-IDF word replacement for text, then train models jointly on labeled cross-entropy loss and unlabeled consistency loss across six text classification tasks and three vision benchmarks.

Experiments used standard architectures including Wide-ResNet, Shake-Shake, PyramidNet, and BERT variants. Labeled data sizes ranged from 20 examples to the full supervised sets, with additional out-of-domain unlabeled data filtered for relevance on ImageNet. Results showed a clear correlation between an augmentation method’s supervised performance and its gains under consistency training.

With only 20 labeled examples, UDA reached 4.20 percent error on IMDb, beating the prior state-of-the-art trained on 25,000 labels. On CIFAR-10 it achieved 5.43 percent error with 250 labels and 4.32 percent with 4,000 labels, outperforming all prior semi-supervised methods. Similar gains appeared on SVHN, and UDA improved ImageNet top-1 accuracy from 58.84 to 68.78 percent with 10 percent labels and from 78.43 to 79.05 percent with the full labeled set plus 1.3 million extra unlabeled images. The method also complemented BERT fine-tuning.

These findings indicate that high-quality, diverse, and label-preserving augmentations allow models to propagate label information across larger regions of the data manifold, sharply reducing the number of labels needed while remaining effective even when labels are plentiful. Organizations can therefore cut annotation costs and still reach competitive accuracy on standard classification tasks.

Practitioners should adopt UDA when labeled data is limited or when additional unlabeled data is available, starting with the reported augmentations and hyperparameters. Further gains are likely from task-specific augmentation tuning and from extending the approach beyond classification. The main limitations are that results rest on specific benchmarks and that five-way sentiment tasks still show a gap relative to fully supervised performance; caution is warranted when moving to new domains or label distributions without additional validation.

Carnegie Mellon UniversityGoogleclassificationdata augmentationdeep learningUnsupervised Data AugmentationConsistency TrainingRandAugment
arXiv: 1904.12848google-research/uda
Cover for Unsupervised Data Augmentation for Consistency Training

Abstract

Semi-supervised learning lately has shown much promise in improving deep learning models when labeled data is scarce. Common among recent approaches is the use of consistency training on a large amount of unlabeled data to constrain model predictions to be invariant to input noise. In this work, we present a new perspective on how to effectively noise unlabeled examples and argue that the quality of noising, specifically those produced by advanced data augmentation methods, plays a crucial role in semi-supervised learning. By substituting simple noising operations with advanced data augmentation methods such as RandAugment and back-translation, our method brings substantial improvements across six language and three vision tasks under the same consistency training framework. On the IMDb text classification dataset, with only 20 labeled examples, our method achieves an error rate of 4.20, outperforming the state-of-the-art model trained on 25,000 labeled examples. On a standard semi-supervised learning benchmark, CIFAR-10, our method outperforms all previous approaches and achieves an error rate of 5.43 with only 250 examples. Our method also combines well with transfer learning, e.g., when finetuning from BERT, and yields improvements in high-data regime, such as ImageNet, whether when there is only 10% labeled data or when a full labeled set with 1.3M extra unlabeled examples is used. Code is available at this https URL.

Topics

classificationdata augmentationdeep learningsemi-supervised learningsupervised machine learningtext classificationtransfer learningunsupervised learning method

Keywords

Unsupervised Data AugmentationConsistency TrainingRandAugmentBack‑TranslationSemi‑Supervised LearningBERTCIFAR‑10ImageNetConfidence‑Based MaskingTraining Signal Annealing

Table of Contents

  • 1 Introduction
  • 2 Unsupervised Data Augmentation (UDA)
  • 2.1 Background: Supervised Data Augmentation
  • 2.2 Unsupervised Data Augmentation
  • 2.3 Augmentation Strategies for Different Tasks
  • 2.4 Additional Training Techniques
  • 3 Theoretical Analysis
  • 4 Experiments
  • 4.1 Correlation between Supervised and Semi-supervised Performances
  • 4.2 Algorithm Comparison on Vision Semi-supervised Learning Benchmarks
  • 4.3 Evaluation on Text Classification Datasets
  • 4.4 Scalability Test on the ImageNet Dataset
  • 5 Related Work
  • 6 Conclusion
  • References
  • A Extended Method Details
  • A.1 Training Signal Annealing for Low-data Regime
  • A.2 Extended Augmentation Strategies for Different Tasks
  • B Extended Experiments
  • B.1 Ablation Studies
  • B.2 More Results on CIFAR-10, SVHN and Text Classification Datasets
  • C Proof for Theoretical Analysis
  • D Extended Related Work
  • E Experiment Details
  • E.1 Text Classifications
  • E.2 Semi-supervised learning benchmarks CIFAR-10 and SVHN
  • E.3 ImageNet

Knowls

  1. Knowl 1 — Unsupervised Data Augmentation Objective and Training Procedure

    model/method

    Unsupervised Data Augmentation (UDA) trains a classifier pθ(y∣x)p_\theta(y \mid x) by jointly minimizing a supervised cross-entropy loss on labeled data and an unsupervised consistency loss on unannotated data transformed with targeted, high-diversity data augmentations.

    Let pL(x)p_L(x) and pU(x)p_U(x) denote the distributions of labeled and unlabeled instances, f∗(x)f^*(x) the ground-truth label of xx, q(x^∣x)q(\hat{x} \mid x) an augmentation transformation distribution, and θ~\tilde{\theta} a fixed copy of parameters θ\theta (enforcing a stop-gradient). The general training objective parameterized by a trade-off weight λ\lambda is:

    min⁡θJ(θ)=Ex1∼pL(x)[−log⁡pθ(f∗(x1)∣x1)]+λ Ex2∼pU(x)Ex^∼q(x^∣x2)[CE(pθ~(y∣x2) ∥ pθ(y∣x^))]\min_\theta \mathcal{J}(\theta) = \mathbb{E}_{x_1 \sim p_L(x)} \left[ -\log p_\theta(f^*(x_1) \mid x_1) \right] + \lambda \, \mathbb{E}_{x_2 \sim p_U(x)} \mathbb{E}_{\hat{x} \sim q(\hat{x} \mid x_2)} \left[ \text{CE}\left(p_{\tilde{\theta}}(y \mid x_2) \,\|\, p_\theta(y \mid \hat{x})\right) \right]

    where CE(p ∥ q)=−∑yp(y)log⁡q(y)\text{CE}(p \,\|\, q) = -\sum_y p(y) \log q(y).

    In practical minibatch training over an unlabeled minibatch BB, the consistency loss incorporates confidence-based masking and prediction sharpening with a low temperature τ\tau:

    Lunsup(θ)=1∣B∣∑x∈BI(max⁡y′pθ~(y′∣x)>β)CE(pθ~(sharp)(y∣x) ∥ pθ(y∣x^))\mathcal{L}_{\text{unsup}}(\theta) = \frac{1}{|B|} \sum_{x \in B} \mathbb{I}\left(\max_{y'} p_{\tilde{\theta}}(y' \mid x) > \beta\right) \text{CE}\left(p_{\tilde{\theta}}^{(\text{sharp})}(y \mid x) \,\|\, p_\theta(y \mid \hat{x})\right)

    where I(⋅)\mathbb{I}(\cdot) is the indicator function, β∈(0,1)\beta \in (0, 1) is a confidence threshold, and the sharpened target distribution is defined from the logit zyz_y for category yy as:

    pθ~(sharp)(y∣x)=exp⁡(zy/τ)∑y′exp⁡(zy′/τ)p_{\tilde{\theta}}^{(\text{sharp})}(y \mid x) = \frac{\exp(z_y / \tau)}{\sum_{y'} \exp(z_{y'} / \tau)}

  2. Knowl 2 — Training Signal Annealing for Semi-Supervised Overfitting Mitigation

    model/method

    In semi-supervised scenarios where the volume of unlabeled data vastly exceeds labeled data, standard training quickly overfits the small labeled dataset. Training Signal Annealing (TSA) addresses this imbalance by dynamically filtering out labeled examples on which the model is already confident.

    At training step t∈[1,T]t \in [1, T] (where TT is total steps), a labeled training sample (x,y∗)(x, y^*) is retained in the supervised cross-entropy objective only if pθ(y∗∣x)≤ηtp_\theta(y^* \mid x) \le \eta_t, and excluded otherwise. For a task with KK classes, the threshold ηt\eta_t is annealed from 1/K1/K to 11 via a progression factor αt∈[0,1]\alpha_t \in [0, 1]:

    ηt=αt(1−1K)+1K\eta_t = \alpha_t \left(1 - \frac{1}{K}\right) + \frac{1}{K}

    Three scheduling functions for αt\alpha_t adapt to different regularization demands:

    1. Logarithmic schedule (for settings with larger labeled pools or strong implicit regularization): αt=1−exp⁡(−tT⋅5)\alpha_t = 1 - \exp\left(-\frac{t}{T} \cdot 5\right)

    2. Linear schedule: αt=tT\alpha_t = \frac{t}{T}

    3. Exponential schedule (for low-data regimes prone to rapid overfitting, withholding most labeled signal until late in training): αt=exp⁡((tT−1)⋅5)\alpha_t = \exp\left(\left(\frac{t}{T} - 1\right) \cdot 5\right)

  3. Knowl 3 — Generalization Error Bound under Augmentation Graph Connectivity

    theoretical result

    Let GpUG_{p_U} be a connectivity graph whose nodes are data points x∈Xx \in \mathcal{X} drawn from the unlabeled marginal distribution pU(x)p_U(x), with an edge between xx and x^\hat{x} if and only if the augmentation probability q(x^∣x)>0q(\hat{x} \mid x) > 0. Under the assumptions that the augmentation is in-domain (pU(x^)>0p_U(\hat{x}) > 0), label-preserving (f∗(x)=f∗(x^)f^*(x) = f^*(\hat{x})), and reversible (q(x^∣x)>0  ⟺  q(x∣x^)>0q(\hat{x} \mid x) > 0 \iff q(x \mid \hat{x}) > 0), the graph partitions into kk disconnected subgraphs (components) {C1,C2,…,Ck}\{C_1, C_2, \dots, C_k\} where k≥Nk \ge N for an NN-class problem.

    Let Pi=∑x∈CipL(x)P_i = \sum_{x \in C_i} p_L(x) be the probability mass of component CiC_i under labeled distribution pLp_L. If mm labeled examples are drawn independently from pLp_L, the probability Pr(A)\text{Pr}(A) that the algorithm cannot infer the label of a new test example (i.e., the test instance lies in a component containing zero labeled training instances) is:

    Pr(A)=∑i=1kPi(1−Pi)m\text{Pr}(A) = \sum_{i=1}^k P_i (1 - P_i)^m

    Maximizing this failure probability under the simplex constraint ∑i=1kPi=1\sum_{i=1}^k P_i = 1 yields uniform mass Pi=1/kP_i = 1/k, establishing the upper bound:

    Pr(A)≤(1−1k)m≤exp⁡(−mk)\text{Pr}(A) \le \left(1 - \frac{1}{k}\right)^m \le \exp\left(-\frac{m}{k}\right)

    Consequently, drawing m=O(k/ϵ)m = O(k/\epsilon) labeled examples guarantees an expected error rate Pr(A)=O(ϵ)\text{Pr}(A) = O(\epsilon). Higher diversity and connectivity in data augmentation directly reduce the component count kk, decreasing the labeled sample complexity required for a target generalization error.

  4. Knowl 4 — Augmentation Transformation Assumptions for Theoretical Consistency Analysis

    assumption

    The theoretical sample complexity guarantees of consistency training under Unsupervised Data Augmentation rely on three properties of the augmentation transformation q(x^∣x)q(\hat{x} \mid x) with respect to the unlabeled data distribution pU(x)p_U(x) and target classification function f∗(x)f^*(x):

    1. In-domain augmentation: Synthetic examples generated by the augmentation distribution have non-zero support under the unlabeled data distribution: pU(x^)>0for x^∼q(x^∣x),  x∼pU(x)p_U(\hat{x}) > 0 \quad \text{for } \hat{x} \sim q(\hat{x} \mid x), \; x \sim p_U(x)

    2. Label-preserving augmentation: The augmentation operator preserves the true underlying class label of the source instance: f∗(x)=f∗(x^)for x^∼q(x^∣x),  x∼pU(x)f^*(x) = f^*(\hat{x}) \quad \text{for } \hat{x} \sim q(\hat{x} \mid x), \; x \sim p_U(x)

    3. Reversible augmentation: The augmentation transition graph is undirected in probability support: q(x^∣x)>0  ⟹  q(x∣x^)>0q(\hat{x} \mid x) > 0 \implies q(x \mid \hat{x}) > 0

  5. Knowl 5 — Text Augmentation via Diverse Back-Translation and TF-IDF Word Replacement

    model/method

    Unsupervised consistency training on text uses two semantic-preserving text augmentation techniques:

    1. Diverse Back-Translation: A source sentence xx in language AA is translated to intermediate language BB and translated back to language AA to yield x^\hat{x}. To ensure lexical and syntactic diversity without semantic degradation, generation utilizes random sampling with a tuned temperature τsample∈[0.7,0.9]\tau_{\text{sample}} \in [0.7, 0.9] instead of deterministic beam search (which yields low diversity) or τ=1.0\tau=1.0 (which produces ungrammatical text). For English text, bidirectional WMT'14 English-French translation models are employed.

    2. TF-IDF Word Replacement: Designed for keyword-sensitive tasks where back-translation might omit key terms. Uninformative words with low TF-IDF are preferentially replaced while high TF-IDF terms are preserved. For a word wiw_i in sentence xx, its replacement probability is: preplace(wi)=min⁡(p⋅(C−TFIDF(wi))Z,1)p_{\text{replace}}(w_i) = \min\left( \frac{p \cdot (C - \text{TFIDF}(w_i))}{Z}, 1 \right) where TFIDF(w)=TF(w)⋅IDF(w)\text{TFIDF}(w) = \text{TF}(w) \cdot \text{IDF}(w), C=max⁡jTFIDF(wj)C = \max_j \text{TFIDF}(w_j), Z=1∣x∣∑j(C−TFIDF(wj))Z = \frac{1}{|x|} \sum_j (C - \text{TFIDF}(w_j)), and pp is an augmentation magnitude hyperparameter. When a token is replaced, a substitute word ww is drawn from the vocabulary inversely proportional to its corpus-level importance S(w)=freq(w)⋅IDF(w)S(w) = \text{freq}(w) \cdot \text{IDF}(w) with probability (max⁡w′S(w′)−S(w))/Z′(\max_{w'} S(w') - S(w)) / Z', avoiding accidental insertion of discriminative keywords.

  6. Knowl 6 — Semi-Supervised Text Classification Benchmark Results

    data/table

    The performance of Unsupervised Data Augmentation across text classification benchmarks (IMDb, Yelp-2, Yelp-5, Amazon-2, Amazon-5, and DBpedia) is evaluated across four model initializations: Random Transformer (6 layers, 8 heads), BERTBASE\text{BERT}_{\text{BASE}}, BERTLARGE\text{BERT}_{\text{LARGE}}, and BERTFINETUNE\text{BERT}_{\text{FINETUNE}} (fine-tuned on in-domain unlabeled text). All inputs are truncated to 512 subwords.

    Supervised Baselines IMDb Yelp-2 Yelp-5 Amazon-2 Amazon-5 DBpedia
    (# Sup examples) (25k) (560k) (650k) (3.6m) (3m) (560k)
    Pre-BERT SOTA 4.32 2.16 29.98 3.32 34.81 0.70
    BERTLARGE\text{BERT}_{\text{LARGE}} 4.51 1.89 29.32 2.63 34.17 0.64
    Semi-Supervised Setting IMDb Yelp-2 Yelp-5 Amazon-2 Amazon-5 DBpedia
    (# Sup examples) (20) (20) (2.5k) (20) (2.5k) (140)
    Random w/o UDA 43.27 40.25 50.80 45.39 55.70 41.14
    Random + UDA 25.23 8.33 41.35 16.16 44.19 7.24
    BERTBASE\text{BERT}_{\text{BASE}} w/o UDA 18.40 13.60 41.00 26.75 44.09 2.58
    BERTBASE\text{BERT}_{\text{BASE}} + UDA 5.45 2.61 33.80 3.96 38.40 1.33
    BERTLARGE\text{BERT}_{\text{LARGE}} w/o UDA 11.72 10.55 38.90 15.54 42.30 1.68
    BERTLARGE\text{BERT}_{\text{LARGE}} + UDA 4.78 2.50 33.54 3.93 37.80 1.09
    BERTFINETUNE\text{BERT}_{\text{FINETUNE}} w/o UDA 6.50 2.94 32.39 12.17 37.32 –
    BERTFINETUNE\text{BERT}_{\text{FINETUNE}} + UDA 4.20 2.05 32.08 3.50 37.12 –

    On IMDb with only 20 labeled examples, UDA combined with BERTFINETUNE\text{BERT}_{\text{FINETUNE}} reaches an error rate of 4.20%, surpassing the fully supervised pre-BERT state-of-the-art (4.32%) trained on all 25,000 labeled instances.

  7. Knowl 7 — Semi-Supervised Vision Benchmark Results on CIFAR-10 and SVHN

    data/table

    UDA utilizing RandAugment is evaluated on standard semi-supervised vision benchmarks against established consistency and pseudo-labeling methods across multiple neural architectures. Error rates (mean ±\pm standard deviation over multiple runs) on CIFAR-10 (4,000 labeled examples) and SVHN (1,000 labeled examples) are summarized below:

    Method Model # Param CIFAR-10 (4k) SVHN (1k)
    Π\Pi-Model Conv-Large 3.1M 12.36±0.3112.36 \pm 0.31 4.82±0.174.82 \pm 0.17
    Mean Teacher Conv-Large 3.1M 12.31±0.2812.31 \pm 0.28 3.95±0.193.95 \pm 0.19
    VAT + EntMin Conv-Large 3.1M 10.55±0.0510.55 \pm 0.05 3.86±0.113.86 \pm 0.11
    SNTG Conv-Large 3.1M 10.93±0.1410.93 \pm 0.14 3.86±0.273.86 \pm 0.27
    ICT Conv-Large 3.1M 7.29±0.027.29 \pm 0.02 3.89±0.043.89 \pm 0.04
    Pseudo-Label WRN-28-2 1.5M 16.21±0.1116.21 \pm 0.11 7.62±0.297.62 \pm 0.29
    LGA + VAT WRN-28-2 1.5M 12.06±0.1912.06 \pm 0.19 6.58±0.366.58 \pm 0.36
    ICT WRN-28-2 1.5M 7.66±0.177.66 \pm 0.17 3.53±0.073.53 \pm 0.07
    MixMatch WRN-28-2 1.5M 6.24±0.066.24 \pm 0.06 2.89±0.062.89 \pm 0.06
    Mean Teacher Shake-Shake 26M 6.28±0.156.28 \pm 0.15 –
    Fast-SWA Shake-Shake 26M 5.0 –
    MixMatch WRN 26M 4.95±0.084.95 \pm 0.08 –
    UDA (RandAugment) WRN-28-2 1.5M 4.32±0.084.32 \pm 0.08 2.23±0.072.23 \pm 0.07
    UDA (RandAugment) Shake-Shake 26M 3.7 –
    UDA (RandAugment) PyramidNet 26M 2.7 –

    With 4,000 labels on CIFAR-10, UDA with PyramidNet+ShakeDrop achieves a 2.7% error rate, matching the error rate of a fully supervised PyramidNet trained on all 50,000 examples without RandAugment. On CIFAR-10 with 250 labeled examples, UDA achieves an error rate of 5.43±0.96%5.43 \pm 0.96\% (compared to 11.08±0.87%11.08 \pm 0.87\% for MixMatch and 36.03±2.82%36.03 \pm 2.82\% for VAT).

  8. Knowl 8 — ImageNet Scalability and Out-of-Domain Unlabeled Data Utilization

    data/table

    UDA scales to high-resolution visual classification on ImageNet using ResNet-50 under two conditions: (1) low in-domain labeled data (10% labeled ImageNet, remaining 90% used as unlabeled in-domain data, image size 224), and (2) full labeled ImageNet (100% labeled, image size 331) supplemented by 1.3 million unlabeled out-of-domain images retrieved from the JFT dataset via domain-relevance data filtering.

    Methods SSL 10% Labeled 100% Labeled
    ResNet-50 Baseline No 55.09 / 77.26 77.28 / 93.73
    ResNet-50 w/ RandAugment No 58.84 / 80.56 78.43 / 94.37
    UDA (RandAugment) Yes 68.78 / 88.80 79.05 / 94.49

    In the 10% regime, UDA improves top-1 accuracy by +9.94 percentage points over the supervised RandAugment baseline (from 58.84% to 68.78%). In the full data regime, UDA improves top-1 accuracy from 78.43% to 79.05% using filtered unlabeled out-of-domain images.

  9. Knowl 9 — Domain-Relevance Data Filtering for Unlabeled Datasets

    model/method

    When acquiring unlabeled data from open, out-of-domain distributions (such as external image or text collections), class mismatch can impair semi-supervised consistency regularization. Domain-relevance data filtering aligns the unlabeled dataset with target domain categories before training.

    1. Train an initial baseline classifier using available in-domain labeled instances.
    2. Run inference with the baseline classifier across the entire candidate out-of-domain unlabeled dataset to compute category probability distributions p(y∣x)p(y \mid x) for each instance.
    3. For each category cc, rank candidate samples in descending order of their predicted class probability p(y=c∣x)p(y=c \mid x).
    4. Select the highest-probability instances per category to populate the curated unlabeled dataset for consistency training.

  10. Knowl 10 — Correlation Between Supervised and Consistency-Training Augmentation Strength

    empirical result

    The effectiveness of a data augmentation strategy in fully supervised learning exhibits a strict positive correlation with its performance boost in semi-supervised consistency training.

    On CIFAR-10, comparing augmentations in fully supervised (50,000 samples) vs semi-supervised (4,000 labeled samples) settings:

    • Standard Crop & Flip: 5.36% supervised error →\rightarrow 10.94% semi-supervised error
    • Cutout: 4.42% supervised error →\rightarrow 5.43% semi-supervised error
    • RandAugment: 4.23% supervised error →\rightarrow 4.32% semi-supervised error

    On Yelp-5, comparing NLP augmentations in fully supervised (650,000 samples) vs semi-supervised (2,500 labeled samples) settings:

    • No augmentation: 38.36% supervised error →\rightarrow 50.80% semi-supervised error
    • Switchout: 37.24% supervised error →\rightarrow 43.38% semi-supervised error
    • Back-translation: 36.71% supervised error →\rightarrow 41.35% semi-supervised error

    Augmentations that provide greater diversity while preserving valid class semantics produce larger gains in consistency training than simple input perturbations.

Coverage note — None was omitted; all main methods, theoretical formulations, algorithms, and primary benchmark results across vision and language are covered.

References

  1. 1.Ben Athiwaratkun, Marc Finzi, Pavel Izmailov, and Andrew Gordon Wilson. There are many consistent explanations of unlabeled data: Why you should average. ICLR, 2019.
  2. 2.Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In Advances in Neural Information Processing Systems, pages 3365–3373, 2014.
  3. 3.David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint arXiv:1905.02249, 2019.
  4. 4.Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C Duchi. Unlabeled data improves adversarial robustness. arXiv preprint arXiv:1905.13736, 2019.
  5. 5.Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews]. IEEE Transactions on Neural Networks, 20(3):542–542, 2009.
  6. 6.François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251–1258, 2017.
  7. 7.Kevin Clark, Minh-Thang Luong, Christopher D Manning, and Quoc V Le. Semi-supervised sequence modeling with cross-view training. arXiv preprint arXiv:1809.08370, 2018.
  8. 8.Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM, 2008.
  9. 9.Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501, 2018.
  10. 10.Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical data augmentation with no separate search. arXiv preprint arXiv:1909.13719, 2019.
  11. 11.Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Advances in neural information processing systems, pages 3079–3087, 2015.
  12. 12.Zihang Dai, Zhilin Yang, Fan Yang, William W Cohen, and Ruslan R Salakhutdinov. Good semi-supervised learning that requires a bad gan. In Advances in Neural Information Processing Systems, pages 6510–6520, 2017.
  13. 13.Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  14. 14.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  15. 15.Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back-translation at scale. arXiv preprint arXiv:1808.09381, 2018.
  16. 16.Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Advances in neural information processing systems, pages 529–536, 2005.
  17. 17.Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567, 2014.
  18. 18.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  19. 19.Xuanli He, Gholamreza Haffari, and Mohammad Norouzi. Sequence to sequence mixture model for diverse machine translation. arXiv preprint arXiv:1810.07391, 2018.
  20. 20.Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:1905.09272, 2019.
  21. 21.Alex Hernández-García and Peter König. Data augmentation instead of explicit regularization. arXiv preprint arXiv:1806.03852, 2018.
  22. 22.Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  23. 23.Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 328–339, 2018.
  24. 24.Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-augmented training. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1558–1567. JMLR. org, 2017.
  25. 25.Jacob Jackson and John Schulman. Semi-supervised learning by label gradient alignment. arXiv preprint arXiv:1902.02336, 2019.
  26. 26.Rie Johnson and Tong Zhang. Deep pyramid convolutional neural networks for text categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 562–570, 2017.
  27. 27.Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pages 3581–3589, 2014.
  28. 28.Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.
  29. 29.Wouter Kool, Herke van Hoof, and Max Welling. Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement. arXiv preprint arXiv:1903.06059, 2019.
  30. 30.Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
  31. 31.Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012.
  32. 32.Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242, 2016.
  33. 33.Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, ICML, volume 3, page 2, 2013.
  34. 34.Davis Liang, Zhiheng Huang, and Zachary C Lipton. Learning noise-invariant representations for robust speech recognition. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 56–63. IEEE, 2018.
  35. 35.Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren, and Bo Zhang. Smooth neighbors on teacher graphs for semi-supervised learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8896–8905, 2018.
  36. 36.Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary deep generative models. arXiv preprint arXiv:1602.05473, 2016.
  37. 37.Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies-volume 1, pages 142–150. Association for Computational Linguistics, 2011.
  38. 38.Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 43–52. ACM, 2015.
  39. 39.Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119, 2013.
  40. 40.Takeru Miyato, Andrew M Dai, and Ian Goodfellow. Adversarial training methods for semi-supervised text classification. arXiv preprint arXiv:1605.07725, 2016.
  41. 41.Takeru Miyato, Shin-ichi Maeda, Shin Ishii, and Masanori Koyama. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 2018.
  42. 42.Amir Najafi, Shin-ichi Maeda, Masanori Koyama, and Takeru Miyato. Robustness to adversarial perturbations in learning from incomplete data. arXiv preprint arXiv:1905.13021, 2019.
  43. 43.Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
  44. 44.Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems, pages 3235–3246, 2018.
  45. 45.Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779, 2019.
  46. 46.Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
  47. 47.Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018.
  48. 48.Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf, 2018.
  49. 49.Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi-supervised learning with ladder networks. In Advances in neural information processing systems, pages 3546–3554, 2015.
  50. 50.Devendra Singh Sachan, Manzil Zaheer, and Ruslan Salakhutdinov. Revisiting lstm networks for semi-supervised text classification via mixed objective function. 2018.
  51. 51.Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In Advances in Neural Information Processing Systems, pages 1163–1171, 2016.
  52. 52.Julian Salazar, Davis Liang, Zhiheng Huang, and Zachary C Lipton. Invariant representation learning for robust deep networks. In Workshop on Integration of Deep Learning Theories, NeurIPS, 2018.
  53. 53.Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pages 2234–2242, 2016.
  54. 54.Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709, 2015.
  55. 55.Tianxiao Shen, Myle Ott, Michael Auli, and Marc’Aurelio Ranzato. Mixture models for diverse machine translation: Tricks of the trade. arXiv preprint arXiv:1902.07816, 2019.
  56. 56.Patrice Y Simard, Yann A LeCun, John S Denker, and Bernard Victorri. Transformation invariance in pattern recognition—tangent distance and tangent propagation. In Neural networks: tricks of the trade, pages 239–274. Springer, 1998.
  57. 57.Robert Stanforth, Alhussein Fawzi, Pushmeet Kohli, et al. Are labels required for improving adversarial robustness? arXiv preprint arXiv:1905.13725, 2019.
  58. 58.Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in neural information processing systems, pages 1195–1204, 2017.
  59. 59.Trieu H Trinh, Minh-Thang Luong, and Quoc V Le. Selfie: Self-supervised pretraining for image embedding. arXiv preprint arXiv:1906.02940, 2019.
  60. 60.Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. arXiv preprint arXiv:1903.03825, 2019.
  61. 61.Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. Switchout: an efficient data augmentation algorithm for neural machine translation. arXiv preprint arXiv:1808.07512, 2018.
  62. 62.Jason Weston, Frédéric Ratle, Hossein Mobahi, and Ronan Collobert. Deep learning via semi-supervised embedding. In Neural Networks: Tricks of the Trade, pages 639–655. Springer, 2012.
  63. 63.Zhilin Yang, William W Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. arXiv preprint arXiv:1603.08861, 2016.
  64. 64.Zhilin Yang, Junjie Hu, Ruslan Salakhutdinov, and William W Cohen. Semi-supervised qa with generative domain-adaptive nets. arXiv preprint arXiv:1702.02206, 2017.
  65. 65.Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6210–6219, 2019.
  66. 66.Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. Qanet: Combining local convolution with global self-attention for reading comprehension. arXiv preprint arXiv:1804.09541, 2018.
  67. 67.Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. BMVC, 2016.
  68. 68.Runtian Zhai, Tianle Cai, Di He, Chen Dan, Kun He, John Hopcroft, and Liwei Wang. Adversarially robust generalization just requires more unlabeled data. arXiv preprint arXiv:1906.00555, 2019.
  69. 69.Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S4l: Self-supervised semi-supervised learning. In Proceedings of the IEEE international conference on computer vision, 2019.
  70. 70.Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.
  71. 71.Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pages 649–657, 2015.
  72. 72.Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In Proceedings of the 20th International conference on Machine learning (ICML-03), pages 912–919, 2003.

Citation

MLA
Xie, Q., et al. “Unsupervised Data Augmentation for Consistency Training”. Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 6256–68, https://proceedings.neurips.cc/paper_files/paper/2020/file/44feb0096faa8326192570788b38c1d1-Paper.pdf.
APA
Xie, Q., Dai, Z., Hovy, E., Luong, T., & Le, Q. (2020). Unsupervised Data Augmentation for Consistency Training. Advances in Neural Information Processing Systems, 33, 6256–6268. https://proceedings.neurips.cc/paper_files/paper/2020/file/44feb0096faa8326192570788b38c1d1-Paper.pdf
Chicago
Xie, Q., Z. Dai, E. Hovy, T. Luong, and Q. Le. 2020. “Unsupervised Data Augmentation for Consistency Training”. Advances in Neural Information Processing Systems 33: 6256–68. https://proceedings.neurips.cc/paper_files/paper/2020/file/44feb0096faa8326192570788b38c1d1-Paper.pdf.
Harvard
Xie, Q. et al. (2020) “Unsupervised Data Augmentation for Consistency Training”, Advances in Neural Information Processing Systems. Curran Associates, Inc., pp. 6256–6268. Available at: https://proceedings.neurips.cc/paper_files/paper/2020/file/44feb0096faa8326192570788b38c1d1-Paper.pdf.
Vancouver
1. Xie Q, Dai Z, Hovy E, Luong T, Le Q (2020) Unsupervised Data Augmentation for Consistency Training. In: Advances in Neural Information Processing Systems. Curran Associates, Inc., pp 6256–6268

BibTeX

@inproceedings{xie2020unsupervised,
  title = {Unsupervised Data Augmentation for Consistency Training},
  author = {Xie, Qizhe and Dai, Zihang and Hovy, Eduard and Luong, Thang and Le, Quoc},
  year = {2020},
  booktitle = {Advances in Neural Information Processing Systems},
  publisher = {Curran Associates, Inc.},
  volume = {33},
  pages = {6256-6268},
  url = {https://proceedings.neurips.cc/paper_files/paper/2020/file/44feb0096faa8326192570788b38c1d1-Paper.pdf}
}
Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Published with permission