Back

A survey on semi-supervised learning

Jesper E. van EngelenHolger H. Hoos

article2019Machine Learning2,553 citations

Presents a comprehensive taxonomy of semi-supervised classification algorithms that systematically organizes inductive, transductive, and deep learning methods while connecting their foundational assumptions to clustering.

Listen

Semi-supervised learning combines small amounts of labeled data with large amounts of unlabeled data to improve classification performance in settings where labels are costly or difficult to obtain. The article surveys two decades of research on this topic to organize methods, clarify their foundations, and guide practical use.

The authors set out to deliver a comprehensive, up-to-date review focused on classification, introduce a clear taxonomy, and connect the core assumptions that justify the use of unlabeled data. They synthesized the literature through structured categorization and analysis rather than new experiments.

The resulting taxonomy divides methods into inductive approaches, which produce a reusable classifier, and transductive approaches, which predict only for the given unlabeled points. Inductive methods further split into wrapper techniques that pseudo-label data, unsupervised preprocessing steps such as feature extraction or pre-training, and intrinsically semi-supervised algorithms that embed unlabeled data directly into the objective. Transductive methods rely on graphs that propagate labels along similarity edges.

Key findings show that the smoothness, low-density, and manifold assumptions all reduce to variants of the cluster assumption and that unlabeled data improves performance only when these assumptions hold and the algorithm can exploit them. Performance can degrade when assumptions are violated or when strong supervised baselines already exist. Recent neural-network methods achieve more consistent gains on image tasks when evaluated under realistic protocols that include properly tuned supervised baselines.

These results matter because they reduce the risk of deploying ineffective or harmful semi-supervised systems and help practitioners select methods suited to their data. The survey also supplies concrete evaluation guidelines that guard against overly optimistic claims.

Organizations should treat semi-supervised learning as one option among many and validate chosen approaches on diverse data sets with varying label quantities. Further work is needed on methods that reliably avoid performance degradation and on extending consistent gains beyond image classification. The main limitations are the focus on influential rather than exhaustive coverage and the inherent difficulty of guaranteeing that any given data set satisfies the required assumptions.

Leiden UniversityUniversity of British Columbiaclassification algorithmengineersneural networksSemi-supervised learningUnlabelled dataClassification
Engelen et al (2019).pdf
Cover for A survey on semi-supervised learning

Abstract

Semi-supervised learning is the branch of machine learning concerned with using labelled as well as unlabelled data to perform certain learning tasks. Conceptually situated between supervised and unsupervised learning, it permits harnessing the large amounts of unlabelled data available in many use cases in combination with typically smaller sets of labelled data. In recent years, research in this area has followed the general trends observed in machine learning, with much attention directed at neural network-based models and generative learning. The literature on the topic has also expanded in volume and scope, now encompassing a broad spectrum of theory, algorithms and applications. However, no recent surveys exist to collect and organize this knowledge, impeding the ability of researchers and engineers alike to utilize it. Filling this void, we present an up-to-date overview of semi-supervised learning methods, covering earlier work as well as more recent advances. We focus primarily on semi-supervised classification, where the large majority of semi-supervised learning research takes place. Our survey aims to provide researchers and practitioners new to the field as well as more advanced readers with a solid understanding of the main approaches and algorithms developed over the past two decades, with an emphasis on the most prominent and currently relevant work. Furthermore, we propose a new taxonomy of semi-supervised classification algorithms, which sheds light on the different conceptual and methodological approaches for incorporating unlabelled data into the training process. Lastly, we show how the fundamental assumptions underlying most semi-supervised learning algorithms are closely connected to each other, and how they relate to the well-known semi-supervised clustering assumption.

Topics

classification algorithmengineersneural networkssemi-supervised clusteringsemi-supervised learningsupervised classificationsupervised machine learningtaxonomyunsupervised learningclassifiers

Keywords

Semi-supervised learningUnlabelled dataClassificationGraph-based methodManifold assumptionSmoothness assumptionLow-density assumptionWrapper methodTransductive methodGenerative adversarial network

Table of Contents

  • A survey on semi-supervised learning
  • 1 Introduction
  • 2 Background
  • 2.1 Assumptions of semi-supervised learning
  • 2.1.1 Smoothness assumption
  • 2.1.2 Low-density assumption
  • 2.1.3 Manifold assumption
  • 2.2 Connection to clustering
  • 2.3 When does semi-supervised learning work?
  • 2.4 Empirical evaluation of semi-supervised learning methods
  • 3 Taxonomy of semi-supervised learning methods
  • 3.1 Inductive methods
  • 3.1.1 Wrapper methods
  • 3.1.2 Unsupervised preprocessing
  • 3.1.3 Intrinsically semi-supervised methods
  • 3.2 Transductive methods
  • 4 Wrapper methods
  • 4.1 Self-training
  • 4.2 Co-training
  • 4.2.1 Multi-view co-training
  • 4.2.2 Single-view co-training
  • 4.2.3 Co-regularization
  • 4.3 Boosting
  • 4.3.1 SSMBoost
  • 4.3.2 ASSEMBLE
  • 4.3.3 SemiBoost
  • 4.3.4 Other semi-supervised boosting methods
  • 5 Unsupervised preprocessing
  • 5.1 Feature extraction
  • 5.2 Cluster-then-label
  • 5.3 Pre-training
  • 6 Intrinsically semi-supervised methods
  • 6.1 Maximum-margin methods
  • 6.1.1 Support vector machines
  • 6.1.2 Gaussian processes
  • 6.1.3 Density regularization
  • 6.1.4 Pseudo-labelling as a form of margin maximization
  • 6.2 Perturbation-based methods
  • 6.2.1 Neural networks
  • 6.2.2 Semi-supervised neural networks
  • 6.2.3 Ladder networks
  • 6.2.4 Pseudo-ensembles
  • 6.2.5 Π\Pi-model
  • 6.2.6 Temporal ensembling
  • 6.2.7 Mean teacher
  • 6.2.8 Virtual adversarial training
  • 6.2.9 Semi-supervised mixup
  • 6.3 Manifolds
  • 6.3.1 Manifold regularization
  • 6.3.2 Manifold approximation
  • 6.4 Generative models
  • 6.4.1 Mixture models
  • 6.4.2 Generative adversarial networks
  • 6.4.3 Variational autoencoders
  • 7 Transductive methods
  • 7.1 A general framework for graph-based methods
  • 7.2 Inference in graphs
  • 7.2.1 Hard label assignments: graph min-cut
  • 7.2.2 Probabilistic label assignments: Markov random fields
  • 7.2.3 Efficient probabilistic label assignments: Gaussian random fields
  • 7.2.4 Handling label noise and irregular graphs: local and global consistency
  • 7.2.5 Further research on graph-based inference
  • 7.3 Graph construction
  • 7.3.1 Adjacency matrix construction
  • 7.3.2 Graph weighting
  • 7.3.3 Simultaneous graph construction and weighting
  • 7.4 Scalable transductive learning
  • 7.5 From transduction to induction
  • 7.6 Classification in network data
  • 8 Related areas
  • 8.1 Semi-supervised regression
  • 8.2 Semi-supervised clustering
  • 9 Conclusions and future perspectives
  • Compliance with ethical standards
  • References

Knowls

  1. Knowl 1 — Taxonomy of Semi-Supervised Classification Methods

    definition

    Semi-supervised classification algorithms can be hierarchically categorized into two top-level branches based on their learning goal:

    1. Transductive Methods: Methods that optimize directly over the predictions y^U\hat{\mathbf{y}}_U for the unlabelled instances XU=(xi)i=l+1l+uX_U = (x_i)_{i=l+1}^{l+u} provided during training, without constructing an explicit classifier for the entire input space X\mathcal{X}. These methods are universally graph-based and comprise three consecutive stages: graph construction (e.g., ϵ\epsilon-neighbourhood, kk-nearest neighbours, bb-matching), graph weighting (e.g., Gaussian weighting, linear neighbourhood propagation, sparse coding), and inference (e.g., min-cut, Gaussian random fields, local and global consistency).

    2. Inductive Methods: Methods that yield an explicit prediction function f:X→Yf: \mathcal{X} \rightarrow \mathcal{Y} capable of evaluating unseen test samples independently after training. Inductive methods are divided into three subclasses:

      • Wrapper Methods: Procedures where unlabelled data is pseudo-labelled iteratively by one or more supervised base learners, remaining agnostic to the specific base classification algorithm (e.g., self-training, co-training, semi-supervised boosting).
      • Unsupervised Preprocessing: Two-stage procedures that utilize unlabelled data prior to supervised training for automated feature extraction (e.g., autoencoders), pre-clustering (cluster-then-label), or parameter pre-training (e.g., stacked autoencoders, deep belief networks).
      • Intrinsically Semi-Supervised Methods: Algorithms that directly incorporate unlabelled data into their objective function or optimization criterion. These are grouped by underlying data assumptions into maximum-margin methods (S3VMs, semi-supervised Gaussian processes, density regularization), perturbation-based methods (ladder networks, pseudo-ensembles, Π\Pi-model, temporal ensembling, mean teacher, virtual adversarial training), manifold-based methods (manifold regularization, Laplacian SVMs, manifold approximation), and generative models (mixture models, semi-supervised GANs, semi-supervised VAEs).

  2. Knowl 2 — Unification of Semi-Supervised Learning Assumptions via the Cluster Assumption

    theoretical result

    A necessary condition for semi-supervised learning to improve over supervised learning is that the marginal distribution p(x)p(x) contains information about the posterior class distribution p(y∣x)p(y|x). The fundamental semi-supervised learning assumptions are instances of the general cluster assumption—which posits that data points belonging to the same cluster share the same class label—differentiated by their chosen definition of sample similarity:

    • Smoothness Assumption: For two points x,x′∈Xx, x' \in \mathcal{X} that are close in input space, their labels y,y′y, y' should be identical. Similarity corresponds to metric proximity in the ambient input space, allowing transitive label propagation across sequences of unlabelled instances.
    • Low-Density Assumption: The decision boundary should not pass through high-density regions of p(x)p(x), but rather through low-density areas. This is the distributional counterpart of the smoothness assumption: dense regions must contain points of the same class to avoid intersecting the boundary.
    • Manifold Assumption: The data distribution is concentrated on lower-dimensional manifolds M⊂X\mathcal{M} \subset \mathcal{X}, and data points on the same manifold share the same label. Similarity corresponds to geodesic proximity along the manifold rather than Euclidean distance in the ambient space.

    Because defining a similarity measure uniquely determines the cluster structure between p(x)p(x) and p(y∣x)p(y|x), the failure of the data to be meaningfully clusterable under a suitable similarity metric renders semi-supervised improvement inherently impossible.

  3. Knowl 3 — General Framework for Transductive Graph-Based Learning

    model/method

    Transductive graph-based semi-supervised learning represents ll labelled data points XL=((x1,y1),…,(xl,yl))X_L = ((x_1, y_1), \dots, (x_l, y_l)) and uu unlabelled data points XU=(xl+1,…,xl+u)X_U = (x_{l+1}, \dots, x_{l+u}) as nodes in a graph V={v1,…,vn}V = \{v_1, \dots, v_n\} of size n=l+un = l + u. Pairwise similarity is encoded by an n×nn \times n symmetric affinity matrix WW with non-negative entries Wij≥0W_{ij} \ge 0.

    The general optimization problem for inferring predicted labels y^=(y^1,…,y^n)T\hat{\mathbf{y}} = (\hat{y}_1, \dots, \hat{y}_n)^T is formulated as:

    min⁡y^λ∑i=1lℓ(y^i,yi)+∑i=1n∑j=1nWij⋅ℓU(y^i,y^j)\min_{\hat{\mathbf{y}}} \lambda \sum_{i=1}^l \ell(\hat{y}_i, y_i) + \sum_{i=1}^n \sum_{j=1}^n W_{ij} \cdot \ell_U(\hat{y}_i, \hat{y}_j)

    where ℓ(⋅,⋅)\ell(\cdot, \cdot) is a supervised loss penalizing deviations from ground-truth labels, ℓU(⋅,⋅)\ell_U(\cdot, \cdot) is an unsupervised pairwise loss penalizing prediction differences across connected nodes, and λ>0\lambda > 0 is a hyperparameter balancing the supervised and graph consistency objectives.

  4. Knowl 4 — Manifold Regularization in Reproducing Kernel Hilbert Spaces

    model/method

    Manifold regularization extends supervised learning in a Reproducing Kernel Hilbert Space (RKHS) HK\mathcal{H}_K associated with Mercer kernel K:X×X→RK: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R} by penalizing function evaluations along the data manifold.

    Given labelled instances XL=((x1,y1),…,(xl,yl))X_L = ((x_1, y_1), \dots, (x_l, y_l)) and unlabelled instances XU=(xl+1,…,xl+u)X_U = (x_{l+1}, \dots, x_{l+u}), let W∈Rn×nW \in \mathbb{R}^{n \times n} denote the symmetric affinity matrix for n=l+un = l + u points, DD denote the diagonal degree matrix (Dii=∑j=1nWijD_{ii} = \sum_{j=1}^n W_{ij}), and L=D−WL = D - W denote the unnormalized graph Laplacian. The empirical manifold regularizer is defined as:

    ∥f∥I2=12∑i=1n∑j=1nWij(f(xi)−f(xj))2=fTLf\|f\|_I^2 = \frac{1}{2} \sum_{i=1}^n \sum_{j=1}^n W_{ij} \left(f(x_i) - f(x_j)\right)^2 = \mathbf{f}^T L \mathbf{f}

    where f=(f(x1),…,f(xn))T∈Rn\mathbf{f} = (f(x_1), \dots, f(x_n))^T \in \mathbb{R}^n.

    The optimal inductive predictor f∈HKf \in \mathcal{H}_K is obtained by solving:

    min⁡f∈HK1l∑i=1lℓ(f(xi),yi)+γ∥f∥K2+γU∥f∥I2\min_{f \in \mathcal{H}_K} \frac{1}{l} \sum_{i=1}^l \ell(f(x_i), y_i) + \gamma \|f\|_K^2 + \gamma_U \|f\|_I^2

    where ℓ\ell is a supervised loss (e.g., hinge loss ℓ(y^,y)=max⁡{0,1−yy^}\ell(\hat{y}, y) = \max\{0, 1 - y\hat{y}\} in Laplacian SVMs), ∥f∥K2\|f\|_K^2 is the ambient RKHS norm enforcing smoothness, and γ,γU>0\gamma, \gamma_U > 0 are regularization hyperparameters.

  5. Knowl 5 — Semi-Supervised Support Vector Machines Formulation

    model/method

    Semi-Supervised Support Vector Machines (S3S^3VMs) incorporate the low-density assumption by maximizing the classification margin on labelled samples while penalizing unlabelled samples that fall within the margin boundary of the hyperplane f(x)=wTx+bf(x) = \mathbf{w}^T x + b.

    For binary classification with labels yi∈{−1,+1}y_i \in \{-1, +1\}, labelled set XL={(xi,yi)}i=1lX_L = \{(x_i, y_i)\}_{i=1}^l, and unlabelled set XU={xi}i=l+1nX_U = \{x_i\}_{i=l+1}^n, the optimization problem is formulated as:

    min⁡w,b,ξ12∥w∥2+C∑i=1lξi+C′∑i=l+1nξi\min_{\mathbf{w}, b, \boldsymbol{\xi}} \frac{1}{2}\|\mathbf{w}\|^2 + C \sum_{i=1}^l \xi_i + C' \sum_{i=l+1}^n \xi_i

    subject to yi(wTxi+b)≥1−ξi,i=1,…,l\text{subject to } y_i (\mathbf{w}^T x_i + b) \ge 1 - \xi_i, \quad i = 1, \dots, l

    ∣wTxi+b∣≥1−ξi,i=l+1,…,n|\mathbf{w}^T x_i + b| \ge 1 - \xi_i, \quad i = l+1, \dots, n

    ξi≥0,i=1,…,n\xi_i \ge 0, \quad i = 1, \dots, n

    where CC and C′C' are penalty multipliers for margin violations on labelled and unlabelled data, respectively.

    Equivalently, it is expressed as the unconstrained loss minimization:

    min⁡w,b12∥w∥2+C∑i=1lmax⁡(0,1−yif(xi))+C′∑i=l+1nmax⁡(0,1−∣f(xi)∣)\min_{\mathbf{w}, b} \frac{1}{2}\|\mathbf{w}\|^2 + C \sum_{i=1}^l \max\left(0, 1 - y_i f(x_i)\right) + C' \sum_{i=l+1}^n \max\left(0, 1 - |f(x_i)|\right)

    Due to the non-convex hat loss on unlabelled points, exact optimization is NP-hard.

  6. Knowl 6 — Gaussian Random Fields and Harmonic Label Propagation

    algorithm

    In Gaussian random fields for graph inference, predicted label variables y^=(y^1,…,y^n)T∈Rn\hat{\mathbf{y}} = (\hat{y}_1, \dots, \hat{y}_n)^T \in \mathbb{R}^n are relaxed to continuous values with quadratic loss ∑i,j=1nWij(y^i−y^j)2=2y^TLy^\sum_{i,j=1}^n W_{ij} (\hat{y}_i - \hat{y}_j)^2 = 2 \hat{\mathbf{y}}^T L \hat{\mathbf{y}}, with labelled nodes clamped to their true values (y^i=yi\hat{y}_i = y_i for i≤li \le l).

    The unlabelled predictions satisfy the harmonic property Ly^=0L \hat{\mathbf{y}} = 0 on unlabelled nodes, meaning each unlabelled prediction is the weighted average of its neighbours: y^i=1Dii∑j∈N(vi)Wijy^j\hat{y}_i = \frac{1}{D_{ii}} \sum_{j \in \mathcal{N}(v_i)} W_{ij} \hat{y}_j.

    Input: Affinity matrix W∈Rn×nW \in \mathbb{R}^{n \times n}, true label vector yL=(y1,…,yl)T\mathbf{y}_L = (y_1, \dots, y_l)^T, tolerance ϵ>0\epsilon > 0
    Output: Soft label predictions y^∈[0,1]n\hat{\mathbf{y}} \in [0, 1]^n
    Compute diagonal degree matrix DD where Dii=∑j=1nWijD_{ii} = \sum_{j=1}^n W_{ij}
    Form row-normalized transition matrix A=D−1WA = D^{-1} W
    Initialize y^0=(y1,…,yl,y^l+1(0),…,y^n(0))T\hat{\mathbf{y}}_0 = (y_1, \dots, y_l, \hat{y}_{l+1}^{(0)}, \dots, \hat{y}_n^{(0)})^T with random initializations for unlabelled nodes
    repeat
        Propagate labels: y^t+1←Ay^t\hat{\mathbf{y}}_{t+1} \leftarrow A \hat{\mathbf{y}}_t
        Reset labelled nodes: y^i,t+1←yi\hat{y}_{i, t+1} \leftarrow y_i for all i∈{1,…,l}i \in \{1, \dots, l\}
    until ∥y^t+1−y^t∥<ϵ\|\hat{\mathbf{y}}_{t+1} - \hat{\mathbf{y}}_t\| < \epsilon
    return y^\hat{\mathbf{y}}

    This algorithm is guaranteed to converge to the closed-form harmonic solution y^U=−(LUU)−1LULyL\hat{\mathbf{y}}_U = -(L_{UU})^{-1} L_{UL} \mathbf{y}_L, where LUUL_{UU} and LULL_{UL} are the corresponding block partitions of the graph Laplacian L=D−WL = D - W.

  7. Knowl 7 — Local and Global Consistency Inference and Label Spreading

    algorithm

    The Local and Global Consistency (LGC) method avoids hard clamping to accommodate label noise and mitigates vertex degree imbalances by regularizing predictions via the normalized graph Laplacian L~=D−1/2LD−1/2=I−D−1/2WD−1/2\tilde{L} = D^{-1/2} L D^{-1/2} = I - D^{-1/2} W D^{-1/2}.

    For binary labels Y={−1,+1}\mathcal{Y} = \{-1, +1\} with labelled vector y∈Rn\mathbf{y} \in \mathbb{R}^n (yi∈{−1,+1}y_i \in \{-1, +1\} for i≤li \le l and yi=0y_i = 0 for i>li > l), the optimization objective is:

    min⁡y^∑i=1l(y^i−yi)2+∑i=l+1ny^i2+λU∑i=1n∑j=1nWij(y^iDii−y^jDjj)2\min_{\hat{\mathbf{y}}} \sum_{i=1}^l (\hat{y}_i - y_i)^2 + \sum_{i=l+1}^n \hat{y}_i^2 + \lambda_U \sum_{i=1}^n \sum_{j=1}^n W_{ij} \left( \frac{\hat{y}_i}{\sqrt{D_{ii}}} - \frac{\hat{y}_j}{\sqrt{D_{jj}}} \right)^2

    Input: Affinity matrix W∈Rn×nW \in \mathbb{R}^{n \times n}, base label vector y∈Rn\mathbf{y} \in \mathbb{R}^n, weighting factor α∈(0,1)\alpha \in (0, 1), tolerance ϵ>0\epsilon > 0
    Output: Prediction vector y^∈Rn\hat{\mathbf{y}} \in \mathbb{R}^n
    Compute diagonal degree matrix DD with Dii=∑j=1nWijD_{ii} = \sum_{j=1}^n W_{ij}
    Compute symmetrically normalized matrix S=D−1/2WD−1/2S = D^{-1/2} W D^{-1/2}
    Initialize y^0=y\hat{\mathbf{y}}_0 = \mathbf{y}
    repeat
        Update predictions: y^t+1←αSy^t+(1−α)y\hat{\mathbf{y}}_{t+1} \leftarrow \alpha S \hat{\mathbf{y}}_t + (1 - \alpha) \mathbf{y}
    until ∥y^t+1−y^t∥<ϵ\|\hat{\mathbf{y}}_{t+1} - \hat{\mathbf{y}}_t\| < \epsilon
    return y^\hat{\mathbf{y}}

    The label spreading iteration converges to the exact closed-form solution y^∗=(1−α)(I−αS)−1y\hat{\mathbf{y}}^* = (1 - \alpha)(I - \alpha S)^{-1}\mathbf{y}.

  8. Knowl 8 — Perturbation-Based Semi-Supervised Learning in Neural Networks

    model/method

    Perturbation-based methods implement the smoothness assumption in deep neural networks by regularizing the network to output consistent predictions when inputs or network representations are perturbed.

    The overall objective combines supervised loss on labelled samples with an unsupervised consistency loss on all samples:

    L=1l∑i=1lℓ(f(xi),yi)+λ(t)∑i=1nD(f1(xi),f2(xi))\mathcal{L} = \frac{1}{l} \sum_{i=1}^l \ell(f(x_i), y_i) + \lambda(t) \sum_{i=1}^n \mathcal{D}\left(f_1(x_i), f_2(x_i)\right)

    where D(⋅,⋅)\mathcal{D}(\cdot, \cdot) is a divergence measure (e.g., mean squared error or Kullback-Leibler divergence), and λ(t)\lambda(t) is a time-dependent weighting ramp-up schedule.

    Key paradigms include:

    • Π\Pi-Model: Compares predictions from two stochastic passes of the network on the same input under independent dropout masks and data augmentation: D(fθ(xi;ξ),fθ(xi;ξ′))\mathcal{D}(f_\theta(x_i; \xi), f_\theta(x_i; \xi')).
    • Temporal Ensembling: Replaces one branch with an exponential moving average (EMA) of network predictions computed over preceding training epochs: zi←αzi+(1−α)fθ(xi;ξ)z_i \leftarrow \alpha z_i + (1 - \alpha) f_\theta(x_i; \xi), penalizing D(fθ(xi;ξ),zi)\mathcal{D}(f_\theta(x_i; \xi), z_i).
    • Mean Teacher: Maintains a student model with weights θt\theta_t updated via gradient descent and a teacher model whose weights Θt\Theta_t are updated via an EMA of student weights: Θt+1=αΘt+(1−α)θt+1\Theta_{t+1} = \alpha \Theta_t + (1 - \alpha) \theta_{t+1}. Consistency is penalized between student output fθ(x′)f_\theta(x') and teacher output fΘ(x′′)f_\Theta(x'') for noisy variants x′,x′′x', x''.
    • Virtual Adversarial Training (VAT): Computes an adversarial perturbation vector rv-adv=arg⁡max⁡r,∥r∥2≤ϵD(fθ(x),fθ(x+r))\mathbf{r}_{\text{v-adv}} = \arg\max_{\mathbf{r}, \|\mathbf{r}\|_2 \le \epsilon} \mathcal{D}(f_\theta(x), f_\theta(x + \mathbf{r})) along the direction of greatest model sensitivity, penalizing D(fθ^(x),fθ(x+rv-adv))\mathcal{D}(f_{\hat{\theta}}(x), f_\theta(x + \mathbf{r}_{\text{v-adv}})) without labels.

  9. Knowl 9 — Semi-Supervised Mixup Regularization

    model/method

    Mixup regularization extends the smoothness assumption across pairs of instances by enforcing that linear interpolations in the input space yield corresponding linear interpolations in the label space.

    For two instances (xi,yi)(x_i, y_i) and (xj,yj)(x_j, y_j), a synthetic instance (x^,y^)(\hat{x}, \hat{y}) is formed by:

    x^=λxi+(1−λ)xj\hat{x} = \lambda x_i + (1 - \lambda) x_j

    y^=λyi+(1−λ)yj\hat{y} = \lambda y_i + (1 - \lambda) y_j

    where λ∼Beta(α,α)\lambda \sim \text{Beta}(\alpha, \alpha) for hyperparameter α>0\alpha > 0.

    In semi-supervised extensions (such as Interpolation Consistency Training and MixMatch):

    • For unlabelled instances xi,xj∈XUx_i, x_j \in X_U, the ground-truth targets yi,yjy_i, y_j are replaced by model predictions y^i=fθ(xi)\hat{y}_i = f_\theta(x_i) and y^j=fθ(xj)\hat{y}_j = f_\theta(x_j) (or predictions from a Mean Teacher network).
    • The network parameters are optimized by minimizing a consistency loss D(fθ(x^),y^)=D(fθ(λxi+(1−λ)xj),λy^i+(1−λ)y^j)\mathcal{D}\left(f_\theta(\hat{x}), \hat{y}\right) = \mathcal{D}\left(f_\theta(\lambda x_i + (1 - \lambda) x_j), \lambda \hat{y}_i + (1 - \lambda) \hat{y}_j\right) on the interpolated unlabelled points.

  10. Knowl 10 — Safe Semi-Supervised Support Vector Machines (S4VM)

    theoretical result

    When semi-supervised assumptions are violated, incorporating unlabelled data can degrade classifier accuracy compared to a supervised classifier. Safe Semi-Supervised Support Vector Machines (S4S^4VM) optimize for the separator that maximizes worst-case performance gain relative to a supervised baseline.

    Let M={y^(1),…,y^(m)}\mathcal{M} = \{\hat{\mathbf{y}}^{(1)}, \dots, \hat{\mathbf{y}}^{(m)}\} be a pool of diverse candidate low-density label assignments for unlabelled points XUX_U. For candidate predictions y^\hat{\mathbf{y}}, true unlabelled labels y\mathbf{y}, and supervised SVM predictions ysvm\mathbf{y}^{\text{svm}}, the performance gain score is defined as:

    J(y^,y,ysvm)=gain(y^,y,ysvm)−λ⋅lose(y^,y,ysvm)J(\hat{\mathbf{y}}, \mathbf{y}, \mathbf{y}^{\text{svm}}) = \text{gain}(\hat{\mathbf{y}}, \mathbf{y}, \mathbf{y}^{\text{svm}}) - \lambda \cdot \text{lose}(\hat{\mathbf{y}}, \mathbf{y}, \mathbf{y}^{\text{svm}})

    where gain\text{gain} and lose\text{lose} denote the number of samples where y^\hat{\mathbf{y}} improves upon or degrades from ysvm\mathbf{y}^{\text{svm}}, respectively, and λ≥0\lambda \ge 0 is a risk parameter.

    The optimal label assignment yˉ\bar{\mathbf{y}} under worst-case ground truth is:

    yˉ∈arg⁡max⁡y∈{±1}u[min⁡y^∈MJ(y^,y,ysvm)]\bar{\mathbf{y}} \in \arg\max_{\mathbf{y} \in \{\pm 1\}^u} \left[ \min_{\hat{\mathbf{y}} \in \mathcal{M}} J(\hat{\mathbf{y}}, \mathbf{y}, \mathbf{y}^{\text{svm}}) \right]

    Under the assumption that the true unlabelled assignment is contained in M\mathcal{M}, setting λ≥1\lambda \ge 1 guarantees that the generalization performance of S4S^4VM is never strictly lower than that of the supervised baseline SVM.

Coverage note — Domain-specific application surveys (such as specialized natural language processing and computer vision sub-tasks) and peripheral fields (active learning, positive-unlabelled learning) mentioned in brief survey sections were omitted as they are outside the core contributed taxonomic and methodological syntheses.

References

  1. 1.Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., & Isard, M., et al. (2016). Tensorflow: A system for large-scale machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16) (pp. 265–283).
  2. 2.Abney, S. (2002). Bootstrapping. In Proceedings of the 40th annual meeting on association for computational linguistics, association for computational linguistics (pp. 360–367).
  3. 3.Anderberg, M. R. (1973). Cluster analysis for applications. Cambridge: Academic Press.
  4. 4.Azran, A. (2007). The rendezvous algorithm: Multiclass semi-supervised learning with Markov random walks. In Proceedings of the 24th international conference on machine learning (pp. 49–56).
  5. 5.Bachman, P., Alsharif, O., & Precup, D. (2014). Learning with pseudo-ensembles. In Advances in neural information processing systems (pp. 3365–3373).
  6. 6.Bair, E. (2013). Semi-supervised clustering methods. Wiley Interdisciplinary Reviews: Computational Statistics, 5(5), 349–361.
  7. 7.Balcan, M. F., Blum, A., & Yang, K. (2005). Co-training and expansion: Towards bridging theory and practice. In Advances in neural information processing systems (pp. 89–96).
  8. 8.Baluja, S., Seth, R., Sivakumar, D., Jing, Y., Yagnik, J., Kumar, S., Ravichandran, D., & Aly, M. (2008). Video suggestion and discovery for youtube: Taking random walks through the view graph. In Proceedings of the 17th international conference on world wide web (pp. 895–904). ACM.
  9. 9.Barabási, A. L. (2016). Network science. Cambridge: Cambridge University Press.
  10. 10.Basu, S., Banerjee, A., & Mooney, R. (2002). Semi-supervised clustering by seeding. In Proceedings of the 19th international conference on machine learning (pp. 27–34).
  11. 11.Belkin, M., Matveeva, I., & Niyogi, P. (2004). Regularization and semi-supervised learning on large graphs. In Proceedings of the international conference on computational learning theory (pp. 624–638). Springer.
  12. 12.Belkin, M., Niyogi, P., & Sindhwani, V. (2005). On manifold regularization. In Proceedings of the 10th international conference on artificial intelligence and statistics (pp. 17–24).
  13. 13.Belkin, M., Niyogi, P., & Sindhwani, V. (2006). Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of Machine Learning Research, 7, 2399–2434.
  14. 14.Ben-David, S., Lu, T., Pál, D., & Sotáková, M. (2009). Learning low density separators. In Proceedings of the 12th international conference on artificial intelligence and statistics (pp. 25–32).
  15. 15.Bengio, Y., Delalleau, O., & Le Roux, N. (2006). Chapter 11. Label propagation and quadratic criterion. In O. Chapelle, B. Schölkopf, & A. Zien (Eds.), Semi-supervised learning (pp. 193–216). Cambridge: The MIT Press.
  16. 16.Bennett, K. P., & Demiriz, A. (1999). Semi-supervised support vector machines. In Advances in neural information processing systems (pp. 368–374).
  17. 17.Bennett, K. P., Demiriz, A., & Maclin, R. (2002). Exploiting unlabeled data in ensemble methods. In Proceedings of the 8th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 289–296). ACM.
  18. 18.Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., & Raffel, C. (2019). Mixmatch: A holistic approach to semi-supervised learning. arXiv:1905.02249.
  19. 19.Bishop, C. M. (2006). Pattern recognition and machine learning. Berlin: Springer.
  20. 20.Blum, A., & Chawla, S. (2001). Learning from labeled and unlabeled data using graph mincuts. In Proceedings of the 18th international conference on machine learning (pp. 19–26).
  21. 21.Blum, A., Lafferty, J., Rwebangira, M. R., & Reddy, R. (2004). Semi-supervised learning using randomized mincuts. In Proceedings of the 21st international conference on machine learning (p. 13).
  22. 22.Blum, A., & Mitchell, T. (1998). Combining labeled and unlabeled data with co-training. In Proceedings of the 11th annual conference on computational learning theory (pp. 92–100). ACM.
  23. 23.Bruna, J., Zaremba, W., Szlam, A., & LeCun, Y. (2014). Spectral networks and locally connected networks on graphs. In International conference on learning, representations.
  24. 24.Chapelle, O., Chi, M., & Zien, A. (2006a). A continuation method for semi-supervised SVMs. In Proceedings of the 23rd international conference on machine learning (pp. 185–192).
  25. 25.Chapelle, O., Schölkopf, B., & Zien, A. (2006b). Semi-supervised learning (1st ed.). Cambridge: The MIT Press.
  26. 26.Chapelle, O., Sindhwani, V., & Keerthi, S. S. (2008). Optimization techniques for semi-supervised support vector machines. Journal of Machine Learning Research, 9, 203–233.
  27. 27.Chapelle, O., & Zien, A. (2005). Semi-supervised classification by low density separation. In Proceedings of the 10th international workshop on artificial intelligence and statistics (pp. 57–64).
  28. 28.Chen, K., & Wang, S. (2011). Semi-supervised learning via regularized boosting working on multiple semi-supervised assumptions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1), 129–143.
  29. 29.Chen, M., Chen, Y., & Weinberger, K. Q. (2011). Automatic feature decomposition for single view co-training. In Proceedings of the 28th international conference on machine learning (pp. 953–960).
  30. 30.Chen, T., & Guestrin, C. (2016). Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 785–794). ACM.
  31. 31.Christoudias, C. M., Urtasun, R., Kapoorz, A., & Darrell, T. (2009). Co-training with noisy perceptual observations. In Proceedings of the 2009 IEEE conference on computer vision and pattern recognition (pp. 2844–2851). IEEE.
  32. 32.Collobert, R., Sinz, F., Weston, J., & Bottou, L. (2006). Large scale transductive SVMs. Journal of Machine Learning Research, 7, 1687–1712.
  33. 33.Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. (2011). Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12, 2493–2537.
  34. 34.Corduneanu, A., & Jaakkola, T. (2003). On information regularization. In Proceedings of the 19th conference on uncertainty in artificial intelligence (pp. 151–158). Morgan Kaufmann Publishers Inc.
  35. 35.Cortes, C., & Mohri, M. (2007). On transductive regression. In Advances in neural information processing systems (pp. 305–312).
  36. 36.Cozman, F. G., Cohen, I., & Cirelo, M. C. (2003) Semi-supervised learning of mixture models. In Proceedings of the 20th international conference on machine learning (pp. 99–106).
  37. 37.Culp, M., & Michailidis, G. (2008). An iterative algorithm for extending learners to a semi-supervised setting. Journal of Computational and Graphical Statistics, 17(3), 545–571.
  38. 38.Dai, Z., Yang, Z., Yang, F., Cohen, W. W., & Salakhutdinov, R.R. (2017). Good semi-supervised learning that requires a bad gan. In Advances in neural information processing systems (pp. 6510–6520).
  39. 39.d’Alché Buc, F., Grandvalet, Y., & Ambroise, C. (2002). Semi-supervised marginboost. Advances in Neural Information Processing Systems, 1, 553–560.
  40. 40.Dara, R., Kremer, S. C., & Stacey, D. A. (2002). Clustering unlabeled data with SOMs improves classification of labeled real-world data. In Proceedings of the international joint conference on neural networks (Vol. 3, pp. 2237–2242). IEEE.
  41. 41.Dasgupta, S., Littman, M. L., & McAllester, D. A. (2002). PAC generalization bounds for co-training. In Advances in neural information processing systems (pp. 375–382).
  42. 42.de Bie, T., & Cristianini, N. (2004). Convex methods for transduction. In Advances in neural information processing systems (pp. 73–80).
  43. 43.de Bie, T., & Cristianini, N. (2006). Semi-supervised learning using semi-definite programming. In O. Chapelle, B. Schölkopf, & A. Zien (Eds.), Semi-supervised learning (pp. 119–135). Cambridge: The MIT Press.
  44. 44.de Sousa, C. A. R., Rezende, S. O., & Batista, G. E. (2013) Influence of graph construction on semi-supervised learning. In Proceedings of the joint European conference on machine learning and knowledge discovery in databases (pp. 160–175). Springer.
  45. 45.Demiriz, A., Bennett, K. P., & Embrechts, M. J. (1999). Semi-supervised clustering using genetic algorithms. In Artificial Neural Networks in Engineering (pp. 809–814).
  46. 46.Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal statistical society, Series B, 39, 1–38.
  47. 47.Deng, C., & Zu Guo, M. (2011). A new co-training-style random forest for computer aided diagnosis. Journal of Intelligent Information Systems, 36(3), 253–281.
  48. 48.Denis, F., Gilleron, R., & Letouzey, F. (2005). Learning from positive and unlabeled examples. Theoretical Computer Science, 348(1), 70–83.
  49. 49.Doersch, C. (2016). Tutorial on variational autoencoders. arXiv:1606.05908.
  50. 50.Dópido, I., Li, J., Marpu, P. R., Plaza, A., Dias, J. M. B., & Benediktsson, J. A. (2013). Semisupervised self-learning for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 51(7), 4032–4044.
  51. 51.Du, J., Ling, C. X., & Zhou, Z. H. (2011). When does cotraining work in real data? IEEE Transactions on Knowledge and Data Engineering, 23(5), 788–799.
  52. 52.Dua, D., & Graff, C. (2019). UCI machine learning repository. Retrieved September 12, 2019 from http://archive.ics.uci.edu/ml.
  53. 53.Duvenaud, D. K., Maclaurin, D., Iparraguirre, J., Bombarell, R., Hirzel, T., Aspuru-Guzik, A., & Adams, R. P. (2015). Convolutional networks on graphs for learning molecular fingerprints. In Advances in neural information processing systems (pp. 2224–2232).
  54. 54.Elkan, C., & Noto, K. (2008). Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 213–220). ACM.
  55. 55.Elsken, T., Metzen, J. H., & Hutter, F. (2019). Neural architecture search: A survey. Journal of Machine Learning Research, 20(55), 1–21.
  56. 56.Erhan, D., Bengio, Y., Courville, A., Manzagol, P. A., Vincent, P., & Bengio, S. (2010). Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 11, 625–660.
  57. 57.Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., & Hutter, F. (2015). Efficient and robust automated machine learning. In Advances in neural information processing systems (pp. 2962–2970).
  58. 58.Freund, Y., & Schapire, R. E. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1), 119–139.
  59. 59.Geng, B., Tao, D., Xu, C., Yang, L., & Hua, X. S. (2012). Ensemble manifold regularization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(6), 1227–1233.
  60. 60.Goldberg, A. B., Zhu, X., Singh, A., Xu, Z., & Nowak, R. D. (2009). Multi-manifold semi-supervised learning. In Proceedings of the 12th international conference on artificial intelligence and statistics (pp. 169–176).
  61. 61.Goldman, S., & Zhou, Y. (2000) Enhancing supervised learning with unlabeled data. In Proceedings of the 17th international conference on machine learning (pp. 327–334).
  62. 62.Goodfellow, I. (2017). NIPS 2016 tutorial: Generative adversarial networks. arXiv:1701.00160.
  63. 63.Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. Cambridge: The MIT Press.
  64. 64.Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014a). Generative adversarial nets. In Advances in neural information processing systems (pp. 2672–2680).
  65. 65.Goodfellow, I., Shlens, J., & Szegedy, C. (2014b). Explaining and harnessing adversarial examples. arXiv:1412.6572.
  66. 66.Grabner, H., Leistner, C., Bischof, H. (2008). Semi-supervised on-line boosting for robust tracking. Proceedings of the 10th European conference on computer vision (pp. 234–247).
  67. 67.Grandvalet, Y., & Bengio, Y. (2005). Semi-supervised learning by entropy minimization. In Advances in neural information processing systems (pp. 529–536).
  68. 68.Grandvalet, Y., d’Alché Buc, F., & Ambroise, C. (2001). Boosting mixture models for semi-supervised learning. International conference on artificial neural networks (pp. 41–48).
  69. 69.Grira, N., Crucianu, M., & Boujemaa, N. (2004). Unsupervised and semisupervised clustering: A brief survey. In 7th ACM SIGMM international workshop on multimedia information retrieval.
  70. 70.Grover, A., & Leskovec, J. (2016). node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 855–864). ACM.
  71. 71.Guyon, I., & Elisseeff, A. (2006). An introduction to feature extraction. In I. Guyon, M. Nikravesh, S. Gunn, & L. A. Zadeh (Eds.), Feature extraction (pp. 1–25). Berlin: Springer.
  72. 72.Haffari, G. R., & Sarkar, A. (2007). Analysis of semi-supervised learning with the Yarowsky algorithm. In Proceedings of the 23rd conference on uncertainty in artificial intelligence (pp. 159–166).
  73. 73.Hammersley, J. M., & Clifford, P. (1971). Markov fields on finite graphs and lattices. Retrieved October 27, 2019 from http://www.statslab.cam.ac.uk/~grg/books/hammfest/hamm-cliff.pdf.
  74. 74.He, R., Zheng, W. S., Hu, B. G., & Kong, X. W. (2011). Nonnegative sparse coding for discriminative semi-supervised learning. In Proceedings of the 2011 IEEE conference on computer vision and pattern recognition (pp. 2849–2856). IEEE.
  75. 75.Hein, M., & Maier, M. (2007). Manifold denoising. In Advances in neural information processing systems (pp. 561–568).
  76. 76.Hinton, G. E., Osindero, S., & Teh, Y. W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18(7), 1527–1554.
  77. 77.Huang, B., & Jebara, T. (2011). Fast b-matching via sufficient selection belief propagation. In Proceedings of the 14th international conference on artificial intelligence and statistics (pp. 361–369).
  78. 78.Jayadeva, K. R., & Chandra, S. (2007). Twin support vector machines for pattern classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(5), 905–910.
  79. 79.Jebara, T., Wang, J., & Chang, S. F. (2009) Graph construction and b-matching for semi-supervised learning. In Proceedings of the 26th annual international conference on machine learning (pp. 441–448).
  80. 80.Joachims, T. (1999). Transductive inference for text classification using support vector machines. In Proceedings of the 16th international conference on machine learning (Vol. 99, pp. 200–209).
  81. 81.Joachims, T. (2003). Transductive learning via spectral graph partitioning. In Proceedings of the 20th international conference on machine learning (pp. 290–297).
  82. 82.Karasuyama, M., & Mamitsuka, H. (2013) Manifold-based similarity adaptation for label propagation. In Advances in neural information processing systems (pp. 1547–1555).
  83. 83.Kingma, D. P., Mohamed, S., Rezende, D. J., & Welling, M. (2014). Semi-supervised learning with deep generative models. In Advances in neural information processing systems (pp. 3581–3589).
  84. 84.Kingma, D. P., & Welling, M. (2013). Auto-encoding variational Bayes. In International conference on learning, representations.
  85. 85.Kipf, T. N., & Welling, M. (2016). Semi-supervised classification with graph convolutional networks. arXiv:1609.02907.
  86. 86.Kiritchenko, S., & Matwin, S. (2001). Email classification with co-training. In Proceedings of the 2001 conference of the centre for advanced studies on collaborative research (P. 8). IBM press.
  87. 87.Kohonen, T. (1998). The self-organizing map. Neurocomputing, 21(1–3), 1–6.
  88. 88.Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. Master’s thesis, University of Toronto, Department of Computer Science.
  89. 89.Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097–1105).
  90. 90.Kveton, B., Valko, M., Rahimi, A., & Huang, L. (2010). Semi-supervised learning with max-margin graph cuts. In Proceedings of the 13th international conference on artificial intelligence and statistics (pp. 421–428).
  91. 91.Laine, S., & Aila, T. (2017). Temporal ensembling for semi-supervised learning. In International conference on learning, representations.
  92. 92.Lange, T., Law, M. H., Jain, A. K., & Buhmann, J. M. (2005). Learning with constrained and unlabelled data. In Proceedings of the 2005 IEEE conference on computer vision and pattern recognition (Vol. 1, pp. 731–738). IEEE.
  93. 93.Lawrence, N. D., & Jordan, M. I. (2005). Semi-supervised learning via Gaussian processes. In Advances in neural information processing systems (pp. 753–760).
  94. 94.LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436.
  95. 95.Lee, D. H. (2013). Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Proceedings of the 30th ICML workshop on challenges in representation learning (Vol. 3, p. 2).
  96. 96.Leistner, C., Saffari, A., Santner, J., Bischof, H. (2009). Semi-supervised random forests. In Proceedings of the IEEE 12th international conference on computer vision (pp. 506–513). IEEE.
  97. 97.Levatić, J., Ceci, M., Kocev, D., & Džeroski, S. (2017). Semi-supervised classification trees. Journal of Intelligent Information Systems, 49(3), 461–486.
  98. 98.Li, C., Xu, K., Zhu, J., & Zhang, B. (2017). Triple generative adversarial nets. arXiv:1703.02291.
  99. 99.Li, M., & Zhou, Z. H. (2007). Improve computer-aided diagnosis with machine learning techniques using undiagnosed samples. IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans, 37(6), 1088–1098.
  100. 100.Li, S., & Fu, Y. (2013). Low-rank coding with b-matching constraint for semi-supervised classification. In Proceedings of the 23rd international joint conference on artificial intelligence (pp. 1472–1478).
  101. 101.Li, S., & Fu, Y. (2015). Learning balanced and unbalanced graphs via low-rank coding. IEEE Transactions on Knowledge and Data Engineering, 27(5), 1274–1287.
  102. 102.Li, Y. F., & Zhou, Z. H. (2015). Towards making unlabeled data never hurt. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(1), 175–188.
  103. 103.Liu, B., Lee, W. S., Yu, P. S., & Li, X. (2002). Partially supervised classification of text documents. In Proceedings of the 19th international conference on machine learning (Vol. 2, pp. 387–394).
  104. 104.Liu, G., Lin, Z., & Yu, Y. (2010a). Robust subspace segmentation by low-rank representation. In Proceedings of the 27th international conference on machine learning (pp. 663–670).
  105. 105.Liu, W., & Chang, S. F. (2009). Robust multi-class transductive learning with graphs. In Proceedings of the 2009 IEEE conference on computer vision and pattern recognition (pp. 381–388). IEEE.
  106. 106.Liu, W., He, J., & Chang, S. F. (2010b). Large graph construction for scalable semi-supervised learning. In Proceedings of the 27th international conference on machine learning (pp. 679–686).
  107. 107.Liu, X., Song, M., Tao, D., Liu, Z., Zhang, L., Chen, C., & Bu, J. (2013). Semi-supervised node splitting for random forest construction. In Proceedings of the 2013 IEEE conference on computer vision and pattern recognition (pp. 492–499). IEEE.
  108. 108.Liu, W., Wang, J., & Chang, S. F. (2012). Robust and scalable graph-based semisupervised learning. Proceedings of the IEEE, 100(9), 2624–2638.
  109. 109.Liu, X., Song, M., Tao, D., Liu, Z., Zhang, L., Chen, C., et al. (2015). Random forest construction with robust semisupervised node splitting. IEEE Transactions on Image Processing, 24(1), 471–483.
  110. 110.Lu, Q., Getoor, L. (2003). Link-based classification. In Proceedings of the 20th international conference on machine learning (pp. 496–503).
  111. 111.Luo, Y., Zhu, J., Li, M., Ren, Y., & Zhang, B. (2018). Smooth neighbors on teacher graphs for semi-supervised learning. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8896–8905).
  112. 112.Maier, M., Luxburg, U. V., & Hein, M. (2009). Influence of graph construction on graph-based clustering measures. In Advances in neural information processing systems (pp. 1025–1032).
  113. 113.Mallapragada, P. K., Jin, R., Jain, A. K., & Liu, Y. (2009). Semiboost: Boosting for semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(11), 2000–2014.
  114. 114.Melacci, S., & Belkin, M. (2011). Laplacian support vector machines trained in the primal. Journal of Machine Learning Research, 12, 1149–1184.
  115. 115.Mihalcea, R. (2004). Co-training and self-training for word sense disambiguation. In Proceedings of the 8th conference on computational natural language learning.
  116. 116.Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013) Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems (pp. 3111–3119).
  117. 117.Miyato, T., Maeda, S. I., Koyama, M., & Ishii, S. (2018). Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(8), 1979–1993.
  118. 118.Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., & Ng, A.Y. (2011). Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning.
  119. 119.Neville, J., & Jensen, D. (2000). Iterative classification in relational data. In Proceedings of the 17th AAAI workshop on learning statistical models from relational data (pp. 13–20).
  120. 120.Nigam, K., & Ghani, R. (2000). Analyzing the effectiveness and applicability of co-training. In Proceedings of the 9th international conference on information and knowledge management (pp. 86–93). ACM.
  121. 121.Nigam, K., McCallum, A., Mitchell, T. (2006). Semi-supervised text classification using EM. In Semi-Supervised Learning (pp. 33–56).
  122. 122.Nigam, K., McCallum, A. K., Thrun, S., & Mitchell, T. (2000). Text classification from labeled and unlabeled documents using EM. Machine Learning, 39(2), 103–134.
  123. 123.Niyogi, P. (2008). Manifold regularization and semi-supervised learning: Some theoretical analyses. Journal of Machine Learning Research, 14(1), 1229–1250.
  124. 124.Odena, A. (2016). Semi-supervised learning with generative adversarial networks. arXiv:1606.01583.
  125. 125.Oliver, A., Odena, A., Raffel, C., Cubuk, E. D., Goodfellow, I. J. (2018). Realistic evaluation of deep semi-supervised learning algorithms. arXiv:1804.09170.
  126. 126.Oshiro, T. M., Perez, P. S., & Baranauskas, J. A. (2012). How many trees in a random forest? In Proceedings of the international workshop on machine learning and data mining in pattern recognition (pp. 154–168). Springer.
  127. 127.Pang, B., & Lee, L. (2004). A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd annual meeting on association for computational linguistics, association for computational linguistics (p. 271).
  128. 128.Park, S., Park, J., Shin, S., & Moon, I. (2018). Adversarial dropout for supervised and semi-supervised learning. In Proceedings of the thirty-second AAAI conference on artificial intelligence (pp. 3917–3924).
  129. 129.Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., & Lerer, A. (2017). Automatic differentiation in pytorch. In NIPS Autodiff workshop.
  130. 130.Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., et al. (2011). Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12, 2825–2830.
  131. 131.Perozzi, B., Al-Rfou, R., & Skiena, S. (2014). Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 701–710). ACM.
  132. 132.Pezeshki, M., Fan, L., Brakel, P., Courville, A., & Bengio, Y. (2016). Deconstructing the ladder network architecture. In Proceedings of the 33rd international conference on machine learning (pp. 2368–2376).
  133. 133.Pitelis, N., Russell, C., & Agapito, L. (2013). Learning a manifold as an atlas. In Proceedings of the 2013 IEEE conference on computer vision and pattern recognition (pp. 1642–1649). IEEE.
  134. 134.Pitelis, N., Russell, C., & Agapito, L. (2014). Semi-supervised learning using an unsupervised atlas. In Proceedings of the joint European conference on machine learning and knowledge discovery in databases (pp. 565–580). Springer.
  135. 135.Prémont-Schwarz, I., Ilin, A., Hao, T., Rasmus, A., Boney, R., & Valpola, H. (2017). Recurrent ladder networks. In: I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett (eds.), Advances in neural information processing systems (pp. 6009–6019).
  136. 136.Provost, F., & Domingos, P. (2003). Tree induction for probability-based ranking. Machine Learning, 52(3), 199–215.
  137. 137.Qi, Z., Tian, Y., & Shi, Y. (2012). Laplacian twin support vector machine for semi-supervised classification. Neural Networks, 35, 46–53.
  138. 138.Rasmus, A., Berglund, M., Honkala, M., Valpola, H., & Raiko, T. (2015). Semi-supervised learning with ladder networks. In Advances in neural information processing systems (pp. 3546–3554).
  139. 139.Ratle, F., Camps-Valls, G., & Weston, J. (2010). Semisupervised neural networks for efficient hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 48(5), 2271–2282.
  140. 140.Rifai, S., Dauphin, Y. N., Vincent, P., Bengio, Y., & Muller, X. (2011a). The manifold tangent classifier. In Advances in neural information processing systems (pp. 2294–2302).
  141. 141.Rifai, S., Vincent, P., Muller, X., Glorot, X., & Bengio, Y. (2011b). Contractive auto-encoders: Explicit invariance during feature extraction. In Proceedings of the 28th international conference on machine learning (pp. 833–840).
  142. 142.Rosenberg, C., Hebert, M., & Schneiderman, H. (2005). Semi-supervised self-training of object detection models. In Proceedings of the 7th IEEE workshop on applications of computer vision (pp. 29–36).
  143. 143.Roweis, S. T., & Saul, L. K. (2000). Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500), 2323–2326.
  144. 144.Sajjadi, M., Javanmardi, M., & Tasdizen, T. (2016). Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In Advances in neural information processing systems (pp. 1163–1171).
  145. 145.Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016). Improved techniques for training gans. In Advances in neural information processing systems (pp. 2234–2242).
  146. 146.Sen, P., Namata, G., Bilgic, M., Getoor, L., Galligher, B., & Eliassi-Rad, T. (2008). Collective classification in network data. AI Magazine, 29(3), 93.
  147. 147.Settles, B. (2012). Active learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 6(1), 1–114.
  148. 148.Sheikhpour, R., Sarram, M. A., Gharaghani, S., & Chahooki, M. A. Z. (2017). A survey on semi-supervised feature selection methods. Pattern Recognition, 64, 141–158.
  149. 149.Shental, N., & Domany, E. (2005). Semi-supervised learning—A statistical physics approach. In Proceedings of the 22nd ICML workshop on learning with partially classified training data.
  150. 150.Sindhwani, V., Niyogi, P., & Belkin, M. (2005). A co-regularization approach to semi-supervised learning with multiple views. In Proceedings of the 22nd ICML workshop on learning with multiple views (pp. 74–79).
  151. 151.Sindhwani, V., & Rosenberg, D. S. (2008). An RKHS for multi-view learning and manifold co-regularization. In Proceedings of the 25th international conference on machine learning (pp. 976–983).
  152. 152.Singh, A., Nowak, R., & Zhu, X. (2009) Unlabeled data: Now it helps, now it doesn’t. In Advances in neural information processing systems (pp. 1513–1520).
  153. 153.Solomon, J., Rustamov, R., Guibas, L., & Butscher, A. (2014) Wasserstein propagation for semi-supervised learning. In Proceedings of the 31st international conference on machine learning (pp. 306–314).
  154. 154.Springenberg, J. T. (2015). Unsupervised and semi-supervised learning with categorical generative adversarial networks. arXiv:1511.06390.
  155. 155.Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1), 1929–1958.
  156. 156.Subramanya, A., & Bilmes, J. (2008). Soft-supervised learning for text classification. In Proceedings of the conference on empirical methods in natural language processing, association for computational linguistics (pp. 1090–1099).
  157. 157.Subramanya, A., & Bilmes, J. (2011). Semi-supervised learning with measure propagation. Journal of Machine Learning Research, 12, 3311–3370.
  158. 158.Subramanya, A., & Talukdar, P. P. (2014). Graph-based semi-supervised learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 8(4), 1–125.
  159. 159.Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., et al. (2013). Intriguing properties of neural networks. arXiv:1312.6199.
  160. 160.Szummer, M., & Jaakkola, T. (2002) Partially labeled classification with Markov random walks. In Advances in neural information processing systems (pp. 945–952).
  161. 161.Szummer, M., & Jaakkola, T. S. (2003) Information regularization with partially labeled data. In Advances in neural information processing systems (pp. 1049–1056).
  162. 162.Talukdar, P. P., & Crammer, K. (2009). New regularized algorithms for transductive learning. In Proceedings of the joint European conference on machine learning and knowledge discovery in databases (pp. 442–457). Springer.
  163. 163.Talukdar, P. P., Reisinger, J., Paşca, M., Ravichandran, D., Bhagat, R., & Pereira, F. (2008). Weakly-supervised acquisition of labeled class instances using graph random walks. In Proceedings of the conference on empirical methods in natural language processing, association for computational linguistics (pp. 582–590).
  164. 164.Tan, C., Lee, L., Tang, J., Jiang, L., Zhou, M., & Li, P. (2011). User-level sentiment analysis incorporating social networks. In Proceedings of the 17th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1397–1405). ACM.
  165. 165.Tang, J., Qu, M., Wang, M., Zhang, M., Yan, J., & Mei, Q. (2015). Line: Large-scale information network embedding. In Proceedings of the 24th international conference on world wide web, international world wide web conferences steering committee (pp. 1067–1077).
  166. 166.Tanha, J., van Someren, M., & Afsarmanesh, H. (2012). An adaboost algorithm for multiclass semi-supervised learning. In Proceedings of the 12th IEEE international conference on data mining (pp. 1116–1121). IEEE.
  167. 167.Tanha, J., van Someren, M., & Afsarmanesh, H. (2017). Semi-supervised self-training for decision tree classifiers. International Journal of Machine Learning and Cybernetics, 8(1), 355–370.
  168. 168.Tarvainen, A., & Valpola, H. (2017) Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in neural information processing systems (pp. 1195–1204).
  169. 169.Thornton, C., Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2013) Auto-weka: Combined selection and hyperparameter optimization of classification algorithms. In Proceedings of the 19th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 847–855). ACM.
  170. 170.Triguero, I., García, S., & Herrera, F. (2015). Self-labeled techniques for semi-supervised learning: Taxonomy, software and empirical study. Knowledge and Information Systems, 42(2), 245–284.
  171. 171.Triguero, I., González, S., Moyano, J. M., García López, S., Alcalá Fernández, J., Luengo Martín, J., et al. (2017). KEEL 3.0: An open source software for multi-stage analysis in data mining. International Journal of Computational Intelligence Systems, 10, 1238–1249.
  172. 172.Urner, R., Ben-David, S., & Shalev-Shwartz, S. (2011). Access to unlabeled data can speed up prediction time. In Proceedings of the 27th international conference on machine learning (pp. 641–648).
  173. 173.Valizadegan, H., Jin, R., & Jain, A. K. (2008). Semi-supervised boosting for multi-class classification. In Joint European conference on machine learning and knowledge discovery in databases (pp. 522–537). Springer.
  174. 174.Vapnik, V. (1998). Statistical learning theory (Vol. 1). New York: Wiley.
  175. 175.Verma, V., Lamb, A., Kannala, J., Bengio, Y., & Lopez-Paz, D. (2019). Interpolation consistency training for semi-supervised learning. arXiv:1903.03825.
  176. 176.Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P. A. (2008). Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on machine learning (pp. 1096–1103).
  177. 177.Wager, S., Wang, S., & Liang, P. S. (2013). Dropout training as adaptive regularization. In Advances in neural information processing systems (pp. 351–359).
  178. 178.Wan, X. (2009). Co-training for cross-lingual sentiment classification. In Proceedings of the 47th annual meeting of the ACL, association for computational linguistics (pp. 235–243).
  179. 179.Wang, D., Cui, P., Zhu, W. (2016). Structural deep network embedding. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1225–1234). ACM.
  180. 180.Wang, F., & Zhang, C. (2008). Label propagation through linear neighborhoods. IEEE Transactions on Knowledge and Data Engineering, 20(1), 55–67.
  181. 181.Wang, J., Jebara, T., & Chang, S. F. (2008a). Graph transduction via alternating minimization. In Proceedings of the 25th international conference on machine learning (pp. 1144–1151).
  182. 182.Wang, J., Jebara, T., & Chang, S. F. (2013). Semi-supervised learning using greedy max-cut. Journal of Machine Learning Research, 14, 771–800.
  183. 183.Wang, J., Luo, S. W., & Zeng. X. H. (2008b). A random subspace method for co-training. In Proceedings of the IEEE international joint conference on neural networks (pp. 195–200). IEEE.
  184. 184.Wang, W., & Zhou, Z. H. (2007). Analyzing co-training style algorithms. In Proceedings of the 18th European conference on machine learning (pp. 454–465). Springer.
  185. 185.Wang, W., Zhou, Z. H. (2010). A new analysis of co-training. In Proceedings of the 27th international conference on machine learning (pp. 1135–1142).
  186. 186.Weston, J., Ratle, F., & Collobert, R. (2008). Deep learning via semi-supervised embedding. In Proceedings of the 25th international conference on machine learning (pp. 1168–1175).
  187. 187.Wold, S., Esbensen, K., & Geladi, P. (1987). Principal component analysis. Chemometrics and Intelligent Laboratory Systems, 2(1–3), 37–52.
  188. 188.Wright, J., Yang, A. Y., Ganesh, A., Sastry, S. S., & Ma, Y. (2009). Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(2), 210–227.
  189. 189.Wu, X. M., Li, Z., So, A. M., Wright, J., & Chang, S. F. (2012a). Learning with partially absorbing random walks. In Advances in neural information processing systems (pp. 3077–3085).
  190. 190.Wu, Z., Wu, J., Cao, J., & Tao, D. (2012b). Hysad: A semi-supervised hybrid shilling attack detector for trustworthy product recommendation. In Proceedings of the 18th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 985–993). ACM.
  191. 191.Xu, C., Tao, D., & Xu, C. (2013). A survey on multi-view learning. arXiv:1304.5634.
  192. 192.Xu, J., He, H., & Man, H. (2012). Dcpe co-training for classification. Neurocomputing, 86, 75–85.
  193. 193.Xu, L., & Schuurmans, D. (2005) Unsupervised and semi-supervised multi-class support vector machines. In Proceedings of the 20th national conference on artificial intelligence (Vol. 5, p. 13).
  194. 194.Yan, S., & Wang, H. (2009). Semi-supervised learning by sparse representation. In Proceedings of the 2009 SIAM international conference on data mining (pp. 792–801). SIAM.
  195. 195.Yang, Z., Cohen, W. W., & Salakhutdinov, R. (2016) Revisiting semi-supervised learning with graph embeddings. In Proceedings of the 33rd international conference on machine learning (pp. 40–48).
  196. 196.Yarowsky, D. (1995). Unsupervised word sense disambiguation rivaling supervised methods. In Proceedings of the 33rd annual meeting of the association for computational linguistics, association for computational linguistics (pp. 189–196).
  197. 197.Yaslan, Y., & Cataltepe, Z. (2010). Co-training with relevant random subspaces. Neurocomputing, 73(10), 1652–1661.
  198. 198.Yu, S., Krishnapuram, B., Rosales, R., & Rao, R. B. (2011). Bayesian co-training. Journal of Machine Learning Research, 12, 2649–2680.
  199. 199.Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2018). mixup: Beyond empirical risk minimization. In International conference on learning representations.
  200. 200.Zhang, K., Kwok, J. T., & Parvin, B. (2009). Prototype vector machine for large scale semi-supervised learning. In Proceedings of the 26th international conference on machine learning (pp. 1233–1240).
  201. 201.Zhang, W., & Zheng, Q. (2009). Tsfs: A novel algorithm for single view co-training. In Proceedings of the 2nd IEEE international joint conference on computational sciences and optimization (Vol. 1, pp. 492–496). IEEE.
  202. 202.Zhou, D., Bousquet, O., Lal, T. N., Weston, J., & Schölkopf, B. (2004). Learning with local and global consistency. In Advances in Neural Information Processing Systems (pp. 321–328).
  203. 203.Zhou, Y., & Goldman, S. (2004) Democratic co-learning. In Proceedings of the 16th IEEE international conference on tools with artificial intelligence (pp. 594–602). IEEE.
  204. 204.Zhou, Z. H. (2012). Ensemble methods: Foundations and algorithms. Boca Raton: CRC Press.
  205. 205.Zhou, Z. H., & Li, M. (2005a) Semi-supervised regression with co-training. In Proceedings of the 19th international joint conference on artificial intelligence (Vol. 5, pp. 908–913).
  206. 206.Zhou, Z. H., & Li,M. (2005b). Tri-training: Exploiting unlabeled data using three classifiers. IEEE Transactions on Knowledge and Data Engineering, 17(11), 1529–1541.
  207. 207.Zhou, Z. H., & Li, M. (2010). Semi-supervised learning by disagreement. Knowledge and Information Systems, 24(3), 415–439.
  208. 208.Zhu, X. (2005). Semi-supervised learning with graphs. Ph.D. thesis, Carnegie Mellon University.
  209. 209.Zhu, X. (2008). Semi-supervised learning literature survey. Technical Report. 1530, University of Wisconsin-Madison.
  210. 210.Zhu, X., & Ghahramani, Z. (2002a). Learning from labeled and unlabeled data with label propagation. Technical Report. CMU-CALD-02-107, Carnegie Mellon University.
  211. 211.Zhu, X., & Ghahramani, Z. (2002b) Towards semi-supervised classification with Markov random fields. Technival Report. CMU-CALD-02-106, Carnegie Mellon University.
  212. 212.Zhu, X., Ghahramani, Z., & Lafferty, J. D. (2003) Semi-supervised learning using Gaussian fields and harmonic functions. In Proceedings of the 20th international conference on machine learning (pp. 912–919).
  213. 213.Zhu, X., & Goldberg, A. B. (2009). Introduction to semi-supervised learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 3(1), 1–130.
  214. 214.Zhu, X., & Lafferty, J. (2005). Harmonic mixtures: Combining mixture models and graph-based methods for inductive and scalable semi-supervised learning. In Proceedings of the 22nd international conference on machine learning (pp. 1052–1059). ACM.
  215. 215.Zhuang, L., Gao, H., Lin, Z., Ma, Y., Zhang, X., & Yu, N. (2012) Non-negative low rank and sparse graph for semi-supervised learning. In Proceedings of the 2012 IEEE conference on computer vision and pattern recognition (pp. 2328–2335). IEEE.

Citation

MLA
van Engelen, J. E., and H. H. Hoos. “A Survey on Semi-supervised Learning”. Machine Learning, vol. 109, no. 2, 2019, pp. 373–440, https://doi.org/10.1007/s10994-019-05855-6.
APA
van Engelen, J. E., & Hoos, H. H. (2019). A survey on semi-supervised learning. Machine Learning, 109(2), 373–440. https://doi.org/10.1007/s10994-019-05855-6
Chicago
van Engelen, J. E., and H. H. Hoos. 2019. “A Survey on Semi-supervised Learning”. Machine Learning 109 (2): 373–440. https://doi.org/10.1007/s10994-019-05855-6.
Harvard
van Engelen, J.E. and Hoos, H.H. (2019) “A survey on semi-supervised learning”, Machine Learning, 109(2), pp. 373–440. Available at: https://doi.org/10.1007/s10994-019-05855-6.
Vancouver
1. van Engelen JE, Hoos HH (2019) A survey on semi-supervised learning. Machine Learning 109:373–440

BibTeX

@article{van_Engelen_2019, title={A survey on semi-supervised learning}, volume={109}, ISSN={1573-0565}, url={http://dx.doi.org/10.1007/s10994-019-05855-6}, DOI={10.1007/s10994-019-05855-6}, number={2}, journal={Machine Learning}, publisher={Springer Science and Business Media LLC}, author={van Engelen, Jesper E. and Hoos, Holger H.}, year={2019}, month=Nov, pages={373–440} }
Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF