Back

Correcting Sample Selection Bias by Unlabeled Data

Jiayuan HuangArthur GrettonKarsten M. BorgwardtBernhard SchölkopfAlex Smola

article2006NeurIPS1,927 citations

Introduces Kernel Mean Matching, a nonparametric method that directly corrects sample selection bias by matching training and test feature distributions in reproducing kernel Hilbert spaces without requiring intermediate density estimation.

Listen

Machine learning models rely heavily on the assumption that training data and future operational data share the exact same statistical distribution. In real-world applications, however, this assumption frequently fails due to sample selection bias or covariate shift. For example, clinical diagnostic tools often train on demographic groups with higher screening rates or under specific laboratory protocols, yet practitioners deploy them across broad populations under differing conditions. Conventional methods to correct this imbalance attempt to estimate entire probability distributions or require prior knowledge of selection probabilities, an intermediate step that introduces substantial estimation errors and degrades predictive performance.

The article introduces and evaluates Kernel Mean Matching, a non-parametric method that directly computes sample reweighting values to align training and test distributions without estimating probability densities. The authors set out to demonstrate that matching distribution means in a high-dimensional feature space using unlabeled target data provides an efficient, robust correction for sample selection bias across various predictive tasks.

To evaluate this framework, the authors conducted theoretical convergence analyses and extensive empirical evaluations across synthetic regression tasks, 18 real-world benchmark datasets spanning 23 experimental configurations, and two medical gene expression microarray datasets for prostate and breast cancer diagnosis. The methodology reformulates sample correction as a standard convex quadratic optimization problem, adjusting the influence of individual training points so that under-represented cases receive higher weights and over-represented cases are down-weighted before training standard regression models or support vector machines.

The findings demonstrate three major outcomes. First, the proposed reweighting method consistently outperformed unweighted baseline models across all evaluated datasets and achieved the lowest error rates in 13 out of 23 benchmark configurations, often surpassing idealized models that had access to the true, known data bias. Second, on the California Housing regression benchmark, the approach reduced normalized mean square error to 1.24 compared to 2.29 for the unweighted baseline, while also cutting error rates by more than half on several computer system benchmarks. Third, in real-world biomedical applications, the method significantly improved diagnostic classification accuracy across independent microarray platforms and remained effective even when selection bias affected target labels directly rather than purely input features.

These results provide immediate operational and risk-mitigation benefits for deploying predictive models in changing environments. By eliminating the need to model complex underlying distributions, organizations can adapt existing classification and regression algorithms to new populations using only unlabeled operational data. This prevents silent performance degradation, reduces model re-training costs across distinct operational platforms, and improves compliance and diagnostic safety in sensitive domains like clinical healthcare.

Organizations facing distribution shifts should integrate distribution matching into their model training pipelines as an automated data pre-processing step. Because the method operates as a standard quadratic program, teams can readily embed it into existing workflows without redesigning core learning architectures. For next steps, technical teams should evaluate decoupling the feature-matching kernel scales from the final model kernel scales to further refine predictive accuracy.

Decision-makers should note certain limitations: the method requires a sufficient volume of training and unlabeled test data, as reweighting inherently reduces effective sample sizes and yields diminishing returns on very small datasets. Furthermore, theoretical convergence slows when the disparity between training and target populations is severe. Nevertheless, for moderately to heavily biased datasets of reasonable size, confidence in the method's ability to boost predictive accuracy remains high.

Ludwig Maximilian University of MunichMax Planck Institute for Biological CyberneticsNICTA, ANUUniversity of Waterloodistribution functionsfeature spacegenetic selectionsample selection biasunlabeled dataimportance sampling
Huang et al (2006).pdf
  • Paper: A Kernel Method for the Two-Sample Problem, Arthur Gretton et al. (2006). This work introduces Maximum Mean Discrepancy (MMD) to measure distribution differences in feature space without intermediate density estimation, providing the foundational kernel two-sample test that sample selection bias correction relies upon.
  • Paper: Analysis of Representations for Domain Adaptation, Shai Ben-David et al. (2006). This paper develops theoretical generalization bounds for domain adaptation based on discrepancy distances between feature representations, providing formal guarantees for feature matching approaches.
  • Paper: A theory of learning from different domains, Shai Ben-David et al. (2010). This work formalizes learning-theoretic bounds under distribution shift via the HΔH-divergence on unlabeled target data, generalizing principles of distribution matching between source and target domains.
  • Paper: A Survey on Transfer Learning, Sinno Jialin Pan et al. (2010). This survey systematically categorizes instance re-weighting and feature-space alignment techniques developed for covariate shift and transfer learning.
  • Paper: Deep Domain Confusion: Maximizing for Domain Invariance, Eric Tzeng et al. (2014). This paper extends kernel distribution matching into deep neural architectures by integrating an MMD domain confusion loss to learn shift-invariant representations.
  • Paper: Return of Frustratingly Easy Domain Adaptation, Baochen Sun et al. (2015). This work simplifies feature-space distribution matching by directly aligning second-order statistics between training and test distributions.
  • Paper: Moment Matching for Multi-Source Domain Adaptation, Xingchao Peng et al. (2018). This study scales moment matching and distribution alignment across multiple source domains and an unlabeled target domain.
  • Paper: A Comprehensive Survey on Transfer Learning, Fuzhen Zhuang et al. (2019). This comprehensive survey provides an expanded taxonomy of modern instance weighting and feature transformation algorithms for domain shift.
  • Paper: Unbiased look at dataset bias, A. Torralba et al. (2011). This empirical investigation quantifies how dataset bias systematically induces severe distribution shifts in visual learning benchmarks.
  • Paper: Detecting and Correcting for Label Shift with Black Box Predictors, Zachary C. Lipton et al. (2018). This paper extends distribution shift correction from covariate shift to label shift using black-box predictor outputs on unlabeled test data.
  • Paper: WILDS: A Benchmark of in-the-Wild Distribution Shifts, Pang Wei Koh et al. (2020). This benchmark evaluates modern distribution shift and domain adaptation algorithms against diverse real-world distribution shifts.
Cover for Correcting Sample Selection Bias by Unlabeled Data

Abstract

We consider the scenario where training and test data are drawn from different distributions, commonly referred to as sample selection bias. Most algorithms for this setting try to first recover sampling distributions and then make appropriate corrections based on the distribution estimate. We present a nonparametric method which directly produces resampling weights without distribution estimation. Our method works by matching distributions between training and testing sets in feature space. Experimental results demonstrate that our method works well in practice.

Topics

distribution functionsfeature spacegenetic selectionestimation method

Keywords

sample selection biasunlabeled dataimportance samplingdistribution matchingsupport vector machinesselection probabilities

Table of Contents

  • 1 Introduction
  • 2 Sample Reweighting
  • 2.1 Sample Correction
  • 2.2 Using the sample reweighting in learning algorithms
  • 3 Distribution Matching
  • 3.1 Kernel Mean Matching and its relation to importance sampling
  • 3.2 Convergence of reweighted means in feature space
  • 3.3 Empirical KMM optimization
  • 4 Experiments
  • 4.1 Toy regression example
  • 4.2 Real world datasets
  • 4.2.1 Breast Cancer Dataset
  • 4.2.2 Further Benchmark Datasets
  • 4.2.3 Tumor Diagnosis using Microarrays
  • References

Knowls

  1. Knowl 1 — Empirical Kernel Mean Matching Optimization

    model/method

    Given an unlabeled training sample X={x1,…,xm}⊂XX = \{x_1, \dots, x_m\} \subset \mathcal{X} drawn i.i.d. from distribution Pr(x)\text{Pr}(x) and an unlabeled test sample X′={x1′,…,xm′′}⊂XX' = \{x'_1, \dots, x'_{m'}\} \subset \mathcal{X} drawn i.i.d. from distribution Pr′(x)\text{Pr}'(x), Kernel Mean Matching (KMM) directly infers training instance weights β=(β1,…,βm)⊤∈Rm\beta = (\beta_1, \dots, \beta_m)^\top \in \mathbb{R}^m by minimizing the discrepancy between the reweighted training mean and test mean in a Reproducing Kernel Hilbert Space (RKHS) F\mathcal{F} with kernel kk. The empirical objective minimizes

    ∥1m∑i=1mβiΦ(xi)−1m′∑i=1m′Φ(xi′)∥F2=1m2β⊤Kβ−2m2κ⊤β+const\left\| \frac{1}{m} \sum_{i=1}^m \beta_i \Phi(x_i) - \frac{1}{m'} \sum_{i=1}^{m'} \Phi(x'_i) \right\|_\mathcal{F}^2 = \frac{1}{m^2} \beta^\top K \beta - \frac{2}{m^2} \kappa^\top \beta + \text{const}

    where K∈Rm×mK \in \mathbb{R}^{m \times m} has entries Kij=k(xi,xj)K_{ij} = k(x_i, x_j) and κ∈Rm\kappa \in \mathbb{R}^m has entries κi=mm′∑j=1m′k(xi,xj′)\kappa_i = \frac{m}{m'} \sum_{j=1}^{m'} k(x_i, x'_j). This is formulated as a quadratic program:

    minimizeβ12β⊤Kβ−κ⊤β\text{minimize}_{\beta} \quad \frac{1}{2} \beta^\top K \beta - \kappa^\top \beta

    subject toβi∈[0,B]for i=1,…,mand∣∑i=1mβi−m∣≤mϵ\text{subject to} \quad \beta_i \in [0, B] \quad \text{for } i = 1, \dots, m \quad \text{and} \quad \left| \sum_{i=1}^m \beta_i - m \right| \le m\epsilon

    where B>0B > 0 bounds the maximum allowable weight Pr′(x)/Pr(x)\text{Pr}'(x)/\text{Pr}(x), and ϵ>0\epsilon > 0 allows slack for finite-sample empirical deviations from E[β(x)]=1\mathbb{E}[\beta(x)] = 1.

  2. Knowl 2 — Population Density Ratio Recovery via Universal Kernel Mean Matching

    theoretical result

    Let P\mathcal{P} denote the set of all Borel probability distributions on X\mathcal{X}, and let Φ:X→F\Phi: \mathcal{X} \to \mathcal{F} be the feature map into an RKHS F\mathcal{F} endowed with a universal kernel k(x,x′)=⟨Φ(x),Φ(x′)⟩Fk(x, x') = \langle \Phi(x), \Phi(x') \rangle_\mathcal{F}. The mean embedding operator μ(Pr):=Ex∼Pr[Φ(x)]\mu(\text{Pr}) := \mathbb{E}_{x \sim \text{Pr}}[\Phi(x)] is bijective from P\mathcal{P} into its image M(Φ)⊂F\mathcal{M}(\Phi) \subset \mathcal{F}.

    Consider the population mean matching optimization problem:

    minimizeβ∥μ(Pr′)−Ex∼Pr(x)[β(x)Φ(x)]∥Fsubject to β(x)≥0 and Ex∼Pr(x)[β(x)]=1\text{minimize}_{\beta} \quad \|\mu(\text{Pr}') - \mathbb{E}_{x \sim \text{Pr}(x)}[\beta(x) \Phi(x)]\|_\mathcal{F} \quad \text{subject to } \beta(x) \ge 0 \text{ and } \mathbb{E}_{x \sim \text{Pr}(x)}[\beta(x)] = 1

    This optimization problem is convex. If Pr′\text{Pr}' is absolutely continuous with respect to Pr\text{Pr} (such that Pr(A)=0  ⟹  Pr′(A)=0\text{Pr}(A) = 0 \implies \text{Pr}'(A) = 0 for any measurable set AA) and kk is a universal kernel, then the unique optimal population solution β(x)\beta(x) satisfies Pr′(x)=β(x)Pr(x)\text{Pr}'(x) = \beta(x)\text{Pr}(x), exactly recovering the density ratio without requiring explicit density estimation.

  3. Knowl 3 — Finite-Sample Deviation Bound for Reweighted RKHS Means

    theoretical result

    Let X={x1,…,xm}X = \{x_1, \dots, x_m\} be drawn i.i.d. from Pr\text{Pr}, and let X′={x1′,…,xm′′}X' = \{x'_1, \dots, x'_{m'}\} be drawn i.i.d. from Pr′\text{Pr}' where Pr′(x)=β(x)Pr(x)\text{Pr}'(x) = \beta(x)\text{Pr}(x). Assume β(x)∈[0,B]\beta(x) \in [0, B] is a fixed function on X\mathcal{X} with finite mean and non-zero variance, and the feature map Φ\Phi satisfies ∥Φ(x)∥F≤R\|\Phi(x)\|_\mathcal{F} \le R for all x∈Xx \in \mathcal{X}.

    With probability at least 1−δ1 - \delta over the random draw of XX and X′X', the RKHS distance between the empirical reweighted training mean and the empirical test mean is bounded by:

    ∥1m∑i=1mβ(xi)Φ(xi)−1m′∑i=1m′Φ(xi′)∥F≤(1+−2ln⁡(δ/2))RB2m+1m′\left\| \frac{1}{m} \sum_{i=1}^m \beta(x_i) \Phi(x_i) - \frac{1}{m'} \sum_{i=1}^{m'} \Phi(x'_i) \right\|_\mathcal{F} \le \left(1 + \sqrt{-2\ln(\delta/2)}\right) R \sqrt{\frac{B^2}{m} + \frac{1}{m'}}

    This bound guarantees convergence at the rate O(RB2/m+1/m′)O\left(R \sqrt{B^2/m + 1/m'}\right) in terms of sample sizes m,m′m, m' and bound BB. If the distribution discrepancy (and thus BB) is large, feature mean convergence requires a correspondingly larger training sample size.

  4. Knowl 4 — Covariate Shift Assumption for Selection Bias Correction

    assumption

    The training distribution Pr(x,y)\text{Pr}(x, y) and target test distribution Pr′(x,y)\text{Pr}'(x, y) over X×Y\mathcal{X} \times \mathcal{Y} are assumed to satisfy covariate shift, wherein the conditional label distribution remains invariant between training and test sets:

    Pr(x,y)=Pr(y∣x)Pr(x)andPr′(x,y)=Pr(y∣x)Pr′(x)\text{Pr}(x, y) = \text{Pr}(y|x)\text{Pr}(x) \quad \text{and} \quad \text{Pr}'(x, y) = \text{Pr}(y|x)\text{Pr}'(x)

    Under this assumption, the discrepancy between the joint distributions reduces strictly to the marginal distributions on inputs xx, implying that the true importance reweighting factor depends exclusively on the inputs:

    β(x,y)=Pr′(x,y)Pr(x,y)=Pr′(x)Pr(x)=β(x)\beta(x, y) = \frac{\text{Pr}'(x, y)}{\text{Pr}(x, y)} = \frac{\text{Pr}'(x)}{\text{Pr}(x)} = \beta(x)

  5. Knowl 5 — Weighted Support Vector Classification under Sample Selection Bias

    model/method

    Given training instances {(xi,yi)}i=1m⊂X×Y\{(x_i, y_i)\}_{i=1}^m \subset \mathcal{X} \times \mathcal{Y}, discrepancy loss Δ(yi,y)\Delta(y_i, y), feature map Φ(x,y)∈F\Phi(x, y) \in \mathcal{F} with inner product k(x,y,x′,y′)=⟨Φ(x,y),Φ(x′,y′)⟩Fk(x, y, x', y') = \langle \Phi(x, y), \Phi(x', y') \rangle_\mathcal{F}, and importance weights βi≥0\beta_i \ge 0, the reweighted regularized classification problem is:

    minimizeθ,ξ12∥θ∥F2+C∑i=1mβiξi\text{minimize}_{\theta, \xi} \quad \frac{1}{2} \|\theta\|_\mathcal{F}^2 + C \sum_{i=1}^m \beta_i \xi_i

    subject to⟨Φ(xi,yi)−Φ(xi,y),θ⟩F≥1−ξiΔ(yi,y)∀y∈Y,ξi≥0\text{subject to} \quad \langle \Phi(x_i, y_i) - \Phi(x_i, y), \theta \rangle_\mathcal{F} \ge 1 - \frac{\xi_i}{\Delta(y_i, y)} \quad \forall y \in \mathcal{Y}, \quad \xi_i \ge 0

    The dual optimization problem is formulated as:

    minimizeα12∑i,j=1m∑y,y′∈Yαiyαjy′k(xi,y,xj,y′)−∑i=1m∑y∈Yαiy\text{minimize}_{\alpha} \quad \frac{1}{2} \sum_{i,j=1}^m \sum_{y, y' \in \mathcal{Y}} \alpha_{iy} \alpha_{jy'} k(x_i, y, x_j, y') - \sum_{i=1}^m \sum_{y \in \mathcal{Y}} \alpha_{iy}

    subject toαiy≥0∀i,yand∑y∈YαiyΔ(yi,y)≤βiC∀i=1,…,m\text{subject to} \quad \alpha_{iy} \ge 0 \quad \forall i, y \quad \text{and} \quad \sum_{y \in \mathcal{Y}} \frac{\alpha_{iy}}{\Delta(y_i, y)} \le \beta_i C \quad \forall i=1,\dots,m

    The sample weight βi\beta_i directly scales the upper bound constraint on the dual variables αiy\alpha_{iy} for each instance.

  6. Knowl 6 — Weighted Penalized Least Mean Squares Kernel Regression

    model/method

    For regression targets yi∈Ry_i \in \mathbb{R}, feature map Φ(x)∈F\Phi(x) \in \mathcal{F}, kernel matrix K∈Rm×mK \in \mathbb{R}^{m \times m} with Kij=k(xi,xj)K_{ij} = k(x_i, x_j), and estimated weights β1,…,βm>0\beta_1, \dots, \beta_m > 0, weighted penalized least mean squares (LMS) regression minimizes the empirical objective:

    ∑i=1mβi(yi−⟨Φ(xi),θ⟩F)2+λ∥θ∥F2\sum_{i=1}^m \beta_i \left(y_i - \langle \Phi(x_i), \theta \rangle_\mathcal{F}\right)^2 + \lambda \|\theta\|_\mathcal{F}^2

    Letting βˉ=diag(β1,…,βm)∈Rm×m\bar{\beta} = \text{diag}(\beta_1, \dots, \beta_m) \in \mathbb{R}^{m \times m}, y=(y1,…,ym)⊤y = (y_1, \dots, y_m)^\top, and representing θ=∑i=1mαiΦ(xi)\theta = \sum_{i=1}^m \alpha_i \Phi(x_i), the objective becomes (y−Kα)⊤βˉ(y−Kα)+λα⊤Kα(y - K\alpha)^\top \bar{\beta} (y - K\alpha) + \lambda \alpha^\top K \alpha. Assuming KK and βˉ\bar{\beta} are full rank, the dual coefficient vector α∈Rm\alpha \in \mathbb{R}^m has the closed-form solution:

    α=(λβˉ−1+K)−1y\alpha = (\lambda \bar{\beta}^{-1} + K)^{-1} y

    This formulation effectively rescales the regularizer per training sample: samples with higher importance weights βi\beta_i are subject to less regularization.

  7. Knowl 7 — Sample Mean Normalization Slack Parameter for Empirical KMM

    theoretical result

    For an i.i.d. sample x1,…,xm∼Prx_1, \dots, x_m \sim \text{Pr} and any fixed weight function β(x)∈[0,B]\beta(x) \in [0, B] with finite mean and non-zero variance, the Central Limit Theorem establishes that the empirical average 1m∑i=1mβ(xi)\frac{1}{m} \sum_{i=1}^m \beta(x_i) converges in distribution to a Gaussian variable with mean ∫β(x)dPr(x)\int \beta(x) d\text{Pr}(x) and standard deviation bounded by B2m\frac{B}{2\sqrt{m}}.

    Because finite sample size induces fluctuations around the true expectation E[β(x)]=1\mathbb{E}[\beta(x)] = 1, the normalization constraint in empirical KMM cannot be set as an exact equality. The slack parameter ϵ\epsilon in ∣1m∑i=1mβi−1∣≤ϵ\left|\frac{1}{m}\sum_{i=1}^m \beta_i - 1\right| \le \epsilon must be chosen on the order of O(B/m)O(B/\sqrt{m}); in practice, setting ϵ=(m−1)/m\epsilon = (\sqrt{m} - 1)/\sqrt{m} provides appropriate slack.

  8. Knowl 8 — Benchmark Comparison of KMM on Biased Regression and Classification Tasks

    data/table

    The table below compares unweighted models, true importance sampling (reweighting using the known inverse sampling probabilities), and KMM across 23 biased sampling configurations derived from 18 datasets. Regression models (marked with *) report Normalized Mean Square Error (NMSE) averaged over 10 trials, while classification models report test error averaged over 30 trials. Gaussian kernel size σ\sigma was shared between KMM and SVM/LMS, with upper bound B=1000B = 1000 and slack ϵ=(m−1)/m\epsilon = (\sqrt{m}-1)/\sqrt{m}.

    DataSet σ\sigma ntrn_{tr} selected ntstn_{tst} unweighted importance samp. KMM
    1. Abalone* 1e-1 2000 853 2177 1.00±0.081.00 \pm 0.08 1.1±0.21.1 \pm 0.2 0.6±0.10.6 \pm 0.1
    2. CA Housing* 1e-1 16512 3470 4128 2.29±0.012.29 \pm 0.01 1.72±0.041.72 \pm 0.04 1.24±0.091.24 \pm 0.09
    3. Delta Ailerons(1)* 1e3 4000 1678 3129 0.51±0.010.51 \pm 0.01 0.51±0.010.51 \pm 0.01 0.401±0.0070.401 \pm 0.007
    4. Ailerons* 1e-5 7154 925 6596 1.50±0.061.50 \pm 0.06 0.7±0.10.7 \pm 0.1 1.2±0.21.2 \pm 0.2
    5. haberman(1) 1e-2 150 52 156 0.50±0.090.50 \pm 0.09 0.37±0.030.37 \pm 0.03 0.30±0.050.30 \pm 0.05
    6. USPS(6vs8)(1) 1/128 500 260 1042 0.13±0.180.13 \pm 0.18 0.1±0.20.1 \pm 0.2 0.1±0.10.1 \pm 0.1
    7. USPS(3vs9)(1) 1/128 500 252 1145 0.016±0.0060.016 \pm 0.006 0.012±0.0050.012 \pm 0.005 0.013±0.0050.013 \pm 0.005
    8. Bank8FM* 1e-1 4500 654 3692 0.5±0.10.5 \pm 0.1 0.45±0.060.45 \pm 0.06 0.47±0.050.47 \pm 0.05
    9. Bank32nh* 1e-2 4500 740 3692 23±4.023 \pm 4.0 19±219 \pm 2 19±219 \pm 2
    10. cpu-act* 1e-12 4000 1462 4192 10±110 \pm 1 4.0±0.24.0 \pm 0.2 1.9±0.21.9 \pm 0.2
    11. cpu-small* 1e-12 4000 1488 4192 9±29 \pm 2 4.0±0.24.0 \pm 0.2 2.0±0.52.0 \pm 0.5
    12. Delta Ailerons(2)* 1e3 4000 634 3129 2±22 \pm 2 1.5±1.51.5 \pm 1.5 1.7±0.91.7 \pm 0.9
    13. Boston house* 1e-4 300 108 206 0.8±0.20.8 \pm 0.2 0.74±0.090.74 \pm 0.09 0.76±0.070.76 \pm 0.07
    14. kin8nm* 1e-1 5000 428 3192 0.85±0.20.85 \pm 0.2 0.81±0.10.81 \pm 0.1 0.81±0.20.81 \pm 0.2
    15. puma8nh* 1e-1 4499 823 3693 1.1±0.11.1 \pm 0.1 0.77±0.050.77 \pm 0.05 0.83±0.030.83 \pm 0.03
    16. haberman(2) 1e-2 150 90 156 0.27±0.010.27 \pm 0.01 0.39±0.040.39 \pm 0.04 0.25±0.20.25 \pm 0.2
    17. USPS(6vs8)(2) 1/128 500 156 1042 0.23±0.20.23 \pm 0.2 0.23±0.20.23 \pm 0.2 0.16±0.080.16 \pm 0.08
    18. USPS(6vs8)(3) 1/128 500 104 1042 0.54±0.00020.54 \pm 0.0002 0.5±0.20.5 \pm 0.2 0.16±0.040.16 \pm 0.04
    19. USPS(3vs9)(2) 1/128 500 252 1145 0.46±0.090.46 \pm 0.09 0.5±0.20.5 \pm 0.2 0.2±0.10.2 \pm 0.1
    20. Breast Cancer 1e-1 280 96 419 0.05±0.010.05 \pm 0.01 0.036±0.0050.036 \pm 0.005 0.033±0.0040.033 \pm 0.004
    21. India diabetes 1e-4 200 97 568 0.32±0.020.32 \pm 0.02 0.30±0.020.30 \pm 0.02 0.30±0.020.30 \pm 0.02
    22. ionosphere 1e-1 150 64 201 0.32±0.060.32 \pm 0.06 0.31±0.070.31 \pm 0.07 0.28±0.060.28 \pm 0.06
    23. German credit 1e-4 400 214 600 0.283±0.0040.283 \pm 0.004 0.282±0.0040.282 \pm 0.004 0.280±0.0040.280 \pm 0.004

    KMM consistently outperforms the unweighted baseline across all 23 settings and yields the best overall performance in 13 out of 23 benchmarks, despite operating without knowledge of the true sampling distribution.

  9. Knowl 9 — KMM Robustness under Label Selection Bias

    empirical result

    When sample selection bias is conditioned directly on the label yy (i.e., P(s=1∣x,y)=P(s=1∣y)P(s=1|x, y) = P(s=1|y)) rather than the input features xx, the covariate shift assumption Pr(y∣x)=Pr′(y∣x)\text{Pr}(y|x) = \text{Pr}'(y|x) is violated. In experiments on the UCI Breast Cancer dataset where P(s=1∣y=1)=0.1P(s=1|y=1)=0.1 and P(s=1∣y=−1)=0.9P(s=1|y=-1)=0.9, KMM still reduces test error relative to unweighted SVM classification across various training/test split sizes (10% to 50% training proportions).

    For large training set sizes, KMM outperforms reweighting based on the true label density ratio. The inferred weights βi\beta_i remain approximately proportional to the inverse sampling probabilities, automatically assigning larger weights to under-sampled positive cases and smaller weights to over-sampled negative cases.

  10. Knowl 10 — Sample Size and Mass Disparity Limitations of KMM

    limitation

    Kernel Mean Matching exhibits two main limitations:

    1. Sensitivity to Small Sample Sizes: Re-weighting training examples reduces the effective sample size of the training dataset. On small datasets (e.g., small initial subsets), KMM fails to improve performance over unweighted models because further effective sample reduction leaves insufficient data for generalization.
    2. Disparity Convergence Slowdown: As shown by the O(RB2/m+1/m′)O(R\sqrt{B^2/m + 1/m'}) bound, the sample complexity for empirical mean matching scales linearly with the upper bound B=sup⁡xPr′(x)Pr(x)B = \sup_x \frac{\text{Pr}'(x)}{\text{Pr}(x)}. When target and source distributions have substantial mass disparities, extremely large sample sizes are required for the empirical KMM weights to reliably approximate the true population density ratio.

Coverage note — Omitted the toy 1D polynomial regression experiment and the specific numerical details of the prostate cancer microarray classification, as both serve as secondary demonstrations of the core theoretical and empirical properties fully captured in the selected knowls.

References

  1. 1.G. Casella and R. Berger. Statistical Inference. Duxbury, Pacific Grove, CA, 2nd edition, 2002.
  2. 2.M. Dudik, R.E. Schapire, and S.J. Phillips. Correcting sample selection bias in maximum entropy density estimation. In Advances in Neural Information Processing Systems 17, 2005.
  3. 3.A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, and A. Smola. A kernel method for the two-sample-problem. In NIPS. MIT Press, 2006.
  4. 4.S. Gruvberger, M. Ringner, Y.Chen, S.Panavally, L.H. Saal, C. Peterson A.Borg, M. Ferno, and P.S.Meltzer. Estrogen receptor status in breast cancer is associated with remarkably distinct gene expression patterns. Cancer Research, 61, 2001.
  5. 5.J. Heckman. Sample selection bias as a specification error. Econometrica, 47(1):153–161, 1979.
  6. 6.W. Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58:13–30, 1963.
  7. 7.J. Huang, A. Smola, A. Gretton, K. Borgwardt, and B. Schölkopf. Correcting sample selection bias by unlabeled data. Technical report, CS-2006-44, University of Waterloo, 2006.
  8. 8.Y. Lin, Y. Lee, and G. Wahba. Support vector machines for classification in nonstandard situations. Machine Learning, 46:191–202, 2002.
  9. 9.S. Rosset, J. Zhu, H. Zou, and T. Hastie. A method for inferring label sampling mechanisms in semi-supervised learning. In Advances in Neural Information Processing Systems 17, 2004.
  10. 10.M. Schmidt and H. Gish. Speaker identification via support vector classifiers. In Proc. ICASSP ’96, pages 105–108, Atlanta, GA, May 1996.
  11. 11.B. Schölkopf, J. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. Williamson. Estimating the support of a high-dimensional distribution. Neural Computation, 13(7):1443–1471, 2001.
  12. 12.H. Shimodaira. Improving predictive inference under convariance shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90, 2000.
  13. 13.D. Singh, P. Febbo, K. Ross, D. Jackson, J. Manola, C. Ladd, P. Tamayo, A. Renshaw, A. DAmico, and J. Richie. Gene expression correlates of clinical prostate cancer behavior. Cancer Cell, 1(2), 2002.
  14. 14.I. Steinwart. On the influence of the kernel on the consistency of support vector machines. Journal of Machine Learning Research, 2:67–93, 2002.
  15. 15.I. Steinwart. Support vector machines are universally consistent. J. Compl., 18:768–791, 2002.
  16. 16.M. Sugiyama and K.-R. Müller. Input-dependent estimation of generalization error under covariate shift. Statistics and Decisions, 23:249–279, 2005.
  17. 17.I. Tsochantaridis, T. Joachims, T. Hofmann, and Y. Altun. Large margin methods for structured and interdependent output variables. Journal of Machine Learning Research, 2005.
  18. 18.P. Warnat, R. Eils, and B. Brors. Cross-platform analysis of cancer microarray data improves gene expression based classification of phenotypes. BMC Bioinformatics, 6:265, Nov 2005.
  19. 19.M. West, C. Blanchette, H. Dressman, E. Huang, S. Ishida, R. Spang, H Zuzan, J.A. Olson Jr, J.R.Marks, and J.R.Nevins. Predicting the clinical status of human breast cancer by using gene expression profiles. PNAS, 98(20), 2001.
  20. 20.B. Zadrozny. Learning and evaluating classifiers under sample selection bias. In International Conference on Machine Learning ICML’04, 2004.

Citation

MLA
Huang, J., et al. “Correcting Sample Selection Bias by Unlabeled Data”. Advances in Neural Information Processing Systems 19, The MIT Press, 2007, pp. 601–08, https://doi.org/10.7551/mitpress/7503.003.0080.
APA
Huang, J., Smola, A. J., Gretton, A., Borgwardt, K. M., & Schölkopf, B. (2007). Correcting Sample Selection Bias by Unlabeled Data. In Advances in Neural Information Processing Systems 19 (pp. 601–608). The MIT Press. https://doi.org/10.7551/mitpress/7503.003.0080
Chicago
Huang, J., A. J. Smola, A. Gretton, K. M. Borgwardt, and B. Schölkopf. 2007. “Correcting Sample Selection Bias by Unlabeled Data”. In Advances in Neural Information Processing Systems 19. The MIT Press. https://doi.org/10.7551/mitpress/7503.003.0080.
Harvard
Huang, J. et al. (2007) “Correcting Sample Selection Bias by Unlabeled Data”, Advances in Neural Information Processing Systems 19. The MIT Press, pp. 601–608. Available at: https://doi.org/10.7551/mitpress/7503.003.0080.
Vancouver
1. Huang J, Smola AJ, Gretton A, Borgwardt KM, Schölkopf B (2007) Correcting Sample Selection Bias by Unlabeled Data. In: Advances in Neural Information Processing Systems 19. The MIT Press, pp 601–608

BibTeX

@inbook{Huang_2007, title={Correcting Sample Selection Bias by Unlabeled Data}, ISBN={9780262256919}, url={http://dx.doi.org/10.7551/mitpress/7503.003.0080}, DOI={10.7551/mitpress/7503.003.0080}, booktitle={Advances in Neural Information Processing Systems 19}, publisher={The MIT Press}, author={Huang, Jiayuan and Smola, Alexander J. and Gretton, Arthur and Borgwardt, Karsten M. and Schölkopf, Bernhard}, year={2007}, month=Sept, pages={601–608} }
Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors