Moment Matching for Multi-Source Domain Adaptation

Xingchao PengQinxun BaiXide XiaZijun HuangKate SaenkoBo Wang

article2018ICCV2,392 citations

Introduces the large-scale DomainNet benchmark alongside M3SDA, a moment-matching framework with theoretical grounding that transfers knowledge from multiple labeled source domains to an unlabeled target by dynamically aligning their feature distributions.

Listen

Modern computer vision models experience severe performance degradation when deployed in operational environments that differ from their training data. While traditional transfer learning methods assume labeled training data comes from a single visual environment, real-world systems must routinely aggregate diverse datasets spanning varied lighting conditions, art styles, photo qualities, and visual modalities. Prior transfer learning techniques struggled because existing benchmark datasets were too small and near performance saturation, and the algorithms overlooked the statistical distribution differences that exist among multiple training sources.

The article establishes a large-scale evaluation benchmark for multi-source domain adaptation and develops a deep learning algorithm designed to transfer knowledge from multiple labeled source domains to an unlabeled target domain.

To evaluate real-world transfer capabilities, the authors created DomainNet, a large visual dataset containing 596,010 images across 345 object categories and six distinct domains (clipart, infograph, painting, quickdraw, real photos, and sketches). Using this benchmark, along with standard digit and object datasets, the authors evaluated baseline models against their newly developed algorithm, Moment Matching for Multi-Source Domain Adaptation (M3SDA), and its enhanced variant (M3SDA-beta). The approach aligns statistical moments (mean and variance profiles) of feature distributions both between each source domain and the target domain, and among the source domains themselves.

The analysis yielded several critical findings. First, M3SDA-beta consistently outperformed prior single- and multi-source methods, achieving 87.65% average accuracy on standard digit recognition tasks, 96.4% on Office-Caltech10, and a leading 42.6% on the challenging DomainNet benchmark. Second, simply combining multiple training sources into a single dataset without specialized alignment caused standard single-source transfer algorithms to underperform even basic source-only baselines. Third, ablation testing demonstrated that explicitly aligning source domains with one another provided a vital secondary performance boost beyond source-to-target alignment. Finally, the evaluation revealed that existing transfer learning algorithms degrade sharply as category counts scale up, with some models that excel on small datasets dropping to the lowest performance tiers on large-scale tasks.

These findings have direct operational implications for engineering risk and model performance. In multi-source production environments, naïve data aggregation creates negative transfer risks that degrade model accuracy. The results demonstrate that alignment among sources is essential to establishing a robust shared feature representation. Furthermore, previous high benchmark scores on small datasets masked fundamental architectural vulnerabilities that emerge when deploying vision models across broad real-world object catalogs.

Organizations developing computer vision systems across disparate operational domains should implement multi-source moment alignment frameworks rather than conventional data merging. Engineering teams should also replace saturated, small-scale test suites with large-scale benchmarks like DomainNet to properly evaluate model viability. Future development should focus on investigating the causes of negative transfer in highly abstract domains (such as quickdraw sketches) and establishing methods to identify and exclude harmful source data before adaptation.

Confidence in these findings is supported by theoretical proofs bounding target error and extensive empirical validation across 714 separate experimental runs totaling over 21,440 GPU hours. However, users should exercise caution when transferring knowledge to highly abstract or low-fidelity target domains, where negative transfer effects remain an active challenge.

Boston University Columbia University Horizon Robotics Peter Munk Cardiac Centre Vector Institute deep learning http matching algorithm Domain Domain Adaptation Source

arXiv: 1812.01754 M3SDA

Paper: Learning Transferable Features with Deep Adaptation Networks, Mingsheng Long et al. (2015). This paper establishes foundational deep domain adaptation techniques using maximum mean discrepancy, which the source extends to the multi-source setting.
Paper: A theory of learning from different domains, Shai Ben-David et al. (2010). This seminal work provides the learning-theoretic generalization bounds for domain adaptation that the source relies on for its theoretical insights.
Paper: Deep Domain Confusion: Maximizing for Domain Invariance, Eric Tzeng et al. (2014). This early study introduces explicit feature distribution alignment via maximum mean discrepancy in neural networks, directly inspiring moment-matching adaptation.

Paper: CyCADA: Cycle-Consistent Adversarial Domain Adaptation, Judy Hoffman et al. (2018). This work extends domain adaptation to both pixel and feature levels using adversarial and cycle-consistent constraints, continuing the push for robust transfer.
Paper: Momentum Contrast for Unsupervised Visual Representation Learning, Kaiming He et al. (2020). This paper advances unsupervised feature learning through momentum contrast, generalizing representation techniques used across transfer learning.
Paper: Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift, Stephan Rabanser et al. (2019). This study investigates dataset shift detection methods, following up on the distribution discrepancies tackled by multi-source adaptation.

Cover for Moment Matching for Multi-Source Domain Adaptation

Abstract

Conventional unsupervised domain adaptation (UDA) assumes that training data are sampled from a single domain. This neglects the more practical scenario where training data are collected from multiple sources, requiring multi-source domain adaptation. We make three major contributions towards addressing this problem. First, we collect and annotate by far the largest UDA dataset, called DomainNet, which contains six domains and about 0.6 million images distributed among 345 categories, addressing the gap in data availability for multi-source UDA research. Second, we propose a new deep learning approach, Moment Matching for Multi-Source Domain Adaptation M3SDA, which aims to transfer knowledge learned from multiple labeled source domains to an unlabeled target domain by dynamically aligning moments of their feature distributions. Third, we provide new theoretical insights specifically for moment matching approaches in both single and multiple source domain adaptation. Extensive experiments are conducted to demonstrate the power of our new dataset in benchmarking state-of-the-art multi-source domain adaptation methods, as well as the advantage of our proposed model. Dataset and Code are available at \url{this http URL}.

Topics

deep learning http matching algorithm matching methods transfer learning supervised machine learning unsupervised learning method parameter adaptation distribution functions

Keywords

Domain Domain Adaptation Source Domains Multi-Source Domain Adaptation Dataset Moment Matching Source Domains Target Pnt Qdr Rel

1 Introduction
2 Related Work
3 The DomainNet dataset
4 Moment Matching for Multi-Source DA
4.1 Theoretical Insight
5 Experiments
5.1 Experiments on Digit Recognition
5.2 Experiments on Office-Caltech10
5.3 Experiments on DomainNet
6 Conclusion
7 Acknowledgements
References
8 Appendix
A Ablation Study
B Cross-moment Divergence
C Proof of Theorem
D Details of Digit Experiments
E Feature visualization
F Effect of Category Number
G ResNet baselines
H Train/Test Split
I Image Samples
J Dataset Statistics
K Toy Experiment
L Time Consumption

Knowls

Knowl 1 — DomainNet Multi-Source Domain Adaptation Benchmark
definition
DomainNet is a large-scale visual dataset designed to benchmark single-source and multi-source unsupervised domain adaptation (MSDA). It comprises $596,010$ images distributed across $345$ object categories organized into $24$ semantic divisions (including furniture, mammal, tool, cloth, building, food, transport, and others).

The dataset spans six distinct visual domains:
- Clipart (clp): $48,837$ stylized vector-like clipart illustrations.
- Infograph (inf): $53,201$ infographic images depicting objects with text and graphic elements.
- Painting (pnt): $75,759$ artistic depictions in the form of paintings.
- Quickdraw (qdr): $172,500$ hand-drawn sketches generated by players of the "Quick, Draw!" game (reconstructed via B-spline curves from stroke coordinates; exactly $500$ images per category).
- Real (rel): $175,327$ natural photographs and real-world images.
- Sketch (skt): $70,386$ monochrome/grayscale sketches of objects.
The dataset uses an official $70\% / 30\%$ train/test split per domain, yielding $416,401$ training images and $179,609$ test images in total.
Knowl 2 — Cross-Moment Divergence Between Probability Distributions
definition
Let $\mathcal{X} \subset \mathbb{R}^n$ be a compact domain, and let $\mu$ and $\mu'$ be two probability measures on $\mathcal{X}$ . The $k$ -th order cross-moment divergence between $\mu$ and $\mu'$ , denoted by $d_{CM^k}(\mu, \mu')$ , is defined as:

$d_{CM^k}(\mu, \mu') = \sum_{\mathbf{i} \in \Delta_k} \left| \int_{\mathcal{X}} \prod_{j=1}^n (x_j)^{i_j} \, d\mu(\mathbf{x}) - \int_{\mathcal{X}} \prod_{j=1}^n (x_j)^{i_j} \, d\mu'(\mathbf{x}) \right|$

where $\mathbf{x} = (x_1, \dots, x_n) \in \mathcal{X}$ , $\mathbf{i} = (i_1, \dots, i_n) \in \mathbb{N}_0^n$ , and the multi-index index set is:

$\Delta_k = \left\{ (i_1, i_2, \dots, i_n) \in \mathbb{N}_0^n \;\middle|\; \sum_{j=1}^n i_j = k \right\}$

For two domains $\mathcal{D} = (\mu, f)$ and $\mathcal{D}' = (\mu', f')$ , $d_{CM^k}(\mathcal{D}, \mathcal{D}')$ denotes $d_{CM^k}(\mu, \mu')$ . This metric quantifies discrepancies across all multivariate moments of total degree $k$ between two feature distributions.
Knowl 3 — Moment Distance for Multi-Source Domain Alignment
equation
Let $\mathcal{D}_S = \{\mathcal{D}_1, \mathcal{D}_2, \dots, \mathcal{D}_N\}$ denote $N$ labeled source domains and $\mathcal{D}_T$ denote an unlabeled target domain. Let $X_1, X_2, \dots, X_N$ and $X_T$ be collections of independent and identically distributed feature vectors extracted from $\mathcal{D}_1, \dots, \mathcal{D}_N$ and $\mathcal{D}_T$ , respectively.

The squared Moment Distance $MD^2(\mathcal{D}_S, \mathcal{D}_T)$ between the source domains and the target domain matches the first ( $k=1$ ) and second ( $k=2$ ) empirical moments across both source-target pairs and source-source pairs:

$MD^2(\mathcal{D}_S, \mathcal{D}_T) = \sum_{k=1}^2 \left( \frac{1}{N} \sum_{i=1}^N \|\mathbb{E}(X_i^k) - \mathbb{E}(X_T^k)\|_2 + \binom{N}{2}^{-1} \sum_{i=1}^{N-1} \sum_{j=i+1}^N \|\mathbb{E}(X_i^k) - \mathbb{E}(X_j^k)\|_2 \right)$

Here, $\|\cdot\|_2$ is the Euclidean norm, the first summation term aligns each source domain directly with the target domain, and the second summation term aligns every pair of source domains with one another to ensure mutual compatibility.
Knowl 4 — M3SDA: Moment Matching for Multi-Source Domain Adaptation
model/method
The Moment Matching for Multi-Source Domain Adaptation ( $M^3SDA$ ) framework consists of three components:
1. A shared deep feature extractor $G$ mapping inputs from all source domains $\mathcal{D}_S = \{\mathcal{D}_1, \dots, \mathcal{D}_N\}$ and the target domain $\mathcal{D}_T$ into a common latent feature space.
2. A moment matching component minimizing the multi-source Moment Distance $MD^2(\mathcal{D}_S, \mathcal{D}_T)$ on the latent representations.
3. A set of $N$ task classifiers $C = \{C_1, C_2, \dots, C_N\}$ , where classifier $C_i$ is trained on labeled samples from domain $\mathcal{D}_i$ .
The end-to-end objective function is:

$\min_{G, C} \sum_{i=1}^N \mathcal{L}_{\mathcal{D}_i} + \lambda \min_G MD^2(\mathcal{D}_S, \mathcal{D}_T)$

where $\mathcal{L}_{\mathcal{D}_i}$ denotes the softmax cross-entropy loss for classifier $C_i$ on domain $\mathcal{D}_i$ , and $\lambda > 0$ is a trade-off hyperparameter balancing classification and moment alignment (set to $\lambda = 0.5$ in practice, with stable performance across $\lambda \in [0.1, 1.0]$ ).
Knowl 5 — M3SDA-beta Dual-Classifier Training Algorithm
algorithm
To align the conditional output distributions $p(y|x)$ in addition to the marginal feature distributions $p(x)$ , $M^3SDA$ - $\beta$ introduces $N$ pairs of classifiers $C' = \{(C_1, C_1'), (C_2, C_2'), \dots, (C_N, C_N')\}$ sharing the feature generator $G$ . The model is trained periodically through a three-step optimization procedure:
Input: Labeled source batches from D_1, ..., D_N; unlabeled target batch from D_T; hyperparameter lambda
Output: Trained generator G and classifier pairs C'
while not converged do
    // Step 1: Train G and C' to classify source domains and match moments
    loss_step1 = sum_{i=1}^N L_{D_i}(C_i, G) + sum_{i=1}^N L_{D_i}(C_i', G) + lambda * MD^2(D_S, D_T)
    Update G and C' to minimize loss_step1
    // Step 2: Fix G, train C' to maximize discrepancy on target domain
    loss_step2 = sum_{i=1}^N (L_{D_i}(C_i, G) + L_{D_i}(C_i', G)) - sum_{i=1}^N |P_{C_i}(D_T) - P_{C_i'}(D_T)|
    Update C' to minimize loss_step2
    // Step 3: Fix C', train G to minimize discrepancy on target domain
    loss_step3 = sum_{i=1}^N |P_{C_i}(D_T) - P_{C_i'}(D_T)|
    Update G to minimize loss_step3
end while
Here, $|P_{C_i}(\mathcal{D}_T) - P_{C_i'}(\mathcal{D}_T)|$ is the $L_1$ -distance between the predicted probability vectors of classifiers $C_i$ and $C_i'$ evaluated on target domain inputs.
Knowl 6 — Target Error Generalization Bound Based on Cross-Moment Divergence
theoretical result
Let $\mathcal{H}$ be a hypothesis space of VC dimension $d$ . Let $m$ be the total number of labeled samples drawn from $N$ source domains $\{\mathcal{D}_1, \dots, \mathcal{D}_N\}$ , where each source subset $S_j$ has size $\beta_j m$ (with $\sum_{j=1}^N \beta_j = 1$ ) sampled according to distribution $\mu_j$ with true labeling function $f_j$ .

Let $\boldsymbol{\alpha} = (\alpha_1, \dots, \alpha_N)$ be a fixed non-negative weight vector such that $\sum_{j=1}^N \alpha_j = 1$ . Let $\hat{h} = \arg\min_{h \in \mathcal{H}} \hat{\epsilon}_\alpha(h)$ be the empirical minimizer of the weighted source error $\hat{\epsilon}_\alpha(h) = \sum_{j=1}^N \alpha_j \hat{\epsilon}_j(h)$ , and let $h_T^* = \arg\min_{h \in \mathcal{H}} \epsilon_T(h)$ be the ideal target hypothesis.

For any $\delta \in (0, 1)$ and any $\epsilon > 0$ , there exist $N$ integers $\{n_\epsilon^j\}_{j=1}^N$ and $N$ positive constants $\{a_{n_\epsilon}^j\}_{j=1}^N$ such that with probability at least $1 - \delta$ :

$\epsilon_T(\hat{h}) \le \epsilon_T(h_T^*) + \eta_{\boldsymbol{\alpha}, \boldsymbol{\beta}, m, \delta} + \epsilon + \sum_{j=1}^N \alpha_j \left( 2\lambda_j + a_{n_\epsilon}^j \sum_{k=1}^{n_\epsilon^j} d_{CM^k}(\mathcal{D}_j, \mathcal{D}_T) \right)$

where:

$\eta_{\boldsymbol{\alpha}, \boldsymbol{\beta}, m, \delta} = 4 \sqrt{\left( \sum_{j=1}^N \frac{\alpha_j^2}{\beta_j} \right) \left( \frac{2d\left(\log\left(\frac{2m}{d}\right) + 1\right) + 2\log\left(\frac{4}{\delta}\right)}{m} \right)}$

and $\lambda_j = \min_{h \in \mathcal{H}} \{\epsilon_T(h) + \epsilon_j(h)\}$ is the combined risk of the optimal joint hypothesis on domain $\mathcal{D}_j$ and target $\mathcal{D}_T$ .
Knowl 7 — Theoretical Justification for Source-Source Alignment in MSDA
theoretical result
In multi-source domain adaptation, aligning each source domain $\mathcal{D}_j$ with the target domain $\mathcal{D}_T$ inherently requires aligning the source domains with one another. Because the $k$ -th order cross-moment divergence $d_{CM^k}(\cdot, \cdot)$ is a metric satisfying the triangle inequality, for any two source domains $\mathcal{D}_1, \mathcal{D}_2$ and a target domain $\mathcal{D}_T$ :

$d_{CM^k}(\mathcal{D}_1, \mathcal{D}_T) + d_{CM^k}(\mathcal{D}_2, \mathcal{D}_T) \ge d_{CM^k}(\mathcal{D}_1, \mathcal{D}_2)$

Consequently, the source-target moment divergence term in the generalization error bound is lower-bounded by pairwise source-source divergences. If source domains possess significant domain shift among themselves, they cannot simultaneously align closely with the same target domain unless source-source distributions are explicitly aligned.
Knowl 8 — Inference and Classifier Aggregation Schemes for MSDA
model/method
At inference time, target samples are passed through the learned feature extractor $G$ and evaluated on all source classifiers $C_1, \dots, C_{N-1}$ (assuming the $N$ -th domain is the target). The final class prediction is obtained via one of two ensembling schemas:
1. Uniform Average ( $M^3SDA^*$ ): Predictions are averaged uniformly across all source classifiers: $P(y|x) = \frac{1}{N-1} \sum_{i=1}^{N-1} P_{C_i}(x)$
2. Source-Only Accuracy Weighting ( $M^3SDA$ ): Predictions are combined using a normalized weight vector $\mathbf{w} = (w_1, \dots, w_{N-1})$ reflecting the domain affinity between source domain $\mathcal{D}_i$ and target domain $\mathcal{D}_T$ : $P(y|x) = \sum_{i=1}^{N-1} w_i P_{C_i}(x), \quad w_i = \frac{\text{acc}_i}{\sum_{j=1}^{N-1} \text{acc}_j}$ where $\text{acc}_i$ is the single-source accuracy obtained when a model trained solely on source domain $\mathcal{D}_i$ is evaluated on target domain $\mathcal{D}_T$ .

Knowl 9 — Multi-Source Domain Adaptation Performance on DomainNet

data/table

The table below compares multi-source domain adaptation methods on DomainNet across six transfer tasks (each column specifies 5 source domains $\to$ 1 target domain). Baselines are evaluated under Single Best (best single-source adaptation result), Source Combine (merging all sources into one single domain for single-source UDA), and Multi-Source settings. Models use ResNet-101 (except AlexNet and ResNet-152 oracle baselines).

Setting	$\to$ clp	$\to$ inf	$\to$ pnt	$\to$ qdr	$\to$ rel	$\to$ skt	Avg
Single Best
Source Only	$39.6\pm0.58$	$8.2\pm0.75$	$33.9\pm0.62$	$11.8\pm0.69$	$41.6\pm0.84$	$23.1\pm0.72$	$26.4\pm0.70$
DAN	$39.1\pm0.51$	$11.4\pm0.81$	$33.3\pm0.62$	$16.2\pm0.38$	$42.1\pm0.73$	$29.7\pm0.93$	$28.6\pm0.63$
RTN	$35.3\pm0.73$	$10.7\pm0.61$	$31.7\pm0.82$	$13.1\pm0.68$	$40.6\pm0.55$	$26.5\pm0.78$	$26.3\pm0.70$
JAN	$35.3\pm0.71$	$9.1\pm0.63$	$32.5\pm0.65$	$14.3\pm0.62$	$43.1\pm0.78$	$25.7\pm0.61$	$26.7\pm0.67$
DANN	$37.9\pm0.69$	$11.4\pm0.91$	$33.9\pm0.60$	$13.7\pm0.56$	$41.5\pm0.67$	$28.6\pm0.63$	$27.8\pm0.68$
ADDA	$39.5\pm0.81$	$14.5\pm0.69$	$29.1\pm0.78$	$14.9\pm0.54$	$41.9\pm0.82$	$30.7\pm0.68$	$28.4\pm0.72$
SE	$31.7\pm0.70$	$12.9\pm0.58$	$19.9\pm0.75$	$7.7\pm0.44$	$33.4\pm0.56$	$26.3\pm0.50$	$22.0\pm0.66$
MCD	$42.6\pm0.32$	$19.6\pm0.76$	$42.6\pm0.98$	$3.8\pm0.64$	$50.5\pm0.43$	$33.8\pm0.89$	$32.2\pm0.66$
Source Combine
Source Only	$47.6\pm0.52$	$13.0\pm0.41$	$38.1\pm0.45$	$13.3\pm0.39$	$51.9\pm0.85$	$33.7\pm0.54$	$32.9\pm0.54$
DAN	$45.4\pm0.49$	$12.8\pm0.86$	$36.2\pm0.58$	$15.3\pm0.37$	$48.6\pm0.72$	$34.0\pm0.54$	$32.1\pm0.59$
RTN	$44.2\pm0.57$	$12.6\pm0.73$	$35.3\pm0.59$	$14.6\pm0.76$	$48.4\pm0.67$	$31.7\pm0.73$	$31.1\pm0.68$
JAN	$40.9\pm0.43$	$11.1\pm0.61$	$35.4\pm0.50$	$12.1\pm0.67$	$45.8\pm0.59$	$32.3\pm0.63$	$29.6\pm0.57$
DANN	$45.5\pm0.59$	$13.1\pm0.72$	$37.0\pm0.69$	$13.2\pm0.77$	$48.9\pm0.65$	$31.8\pm0.62$	$32.6\pm0.68$
ADDA	$47.5\pm0.76$	$11.4\pm0.67$	$36.7\pm0.53$	$14.7\pm0.50$	$49.1\pm0.82$	$33.5\pm0.49$	$32.2\pm0.63$
SE	$24.7\pm0.32$	$3.9\pm0.47$	$12.7\pm0.35$	$7.1\pm0.46$	$22.8\pm0.51$	$9.1\pm0.49$	$16.1\pm0.43$
MCD	$54.3\pm0.64$	$22.1\pm0.70$	$45.7\pm0.63$	$7.6\pm0.49$	$58.4\pm0.65$	$43.5\pm0.57$	$38.5\pm0.61$
Multi-Source
DCTN	$48.6\pm0.73$	$23.5\pm0.59$	$48.8\pm0.63$	$7.2\pm0.46$	$53.5\pm0.56$	$47.3\pm0.47$	$38.2\pm0.57$
$M^3SDA^*$ (ours)	$57.0\pm0.79$	$22.1\pm0.68$	$50.5\pm0.45$	$4.4\pm0.21$	$62.0\pm0.45$	$48.5\pm0.56$	$40.8\pm0.52$
$M^3SDA$ (ours)	$57.2\pm0.98$	$24.2\pm1.21$	$51.6\pm0.44$	$5.2\pm0.45$	$61.6\pm0.89$	$49.6\pm0.56$	$41.5\pm0.74$
$M^3SDA$ - $\beta$ (ours)	$\mathbf{58.6\pm0.53}$	$\mathbf{26.0\pm0.89}$	$\mathbf{52.3\pm0.55}$	$6.3\pm0.58$	$\mathbf{62.7\pm0.51}$	$\mathbf{49.5\pm0.76}$	$\mathbf{42.6\pm0.64}$

$M^3SDA$ - $\beta$ achieves the highest overall accuracy ( $42.6\%$ ), outperforming the previous multi-source state-of-the-art DCTN ( $38.2\%$ ) and the single-source combine baseline MCD ( $38.5\%$ ). In the source combine setting, traditional single-source UDA methods (DAN, RTN, JAN, DANN) achieve lower accuracy than the unadapted source-only baseline ( $32.9\%$ ), demonstrating negative transfer when multi-domain sources are merged without multi-source alignment.

Knowl 10 — Ablation of Source-Source Versus Source-Target Moment Alignment

empirical result

An ablation study on Digit-Five, Office-Caltech10, and DomainNet isolates the effects of source-source (S-S) alignment versus source-target (S-T) alignment within $M^3SDA$ - $\beta$ :

Schema	Digit-Five	Office-Caltech10	DomainNet
S-S only	$81.5\;(+4.1)$	$94.5\;(+1.6)$	$34.4\;(+1.5)$
S-T only	$85.8\;(+8.1)$	$96.2\;(+3.3)$	$39.7\;(+6.8)$
$M^3SDA$ - $\beta$ (S-S + S-T)	$\mathbf{87.7\;(+10.0)}$	$\mathbf{96.4\;(+3.5)}$	$\mathbf{42.6\;(+9.7)}$

Values in parentheses represent accuracy gains over the corresponding unadapted source-only baselines ( $77.71\%$ on Digit-Five, $92.9\%$ on Office-Caltech10, and $32.9\%$ on DomainNet). Aligning source distributions directly to the target distribution (S-T only) provides the majority of the gain ( $+6.8\%$ on DomainNet), while additionally aligning source distributions with each other (S-S) provides an extra $+2.9\%$ gain on DomainNet (reaching $+9.7\%$ total improvement), demonstrating that joint S-S and S-T alignment is critical for MSDA.

Knowl 11 — Digit-Five and Office-Caltech10 Multi-Source Classification Performance
empirical result
On the Digit-Five benchmark (MNIST $\to$ mt, USPS $\to$ up, SVHN $\to$ sv, Synthetic Digits $\to$ sy, MNIST-M $\to$ mm), $M^3SDA$ and $M^3SDA$ - $\beta$ outperform existing discrepancy-based and adversarial-based methods across 5 leaves-one-domain-out adaptation settings:
- Source Only (multi-source): $77.71\% \pm 0.81$
- DAN: $80.44\% \pm 0.72$
- CORAL: $80.07\% \pm 0.75$
- DANN: $82.01\% \pm 0.76$
- JAN: $84.07\% \pm 0.71$
- ADDA: $84.84\% \pm 0.64$
- DCTN: $84.79\% \pm 0.72$
- MEDA: $85.60\% \pm 0.78$
- MCD: $86.10\% \pm 0.73$
- $M^3SDA$ (ours): $86.13\% \pm 0.64$
- $M^3SDA$ - $\beta$ (ours): $\mathbf{87.65\% \pm 0.75}$
On the Office-Caltech10 dataset (Amazon $\to$ A, Caltech $\to$ C, DSLR $\to$ D, Webcam $\to$ W using ResNet-101):
- Source Only (multi-source): $92.9\%$
- DAN: $94.8\%$
- DCTN: $95.3\%$
- JAN: $95.5\%$
- MEDA: $95.7\%$
- MCD: $95.6\%$
- $M^3SDA$ (ours): $96.1\%$
- $M^3SDA$ - $\beta$ (ours): $\mathbf{96.4\%}$
Knowl 12 — Degradation of Domain Adaptation Methods with Increasing Category Scale
empirical result
When the number of object classes is varied from $20$ up to $345$ (evaluated on DomainNet transfer tasks including painting $\to$ real, infograph $\to$ real, sketch $\to$ clipart, and quickdraw $\to$ clipart), all single-source UDA methods experience severe accuracy degradation as category scale expands.

While most models achieve competitive accuracy when restricted to small numbers of classes (e.g. 20 classes), their performance drops steeply and at divergent rates:
- Methods such as Self-Ensembling (SE) achieve superior accuracy at small category counts ( $20$ -- $50$ categories) but degrade fastest, exhibiting the worst performance among all baselines when category count exceeds $150$ .
- S-T and S-S moment-matching and discrepancy methods maintain substantially higher stability and retain superior classification accuracy at large category scale ( $345$ classes) compared to pseudo-labeling or self-ensembling methods.

Coverage note — No substantial contributed material was omitted. Detailed per-class counts across all 345 individual classes and single-source 30x30 baseline transfer matrices are summarized via the DomainNet benchmark and MSDA result tables.

References

1.Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1-2):151–175, 2010. 1, 2, 3, 5, 12
2.Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira, et al. Analysis of representations for domain adaptation. Advances in neural information processing systems, pages 137–144, 2007. 12
3.Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In Advances in Neural Information Processing Systems, pages 343–351, 2016. 3
4.Koby Crammer, Michael Kearns, and Jennifer Wortman. Learning from multiple sources. Journal of Machine Learning Research, 9(Aug):1757–1774, 2008. 1, 3, 12
5.Carl De Boor, Carl De Boor, Etats-Unis Mathematicien, Carl De Boor, and Carl De Boor. A practical guide to splines, volume 27. Springer-Verlag New York, 1978. 4
6.Lixin Duan, Dong Xu, and Shih-Fu Chang. Exploiting web images for event recognition in consumer videos: A multiple source domain adaptation approach. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 1338–1345. IEEE, 2012. 2, 3
7.Geoff French, Michal Mackiewicz, and Mark Fisher. Selfensembling for visual domain adaptation. In International Conference on Learning Representations, 2018. 2, 7, 8, 13, 14
8.Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1180–1189, Lille, France, 07–09 Jul 2015. PMLR. 1, 2, 6, 7, 8
9.Muhammad Ghifary, W Bastiaan Kleijn, and Mengjie Zhang. Domain adaptive neural networks for object recognition. In Pacific Rim international conference on artificial intelligence, pages 898–904. Springer, 2014. 2
10.Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li. Deep reconstruction-classification networks for unsupervised domain adaptation. In European Conference on Computer Vision, pages 597–613. Springer, 2016. 3
11.Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 2066–2073. IEEE, 2012. 2, 6
12.Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Scholkopf, and Alex J Smola. A kernel method for the two-sample-problem. In Advances in neural information processing systems, pages 513–520, 2007. 2
13.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 7, 14
14.Judy Hoffman, Mehryar Mohri, and Ningshan Zhang. Algorithms and theory for multiple-source adaptation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 8246–8256. Curran Associates, Inc., 2018. 1, 2, 3, 5
15.Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. CyCADA: Cycle-consistent adversarial domain adaptation. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1989–1998, Stockholmsmssan, Stockholm Sweden, 10–15 Jul 2018. PMLR. 1, 3
16.Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1857–1865, International Convention Centre, Sydney, Australia, 06–11 Aug 2017. PMLR. 3
17.Piotr Koniusz, Yusuf Tas, Hongguang Zhang, Mehrtash Harandi, Fatih Porikli, and Rui Zhang. Museum exhibit identification challenge for the supervised domain adaptation and beyond. In The European Conference on Computer Vision (ECCV), September 2018. 2
18.Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012. 7
19.Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. 2, 6
20.Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabas P'oczos. Mmd gan: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Systems, pages 2203–2213, 2017. 3
21.Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Deeper, broader and artier domain generalization. In International Conference on Computer Vision, 2017. 2
22.Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In International Conference on Machine Learning, pages 1718–1727, 2015. 3
23.Ming-Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Advances in Neural Information Processing Systems, pages 700–708, 2017. 1
24.Ming-Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. In Advances in neural information processing systems, pages 469–477, 2016. 2
25.Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 97–105, Lille, France, 07–09 Jul 2015. PMLR. 1, 2, 6, 7, 8, 13
26.Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Unsupervised domain adaptation with residual transfer networks. In Advances in Neural Information Processing Systems, pages 136–144, 2016. 7, 8
27.Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I. Jordan. Deep transfer learning with joint adaptation networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 2208–2217, 2017. 1, 2, 6, 7, 8
28.Yishay Mansour, Mehryar Mohri, Afshin Rostamizadeh, and A R. Domain adaptation with multiple sources. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems 21, pages 1041–1048. Curran Associates, Inc., 2009. 1, 3
29.Youssef Mroueh, Tom Sercu, and Vaibhava Goel. McGan: Mean and covariance feature matching GAN. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2527–2535, International Convention Centre, Sydney, Australia, 06–11 Aug 2017. PMLR. 3
30.OA Muradyan and S Ya Khavinson. Absolute values of the coefficients of the polynomials in weierstrass’s approximation theorem. Mathematical notes of the Academy of Sciences of the USSR, 22(2):641–645, 1977. 12
31.Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345–1359, 2010. 6, 8, 14
32.Xingchao Peng and Kate Saenko. Synthetic to real adaptation with generative correlation alignment networks. In 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018, Lake Tahoe, NV, USA, March 12-15, 2018, pages 1982–1991, 2018. 1, 2
33.Xingchao Peng, Baochen Sun, Karim Ali, and Kate Saenko. Learning deep object detectors from 3d models. In Proceedings of the IEEE International Conference on Computer Vision, pages 1278–1286, 2015. 2
34.Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924, 2017. 2
35.Xingchao Peng, Ben Usman, Kuniaki Saito, Neela Kaushik, Judy Hoffman, and Kate Saenko. Syn2real: A new benchmark forsynthetic-to-real visual domain adaptation. CoRR, abs/1806.09755, 2018. 2
36.Joaquin Quionero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence. Dataset Shift in Machine Learning. The MIT Press, 2009. 1
37.Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference on computer vision, pages 213–226. Springer, 2010. 2, 6
38.Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018. 1, 4, 5, 6, 7, 8, 14
39.Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In AAAI, volume 6, page 8, 2016. 1, 2, 3, 6
40.Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Computer Vision and Pattern Recognition (CVPR), volume 1, page 4, 2017. 1, 2, 6, 7, 8
41.Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474, 2014. 1, 2
42.Vladimir N Vapnik and A Ya Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. In Measures of complexity, pages 11–30. Springer, 2015. 13
43.Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In (IEEE) Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 2
44.Jindong Wang, Wenjie Feng, Yiqiang Chen, Han Yu, Meiyu Huang, and Philip S Yu. Visual domain adaptation with manifold embedded distribution alignment. In ACM Multimedia Conference, 2018. 2, 6, 7
45.Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3964–3973, 2018. 2, 3, 6, 7, 8
46.Zili Yi, Hao (Richard) Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-image translation. In ICCV, pages 2868–2876, 2017. 3
47.Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschlager, and Susanne Saminger-Platz. Central moment discrepancy (CMD) for domain-invariant representation learning. CoRR, abs/1702.08811, 2017. 1, 3
48.Zhen Zhang, Mianzhi Wang, Yan Huang, and Arye Nehorai. Aligning infinite-dimensional covariance matrices in reproducing kernel hilbert spaces for domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3437–3445, 2018. 3
49.Han Zhao, Shanghang Zhang, Guanhang Wu, Jose MF Moura, Joao P Costeira, and Geoffrey J Gordon. Adversarial multiple source domain adaptation. In Advances in Neural Information Processing Systems, pages 8568–8579, 2018. 1, 2, 5
50.Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017. 1, 3
51.Fuzhen Zhuang, Xiaohu Cheng, Ping Luo, Sinno Jialin Pan, and Qing He. Supervised representation learning: Transfer learning with deep autoencoders. In IJCAI, pages 4119–4125, 2015. 2

Citation

MLA

Peng, X., et al. “Moment Matching for Multi-Source Domain Adaptation”. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 1406–15, https://doi.org/10.1109/ICCV.2019.00149.

APA

Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., & Wang, B. (2019). Moment Matching for Multi-Source Domain Adaptation. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 1406–1415. https://doi.org/10.1109/ICCV.2019.00149

Chicago

Peng, X., Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang. 2019. “Moment Matching for Multi-Source Domain Adaptation”. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 1406–15. https://doi.org/10.1109/ICCV.2019.00149.

Harvard

Peng, X. et al. (2019) “Moment Matching for Multi-Source Domain Adaptation”, 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, pp. 1406–1415. Available at: https://doi.org/10.1109/ICCV.2019.00149.

Vancouver

1. Peng X, Bai Q, Xia X, Huang Z, Saenko K, Wang B (2019) Moment Matching for Multi-Source Domain Adaptation. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, pp 1406–1415

BibTeX

@inproceedings{Peng_2019, title={Moment Matching for Multi-Source Domain Adaptation}, url={http://dx.doi.org/10.1109/ICCV.2019.00149}, DOI={10.1109/iccv.2019.00149}, booktitle={2019 IEEE/CVF International Conference on Computer Vision (ICCV)}, publisher={IEEE}, author={Peng, Xingchao and Bai, Qinxun and Xia, Xide and Huang, Zijun and Saenko, Kate and Wang, Bo}, year={2019}, month=Oct, pages={1406–1415} }

Metadata:Crossref

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE

Moment Matching for Multi-Source Domain Adaptation

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — DomainNet Multi-Source Domain Adaptation Benchmark

Knowl 2 — Cross-Moment Divergence Between Probability Distributions

Knowl 3 — Moment Distance for Multi-Source Domain Alignment

Knowl 4 — M3SDA: Moment Matching for Multi-Source Domain Adaptation

Knowl 5 — M3SDA-beta Dual-Classifier Training Algorithm

Knowl 6 — Target Error Generalization Bound Based on Cross-Moment Divergence

Knowl 7 — Theoretical Justification for Source-Source Alignment in MSDA

Knowl 8 — Inference and Classifier Aggregation Schemes for MSDA