Back

Deep Transfer Learning with Joint Adaptation Networks

Mingsheng LongHanhua ZhuJianmin WangMichael I. Jordan

article2016ICML2,816 citations

Proposes Joint Adaptation Networks, an adversarial framework that aligns the joint distributions of multiple domain-specific layers via joint maximum mean discrepancy to achieve accurate deep domain adaptation with linear-time computation.

Listen

The article addresses the challenge of adapting deep learning models trained on one domain with abundant labels to a different target domain that lacks labels, where shifts in data distributions often degrade performance. This issue is increasingly relevant as organizations seek to deploy models across varied real-world settings without incurring high costs for new data labeling.

The article set out to develop and evaluate a method for aligning the joint distributions of features and labels across domains within deep networks to enable effective unsupervised domain adaptation.

The approach involved extending convolutional neural networks such as AlexNet and ResNet into Joint Adaptation Networks that minimize a Joint Maximum Mean Discrepancy criterion across multiple task-specific layers. Experiments used two standard benchmarks, Office-31 with six transfer tasks and ImageCLEF-DA with six tasks, comparing against prior transfer and deep learning baselines through classification accuracy on held-out target data.

The analysis shows that the proposed networks achieved higher average accuracy than previous methods, reaching 76.0 percent on Office-31 with AlexNet compared to 74.3 percent for the strongest baseline and 84.3 percent with ResNet compared to 82.2 percent. Gains were largest on difficult transfers where domains differed substantially, such as 58.3 percent versus 53.4 percent on one such task. The method also produced smaller measured distribution discrepancies than alternatives. An adversarial variant further improved results by maximizing distinguishability of the domains during training.

These outcomes indicate that explicitly matching joint distributions rather than marginal distributions alone yields more transferable features and classifiers, reducing the performance drop when moving models between domains. This can lower labeling expenses and accelerate deployment in new settings.

Further validation on additional datasets and tasks would strengthen confidence before broad adoption. The linear-time implementation supports scaling, yet performance remains sensitive to the weighting parameter between classification loss and distribution alignment.

NEL-BDSTsinghua UniversityUniversity of California Berkeleybackpropagation algorithmnetwork architecturestochasticJoint Adaptation NetworksDeep transfer learningDomain adaptation
arXiv: 1605.06636thuml/Transfer-Learning-Library
  • Paper: Learning Transferable Features with Deep Adaptation Networks, Mingsheng Long et al. (2015). This paper establishes the Deep Adaptation Network architecture using multi-kernel maximum mean discrepancy to align marginal feature distributions in deep neural networks, providing the immediate foundational framework that Joint Adaptation Networks extend to joint distributions.
  • Paper: Deep Domain Confusion: Maximizing for Domain Invariance, Eric Tzeng et al. (2014). This work introduces deep domain confusion by optimizing an MMD loss directly within CNN representations, pioneering the discrepancy-based deep adaptation methodology built upon by JAN.
  • Book: Domain-Adversarial Training of Neural Networks, Yaroslav Ganin et al. (2016). This work introduces domain-adversarial neural networks and the gradient reversal layer for domain adaptation, establishing the adversarial alignment concepts utilized in JAN's adversarial variant.
  • Paper: How transferable are features in deep neural networks?, Jason Yosinski et al. (2014). This empirical study quantifies the transition from general to task-specific features across CNN layers, motivating the layer-specific adaptation strategy central to Joint Adaptation Networks.
  • Paper: A theory of learning from different domains, Shai Ben-David et al. (2010). This foundational paper provides the learning-theoretic bounds relating target error to source error and domain divergence, supplying the core theoretical basis for deep discrepancy minimization.
  • Paper: Adapting Visual Category Models to New Domains, Kate Saenko et al. (2010). This paper introduces the standard Office-31 benchmark and early metric-learning formulations for domain adaptation, establishing the canonical evaluation testbed used in JAN.
  • Paper: Conditional Adversarial Domain Adaptation, Mingsheng Long et al. (2017). This paper extends joint feature-label alignment by conditioning adversarial discriminators on classifier prediction statistics via multilinear conditioning, advancing the joint adaptation paradigm of JAN.
  • Paper: Adversarial Discriminative Domain Adaptation, Eric Tzeng et al. (2017). This work formulates an untied, adversarial discriminative domain adaptation framework that offers an alternative adversarial paradigm to the joint discrepancy minimization in JAN.
  • Paper: Moment Matching for Multi-Source Domain Adaptation, Xingchao Peng et al. (2018). This study generalizes moment-matching and distribution alignment techniques from single-source settings to multi-source domain adaptation.
  • Paper: Deep Hashing Network for Unsupervised Domain Adaptation, Hemanth Venkateswara et al. (2017). This work integrates multi-kernel maximum mean discrepancy domain alignment with deep hashing layers for scalable, fast cross-domain retrieval.
  • Paper: Deep Visual Domain Adaptation: A Survey, Mei Wang et al. (2018). This survey provides a comprehensive synthesis of deep visual domain adaptation methods, framing discrepancy-based networks like JAN within the wider taxonomy of adversarial and reconstruction-based strategies.
  • Paper: A Survey on Deep Transfer Learning, Chuanqi Tan et al. (2018). This survey provides a systematic taxonomy and evaluation of deep transfer learning paradigms, categorizing mapping-based architectures including JAN.
  • Paper: A Comprehensive Survey on Transfer Learning, Fuzhen Zhuang et al. (2019). This comprehensive survey contextualizes joint distribution alignment methods alongside homogeneous and heterogeneous transfer learning approaches across multiple domains.
Cover for Deep Transfer Learning with Joint Adaptation Networks

Abstract

Deep networks have been successfully applied to learn transferable features for adapting models from a source domain to a different target domain. In this paper, we present joint adaptation networks (JAN), which learn a transfer network by aligning the joint distributions of multiple domain-specific layers across domains based on a joint maximum mean discrepancy (JMMD) criterion. Adversarial training strategy is adopted to maximize JMMD such that the distributions of the source and target domains are made more distinguishable. Learning can be performed by stochastic gradient descent with the gradients computed by back-propagation in linear-time. Experiments testify that our model yields state of the art results on standard datasets.

Topics

backpropagation algorithmnetwork architecturestochasticstochastic gradient descenttransfer learningconvolutional neural networksdata augmentationdeep learningprobability distributions

Keywords

Joint Adaptation NetworksDeep transfer learningDomain adaptationTransferable representationsFeature transferabilityConvolutional neural networksDomain-specific layers

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 Preliminary
  • 3.1 Hilbert Space Embedding
  • 3.2 Maximum Mean Discrepancy
  • 4 Joint Adaptation Networks
  • 4.1 Joint Maximum Mean Discrepancy
  • 4.2 Joint Adaptation Networks
  • 4.3 Adversarial Training for Optimal MMD
  • 5 Experiments
  • 5.1 Setup
  • 5.2 Results
  • 5.3 Analysis
  • 6 Conclusion
  • References

Knowls

  1. Knowl 1 — Joint Maximum Mean Discrepancy

    definition

    Joint Maximum Mean Discrepancy (JMMD) measures the divergence between two joint distributions P(Zs1,…,Zs∣L∣)P(Z^{s1}, \dots, Z^{s|\mathcal{L}|}) and Q(Zt1,…,Zt∣L∣)Q(Z^{t1}, \dots, Z^{t|\mathcal{L}|}) over multiple domain-specific layer activations L\mathcal{L}. Let each layer activation variable ZℓZ^\ell be mapped into a Reproducing Kernel Hilbert Space (RKHS) Hℓ\mathcal{H}^\ell with characteristic kernel kℓ(⋅,⋅)k^\ell(\cdot, \cdot) and feature map ϕℓ(⋅)\phi^\ell(\cdot). The joint distribution is embedded into the tensor product Hilbert space ⊗ℓ=1∣L∣Hℓ\otimes_{\ell=1}^{|\mathcal{L}|} \mathcal{H}^\ell via the uncentered cross-covariance operator:

    CZ1:∣L∣(P)≜EZ1:∣L∣[⨂ℓ=1∣L∣ϕℓ(Zℓ)]\mathcal{C}_{Z^{1:|\mathcal{L}|}}(P) \triangleq \mathbb{E}_{Z^{1:|\mathcal{L}|}} \left[ \bigotimes_{\ell=1}^{|\mathcal{L}|} \phi^\ell(Z^\ell) \right]

    where the inner product in the tensor product space satisfies ⟨⨂ℓ=1∣L∣ϕℓ(zℓ),⨂ℓ=1∣L∣ϕℓ(z′ℓ)⟩=∏ℓ=1∣L∣kℓ(zℓ,z′ℓ)\langle \bigotimes_{\ell=1}^{|\mathcal{L}|} \phi^\ell(z^\ell), \bigotimes_{\ell=1}^{|\mathcal{L}|} \phi^\ell({z'}^\ell) \rangle = \prod_{\ell=1}^{|\mathcal{L}|} k^\ell(z^\ell, {z'}^\ell).

    The population JMMD is defined as the squared Hilbert-Schmidt distance between the joint embeddings:

    DL(P,Q)≜∥CZs,1:∣L∣(P)−CZt,1:∣L∣(Q)∥⊗ℓ=1∣L∣Hℓ2D_\mathcal{L}(P, Q) \triangleq \left\| \mathcal{C}_{Z^{s, 1:|\mathcal{L}|}}(P) - \mathcal{C}_{Z^{t, 1:|\mathcal{L}|}}(Q) \right\|^2_{\otimes_{\ell=1}^{|\mathcal{L}|} \mathcal{H}^\ell}

    Given source samples {(zis1,…,zis∣L∣)}i=1ns\{(\mathbf{z}_i^{s1}, \dots, \mathbf{z}_i^{s|\mathcal{L}|})\}_{i=1}^{n_s} and target samples {(zjt1,…,zjt∣L∣)}j=1nt\{(\mathbf{z}_j^{t1}, \dots, \mathbf{z}_j^{t|\mathcal{L}|})\}_{j=1}^{n_t}, the empirical quadratic estimator is:

    D^L(P,Q)=1ns2∑i=1ns∑j=1ns∏ℓ∈Lkℓ(zisℓ,zjsℓ)+1nt2∑i=1nt∑j=1nt∏ℓ∈Lkℓ(zitℓ,zjtℓ)−2nsnt∑i=1ns∑j=1nt∏ℓ∈Lkℓ(zisℓ,zjtℓ)\hat{D}_\mathcal{L}(P, Q) = \frac{1}{n_s^2} \sum_{i=1}^{n_s} \sum_{j=1}^{n_s} \prod_{\ell \in \mathcal{L}} k^\ell(\mathbf{z}_i^{s\ell}, \mathbf{z}_j^{s\ell}) + \frac{1}{n_t^2} \sum_{i=1}^{n_t} \sum_{j=1}^{n_t} \prod_{\ell \in \mathcal{L}} k^\ell(\mathbf{z}_i^{t\ell}, \mathbf{z}_j^{t\ell}) - \frac{2}{n_s n_t} \sum_{i=1}^{n_s} \sum_{j=1}^{n_t} \prod_{\ell \in \mathcal{L}} k^\ell(\mathbf{z}_i^{s\ell}, \mathbf{z}_j^{t\ell})

    Unlike standard marginal Maximum Mean Discrepancy (MMD) which applies uniform weights, JMMD computes the product of kernel similarities across all layers in L\mathcal{L}, capturing interactions between all domain-specific layer representations.

  2. Knowl 2 — Linear-Time Unbiased Estimator of JMMD

    equation

    To enable mini-batch stochastic gradient descent without incurring the quadratic O(n2)O(n^2) cost of the standard empirical estimator, an unbiased estimator of Joint Maximum Mean Discrepancy (JMMD) with linear computational complexity O(n)O(n) is formulated. For a mini-batch with equal numbers of source and target samples n=ns=ntn = n_s = n_t, the linear-time estimator evaluates pairwise interactions across consecutive sample pairs:

    D^L(P,Q)=2n∑i=1n/2(∏ℓ∈Lkℓ(z2i−1sℓ,z2isℓ)+∏ℓ∈Lkℓ(z2i−1tℓ,z2itℓ))−2n∑i=1n/2(∏ℓ∈Lkℓ(z2i−1sℓ,z2itℓ)+∏ℓ∈Lkℓ(z2i−1tℓ,z2isℓ))\hat{D}_\mathcal{L}(P, Q) = \frac{2}{n} \sum_{i=1}^{n/2} \left( \prod_{\ell \in \mathcal{L}} k^\ell(\mathbf{z}_{2i-1}^{s\ell}, \mathbf{z}_{2i}^{s\ell}) + \prod_{\ell \in \mathcal{L}} k^\ell(\mathbf{z}_{2i-1}^{t\ell}, \mathbf{z}_{2i}^{t\ell}) \right) - \frac{2}{n} \sum_{i=1}^{n/2} \left( \prod_{\ell \in \mathcal{L}} k^\ell(\mathbf{z}_{2i-1}^{s\ell}, \mathbf{z}_{2i}^{t\ell}) + \prod_{\ell \in \mathcal{L}} k^\ell(\mathbf{z}_{2i-1}^{t\ell}, \mathbf{z}_{2i}^{s\ell}) \right)

    where zisℓ\mathbf{z}_i^{s\ell} and zjtℓ\mathbf{z}_j^{t\ell} denote the activations at layer ℓ∈L\ell \in \mathcal{L} for the ii-th source and jj-th target examples, and kℓk^\ell is the kernel function associated with layer ℓ\ell. Sampling identical numbers of source and target points per mini-batch removes domain size imbalance biases and allows back-propagation training in O(n)O(n) time.

  3. Knowl 3 — Joint Adaptation Networks Architecture and Objective

    model/method

    Joint Adaptation Networks (JAN) adapt deep neural networks across a labeled source domain Ds={(xis,yis)}i=1ns\mathcal{D}_s = \{(\mathbf{x}_i^s, y_i^s)\}_{i=1}^{n_s} and an unlabeled target domain Dt={xjt}j=1nt\mathcal{D}_t = \{\mathbf{x}_j^t\}_{j=1}^{n_t}. Recognizing that lower convolutional layers extract general domain-transferable features while higher layers specialize to domain-specific features and output classifications, JAN fine-tunes generic convolutional layers and aligns the joint distribution of activations across higher task-specific layers L\mathcal{L}.

    The optimization objective minimizes the empirical cross-entropy loss J\mathcal{J} on source data alongside the empirical Joint Maximum Mean Discrepancy D^L(P,Q)\hat{D}_\mathcal{L}(P, Q) across domain-specific layers L\mathcal{L}:

    min⁡f1ns∑i=1nsJ(f(xis),yis)+λD^L(P,Q)\min_f \frac{1}{n_s} \sum_{i=1}^{n_s} \mathcal{J}(f(\mathbf{x}_i^s), y_i^s) + \lambda \hat{D}_\mathcal{L}(P, Q)

    where λ>0\lambda > 0 balances task loss and domain discrepancy alignment. The set of domain-specific layers is specified according to the network backbone:

    • For AlexNet: L={fc6,fc7,fc8}\mathcal{L} = \{fc6, fc7, fc8\} (last two feature layers and the classifier layer).
    • For ResNet: L={pool5,fc}\mathcal{L} = \{pool5, fc\} (the final average pooling layer and the classifier layer).

  4. Knowl 4 — Adversarial Joint Adaptation Network (JAN-A)

    model/method

    To enhance the distinguishability of source and target distributions and avoid potential vanishing gradients associated with fixed-bandwidth kernel embeddings in high-dimensional spaces, the Adversarial Joint Adaptation Network (JAN-A) incorporates learnable fully-connected layers parameterized by θ\theta directly preceding the kernel embeddings in D^L(P,Q;θ)\hat{D}_\mathcal{L}(P, Q; \theta).

    JAN-A is formulated as a minimax optimization problem:

    min⁡fmax⁡θ1ns∑i=1nsJ(f(xis),yis)+λD^L(P,Q;θ)\min_f \max_\theta \frac{1}{n_s} \sum_{i=1}^{n_s} \mathcal{J}(f(\mathbf{x}_i^s), y_i^s) + \lambda \hat{D}_\mathcal{L}(P, Q; \theta)

    Here, the parameters θ\theta are trained to maximize D^L(P,Q;θ)\hat{D}_\mathcal{L}(P, Q; \theta), which maximizes the statistical test power to distinguish source activations from target activations. Simultaneously, the base feature and classification network ff is trained to minimize the classification loss and minimize the maximized JMMD. Unlike standard domain-adversarial methods that employ binary logistic discriminators, JAN-A employs the multi-layer joint kernel discrepancy as the adversarial objective.

  5. Knowl 5 — Optimization and Progressive Parameter Scheduling in JAN

    model/method

    JAN models are optimized using mini-batch stochastic gradient descent (SGD) with momentum of 0.90.9. Base networks (such as AlexNet or ResNet-50) are initialized from ImageNet pre-trained weights, while classifier layers trained from scratch receive a learning rate 10×10\times larger than lower layers.

    Training employs two dynamic parameter schedules based on the normalized training progress p∈[0,1]p \in [0, 1]:

    1. Learning Rate Annealing Schedule: ηp=η0(1+αp)β\eta_p = \frac{\eta_0}{(1 + \alpha p)^\beta} with initial learning rate η0=0.01\eta_0 = 0.01, α=10\alpha = 10, and β=0.75\beta = 0.75.

    2. Progressive Adaptation Factor Schedule: To avoid noisy activations during early iterations, the trade-off coefficient λ\lambda progressively increases from 0 to 1 according to: λp=21+exp⁡(−γp)−1\lambda_p = \frac{2}{1 + \exp(-\gamma p)} - 1 with γ=10\gamma = 10 held constant across experiments.

    Kernel parameters are chosen using Gaussian kernels kℓ(zi,zj)=exp⁡(−∥zi−zj∥2/σℓ2)k^\ell(\mathbf{z}_i, \mathbf{z}_j) = \exp(-\|\mathbf{z}_i - \mathbf{z}_j\|^2 / \sigma_\ell^2), where the bandwidth σℓ2\sigma_\ell^2 is set to the median pairwise squared distance over the training data.

  6. Knowl 6 — Unsupervised Domain Adaptation Performance on Office-31

    data/table

    Classification accuracy (mean ±\pm standard error over three random runs) on the Office-31 dataset evaluates transfer across three domains: Amazon (A), Webcam (W), and DSLR (D) using AlexNet and ResNet-50 architectures.

    Method A →\rightarrow W D →\rightarrow W W →\rightarrow D A →\rightarrow D D →\rightarrow A W →\rightarrow A Avg
    AlexNet 61.6±\pm0.5 95.4±\pm0.3 99.0±\pm0.2 63.8±\pm0.5 51.1±\pm0.6 49.8±\pm0.4 70.1
    TCA 61.0±\pm0.0 93.2±\pm0.0 95.2±\pm0.0 60.8±\pm0.0 51.6±\pm0.0 50.9±\pm0.0 68.8
    GFK 60.4±\pm0.0 95.6±\pm0.0 95.0±\pm0.0 60.6±\pm0.0 52.4±\pm0.0 48.1±\pm0.0 68.7
    DDC 61.8±\pm0.4 95.0±\pm0.5 98.5±\pm0.4 64.4±\pm0.3 52.1±\pm0.6 52.2±\pm0.4 70.6
    DAN 68.5±\pm0.5 96.0±\pm0.3 99.0±\pm0.3 67.0±\pm0.4 54.0±\pm0.5 53.1±\pm0.5 72.9
    RTN 73.3±\pm0.3 96.8±\pm0.2 99.6±\pm0.1 71.0±\pm0.2 50.5±\pm0.3 51.0±\pm0.1 73.7
    RevGrad 73.0±\pm0.5 96.4±\pm0.3 99.2±\pm0.3 72.3±\pm0.3 53.4±\pm0.4 51.2±\pm0.5 74.3
    JAN 74.9±\pm0.3 96.6±\pm0.2 99.5±\pm0.2 71.8±\pm0.2 58.3±\pm0.3 55.0±\pm0.4 76.0
    JAN-A 75.2±\pm0.4 96.6±\pm0.2 99.6±\pm0.1 72.8±\pm0.3 57.5±\pm0.2 56.3±\pm0.2 76.3
    ResNet 68.4±\pm0.2 96.7±\pm0.1 99.3±\pm0.1 68.9±\pm0.2 62.5±\pm0.3 60.7±\pm0.3 76.1
    TCA 72.7±\pm0.0 96.7±\pm0.0 99.6±\pm0.0 74.1±\pm0.0 61.7±\pm0.0 60.9±\pm0.0 77.6
    GFK 72.8±\pm0.0 95.0±\pm0.0 98.2±\pm0.0 74.5±\pm0.0 63.4±\pm0.0 61.0±\pm0.0 77.5
    DDC 75.6±\pm0.2 96.0±\pm0.2 98.2±\pm0.1 76.5±\pm0.3 62.2±\pm0.4 61.5±\pm0.5 78.3
    DAN 80.5±\pm0.4 97.1±\pm0.2 99.6±\pm0.1 78.6±\pm0.2 63.6±\pm0.3 62.8±\pm0.2 80.4
    RTN 84.5±\pm0.2 96.8±\pm0.1 99.4±\pm0.1 77.5±\pm0.3 66.2±\pm0.2 64.8±\pm0.3 81.6
    RevGrad 82.0±\pm0.4 96.9±\pm0.2 99.1±\pm0.1 79.7±\pm0.4 68.2±\pm0.4 67.4±\pm0.5 82.2
    JAN 85.4±\pm0.3 97.4±\pm0.2 99.8±\pm0.2 84.7±\pm0.3 68.6±\pm0.3 70.0±\pm0.4 84.3
    JAN-A 86.0±\pm0.4 96.7±\pm0.3 99.7±\pm0.1 85.1±\pm0.4 69.2±\pm0.4 70.7±\pm0.5 84.6

    JAN and JAN-A outperform prior methods, producing substantial improvements on challenging domain adaptation tasks (D →\rightarrow A and W →\rightarrow A) where source and target distributions differ markedly and source sample sizes are smaller than target domain sizes.

  7. Knowl 7 — Unsupervised Domain Adaptation Performance on ImageCLEF-DA

    data/table

    Classification accuracy (mean ±\pm standard error over three runs) on the ImageCLEF-DA benchmark evaluates 6 transfer tasks across three equal-sized domains (600 images across 12 classes each): Caltech-256 (C), ImageNet ILSVRC 2012 (I), and Pascal VOC 2012 (P).

    Method I →\rightarrow P P →\rightarrow I I →\rightarrow C C →\rightarrow I C →\rightarrow P P →\rightarrow C Avg
    AlexNet 66.2±\pm0.2 70.0±\pm0.2 84.3±\pm0.2 71.3±\pm0.4 59.3±\pm0.5 84.5±\pm0.3 73.9
    DAN 67.3±\pm0.2 80.5±\pm0.3 87.7±\pm0.3 76.0±\pm0.3 61.6±\pm0.3 88.4±\pm0.2 76.9
    RTN 67.4±\pm0.3 81.3±\pm0.3 89.5±\pm0.4 78.0±\pm0.2 62.0±\pm0.2 89.1±\pm0.1 77.9
    JAN 67.2±\pm0.5 82.8±\pm0.4 91.3±\pm0.5 80.0±\pm0.5 63.5±\pm0.4 91.0±\pm0.4 79.3
    ResNet 74.8±\pm0.3 83.9±\pm0.1 91.5±\pm0.3 78.0±\pm0.2 65.5±\pm0.3 91.2±\pm0.3 80.7
    DAN 74.5±\pm0.4 82.2±\pm0.2 92.8±\pm0.2 86.3±\pm0.4 69.2±\pm0.4 89.8±\pm0.4 82.5
    RTN 74.6±\pm0.3 85.8±\pm0.1 94.3±\pm0.1 85.9±\pm0.3 71.7±\pm0.3 91.2±\pm0.4 83.9
    JAN 76.8±\pm0.4 88.0±\pm0.2 94.7±\pm0.2 89.5±\pm0.3 74.2±\pm0.3 91.7±\pm0.3 85.8

    JAN achieves superior performance over marginal-distribution alignment methods (DAN, RTN) across balanced domains on both AlexNet (79.3% vs. 76.9% DAN) and ResNet (85.8% vs. 82.5% DAN).

  8. Knowl 8 — Discrepancy and Sensitivity Analysis of Joint Distribution Adaptation

    empirical result

    Empirical evaluations of domain discrepancy, hyperparameter sensitivity, and optimization behavior demonstrate key properties of Joint Adaptation Networks:

    1. Distribution Discrepancy Reduction: Measuring domain discrepancy via the proxy A\mathcal{A}-distance dA=2(1−2ϵ)d_\mathcal{A} = 2(1 - 2\epsilon) (where ϵ\epsilon is the generalization error of a kernel SVM distinguishing source from target) and via empirical JMMD across representations in fc7fc7 and ground-truth labels in fc8fc8 shows that JAN activations produce substantially lower dAd_\mathcal{A} and lower JMMD than standard CNN and DAN activations on both A→WA \rightarrow W and W→DW \rightarrow D.

    2. Sensitivity to Trade-off Weight λ\lambda: Varying the maximum JMMD penalty weight λ∈{0.01,0.02,0.05,0.1,0.2,0.5,1}\lambda \in \{0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1\} reveals a bell-shaped accuracy curve on A→WA \rightarrow W, confirming that an optimal intermediate value of λ\lambda properly balances supervised source discrimination and cross-domain joint alignment.

    3. Convergence Behavior: Nonparametric JAN exhibits faster test error convergence than adversarial baselines (RevGrad), while JAN-A attains equivalent convergence rate to RevGrad with consistently lower test error throughout training.

Coverage note — None was omitted; all key theoretical definitions, estimator formulations, models, training dynamics, benchmarks (Office-31 and ImageCLEF-DA), and empirical analyses were captured.

References

  1. 1.Arjovsky, Martin, Chintala, Soumith, and Bottou, Léon. Wasserstein gan. arXiv preprint arXiv:1701.07875, 2017.
  2. 2.Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. A theory of learning from different domains. Machine Learning, 79(1-2):151–175, 2010.
  3. 3.Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 35(8):1798–1828, 2013.
  4. 4.Bousmalis, Konstantinos, Trigeorgis, George, Silberman, Nathan, Krishnan, Dilip, and Erhan, Dumitru. Domain separation networks. In Advances in Neural Information Processing Systems (NIPS), pp. 343–351, 2016.
  5. 5.Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa, P. Natural language processing (almost) from scratch. Journal of Machine Learning Research (JMLR), 12:2493–2537, 2011.
  6. 6.Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T. Decaf: A deep convolutional activation feature for generic visual recognition. In International Conference on Machine Learning (ICML), 2014.
  7. 7.Duan, L., Tsang, I. W., and Xu, D. Domain transfer multiple kernel learning. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 34(3):465–479, 2012.
  8. 8.Ganin, Y. and Lempitsky, V. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning (ICML), 2015.
  9. 9.Glorot, X., Bordes, A., and Bengio, Y. Domain adaptation for large-scale sentiment classification: A deep learning approach. In International Conference on Machine Learning (ICML), 2011.
  10. 10.Gong, B., Shi, Y., Sha, F., and Grauman, K. Geodesic flow kernel for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
  11. 11.Gong, B., Grauman, K., and Sha, F. Connecting the dots with landmarks: Discriminatively learning domain-invariant features for unsupervised domain adaptation. In International Conference on Machine Learning (ICML), 2013.
  12. 12.Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), 2014.
  13. 13.Gopalan, R., Li, R., and Chellappa, R. Domain adaptation for object recognition: An unsupervised approach. In IEEE International Conference on Computer Vision (ICCV), 2011.
  14. 14.Gretton, A., Borgwardt, K., Rasch, M., Schölkopf, B., and Smola, A. A kernel two-sample test. Journal of Machine Learning Research (JMLR), 13:723–773, 2012.
  15. 15.He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  16. 16.Hoffman, J., Guadarrama, S., Tzeng, E., Hu, R., Donahue, J., Girshick, R., Darrell, T., and Saenko, K. LSDA: Large scale detection through adaptation. In Advances in Neural Information Processing Systems (NIPS), 2014.
  17. 17.Huang, J., Smola, A. J., Gretton, A., Borgwardt, K. M., and Schölkopf, B. Correcting sample selection bias by unlabeled data. In Advances in Neural Information Processing Systems (NIPS), 2006.
  18. 18.Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS), 2012.
  19. 19.Long, Mingsheng, Cao, Yue, Wang, Jianmin, and Jordan, Michael I. Learning transferable features with deep adaptation networks. In International Conference on Machine Learning (ICML), 2015.
  20. 20.Long, Mingsheng, Zhu, Han, Wang, Jianmin, and Jordan, Michael I. Unsupervised domain adaptation with residual transfer networks. In Advances in Neural Information Processing Systems (NIPS), pp. 136–144, 2016.
  21. 21.Mansour, Y., Mohri, M., and Rostamizadeh, A. Domain adaptation: Learning bounds and algorithms. In Conference on Computational Learning Theory (COLT), 2009.
  22. 22.Oquab, M., Bottou, L., Laptev, I., and Sivic, J. Learning and transferring mid-level image representations using convolutional neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.
  23. 23.Pan, S. J. and Yang, Q. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering (TKDE), 22(10):1345–1359, 2010.
  24. 24.Pan, S. J., Tsang, I. W., Kwok, J. T., and Yang, Q. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks (TNN), 22(2):199–210, 2011.
  25. 25.Quionero-Candela, J., Sugiyama, M., Schwaighofer, A., and Lawrence, N. D. Dataset shift in machine learning. The MIT Press, 2009.
  26. 26.Reddi, Sashank J, Ramdas, Aaditya, Póczos, Barnabás, Singh, Aarti, and Wasserman, Larry A. On the high dimensional power of a linear-time two sample test under mean-shift alternatives. In Artificial Intelligence and Statistics Conference (AISTATS), 2015.
  27. 27.Saenko, K., Kulis, B., Fritz, M., and Darrell, T. Adapting visual category models to new domains. In European Conference on Computer Vision (ECCV), 2010.
  28. 28.Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015 (arXiv:1409.1556v6), 2015.
  29. 29.Smola, Alex, Gretton, Arthur, Song, Le, and Schölkopf, Bernhard. A hilbert space embedding for distributions. In International Conference on Algorithmic Learning Theory (ALT), pp. 13–31. Springer, 2007.
  30. 30.Song, L., Huang, J., Smola, A., and Fukumizu, K. Hilbert space embeddings of conditional distributions with applications to dynamical systems. In International Conference on Machine Learning (ICML), 2009.
  31. 31.Song, Le and Dai, Bo. Robust low rank kernel embeddings of multivariate distributions. In Advances in Neural Information Processing Systems (NIPS), pp. 3228–3236, 2013.
  32. 32.Song, Le, Boots, Byron, Siddiqi, Sajid M, Gordon, Geoffrey J, and Smola, Alex. Hilbert space embeddings of hidden markov models. In International Conference on Machine Learning (ICML), 2010.
  33. 33.Song, Le, Fukumizu, Kenji, and Gretton, Arthur. Kernel embeddings of conditional distributions: A unified kernel framework for nonparametric inference in graphical models. IEEE Signal Processing Magazine, 30(4):98–111, 2013.
  34. 34.Sriperumbudur, B. K., Fukumizu, K., Gretton, A., Lanckriet, G., and Schölkopf, B. Kernel choice and classifiability for rkhs embeddings of probability distributions. In Advances in Neural Information Processing Systems (NIPS), 2009.
  35. 35.Sriperumbudur, Bharath K, Gretton, Arthur, Fukumizu, Kenji, Schölkopf, Bernhard, and Lanckriet, Gert RG. Hilbert space embeddings and metrics on probability measures. Journal of Machine Learning Research (JMLR), 11(Apr):1517–1561, 2010.
  36. 36.Sugiyama, M., Nakajima, S., Kashima, H., Buenau, P. V., and Kawanabe, M. Direct importance estimation with model selection and its application to covariate shift adaptation. In Advances in Neural Information Processing Systems (NIPS), 2008.
  37. 37.Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  38. 38.Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., and Darrell, T. Deep domain confusion: Maximizing for domain invariance. CoRR, abs/1412.3474, 2014.
  39. 39.Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., and Darrell, T. Simultaneous deep transfer across domains and tasks. In IEEE International Conference on Computer Vision (ICCV), 2015.
  40. 40.Tzeng, Eric, Hoffman, Judy, Saenko, Kate, and Darrell, Trevor. Adversarial discriminative domain adaptation. arXiv preprint arXiv:1702.05464, 2017.
  41. 41.Wang, X. and Schneider, J. Flexible transfer learning under support and model shift. In Advances in Neural Information Processing Systems (NIPS), 2014.
  42. 42.Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems (NIPS), 2014.
  43. 43.Zhang, K., Schölkopf, B., Muandet, K., and Wang, Z. Domain adaptation under target and conditional shift. In International Conference on Machine Learning (ICML), 2013.
  44. 44.Zhong, E., Fan, W., Yang, Q., Verscheure, O., and Ren, J. Cross validation framework to choose amongst models and datasets for transfer learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML/PKDD), pp. 547–562. Springer, 2010.

Citation

MLA
Long, M., et al. “Deep Transfer Learning with Joint Adaptation Networks”. arXiv, 2016, http://arxiv.org/abs/1605.06636v2.
APA
Long, M., Zhu, H., Wang, J., & Jordan, M. I. (2016). Deep Transfer Learning with Joint Adaptation Networks. arXiv. http://arxiv.org/abs/1605.06636v2
Chicago
Long, M., H. Zhu, J. Wang, and M. I. Jordan. 2016. “Deep Transfer Learning with Joint Adaptation Networks”. arXiv. http://arxiv.org/abs/1605.06636v2.
Harvard
Long, M. et al. (2016) “Deep Transfer Learning with Joint Adaptation Networks”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1605.06636v2.
Vancouver
1. Long M, Zhu H, Wang J, Jordan MI (2016) Deep Transfer Learning with Joint Adaptation Networks. arXiv

BibTeX

@article{long2016deep,
  title = {Deep Transfer Learning with Joint Adaptation Networks},
  author = {Long, Mingsheng and Zhu, Han and Wang, Jianmin and Jordan, Michael I.},
  year = {2016},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1605.06636v2},
  eprint = {1605.06636}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors