Back

A Survey on Deep Transfer Learning

Chuanqi TanFuchun SunTao KongWenchang ZhangChao YangChunfang Liu

article2018ICANN2,957 citations

Categorizes deep transfer learning approaches and their practical applications, offering a structured framework to help researchers overcome data scarcity and costly annotation constraints.

Listen

Deep learning has achieved strong results across many applications by automatically learning features from large datasets, yet it faces a fundamental barrier in domains such as bioinformatics and robotics where collecting and annotating sufficient high-quality data is prohibitively expensive or slow. The mismatch between the need for massive training data and its limited availability in these fields slows progress and raises costs. Transfer learning offers a way forward by relaxing the usual assumption that training and test data must come from identical distributions, allowing knowledge from a data-rich source domain to support learning in a data-scarce target domain.

This survey set out to define deep transfer learning, organize existing methods into coherent categories, and summarize the main techniques and representative studies that apply deep neural networks to transfer tasks. The authors conducted a structured literature review covering work published through mid-2018, drawing on foundational papers and recent advances to create a consistent taxonomy and set of diagrams that illustrate each category.

The review identifies four primary approaches. Instance-based methods reweight selected examples from the source domain so they better match the target distribution. Mapping-based methods project data from both domains into a shared space where distributions become more similar. Network-based methods reuse portions of a neural network already trained on a large source dataset, typically the early layers that extract general features. Adversarial-based methods add a domain discriminator that forces the network to learn representations that are useful for the main task yet indistinguishable across domains. In practice, these approaches are frequently combined.

These categories matter because they provide practitioners with clear options for reducing the amount of labeled target data and training time required while still achieving useful performance. The survey shows that network-based and adversarial methods have seen particularly rapid recent development and often deliver stronger results when source and target tasks are related but not identical. At the same time, the work highlights that most current methods still assume supervised settings and that issues such as negative transfer—where transferred knowledge harms performance—remain underexplored in deep models.

Future work should therefore examine unsupervised and semi-supervised transfer more systematically, develop reliable measures of transferability, and investigate the physical or neurological basis for what features transfer well. Decision makers considering deployment should first verify that the chosen source and target domains share enough structure, then plan for additional validation or fine-tuning once target data become available. The survey itself draws only on published methods up to 2018 and does not include new empirical comparisons, so readers should treat the taxonomy as a useful organizing framework rather than a definitive ranking of performance.

Tsinghua Universitybioinformaticsclassificationconvolutional neural networksdeep transfer learningdomain adaptationtransferable representations
arXiv: 1808.01974
  • Paper: A Survey on Transfer Learning, Sinno Jialin Pan et al. (2010). Reading this foundational survey on transfer learning provides the essential taxonomy and problem definitions that the source paper expands upon for deep learning settings.
  • Paper: Representation Learning: A Review and New Perspectives, Yoshua Bengio et al. (2012). This review on representation learning establishes the theoretical necessity and historical progression of unsupervised feature discovery that underpins deep transfer mechanisms.
  • Paper: Deep learning in neural networks: An overview, Juergen Schmidhuber (2014). This historical overview of deep neural networks supplies crucial context on credit assignment and network architectures that preceded modern deep transfer learning methodologies.
  • Paper: How transferable are features in deep neural networks?, Jason Yosinski et al. (2014). This empirical study on layer-by-layer feature transferability provides critical foundational insights into how convolutional representations behave when adapted to new tasks.
  • Paper: Unsupervised Domain Adaptation by Backpropagation, Yaroslav Ganin et al. (2015). This paper introduces foundational adversarial domain adaptation techniques via gradient reversal, which the source paper reviews as a core deep transfer learning strategy.
  • Paper: Learning Transferable Features with Deep Adaptation Networks, Mingsheng Long et al. (2015). This study on Deep Adaptation Networks presents essential methodology for minimizing domain discrepancy in deep task-specific layers, directly discussed within the source survey.
  • Paper: Adversarial Discriminative Domain Adaptation, Eric Tzeng et al. (2017). This paper details Adversarial Discriminative Domain Adaptation, providing a key technical framework for feature-space alignment reviewed extensively in the source survey.
Cover for A Survey on Deep Transfer Learning

Abstract

As a new classification platform, deep learning has recently received increasing attention from researchers and has been successfully applied to many domains. In some domains, like bioinformatics and robotics, it is very difficult to construct a large-scale well-annotated dataset due to the expense of data acquisition and costly annotation, which limits its development. Transfer learning relaxes the hypothesis that the training data must be independent and identically distributed (i.i.d.) with the test data, which motivates us to use transfer learning to solve the problem of insufficient training data. This survey focuses on reviewing the current researches of transfer learning by using deep neural network and its applications. We defined deep transfer learning, category and review the recent research works based on the techniques used in deep transfer learning.

Topics

bioinformaticsclassificationconvolutional neural networksdeep learningroboticstransfer learningclassification models

Keywords

deep transfer learningdomain adaptationtransferable representationsadversarial lossmetric transfer

Table of Contents

  • 1 Introduction
  • 2 Deep Transfer Learning
  • 3 Categories
  • 3.1 Instances-based deep transfer learning
  • 3.2 Mapping-based deep transfer learning
  • 3.3 Network-based deep transfer learning
  • 3.4 Adversarial-based deep transfer learning
  • 4 Conclusion
  • References

Knowls

  1. Knowl 1 — Formal Definitions of Transfer Learning and Deep Transfer Learning

    definition

    In transfer learning, domains and tasks are defined as follows:

    • A domain D={χ,P(X)}\mathcal{D} = \{\chi, P(X)\} comprises a feature space χ\chi and a marginal probability distribution P(X)P(X) over instances X={x1,…,xn}∈χX = \{x_1, \dots, x_n\} \in \chi.
    • A task T={Y,f(x)}\mathcal{T} = \{\mathcal{Y}, f(x)\} comprises a label space Y\mathcal{Y} and a target predictive function f(x)f(x), which can be interpreted as a conditional probability distribution P(y∣x)P(y|x) for y∈Yy \in \mathcal{Y} and x∈χx \in \chi.

    Transfer Learning: Given a learning task Tt\mathcal{T}_t based on target domain Dt\mathcal{D}_t and auxiliary data from source domain Ds\mathcal{D}_s with task Ts\mathcal{T}_s, transfer learning aims to improve the performance of the target prediction function fT(⋅)f_T(\cdot) for Tt\mathcal{T}_t by transferring latent knowledge from Ds\mathcal{D}_s and Ts\mathcal{T}_s, where Ds≠Dt\mathcal{D}_s \neq \mathcal{D}_t and/or Ts≠Tt\mathcal{T}_s \neq \mathcal{T}_t. Typically, the source sample size exceeds the target sample size (Ns=∣Ds∣≫Nt=∣Dt∣N_s = |\mathcal{D}_s| \gg N_t = |\mathcal{D}_t|).

    Deep Transfer Learning: Given a transfer learning task characterized by ⟨Ds,Ts,Dt,Tt,fT(⋅)⟩\langle \mathcal{D}_s, \mathcal{T}_s, \mathcal{D}_t, \mathcal{T}_t, f_T(\cdot) \rangle, it is defined as a deep transfer learning task when the predictive function fT(⋅)f_T(\cdot) is a non-linear function parameterized by a deep neural network.

  2. Knowl 2 — Taxonomy of Deep Transfer Learning Approaches

    model/method

    Deep transfer learning methods are classified into four foundational categories based on the underlying transfer mechanism:

    Category Brief Description Representative Approaches
    Instances-based Selects and reweights source domain instances to serve as supplemental training data for the target domain. TrAdaBoost, TaskTrAdaBoost, ExpBoost.R2, BIW, Metric TL
    Mapping-based Maps source and target domain instances into a shared latent space where cross-domain distribution discrepancy is minimized. MMD, MK-MMD, JMMD, Deep Domain Confusion, Wasserstein distance
    Network-based Reuses a partial neural network pre-trained on a large source domain (architecture and weights) as part of the target model. ImageNet pre-trained CNNs, fine-tuning, residual transfer, multi-scale sparse coding
    Adversarial-based Employs adversarial objectives to learn feature representations that are discriminative for the primary task but domain-invariant. DANN, ADDA, Gradient Reversal Layer, multi-linear adversarial nets

    In complex practical applications, these four technical paradigms are frequently combined.

  3. Knowl 3 — Instances-Based Deep Transfer Learning

    model/method

    Instances-based deep transfer learning employs a sample weight adjustment strategy to select and weight partial source domain instances, incorporating them into the target domain training set.

    • Foundational Assumption: "Although there are differences between two domains, partial instances in the source domain can be utilized by the target domain with appropriate weights."
    • Mechanism: Instances in the source domain that share strong similarity with the target domain distribution receive higher positive weights and are integrated into training, while dissimilar source instances are downweighted or discarded.
    • Representative Techniques:
      • TrAdaBoost and TaskTrAdaBoost: Adapt AdaBoost boosting principles to iteratively lower weights of source instances that disagree with target labels, enabling rapid retraining on new targets while bounding cross-domain weighted training error.
      • ExpBoost.R2 and TrAdaBoost.R2: Adapt the boosting instance-weighting approach to regression tasks across domains.
      • Bi-Weighting Domain Adaptation (BIW): Aligns source and target feature spaces into a shared coordinate system prior to assigning instance weights.
      • Metric and Ensemble Transfer: Jointly optimizes instance weights with cross-domain distance metrics or incorporates instance reweighting within deep ensemble networks.

  4. Knowl 4 — Mapping-Based Deep Transfer Learning and Maximum Mean Discrepancy

    model/method

    Mapping-based deep transfer learning maps instances from both the source domain Ds\mathcal{D}_s and target domain Dt\mathcal{D}_t into a shared representation space where the two domain distributions become closely aligned.

    • Foundational Assumption: "Although there are differences between two origin domains, they can be more similar in an elaborate new data space."
    • Maximum Mean Discrepancy (MMD): To align marginal distributions in deep networks, domain discrepancy is quantified via MMD in a reproducing kernel Hilbert space (RKHS) with feature map ϕ(⋅)\phi(\cdot): DMMD(XS,XT)=∥1∣XS∣∑xs∈XSϕ(xs)−1∣XT∣∑xt∈XTϕ(xt)∥D_{\text{MMD}}(X_S, X_T) = \left\| \frac{1}{|X_S|} \sum_{x_s \in X_S} \phi(x_s) - \frac{1}{|X_T|} \sum_{x_t \in X_T} \phi(x_t) \right\| where XSX_S and XTX_T denote sample sets from the source and target domains, respectively.
    • Objective Loss Function: Deep domain confusion models jointly optimize labeled classification loss and MMD domain alignment loss: L=LC(XL,y)+λDMMD2(XS,XT)\mathcal{L} = \mathcal{L}_C(X_L, y) + \lambda D^2_{\text{MMD}}(X_S, X_T) where LC(XL,y)\mathcal{L}_C(X_L, y) is the cross-entropy loss on labeled data XLX_L, and λ>0\lambda > 0 is a domain confusion trade-off coefficient.
    • Extensions:
      • Multiple Kernel MMD (MK-MMD): Optimizes a linear combination of kernels in RKHS across task-specific CNN layers.
      • Joint Maximum Mean Discrepancy (JMMD): Aligns joint distributions across multiple network hidden layers.
      • Wasserstein Distance: Employs optimal transport metrics between domain distributions to guide deep feature projection.

  5. Knowl 5 — Network-Based Deep Transfer Learning

    model/method

    Network-based deep transfer learning transfers sub-structures and pre-trained connection weights of a neural network trained on a source domain directly into the network intended for the target domain.

    • Foundational Assumption: "Neural network is similar to the processing mechanism of the human brain, and it is an iterative and continuous abstraction process. The front-layers of the network can be treated as a feature extractor, and the extracted features are versatile."
    • Operational Workflow:
      1. Train a deep neural network on a large-scale source dataset (e.g., ImageNet classification).
      2. Extract the front and intermediate layers to act as generic feature extractors for the target task.
      3. Replace the final output classification or regression layers with target-specific layers.
      4. Fine-tune the entire target network or a subset of its layers using available target domain data, or leave transferred front layers frozen.
    • Architectural Transferability: Specific network modules influence cross-domain transferability independently of in-domain accuracy. Standard architectures including LeNet, AlexNet, VGG, Inception, and ResNet provide strong base transferability for visual and perceptual tasks.

  6. Knowl 6 — Adversarial-Based Deep Transfer Learning

    model/method

    Adversarial-based deep transfer learning introduces domain discrimination objectives inspired by Generative Adversarial Networks (GANs) to discover feature representations that are transferable across domains.

    • Foundational Assumption: "For effective transfer, good representation should be discriminative for the main learning task and indiscriminate between the source domain and target domain."
    • Structure and Optimization: The network consists of a deep feature extractor, a task prediction classifier, and an adversarial domain discriminator layer. The total joint loss is formalized as: LD=Lc+λLadver\mathcal{L}_D = \mathcal{L}_c + \lambda \mathcal{L}_{\text{adver}} where Lc\mathcal{L}_c is the task classification loss on labeled data, Ladver\mathcal{L}_{\text{adver}} is the adversarial domain loss, and λ\lambda controls the trade-off.
    • Adversarial Training: The domain discriminator attempts to distinguish whether extracted feature vectors originate from the source domain or the target domain. Concurrently, the feature extractor is optimized (via gradient reversal layers or alternating minimax optimization) to maximize domain confusion, forcing intermediate representations to become domain-invariant while preserving task discriminability.
    • Advanced Paradigms: Include Adversarial Discriminative Domain Adaptation (ADDA), multi-linear randomized adversarial networks spanning multiple intermediate layers, and domain-adversarial metric embeddings.

  7. Knowl 7 — Open Challenges in Deep Transfer Learning

    limitation

    Several fundamental theoretical and practical challenges remain unresolved in deep transfer learning:

    1. Negative Transfer: When domain divergence is large or unrelated knowledge is transferred, target task performance can degrade relative to non-transfer baselines. Mechanisms to detect and mitigate negative transfer in deep architectures are an open research problem.
    2. Transferability Measurement: There is a lack of rigorous, generalizable metrics to quantify transferability between arbitrary domain pairs and deep network architectures before training.
    3. Unsupervised and Semi-Supervised Knowledge Transfer: The majority of existing works focus on supervised transfer; transferring deep representations effectively with minimal or zero target labels across heterogeneous tasks remains challenging.
    4. Theoretical and Physical Foundations: Deep transfer learning remains predominantly empirical; formalizing the mathematical, statistical, physical, and neuroscientific principles underpinning cross-domain feature reuse in deep non-linear networks requires further cross-disciplinary study.

Coverage note — None was omitted; all core definitions, taxonomy categories, mathematical formulations (MMD, adversarial loss), and survey conclusions/challenges were fully covered.

References

  1. 1.Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M.: Domain-adversarial neural networks. arXiv preprint arXiv:1412.4446 (2014)
  2. 2.Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein gan. arXiv preprint arXiv:1701.07875 (2017)
  3. 3.Chang, H., Han, J., Zhong, C., Snijders, A., Mao, J.H.: Unsupervised transfer learning via multi-scale convolutional sparse coding for biomedical applications. IEEE transactions on pattern analysis and machine intelligence (2017)
  4. 4.Dai, W., Yang, Q., Xue, G.R., Yu, Y.: Boosting for transfer learning. In: Proceedings of the 24th international conference on Machine learning. pp. 193–200. ACM (2007)
  5. 5.Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation. arXiv preprint arXiv:1409.7495 (2014)
  6. 6.George, D., Shen, H., Huerta, E.: Deep transfer learning: A new deep learning glitch classification method for advanced ligo. arXiv preprint arXiv:1706.07446 (2017)
  7. 7.Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in neural information processing systems. pp. 2672–2680 (2014)
  8. 8.Gretton, A., Sejdinovic, D., Strathmann, H., Balakrishnan, S., Pontil, M., Fukumizu, K., Sriperumbudur, B.K.: Optimal kernel choice for large-scale two-sample tests. In: Advances in neural information processing systems. pp. 1205–1213 (2012)
  9. 9.Huang, J.T., Li, J., Yu, D., Deng, L., Gong, Y.: Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers. In: Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. pp. 7304–7308. IEEE (2013)
  10. 10.Li, N., Hao, H., Gu, Q., Wang, D., Hu, X.: A transfer learning method for automatic identification of sandstone microscopic images. Computers & Geosciences 103, 111–121 (2017)
  11. 11.Liu, X., Liu, Z., Wang, G., Cai, Z., Zhang, H.: Ensemble transfer learning algorithm. IEEE Access 6, 2389–2396 (2018)
  12. 12.Long, M., Cao, Y., Wang, J., Jordan, M.: Learning transferable features with deep adaptation networks. In: International Conference on Machine Learning. pp. 97–105 (2015)
  13. 13.Long, M., Cao, Z., Wang, J., Jordan, M.I.: Domain adaptation with randomized multilinear adversarial networks. arXiv preprint arXiv:1705.10667 (2017)
  14. 14.Long, M., Wang, J., Jordan, M.I.: Deep transfer learning with joint adaptation networks. arXiv preprint arXiv:1605.06636 (2016)
  15. 15.Long, M., Zhu, H., Wang, J., Jordan, M.I.: Unsupervised domain adaptation with residual transfer networks. In: Advances in Neural Information Processing Systems. pp. 136–144 (2016)
  16. 16.Luo, Z., Zou, Y., Hoffman, J., Fei-Fei, L.F.: Label efficient learning of transferable representations acrosss domains and tasks. In: Advances in Neural Information Processing Systems. pp. 164–176 (2017)
  17. 17.Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and transferring mid-level image representations using convolutional neural networks. In: Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on. pp. 1717–1724. IEEE (2014)
  18. 18.Pan, S.J., Tsang, I.W., Kwok, J.T., Yang, Q.: Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks 22(2), 199–210 (2011)
  19. 19.Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Transactions on knowledge and data engineering 22(10), 1345–1359 (2010)
  20. 20.Pardoe, D., Stone, P.: Boosting for regression transfer. In: Proceedings of the 27th International Conference on International Conference on Machine Learning. pp. 863–870. Omnipress (2010)
  21. 21.Tzeng, E., Hoffman, J., Darrell, T., Saenko, K.: Simultaneous deep transfer across domains and tasks. In: Computer Vision (ICCV), 2015 IEEE International Conference on. pp. 4068–4076. IEEE (2015)
  22. 22.Tzeng, E., Hoffman, J., Saenko, K., Darrell, T.: Adversarial discriminative domain adaptation. In: Computer Vision and Pattern Recognition (CVPR). vol. 1, p. 4 (2017)
  23. 23.Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., Darrell, T.: Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474 (2014)
  24. 24.Wan, C., Pan, R., Li, J.: Bi-weighting domain adaptation for cross-language text classification. In: IJCAI Proceedings-International Joint Conference on Artificial Intelligence. vol. 22, p. 1535 (2011)
  25. 25.Weiss, K., Khoshgoftaar, T.M., Wang, D.: A survey of transfer learning. Journal of Big Data 3(1), 9 (2016)
  26. 26.Xu, Y., Pan, S.J., Xiong, H., Wu, Q., Luo, R., Min, H., Song, H.: A unified framework for metric transfer learning. IEEE Transactions on Knowledge and Data Engineering 29(6), 1158–1171 (2017)
  27. 27.Yao, Y., Doretto, G.: Boosting for transfer learning with multiple sources. In: Computer vision and pattern recognition (CVPR), 2010 IEEE conference on. pp. 1855–1862. IEEE (2010)
  28. 28.Yosinski, J., Clune, J., Bengio, Y., Lipson, H.: How transferable are features in deep neural networks? In: Advances in neural information processing systems. pp. 3320–3328 (2014)
  29. 29.Zhang, J., Li, W., Ogunbona, P.: Joint geometrical and statistical alignment for visual domain adaptation. In: CVPR (2017)
  30. 30.Zhu, H., Long, M., Wang, J., Cao, Y.: Deep hashing network for efficient similarity retrieval. In: AAAI. pp. 2415–2421 (2016)

Citation

MLA
Tan, C., et al. “A Survey on Deep Transfer Learning”. Lecture Notes in Computer Science, Springer International Publishing, 2018, pp. 270–79, https://doi.org/10.1007/978-3-030-01424-7_27.
APA
Tan, C., Sun, F., Kong, T., Zhang, W., Yang, C., & Liu, C. (2018). A Survey on Deep Transfer Learning. In Lecture Notes in Computer Science (pp. 270–279). Springer International Publishing. https://doi.org/10.1007/978-3-030-01424-7_27
Chicago
Tan, C., F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu. 2018. “A Survey on Deep Transfer Learning”. In Lecture Notes in Computer Science. Springer International Publishing. https://doi.org/10.1007/978-3-030-01424-7_27.
Harvard
Tan, C. et al. (2018) “A Survey on Deep Transfer Learning”, Lecture Notes in Computer Science. Springer International Publishing, pp. 270–279. Available at: https://doi.org/10.1007/978-3-030-01424-7_27.
Vancouver
1. Tan C, Sun F, Kong T, Zhang W, Yang C, Liu C (2018) A Survey on Deep Transfer Learning. In: Lecture Notes in Computer Science. Springer International Publishing, pp 270–279

BibTeX

@inbook{Tan_2018, title={A Survey on Deep Transfer Learning}, ISBN={9783030014247}, ISSN={1611-3349}, url={http://dx.doi.org/10.1007/978-3-030-01424-7_27}, DOI={10.1007/978-3-030-01424-7_27}, booktitle={Artificial Neural Networks and Machine Learning – ICANN 2018}, publisher={Springer International Publishing}, author={Tan, Chuanqi and Sun, Fuchun and Kong, Tao and Zhang, Wenchang and Yang, Chao and Liu, Chunfang}, year={2018}, pages={270–279} }
Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF