Learning to Discover Cross-Domain Relations with Generative Adversarial Networks

Taeksoo KimMoonsu ChaHyunsoo KimJung Kwon LeeJiwon Kim

article2017ICML2,105 citations

Introduces DiscoGAN, an unsupervised generative adversarial network framework that discovers cross-domain relations from unpaired data to perform style transfer while maintaining key structural attributes and object identity.

Listen

Transferring attributes, styles, or concepts across different visual domains typically requires large datasets of explicitly paired and human-annotated examples. In practical applications, acquiring such paired training data is costly, labor-intensive, or infeasible when direct one-to-one correspondences do not exist naturally. The article evaluates a framework to automatically discover relations between distinct visual domains and translate images between them without any paired supervision or pre-training.

The authors develop and test DiscoGAN, a generative framework based on Generative Adversarial Networks (GANs—machine learning models where a generator creates synthetic data and a discriminator evaluates its realism). The architecture couples two generators and two discriminators simultaneously. It enforces a two-way reconstruction constraint ensuring that an image translated from domain A into domain B can be successfully translated back into the original image in domain A, and vice versa. This formulation establishes a bijective (one-to-one) mapping across datasets. The authors benchmark DiscoGAN against standard GANs and single-direction reconstruction models using synthetic two-dimensional datasets and multiple real-world image datasets, including 3D car models, human faces (CelebA and Face datasets), chairs, edges, shoes, and handbags.

The evaluations yielded several key findings. First, DiscoGAN resolves the common "mode collapse" failure, where baseline models map diverse inputs into a small cluster of identical outputs; DiscoGAN successfully mapped into all target modes across both synthetic tests and azimuth rotation benchmarks for cars and faces. Second, the model successfully isolated and modified specific facial attributes—such as changing gender, hair color, or adding eyeglasses—while preserving core identity features and backgrounds, even across sequential and repeated transformations. Third, the system maintained cross-category structural features, accurately aligning orientation when translating between completely different object classes, such as mapping 3D chairs to cars and cars to faces. Finally, the framework discovered abstract, unannotated relationships between dissimilar product domains, translating handbags to shoes while matching color, texture pattern, and degree of formal fashion style.

These findings indicate that generative vision models can learn complex, high-level cross-domain mappings entirely unsupervised. For organizations utilizing visual AI, this approach substantially reduces the operational cost, labor, and timeline risks associated with manual data labeling. It also demonstrates that bidirectional cycle constraints provide stable mathematical regularization against mode collapse in generative modeling.

Based on these results, engineering teams exploring image-to-image translation, digital asset transformation, or style synthesis should consider bidirectional cycle consistency architectures to circumvent paired data bottlenecks. The authors recommend extending the DiscoGAN framework to cross-modal tasks, such as translating between text descriptions and images. Readers should note that the article's empirical validation was conducted on relatively low-resolution images (64 by 64 pixels) under controlled laboratory conditions, and performance on complex, high-resolution production environments will require further testing.

SK T-Brain generative adversarial networks network architecture source codes cross-domain relations generative adversarial networks DiscoGAN model

arXiv: 1703.05192 SKTBrain/DiscoGAN

Paper: Generative Adversarial Networks, Ian J. Goodfellow et al. (2014). Understanding the foundational generative adversarial network framework is essential before exploring multi-domain translation extensions like DiscoGAN.
Paper: Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks, Jun-Yan Zhu et al. (2017). Reviewing CycleGAN provides immediate context for unpaired cross-domain mapping and cycle-consistency principles relied upon by the source.

Paper: StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation, Yunjey Choi et al. (2018). StarGAN extends the source's pairwise cross-domain concept into a unified multi-domain image-to-image translation framework.
Paper: Multimodal Unsupervised Image-to-Image Translation, Xun Huang et al. (2018). Multimodal Unsupervised Image-to-Image Translation builds directly on unpaired translation methods like DiscoGAN to generate diverse, multi-modal outputs.

Cover for Learning to Discover Cross-Domain Relations with Generative Adversarial Networks

Abstract

While humans easily recognize relations between data from different domains without any supervision, learning to automatically discover them is in general very challenging and needs many ground-truth pairs that illustrate the relations. To avoid costly pairing, we address the task of discovering cross-domain relations given unpaired data. We propose a method based on generative adversarial networks that learns to discover relations between different domains (DiscoGAN). Using the discovered relations, our proposed network successfully transfers style from one domain to another while preserving key attributes such as orientation and face identity. Source code for official implementation is publicly available this https URL

Topics

generative adversarial networks network architecture source codes

Keywords

cross-domain relations generative adversarial networks DiscoGAN model image translation image synthesis explicit pair labels

1 Introduction
2 Model
2.1 Formulation
2.2 Notation and Architecture
2.3 GAN with a Reconstruction Loss
2.4 Our Proposed Model: Discovery GAN
3 Experiments
3.1 Toy Experiment
3.2 Real Domain Experiment
3.2.1 Car to Car, Face to Face
3.2.2 Face Conversion
3.2.3 Chair to Car, Car to Face
3.2.4 Edges-to-Photos
3.2.5 Handbag to Shoes, Shoes to Handbag
4 Related Work
5 Conclusion
References

Knowls

Knowl 1 — Bijective Cross-Domain Relation Learning Formulation
definition
Cross-domain relation discovery between two unpaired datasets from domains $A$ and $B$ is formulated as learning a pair of mapping functions: $G_{AB}: A \to B$ and $G_{BA}: B \to A$ . In an unsupervised setting without paired supervision, a meaningful cross-domain relation is defined as a one-to-one correspondence (a bijective mapping), where $G_{AB}$ is the inverse mapping of $G_{BA}$ .

Because the exact bijection equality $G_{BA} \circ G_{AB}(x_A) = x_A$ for all $x_A \in A$ and $G_{AB} \circ G_{BA}(x_B) = x_B$ for all $x_B \in B$ is difficult to optimize directly, it is relaxed into two soft constraints during training:
1. Reconstruction consistency: Minimizing the distance metric $d(G_{BA}(G_{AB}(x_A)), x_A)$ and $d(G_{AB}(G_{BA}(x_B)), x_B)$ , where $d$ can be mean squared error ( $L_2$ ), mean absolute error ( $L_1$ ), Huber loss, or cosine distance.
2. Domain distribution matching: Ensuring that $G_{AB}(x_A)$ falls into the data distribution of domain $B$ and $G_{BA}(x_B)$ falls into domain $A$ by minimizing generative adversarial losses.
Knowl 2 — DiscoGAN Generator and Discriminator Loss Formulation
equation
The DiscoGAN framework couples two generative adversarial networks ( $A \to B \to A$ and $B \to A \to B$ ) that share parameters across identical mapping directions. The full objective optimizes two generators ( $G_{AB}$ and $G_{BA}$ ) and two discriminators ( $D_A$ and $D_B$ ) using the total generator loss $\mathcal{L}_G$ and total discriminator loss $\mathcal{L}_D$ :

$\mathcal{L}_G = \mathcal{L}_{G_{AB}} + \mathcal{L}_{G_{BA}} = \mathcal{L}_{\text{GAN}_B} + \mathcal{L}_{\text{CONST}_A} + \mathcal{L}_{\text{GAN}_A} + \mathcal{L}_{\text{CONST}_B}$

$\mathcal{L}_D = \mathcal{L}_{D_A} + \mathcal{L}_{D_B}$

For data distributions $P_A$ and $P_B$ over domains $A$ and $B$ , the individual generator components are:

$\mathcal{L}_{\text{GAN}_B} = -\mathbb{E}_{x_A \sim P_A}[\log D_B(G_{AB}(x_A))]$

$\mathcal{L}_{\text{GAN}_A} = -\mathbb{E}_{x_B \sim P_B}[\log D_A(G_{BA}(x_B))]$

$\mathcal{L}_{\text{CONST}_A} = \mathbb{E}_{x_A \sim P_A}[d(G_{BA}(G_{AB}(x_A)), x_A)]$

$\mathcal{L}_{\text{CONST}_B} = \mathbb{E}_{x_B \sim P_B}[d(G_{AB}(G_{BA}(x_B)), x_B)]$

where $d$ is a distance metric. The discriminator loss terms are:

$\mathcal{L}_{D_A} = -\mathbb{E}_{x_A \sim P_A}[\log D_A(x_A)] - \mathbb{E}_{x_B \sim P_B}[\log(1 - D_A(G_{BA}(x_B)))]$

$\mathcal{L}_{D_B} = -\mathbb{E}_{x_B \sim P_B}[\log D_B(x_B)] - \mathbb{E}_{x_A \sim P_A}[\log(1 - D_B(G_{AB}(x_A)))]$
Knowl 3 — Mode Collapse in Asymmetric versus Symmetric Reconstruction GANs
theoretical result
In unpaired domain translation between multimodal distributions $A$ and $B$ , standard Generative Adversarial Networks suffer from mode collapse, mapping multiple distinct modes of $A$ into a single mode of $B$ .

Introducing an asymmetric, one-way reconstruction constraint $\mathcal{L}_{\text{CONST}_A} = d(G_{BA}(G_{AB}(x_A)), x_A)$ encourages the reconstruction of domain $A$ samples, but because domain $B$ is not constrained to reconstruct back to $B$ , the learned mapping remains an injection rather than a bijection. When two modes in $A$ map to a single mode in $B$ , the backward generator $G_{BA}$ can only map that single $B$ mode to one of the two $A$ modes; this produces optimization oscillations between states and fails to resolve mode collapse.

Enforcing symmetric cycle-consistency via two simultaneous reconstruction constraints ( $\mathcal{L}_{\text{CONST}_A}$ and $\mathcal{L}_{\text{CONST}_B}$ ) alongside two adversarial losses ( $\mathcal{L}_{\text{GAN}_A}$ and $\mathcal{L}_{\text{GAN}_B}$ ) constrains the system to a bijective correspondence, which prevents many-to-one mapping collapses across both domains.
Knowl 4 — DiscoGAN Architecture and Training Hyperparameters
experimental setup
The image-to-image translation architecture and training configuration for DiscoGAN operate on $64 \times 64 \times 3$ images:
- Generator: Encoder-decoder architecture. The encoder consists of 4 to 5 convolutional layers with $4 \times 4$ filter kernels, each followed by LeakyReLU activations. The decoder consists of 4 to 5 deconvolutional layers with $4 \times 4$ filter kernels, each followed by ReLU activations.
- Discriminator: Matches the generator encoder with 4 to 5 convolutional layers ( $4 \times 4$ kernels) and LeakyReLU activations, followed by an additional convolutional layer with $4 \times 4$ filters and a final sigmoid activation producing a scalar output in $[0, 1]$ .
- Normalization and Regularization: Batch Normalization is applied to all convolutional and deconvolutional layers except the input and final output layers. The $L_2$ weight decay regularization coefficient is $10^{-4}$ .
- Optimization: Adam optimizer with learning rate $\alpha = 0.0002$ , momentum decay parameters $\beta_1 = 0.5$ and $\beta_2 = 0.999$ , trained with a minibatch size of 200.
Knowl 5 — Mode Collapse Prevention in 2D Gaussian Mixture Mapping
empirical result
In a synthetic 2D experiment where source domain $A$ contains 5 Gaussian mixture modes and target domain $B$ contains 10 Gaussian mixture modes distributed along a circular arc:
- Standard GAN: Translated points from multiple source modes map onto the same target mode (e.g., navy and light-blue points overlap, green and orange points overlap). Regions around target modes form a single plateau in the discriminator output, causing translated samples to oscillate and fail to cover all target modes.
- GAN with One-Way Reconstruction Loss: Points from different source modes still overlap at a subset of target modes, failing to cover all 10 target modes because the mapping is purely injective.
- DiscoGAN: Translates all 5 source modes into distinct, well-bounded, non-overlapping regions in target domain $B$ while generating samples across target modes and fooling discriminator $D_B$ .
Knowl 6 — Unpaired Viewpoint Alignment and Azimuth Regression in Car Images
empirical result
On a dataset of 3D rendered car models varying in azimuth angles from $-75^\circ$ to $+75^\circ$ at $15^\circ$ increments, an independent regressor was trained on the training set to predict the azimuth angle of input and translated test images:
- A Standard GAN achieves a root-mean-square error (RMSE) of $0.16$ relative to ground-truth angle correspondence, clustering predictions around a few fixed angles due to mode collapse.
- A GAN with one-way reconstruction loss achieves an RMSE of $0.18$ , similarly clustering outputs into a small number of discrete azimuth angles.
- DiscoGAN achieves an RMSE of $0.11$ , demonstrating a strong linear correlation between input and translated azimuth angles and discovering the continuous rotational relation (or its mirror opposite) without angle supervision.
Knowl 7 — Continuous Azimuth Angle Discovery in 3D Face Images
empirical result
When translating between unpaired sets of 3D face images with azimuth rotations varying between $-90^\circ$ and $+90^\circ$ :
- Standard GAN and GAN with a one-way reconstruction loss fail to preserve rotation variance, collapsing generated outputs to narrow ranges of face orientation regardless of input angle.
- DiscoGAN discovers the cross-domain orientation correspondence without supervision, generating output faces whose azimuth angles vary continuously across the full range (either preserving the $-90^\circ$ to $+90^\circ$ range or consistently inverting it to $+90^\circ$ to $-90^\circ$ ).
Knowl 8 — Unpaired Facial Attribute Manipulation and Invariance
empirical result
On CelebA and FaceScrub datasets where domain pairs share most facial characteristics but differ in a specific attribute (such as gender, hair color between blond and black, or wearing eyeglasses):
- DiscoGAN modifies the target attribute (e.g., converting male to female, blond to black hair, or adding/removing eyeglasses) while preserving facial identity, facial structure, expression, and background details.
- Sequential attribute translation (e.g., altering gender followed by altering hair color) and repeated applications of the same transformation (such as alternating gender flips) retain image quality and consistency without compounding distortion.
Knowl 9 — Unsupervised Alignment of Shared Latent Rotation Across Visually Disparate Categories
empirical result
When trained on unpaired datasets from visually distinct object categories that share only a single underlying degree of freedom—specifically Chair-to-Car and Car-to-Face translations using 3D rendered models varying by azimuth angle:
- DiscoGAN automatically identifies and aligns the shared azimuth rotation angle across categories.
- The generated cars and faces adopt the 3D orientation of the input chairs and cars while fully adopting the texture, geometry, and visual style of the target category.
Knowl 10 — Unpaired Edges-to-Photos Translation and Cross-Product Style Discovery
empirical result
In unpaired image translation tasks on handbags and shoes:
- Edges-to-Photos (1-to- $N$ translation): When trained on unpaired black-and-white edge sketches and color photographs, DiscoGAN successfully generates realistic, colored photos from sketches and sharp sketches from color photos for both handbags and shoes.
- Handbag-to-Shoe Cross-Domain Translation: Without predefined attribute labels or explicit pairing, DiscoGAN discovers implicit fashion style relations, generating shoes from handbags (and vice versa) that match the color palette, material texture, pattern, and formality level of the input item.

Coverage note — None was omitted; all core theoretical models, loss formulations, architectural configurations, and empirical experiments across 2D synthetic data, car/face azimuth alignments, facial attribute manipulations, cross-category alignments, and fashion translations are fully covered.

References

1.Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein GAN. In arXiv preprint arXiv:1701.07875, 2017.
2.Aubry, M., Maturana, D., Efros, A. A., Russell, B., and Sivic, J. Seeing 3d chairs: Exemplar part-based 2d-3d alignment using a large dataset of cad models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
3.Denton, E. L., Chintala, S., Szlam, A., and Fergus, R. Deep generative image models using a laplacian pyramid of adversarial networks. In Advances in Neural Information Processing Systems (NIPS), 2015.
4.Dosovitskiy, A., Springenberg, J. T., and Brox, T. Learning to generate chairs with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
5.Fidler, S., Dickinson, S., and Urtasun, R. 3d object detection and viewpoint estimation with a deformable 3d cuboid model. In Advances in Neural Information Processing Systems (NIPS), 2012.
6.Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), 2014.
7.Goodfellow, Ian J. NIPS 2016 tutorial: Generative adversarial networks. In arXiv preprint arXiv:1701.00160, 2017.
8.Ioffe, S. and Szegedy, C. Batch normalization: Accerlerating deep network training by reducing internal covariate shift. In arXiv preprint arXiv:1502.03167, 2015.
9.Isola, P., Zhu, J., Zhou, T., and Efros, A. A. Image-to-image translation with conditional adversarial networks. In arXiv preprint arXiv:1611.07004, 2016.
10.Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2015.
11.Liu, M. and Tuzel, O. Coupled generative adversarial networks. In Advances in Neural Information Processing Systems (NIPS), 2016.
12.Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015.
13.Maas, A. L., Hannun, A. Y., and Ng, A. Y. Rectifier nonlinearities improve neural network acoustic models. In Proceedings of the 30th International Conference on Machine Learning (ICML), 2013.
14.Mirza, M. and Osindero, S. Conditional generative adversarial nets. In arXiv preprint arXiv:1411.1784, 2014.
15.Paysan, P., Knothe, R., Amberg, B., Romdhani, S., and Vetter, T. A 3d face model for pose and illumination invariant face recognition. In Proceedings of the 6th IEEE International Conference on Advanced Video and Signal based Surveillance (AVSS) for Security, Safety and Monitoring in Smart Environments, 2009.
16.Perarnau, G., van de Weijer, J., Raducanu, B., and Álvarez, J. M. Invertible conditional gans for image editing. In arXiv preprint arXiv:1611.06355, 2016.
17.Radford, A., Metz, L., and Chintala, S. Unsupervised representation learning with deep convolutional generative adversarial networks. In Proceedings of the 4th International Conference on Learning Representations (ICLR), 2016.
18.Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., and Lee, H. Generative adversarial text to image synthesis. In Proceedings of the 33rd International Conference on Machine Learning (ICML), 2016.
19.Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. In Advances in Neural Information Processing Systems (NIPS), 2016.
20.T. Kulkarni, W. Whitney and P. Kohli, J. Tenenbaum. Deep convolutional inverse graphics network. In Advances in Neural Information Processing Systems (NIPS), 2015.
21.Taigman, Y., Polyak, A., and Wolf, L. Unsupervised cross-domain image generation. In arXiv preprint arXiv:1611.02200, 2016.
22.Tong, C., Li, Y., Jacob, A. P., Bengio, Y., and Li, W. Mode regularized generative adversarial networks. In Proceedings of the 5rd International Conference on Learning Representations (ICLR), 2017.
23.Wang, X. and Gupta, A. Generative image modeling using style and structure adversarial networks. In European Conference on Computer Vision (ECCV), 2016.
24.Xu, B., Wang, N., T., Chen, and Li, M. Empirical evaluation of rectified activations in convolutional network. In arXiv preprint arXiv:1505:00853, 2015.
25.Yu, A. and Grauman, K. Fine-grained visual comparisons with local learning. In Computer Vision and Pattern Recognition (CVPR), June 2014.
26.Zhang, H., Xu, Tao., Li, H., Zhang, S., Huang, X., Wang, X., and Metaxas, D. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In arXiv preprint arXiv:1612.03242, 2016.
27.Zhu, Jun-Yan, Krähenbühl, Philipp, Shechtman, Eli, and Efros, Alexei A. Generative visual manipulation on the natural image manifold. In Proceedings of European Conference on Computer Vision (ECCV), 2016.

Citation

MLA

Kim, T., et al. “Learning to Discover Cross-Domain Relations with Generative Adversarial Networks”. arXiv, 2017, http://arxiv.org/abs/1703.05192v2.

APA

Kim, T., Cha, M., Kim, H., Lee, J. K., & Kim, J. (2017). Learning to Discover Cross-Domain Relations with Generative Adversarial Networks. arXiv. http://arxiv.org/abs/1703.05192v2

Chicago

Kim, T., M. Cha, H. Kim, J. K. Lee, and J. Kim. 2017. “Learning to Discover Cross-Domain Relations with Generative Adversarial Networks”. arXiv. http://arxiv.org/abs/1703.05192v2.

Harvard

Kim, T. et al. (2017) “Learning to Discover Cross-Domain Relations with Generative Adversarial Networks”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1703.05192v2.

Vancouver

1. Kim T, Cha M, Kim H, Lee JK, Kim J (2017) Learning to Discover Cross-Domain Relations with Generative Adversarial Networks. arXiv

BibTeX

@article{kim2017learning,
  title = {Learning to Discover Cross-Domain Relations with Generative Adversarial Networks},
  author = {Kim, Taeksoo and Cha, Moonsu and Kim, Hyunsoo and Lee, Jung Kwon and Kim, Jiwon},
  year = {2017},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1703.05192v2},
  eprint = {1703.05192}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Learning to Discover Cross-Domain Relations with Generative Adversarial Networks

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Bijective Cross-Domain Relation Learning Formulation

Knowl 2 — DiscoGAN Generator and Discriminator Loss Formulation

Knowl 3 — Mode Collapse in Asymmetric versus Symmetric Reconstruction GANs

Knowl 4 — DiscoGAN Architecture and Training Hyperparameters

Knowl 5 — Mode Collapse Prevention in 2D Gaussian Mixture Mapping

Knowl 6 — Unpaired Viewpoint Alignment and Azimuth Regression in Car Images

Knowl 7 — Continuous Azimuth Angle Discovery in 3D Face Images

Knowl 8 — Unpaired Facial Attribute Manipulation and Invariance

Knowl 9 — Unsupervised Alignment of Shared Latent Rotation Across Visually Disparate Categories

Knowl 10 — Unpaired Edges-to-Photos Translation and Cross-Product Style Discovery

References

Citation

BibTeX

Source Code

Access the Paper