Multimodal Unsupervised Image-to-Image Translation

Xun HuangMing-Yu LiuSerge J. BelongieJan Kautz

article2018ECCV2,762 citations

Presents the MUNIT framework, which overcomes the one-to-one mapping limitation of unsupervised image translation by disentangling domain-invariant content from domain-specific style to generate diverse, user-controllable outputs.

Listen

The article addresses the challenge of unsupervised image-to-image translation, where models must map images from a source domain to a target domain without paired examples. Existing methods typically produce only a single deterministic output per input, which fails to capture the inherent multimodal nature of many real-world mappings, such as varying appearances of a scene under different weather or lighting conditions.

The work sets out to create a framework that learns conditional distributions allowing diverse, realistic outputs from a single source image while remaining fully unsupervised.

The approach decomposes image representations into a shared content code across domains and domain-specific style codes. Translation recombines an input's content code with a randomly sampled style code from the target domain. Training relies on adversarial losses to match target distributions and bidirectional reconstruction losses to ensure encoders and decoders act as inverses, with experiments conducted across datasets including edges-to-shoes, animal categories, and street scenes using both quantitative metrics like LPIPS diversity and human preference studies.

The framework produces outputs that are both significantly more diverse and higher quality than prior unsupervised methods such as UNIT and CycleGAN, achieving LPIPS diversity scores several times higher while matching or approaching the performance of the supervised BicycleGAN model. It also supports example-guided translation by extracting style codes from reference images rather than random sampling.

These results indicate that explicit disentanglement of content and style enables practical control over translation variability without requiring paired data, reducing the risk of mode collapse and improving applicability to tasks where multiple valid outputs exist.

Further validation on additional high-resolution or multi-domain scenarios would strengthen deployment decisions, along with analysis of sensitivity to loss weighting hyperparameters.

The primary limitations include dependence on the validity of the content-style decomposition assumption and the need for careful tuning of reconstruction loss weights, with results potentially varying across domains not represented in the tested collections.

Cornell University NVIDIA computer vision deterministic image representations multimodal unsupervised image-to-image translation image-to-image translation generative adversarial nets

arXiv: 1804.04732 nvlabs/MUNIT

Paper: Unsupervised Image-to-Image Translation Networks, Ming-Yu Liu et al. (2017). Establishes the shared-latent-space assumption for unsupervised image-to-image translation that MUNIT directly extends by decomposing latent representations into shared content and domain-specific style codes.
Paper: Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks, Jun-Yan Zhu et al. (2017). Introduces cycle-consistent adversarial training for unpaired cross-domain mapping, providing the foundational deterministic baseline and reconstruction principles that MUNIT generalizes for multimodal outputs.
Paper: Image-to-Image Translation with Conditional Adversarial Networks, Phillip Isola et al. (2017). Defines the core conditional generative adversarial framework for image-to-image translation upon which subsequent unpaired and multimodal formulations build.
Paper: Learning to Discover Cross-Domain Relations with Generative Adversarial Networks, Taeksoo Kim et al. (2017). Demonstrates how dual-generator cross-domain reconstruction losses discover relations between unpaired visual collections without supervision.
Paper: beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework, Irina Higgins et al. (2016). Provides fundamental principles for learning factorized, disentangled latent representations in variational autoencoders that motivate MUNIT's content-style decomposition.
Paper: Learning Structured Output Representation using Deep Conditional Generative Models, Kihyuk Sohn et al. (2015). Introduces conditional variational autoencoders to generate structured, diverse outputs for one-to-many visual mapping problems.

Paper: Semantic Image Synthesis With Spatially-Adaptive Normalization, Taesung Park et al. (2019). Builds upon multimodal image synthesis principles by using spatially adaptive normalization to inject semantic layout and style features without erasing spatial detail.
Paper: A Style-Based Generator Architecture for Generative Adversarial Networks, Tero Karras et al. (2019). Extends the disentanglement of style and content to unconditional synthesis by injecting layer-wise style codes via adaptive normalization.
Paper: Palette: Image-to-Image Diffusion Models, Chitwan Saharia et al. (2021). Replaces GAN-based translation frameworks with conditional diffusion models to achieve diverse and stable image-to-image translation across diverse tasks.
Paper: Taming Transformers for High-Resolution Image Synthesis, Patrick Esser et al. (2020). Advances high-resolution multimodal image translation and synthesis by combining discrete perceptual codebooks with autoregressive transformers.
Paper: CyCADA: Cycle-Consistent Adversarial Domain Adaptation, Judy Hoffman et al. (2018). Applies cycle-consistent cross-domain image translation to task-driven unsupervised visual domain adaptation.

Cover for Multimodal Unsupervised Image-to-Image Translation

Abstract

Unsupervised image-to-image translation is an important and challenging problem in computer vision. Given an image in the source domain, the goal is to learn the conditional distribution of corresponding images in the target domain, without seeing any pairs of corresponding images. While this conditional distribution is inherently multimodal, existing approaches make an overly simplified assumption, modeling it as a deterministic one-to-one mapping. As a result, they fail to generate diverse outputs from a given source domain image. To address this limitation, we propose a Multimodal Unsupervised Image-to-image Translation (MUNIT) framework. We assume that the image representation can be decomposed into a content code that is domain-invariant, and a style code that captures domain-specific properties. To translate an image to another domain, we recombine its content code with a random style code sampled from the style space of the target domain. We analyze the proposed framework and establish several theoretical results. Extensive experiments with comparisons to the state-of-the-art approaches further demonstrates the advantage of the proposed framework. Moreover, our framework allows users to control the style of translation outputs by providing an example style image. Code and pretrained models are available at this https URL

Topics

computer vision deterministic image representations supervised machine learning theoretical result unsupervised learning method

Keywords

multimodal unsupervised image-to-image translation image-to-image translation generative adversarial nets conditional GANs style transfer domain adaptation image synthesis adversarial loss perceptual similarity Inception Score

1 Introduction
2 Related Works
3 Multimodal Unsupervised Image-to-image Translation
3.1 Assumptions
3.2 Model
4 Theoretical Analysis
Latent Distribution Matching
Joint Distribution Matching
Style-augmented Cycle Consistency
5 Experiments
5.1 Implementation Details
5.2 Evaluation Metrics
5.3 Baselines
5.4 Datasets
5.5 Results
6 Conclusions
References
0.A Proofs
0.B Training Details
0.B.1 Hyperparameters
0.B.2 Network Architectures
0.C Domain-invariant Perceptual Loss

Knowls

Knowl 1 — Partially Shared Latent Space Assumption
assumption
In unsupervised cross-domain image-to-image translation between domains $\mathcal{X}_1$ and $\mathcal{X}_2$ , each image $x_i \in \mathcal{X}_i$ is assumed to be generated by a deterministic generator $G_i^*(c, s_i)$ from two disentangled latent representations: a domain-invariant content code $c \in \mathcal{C}$ shared between both domains (representing spatial structure), and a domain-specific style code $s_i \in \mathcal{S}_i$ (representing domain rendering properties). Each underlying generator $G_i^*$ is assumed to possess a deterministic inverse encoder $E_i^* = (G_i^*)^{-1}$ .

While encoders and generators are deterministic, the resulting conditional translation distribution $p(x_2|x_1)$ is continuous and multimodal because it depends on the stochastic style variable $s_2$ sampled from the target domain's style space.
Knowl 2 — Multimodal Unsupervised Image-to-Image Translation (MUNIT) Framework
model/method
The MUNIT model consists of an encoder $E_i = (E_i^c, E_i^s)$ and a decoder $G_i$ for each domain $\mathcal{X}_i$ ( $i \in \{1, 2\}$ ), where $E_i^c$ extracts the domain-invariant content code $c_i = E_i^c(x_i)$ and $E_i^s$ extracts the domain-specific style code $s_i = E_i^s(x_i)$ .

Cross-domain translation from $\mathcal{X}_1$ to $\mathcal{X}_2$ is performed by recombining the extracted content code $c_1 = E_1^c(x_1)$ with a target style code $s_2$ via the target decoder:

$x_{1 \to 2} = G_2(c_1, s_2)$

The target style code $s_2$ can be randomly sampled from a standard Gaussian prior $q(s_2) \sim \mathcal{N}(0, \mathbf{I})$ to produce diverse stochastic outputs, or extracted from an exemplar reference image $x_2 \in \mathcal{X}_2$ via $s_2 = E_2^s(x_2)$ for example-guided translation. Translation from $\mathcal{X}_2$ to $\mathcal{X}_1$ operates symmetrically.
Knowl 3 — MUNIT Bidirectional Reconstruction and Adversarial Objectives
equation
The MUNIT model is trained by jointly optimizing encoders $E_1, E_2$ , decoders $G_1, G_2$ , and domain discriminators $D_1, D_2$ with the minimax objective:

$\min_{E_1,E_2,G_1,G_2} \max_{D_1,D_2} \mathcal{L}(E_1, E_2, G_1, G_2, D_1, D_2) = \mathcal{L}_{\text{GAN}}^{x_1} + \mathcal{L}_{\text{GAN}}^{x_2} + \lambda_x (\mathcal{L}_{\text{recon}}^{x_1} + \mathcal{L}_{\text{recon}}^{x_2}) + \lambda_c (\mathcal{L}_{\text{recon}}^{c_1} + \mathcal{L}_{\text{recon}}^{c_2}) + \lambda_s (\mathcal{L}_{\text{recon}}^{s_1} + \mathcal{L}_{\text{recon}}^{s_2})$

where $\lambda_x, \lambda_c, \lambda_s$ are positive weighting hyperparameters.

The individual loss components for domain $\mathcal{X}_1 \to \mathcal{X}_2$ are:

$\mathcal{L}_{\text{recon}}^{x_1} = \mathbb{E}_{x_1 \sim p(x_1)} \left[ \| G_1(E_1^c(x_1), E_1^s(x_1)) - x_1 \|_1 \right]$

$\mathcal{L}_{\text{recon}}^{c_1} = \mathbb{E}_{c_1 \sim p(c_1), s_2 \sim q(s_2)} \left[ \| E_2^c(G_2(c_1, s_2)) - c_1 \|_1 \right]$

$\mathcal{L}_{\text{recon}}^{s_2} = \mathbb{E}_{c_1 \sim p(c_1), s_2 \sim q(s_2)} \left[ \| E_2^s(G_2(c_1, s_2)) - s_2 \|_1 \right]$

$\mathcal{L}_{\text{GAN}}^{x_2} = \mathbb{E}_{c_1 \sim p(c_1), s_2 \sim q(s_2)} [\log(1 - D_2(G_2(c_1, s_2)))] + \mathbb{E}_{x_2 \sim p(x_2)} [\log D_2(x_2)]$

where $q(s_2) \sim \mathcal{N}(0, \mathbf{I})$ and $p(c_1)$ denotes the distribution of $c_1 = E_1^c(x_1)$ under $x_1 \sim p(x_1)$ .
Knowl 4 — Implicit Latent Distribution Matching at Optimality
theoretical result
When the MUNIT total objective reaches global optimality, the encoded latent distributions match their corresponding generation-time distributions without explicit latent-space divergence constraints (such as Kullback-Leibler divergence or adversarial latent matching):

$p(c_1) = p(c_2), \quad p(s_1) = q(s_1), \quad p(s_2) = q(s_2)$

where $p(c_i)$ and $p(s_i)$ denote the empirical distributions of $E_i^c(x_i)$ and $E_i^s(x_i)$ for $x_i \sim p(x_i)$ , and $q(s_i) \sim \mathcal{N}(0, \mathbf{I})$ is the Gaussian prior. Consequently, the content latent space becomes domain-invariant and the encoded style spaces align with the prior distribution.
Knowl 5 — Style-Augmented Cycle Consistency
theoretical result
Standard cycle consistency on image spaces forces cross-domain translation to degenerate into a deterministic mapping. MUNIT instead satisfies style-augmented cycle consistency over joint image-style spaces $\mathcal{H}_1 = \mathcal{X}_1 \times \mathcal{S}_2$ and $\mathcal{H}_2 = \mathcal{X}_2 \times \mathcal{S}_1$ .

Defining the mapping $F_{1 \to 2}: \mathcal{H}_1 \to \mathcal{H}_2$ by:

$F_{1 \to 2}(x_1, s_2) = \left( G_2(E_1^c(x_1), s_2), \, E_1^s(x_1) \right)$

and $F_{2 \to 1}(x_2, s_1) = (G_1(E_2^c(x_2), s_1), E_2^s(x_2))$ , global optimality under the MUNIT bidirectional reconstruction loss implies:

$F_{1 \to 2} = F_{2 \to 1}^{-1}$

Translating an image $x_1$ to domain $\mathcal{X}_2$ with style $s_2$ , and then translating the generated image back to $\mathcal{X}_1$ using the original extracted style $s_1 = E_1^s(x_1)$ , exactly recovers $(x_1, s_2)$ .
Knowl 6 — Joint Distribution Matching and Global Minima
theoretical result
Let $p(x_1, x_{1 \to 2}) = p(x_1)p(x_{1 \to 2}|x_1)$ and $p(x_{2 \to 1}, x_2) = p(x_2)p(x_{2 \to 1}|x_2)$ be the joint distributions defined by the MUNIT framework.
1. If there exist functions $E_1^*, E_2^*, G_1^*, G_2^*$ such that $E_1^* = (G_1^*)^{-1}$ , $E_2^* = (G_2^*)^{-1}$ , and the translated marginal distributions match the true data distributions ( $p(x_{1 \to 2}) = p(x_2)$ and $p(x_{2 \to 1}) = p(x_1)$ ), then $(E_1^*, E_2^*, G_1^*, G_2^*)$ minimizes $\max_{D_1, D_2} \mathcal{L}(E_1, E_2, G_1, G_2, D_1, D_2)$ .
2. When global optimality is achieved, the induced joint distributions match:
$p(x_1, x_{1 \to 2}) = p(x_{2 \to 1}, x_2)$
Knowl 7 — MUNIT Auto-Encoder Architecture with AdaIN and Instance Normalization
model/method
The MUNIT auto-encoder separates content and style processing across distinct sub-networks:
1. Content Encoder ( $E_i^c$ ): Consists of strided convolutional downsampling layers followed by residual blocks. All convolutional layers use Instance Normalization (IN) to eliminate feature mean and variance, thereby stripping style information.
2. Style Encoder ( $E_i^s$ ): Consists of strided convolutional downsampling layers, a global average pooling layer, and a fully connected layer. IN is omitted entirely to preserve the channel-wise feature statistics that represent style.
3. Decoder ( $G_i$ ): Reconstructs images from content and style codes. It processes content features via residual blocks equipped with Adaptive Instance Normalization (AdaIN), followed by convolutional upsampling layers. A multilayer perceptron (MLP) dynamically generates affine transformation parameters $(\gamma, \beta)$ from the style code $s_i$ :
$\text{AdaIN}(z, \gamma, \beta) = \gamma \left( \frac{z - \mu(z)}{\sigma(z)} \right) + \beta$

where $z$ is the intermediate feature activation, and $\mu(z)$ and $\sigma(z)$ are its channel-wise mean and standard deviation.
Knowl 8 — Conditional Inception Score (CIS)
definition
The Conditional Inception Score (CIS) evaluates the sample diversity and quality produced by a multimodal image-to-image translation model conditioned on individual source images. Given a fine-tuned mode classifier $p(y_2|x_2)$ for target domain $\mathcal{X}_2$ :

$\text{CIS} = \mathbb{E}_{x_1 \sim p(x_1)} \left[ \mathbb{E}_{x_{1 \to 2} \sim p(x_{1 \to 2}|x_1)} \left[ \text{KL}\left( p(y_2|x_{1 \to 2}) \,\|\, p(y_2|x_1) \right) \right] \right]$

where $p(y_2|x_1) = \int p(y_2|x_{1 \to 2}) p(x_{1 \to 2}|x_1) \, dx_{1 \to 2}$ represents the marginal class distribution conditioned on a single image $x_1$ .

High CIS requires that outputs conditioned on a single input $x_1$ cover multiple modes (high entropy of $p(y_2|x_1)$ ) while each individual output belongs clearly to a specific mode (low entropy of $p(y_2|x_{1 \to 2})$ ). A model that produces deterministic translations for any input receives $\text{CIS} = 0$ , even if its unconditional Inception Score (IS) is high.
Knowl 9 — Domain-Invariant Perceptual Loss
model/method
To provide perceptual regularization without requiring paired target reference images, a domain-invariant perceptual loss is computed between input image $x_1$ and translated output $x_{1 \to 2}$ .

Instance Normalization (IN) without affine parameters is applied to the deep VGG feature representations of both images before computing their Euclidean distance. Applying IN strips feature mean and variance (which encode domain-specific style) and retains domain-invariant spatial structure, enabling stable training on high-resolution images (resolution $\ge 512 \times 512$ ).

Knowl 10 — Quantitative Quality and Diversity Benchmarks on Edges-to-Shoes/Handbags

data/table

Performance of MUNIT, baseline methods, and ablation variants evaluated on edges $\to$ shoes and edges $\to$ handbags. Diversity is quantified using average LPIPS distance across 1900 randomly sampled output pairs (100 inputs, 19 pairs each; higher is more diverse). Quality is measured via human preference on Amazon Mechanical Turk (percentage preferred over MUNIT; higher is better):

	Edges $\to$ shoes		Edges $\to$ handbags
Method	Quality	Diversity	Quality	Diversity
UNIT	37.4%	0.011	37.3%	0.023
CycleGAN	36.0%	0.010	40.8%	0.012
CycleGAN* with noise	29.5%	0.016	45.1%	0.011
MUNIT w/o $\mathcal{L}_{\text{recon}}^x$	6.0%	0.213	29.0%	0.191
MUNIT w/o $\mathcal{L}_{\text{recon}}^c$	20.7%	0.172	9.3%	0.185
MUNIT w/o $\mathcal{L}_{\text{recon}}^s$	28.6%	0.070	24.6%	0.139
MUNIT	50.0%	0.109	50.0%	0.175
BicycleGAN (paired supervision)	56.7%	0.104	51.2%	0.140
Real data	N/A	0.293	N/A	0.371

MUNIT achieves diversity comparable to the fully supervised BicycleGAN and significantly exceeds all unsupervised baselines (which suffer mode collapse despite dropout or noise injection). Ablation shows that removing image reconstruction ( $\mathcal{L}_{\text{recon}}^x$ ) or content reconstruction ( $\mathcal{L}_{\text{recon}}^c$ ) degrades quality, while removing style reconstruction ( $\mathcal{L}_{\text{recon}}^s$ ) causes partial mode collapse and reduces diversity.

Knowl 11 — Multimodal Translation Performance on Animal Image Benchmarks

data/table

Quantitative evaluation across 6 bidirectional translation tasks involving three animal domains (house cats, big cats, dogs), each containing 4 fine-grained modes. Diversity and quality are evaluated via Conditional Inception Score (CIS) and Inception Score (IS) using fine-tuned Inception-v3 models (100 inputs, 100 samples per input):

	CycleGAN		CycleGAN* w/ noise		UNIT		MUNIT
Task	CIS	IS	CIS	IS	CIS	IS	CIS	IS
house cats $\to$ big cats	0.078	0.795	0.034	0.701	0.096	0.666	0.911	0.923
big cats $\to$ house cats	0.109	0.887	0.124	0.848	0.164	0.817	0.956	0.954
house cats $\to$ dogs	0.044	0.895	0.070	0.901	0.045	0.827	1.231	1.255
dogs $\to$ house cats	0.121	0.921	0.137	0.978	0.193	0.982	1.035	1.034
big cats $\to$ dogs	0.058	0.762	0.019	0.589	0.094	0.910	1.205	1.233
dogs $\to$ big cats	0.047	0.620	0.022	0.558	0.096	0.754	0.897	0.901
Average	0.076	0.813	0.068	0.762	0.115	0.826	1.039	1.050

MUNIT achieves an average CIS of 1.039 compared to 0.068–0.115 for the baselines, demonstrating that baseline translations are essentially deterministic given a single input, whereas MUNIT successfully covers multiple fine-grained categories.

Coverage note — Qualitative visual experiments on street scenes (SYNTHIA/Cityscapes) and the Yosemite summer/winter dataset were omitted as standalone knowls because they demonstrate the qualitative properties already captured quantitatively in the benchmark tables.

References

1.Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10593-2_13
2.Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 649–666. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_40
3.Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: feature learning by inpainting. In: CVPR (2016)
4.Laffont, P.Y., Ren, Z., Tao, X., Qian, C., Hays, J.: Transient attributes for high-level understanding and editing of outdoor scenes. TOG 34, 149 (2014)
5.Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: CVPR (2016)
6.Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: CVPR (2017)
7.Yi, Z., Zhang, H., Tan, P., Gong, M.: DualGAN: unsupervised dual learning for image-to-image translation. In: ICCV (2017)
8.Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: ICCV (2017)
9.Kim, T., Cha, M., Kim, H., Lee, J., Kim, J.: Learning to discover cross-domain relations with generative adversarial networks. In: ICML (2017)
10.Taigman, Y., Polyak, A., Wolf, L.: Unsupervised cross-domain image generation. In: ICLR (2017)
11.Zhu, J.Y., Zhang, R., Pathak, D., Darrell, T., Efros, A.A., Wang, O., Shechtman, E.: Toward multimodal image-to-image translation. In: NIPS (2017)
12.Liu, M.Y., Tuzel, O.: Coupled generative adversarial networks. In: NIPS (2016)
13.Chen, Q., Koltun, V.: Photographic image synthesis with cascaded refinement networks. In: ICCV (2017)
14.Liang, X., Zhang, H., Xing, E.P.: Generative semantic manipulation with contrasting GAN. arXiv preprint arXiv:1708.00315 (2017)
15.Liu, M.Y., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. In: NIPS (2017)
16.Benaim, S., Wolf, L.: One-sided unsupervised domain mapping. In: NIPS (2017)
17.Royer, A., et al.: XGAN: unsupervised image-to-image translation for many-to-many mappings. arXiv preprint arXiv:1711.05139 (2017)
18.Gan, Z., et al.: Triangle generative adversarial networks. In: NIPS, pp. 5253–5262 (2017)
19.Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., Choo, J.: StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. In: CVPR (2018)
20.Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., Catanzaro, B.: High-resolution image synthesis and semantic manipulation with conditional GANs. In: CVPR (2018)
21.Shrivastava, A., Pfister, T., Tuzel, O., Susskind, J., Wang, W., Webb, R.: Learning from simulated and unsupervised images through adversarial training. In: CVPR (2017)
22.Bousmalis, K., Silberman, N., Dohan, D., Erhan, D., Krishnan, D.: Unsupervised pixel-level domain adaptation with generative adversarial networks. In: CVPR (2017)
23.Wolf, L., Taigman, Y., Polyak, A.: Unsupervised creation of parameterized avatars. In: ICCV (2017)
24.Tau, T.G., Wolf, L., Tau, S.B.: The role of minimal complexity functions in unsupervised learning of semantic mappings. In: ICLR (2018)
25.Hoshen, Y., Wolf, L.: Identifying analogies across domains. In: ICLR (2018)
26.Mathieu, M., Couprie, C., LeCun, Y.: Deep multi-scale video prediction beyond mean square error. In: ICLR (2016)
27.Goodfellow, I., et al.: Generative adversarial nets. In: NIPS (2014)
28.Denton, E.L., Chintala, S., Fergus, R.: Deep generative image models using a Laplacian pyramid of adversarial networks. In: NIPS (2015)
29.Wang, X., Gupta, A.: Generative image modeling using style and structure adversarial networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 318–335. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_20
30.Yang, J., Kannan, A., Batra, D., Parikh, D.: LR-GAN: layered recursive generative adversarial networks for image generation. In: ICLR (2017)
31.Huang, X., Li, Y., Poursaeed, O., Hopcroft, J., Belongie, S.: Stacked generative adversarial networks. In: CVPR (2017)
32.Zhang, H., et al.: StackGAN: text to photo-realistic image synthesis with stacked generative adversarial networks. In: ICCV (2017)
33.Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: ICLR (2018)
34.Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training GANs. In: NIPS (2016)
35.Zhao, J., Mathieu, M., LeCun, Y.: Energy-based generative adversarial network. In: ICLR (2017)
36.Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: ICML (2017)
37.Berthelot, D., Schumm, T., Metz, L.: BEGAN: boundary equilibrium generative adversarial networks. arXiv preprint arXiv:1703.10717 (2017)
38.Mao, X., Li, Q., Xie, H., Lau, Y.R., Wang, Z., Smolley, S.P.: Least squares generative adversarial networks. In: ICCV (2017)
39.Tolstikhin, I., Bousquet, O., Gelly, S., Schoelkopf, B.: Wasserstein auto-encoders. In: ICLR (2018)
40.Larsen, A.B.L., Sønderby, S.K., Larochelle, H., Winther, O.: Autoencoding beyond pixels using a learned similarity metric. In: ICML (2016)
41.Dosovitskiy, A., Brox, T.: Generating images with perceptual similarity metrics based on deep networks. In: NIPS (2016)
42.Rosca, M., Lakshminarayanan, B., Warde-Farley, D., Mohamed, S.: Variational approaches for auto-encoding generative adversarial networks. arXiv preprint arXiv:1706.04987 (2017)
43.Li, C., et al.: Alice: towards understanding adversarial learning for joint distribution matching. In: NIPS (2017)
44.Srivastava, A., Valkoz, L., Russell, C., Gutmann, M.U., Sutton, C.: VEEGAN: reducing mode collapse in gans using implicit variational learning. In: NIPS (2017)
45.Ghosh, A., Kulharia, V., Namboodiri, V., Torr, P.H., Dokania, P.K.: Multi-agent diverse generative adversarial networks. arXiv preprint arXiv:1704.02906 (2017)
46.Bansal, A., Sheikh, Y., Ramanan, D.: PixeLNN: example-based image synthesis. In: ICLR (2018)
47.Almahairi, A., Rajeswar, S., Sordoni, A., Bachman, P., Courville, A.: Augmented cycleGAN: learning many-to-many mappings from unpaired data. arXiv preprint arXiv:1802.10151 (2018)
48.Lee, H.Y., Tseng, H.Y., Huang, J.B., Singh, M.K., Yang, M.H.: Diverse image-to-image translation via disentangled representation. In: Ferrari, V. (ed.) ECCV 2018, Part I. LNCS, vol. 11207, pp. 36–52. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_3
49.Anoosheh, A., Agustsson, E., Timofte, R., Van Gool, L.: ComboGAN: unrestrained scalability for image domain translation. arXiv preprint arXiv:1712.06909 (2017)
50.Hui, L., Li, X., Chen, J., He, H., Yang, J., et al.: Unsupervised multi-domain image translation with domain-specific encoders/decoders. arXiv preprint arXiv:1712.02050 (2017)
51.Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., Salesin, D.H.: Image analogies. In: SIGGRAPH (2001)
52.Li, C., Wand, M.: Combining markov random fields and convolutional neural networks for image synthesis. In: CVPR (2016)
53.Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43
54.Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: ICCV (2017)
55.Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., Yang, M.H.: Universal style transfer via feature transforms. In: NIPS, pp. 385–395 (2017)
56.Li, Y., Liu, M.Y., Li, X., Yang, M.H., Kautz, J.: A closed-form solution to photorealistic image stylization. In: Ferrari, V., et al. (eds.) ECCV 2018, Part III. LNCS, vol. 11207, pp. 469–486. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_28
57.Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P.: InfoGAN: interpretable representation learning by information maximizing generative adversarial nets. In: NIPS (2016)
58.Higgins, I., et al.: beta-VAE: learning basic visual concepts with a constrained variational framework. In: ICLR (2017)
59.Tenenbaum, J.B., Freeman, W.T.: Separating style and content. In: NIPS (1997)
60.Bousmalis, K., Trigeorgis, G., Silberman, N., Krishnan, D., Erhan, D.: Domain separation networks. In: NIPS (2016)
61.Villegas, R., Yang, J., Hong, S., Lin, X., Lee, H.: Decomposing motion and content for natural video sequence prediction. In: ICLR (2017)
62.Mathieu, M.F., Zhao, J.J., Zhao, J., Ramesh, A., Sprechmann, P., LeCun, Y.: Disentangling factors of variation in deep representation using adversarial training. In: NIPS (2016)
63.Denton, E.L., et al.: Unsupervised learning of disentangled representations from video. In: NIPS (2017)
64.Tulyakov, S., Liu, M.Y., Yang, X., Kautz, J.: MocoGAN: decomposing motion and content for video generation. In: CVPR (2018)
65.Donahue, C., Balsubramani, A., McAuley, J., Lipton, Z.C.: Semantically decomposing the latent spaces of generative adversarial networks. In: ICLR (2018)
66.Shen, T., Lei, T., Barzilay, R., Jaakkola, T.: Style transfer from non-parallel text by cross-alignment. In: Advances in Neural Information Processing Systems, pp. 6833–6844 (2017)
67.Donahue, J., Krähenbühl, P., Darrell, T.: Adversarial feature learning. In: ICLR (2017)
68.Dumoulin, V., et al.: Adversarially learned inference. In: ICLR (2017)
69.Automatic differentiation in PyTorch. In: NIPS Autodiff Workshop (2017)
70.He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
71.Ulyanov, D., Vedaldi, A., Lempitsky, V.: Improved texture networks: maximizing quality and diversity in feed-forward stylization and texture synthesis. In: CVPR (2017)
72.Dumoulin, V., Shlens, J., Kudlur, M.: A learned representation for artistic style. In: ICLR (2017)
73.Wang, H., Liang, X., Zhang, H., Yeung, D.Y., Xing, E.P.: ZM-Net: real-time zero-shot image manipulation network. arXiv preprint arXiv:1703.07255 (2017)
74.Ghiasi, G., Lee, H., Kudlur, M., Dumoulin, V., Shlens, J.: Exploring the structure of a real-time, arbitrary neural artistic stylization network. In: BMVC (2017)
75.Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)
76.Li, Y., Wang, N., Shi, J., Liu, J., Hou, X.: Revisiting batch normalization for practical domain adaptation. arXiv preprint arXiv:1603.04779 (2016)
77.Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR (2018)
78.Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in neural information processing systems (2012)
79.Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: CVPR (2016)
80.Yu, A., Grauman, K.: Fine-grained visual comparisons with local learning. In: CVPR (2014)
81.Zhu, J.-Y., Krähenbühl, P., Shechtman, E., Efros, A.A.: Generative visual manipulation on the natural image manifold. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9909, pp. 597–613. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46454-1_36
82.Xie, S., Tu, Z.: Holistically-nested edge detection. In: ICCV (2015)
83.Ros, G., Sellart, L., Materzynska, J., Vazquez, D., Lopez, A.M.: The synthia dataset: a large collection of synthetic images for semantic segmentation of urban scenes. In: CVPR (2016)
84.Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding. In: CVPR (2016)

Citation

MLA

Huang, X., et al. “Multimodal Unsupervised Image-to-Image Translation”. Lecture Notes in Computer Science, Springer International Publishing, 2018, pp. 179–96, https://doi.org/10.1007/978-3-030-01219-9_11.

APA

Huang, X., Liu, M.-Y., Belongie, S., & Kautz, J. (2018). Multimodal Unsupervised Image-to-Image Translation. In Lecture Notes in Computer Science (pp. 179–196). Springer International Publishing. https://doi.org/10.1007/978-3-030-01219-9_11

Chicago

Huang, X., M.-Y. Liu, S. Belongie, and J. Kautz. 2018. “Multimodal Unsupervised Image-to-Image Translation”. In Lecture Notes in Computer Science. Springer International Publishing. https://doi.org/10.1007/978-3-030-01219-9_11.

Harvard

Huang, X. et al. (2018) “Multimodal Unsupervised Image-to-Image Translation”, Lecture Notes in Computer Science. Springer International Publishing, pp. 179–196. Available at: https://doi.org/10.1007/978-3-030-01219-9_11.

Vancouver

1. Huang X, Liu M-Y, Belongie S, Kautz J (2018) Multimodal Unsupervised Image-to-Image Translation. In: Lecture Notes in Computer Science. Springer International Publishing, pp 179–196

BibTeX

@inbook{Huang_2018, title={Multimodal Unsupervised Image-to-Image Translation}, ISBN={9783030012199}, ISSN={1611-3349}, url={http://dx.doi.org/10.1007/978-3-030-01219-9_11}, DOI={10.1007/978-3-030-01219-9_11}, booktitle={Computer Vision – ECCV 2018}, publisher={Springer International Publishing}, author={Huang, Xun and Liu, Ming-Yu and Belongie, Serge and Kautz, Jan}, year={2018}, pages={179–196} }

Metadata:Crossref

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Multimodal Unsupervised Image-to-Image Translation

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Partially Shared Latent Space Assumption

Knowl 2 — Multimodal Unsupervised Image-to-Image Translation (MUNIT) Framework

Knowl 3 — MUNIT Bidirectional Reconstruction and Adversarial Objectives

Knowl 4 — Implicit Latent Distribution Matching at Optimality

Knowl 5 — Style-Augmented Cycle Consistency

Knowl 6 — Joint Distribution Matching and Global Minima

Knowl 7 — MUNIT Auto-Encoder Architecture with AdaIN and Instance Normalization

Knowl 8 — Conditional Inception Score (CIS)

Knowl 9 — Domain-Invariant Perceptual Loss