Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Jiajun WuChengkai ZhangTianfan XueBill FreemanJoshua B. Tenenbaum

article2016NeurIPS2,169 citations

Introduces 3D-GAN, a framework that generates realistic 3D volumetric shapes from a probabilistic latent space and produces unsupervised shape descriptors that rival supervised methods in 3D object recognition.

Listen

Generating realistic and novel three-dimensional shapes has long challenged computer vision and graphics. Traditional synthesis methods typically rely on combining parts from existing design libraries, which restricts novelty. Meanwhile, early deep learning methods that operate directly on three-dimensional grids often generate blurred shapes with structural defects such as missing parts or holes. In addition, existing approaches frequently require extensive human-labeled supervision to learn useful shape representations for downstream recognition and reconstruction tasks.

The article establishes and evaluates a framework called 3D Generative Adversarial Network (3D-GAN) to synthesize realistic, high-resolution three-dimensional objects from a probabilistic latent space without human supervision. It also demonstrates an extended architecture, 3D-VAE-GAN, designed to reconstruct full three-dimensional volumetric shapes directly from single two-dimensional images.

To achieve this, the authors combined volumetric convolutional networks with generative adversarial modeling. In 3D-GAN, a generator maps a random 200-dimensional vector into a 64x64x64 voxel cube, while an adversarial discriminator learns to distinguish synthetic shapes from real shapes. An adaptive training rule pauses discriminator updates when its accuracy exceeds 80 percent to prevent training instability. For image-based reconstruction, an encoder network pairs with the generator to infer shape vectors from 2D photos. The models were evaluated using ShapeNet models for generation, the ModelNet benchmark for unsupervised shape classification, and the real-world IKEA dataset for single-image 3D reconstruction.

The evaluation yielded several key findings. First, 3D-GAN synthesized high-resolution 3D objects with fine surface details and realistic variations across multiple categories without merely memorizing training examples. Second, the unsupervised features extracted from the discriminator achieved 83.3 percent classification accuracy on ModelNet40 and 91.0 percent on ModelNet10, outperforming previous unsupervised methods by 7.8 and 10.5 percentage points respectively while matching competitive fully supervised baselines. Third, the model demonstrated strong sample efficiency, matching the performance of prior unsupervised techniques using roughly 25 training samples per class compared to 80 or more. Finally, for single-image 3D reconstruction on the IKEA benchmark, 3D-VAE-GAN achieved a mean average precision of 53.1 percent when trained per category, substantially outperforming prior state-of-the-art baselines.

These findings indicate that generative adversarial training naturally captures complex structural constraints without requiring voxel-level manual annotations. Disentangling generative and discriminative representations allows organizations to generate novel 3D digital assets on demand and perform robust 3D shape recognition at lower labeling costs. Furthermore, the demonstrated ability to infer 3D geometry from single 2D images offers strong utility for automated scene understanding, robotics, and augmented reality workflows.

Organizations developing 3D vision pipelines should evaluate unsupervised adversarial representations to reduce annotation overhead. Future development should focus on testing the framework on higher-resolution voxel grids, evaluating broader object categories beyond standard indoor furniture, and piloting end-to-end single-image reconstruction pipelines in production settings.

Confidence in these findings is high based on strong empirical gains across standard benchmarks. However, stakeholders should note current technical limitations: the experiments operate at a 64x64x64 voxel resolution, which limits ultra-fine surface fidelity, and real-world image reconstruction can still be constrained by heavy visual occlusions.

Google Massachusetts Institute of Technology 3d object recognition cad models gan 3D generative-adversarial modeling probabilistic latent space object shape generation

arXiv: 1610.07584 zck119/3dgan-release

Paper: Generative Adversarial Networks, Ian J. Goodfellow et al. (2014). Introduces the core generative adversarial network framework and minimax optimization that 3D-GAN directly adapts for volumetric 3D shape generation.
Paper: Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, Alec Radford et al. (2016). Establishes the architectural foundations for deep convolutional generative adversarial networks and unsupervised feature representation learning that 3D-GAN extends to 3D voxel grids.
Paper: 3D ShapeNets: A deep representation for volumetric shapes, Zhirong Wu et al. (2014). Pioneers volumetric deep learning and 3D voxel representations on ModelNet, providing the benchmark datasets and representation paradigm used in 3D-GAN.
Paper: Autoencoding beyond pixels using a learned similarity metric, Anders Boesen Lindbo Larsen et al. (2015). Combines autoencoders with adversarial training using discriminator feature representations, influencing 3D-GAN's 3D-VAE-GAN hybrid variant.
Paper: Improved Techniques for Training GANs, Tim Salimans et al. (2016). Presents essential training heuristics and stabilization methods for generative adversarial networks that facilitate stable volumetric GAN optimization.
Paper: Multi-view Convolutional Neural Networks for 3D Shape Recognition, Hang Su et al. (2015). Establishes a key standard for 3D shape recognition baselines and evaluations against which 3D-GAN benchmarks its unsupervised shape representations.

Paper: A Point Set Generation Network for 3D Object Reconstruction from a Single Image, Haoqiang Fan et al. (2017). Moves beyond the rigid volumetric grids of 3D-GAN to directly generate unordered 3D point cloud sets from single images.
Paper: Occupancy Networks: Learning 3D Reconstruction in Function Space, Lars Mescheder et al. (2018). Overcomes the resolution and memory limitations of discrete 3D voxel grids by learning continuous implicit surface representations for 3D reconstruction and generation.
Paper: DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation, Jeong Joon Park et al. (2019). Advances continuous shape latent spaces by formulating class-conditioned signed distance functions rather than volumetric voxel occupancies.
Paper: Efficient Geometry-aware 3D Generative Adversarial Networks, Eric R. Chan et al. (2022). Extends 3D generative adversarial modeling to high-resolution neural rendering and view-consistent 3D-aware image synthesis from single 2D collections.
Paper: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields, Michael Niemeyer et al. (2021). Builds on 3D generative modeling concepts to synthesize multi-object scenes with compositional, disentangled 3D neural feature fields.
Paper: DreamFusion: Text-to-3D using 2D Diffusion, Ben Poole et al. (2023). Evolves 3D shape generation from native 3D voxel GANs to text-driven 3D synthesis via Score Distillation Sampling on 2D diffusion priors.
Paper: Structured 3D Latents for Scalable and Versatile 3D Generation, Jianfeng Xiang et al. (2025). Generalizes 3D generative latent modeling to versatile structured latents that decode across multiple representations including meshes, radiance fields, and Gaussians.

Cover for Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Abstract

We study the problem of 3D object generation. We propose a novel framework, namely 3D Generative Adversarial Network (3D-GAN), which generates 3D objects from a probabilistic space by leveraging recent advances in volumetric convolutional networks and generative adversarial nets. The benefits of our model are three-fold: first, the use of an adversarial criterion, instead of traditional heuristic criteria, enables the generator to capture object structure implicitly and to synthesize high-quality 3D objects; second, the generator establishes a mapping from a low-dimensional probabilistic space to the space of 3D objects, so that we can sample objects without a reference image or CAD models, and explore the 3D object manifold; third, the adversarial discriminator provides a powerful 3D shape descriptor which, learned without supervision, has wide applications in 3D object recognition. Experiments demonstrate that our method generates high-quality 3D objects, and our unsupervisedly learned features achieve impressive performance on 3D object recognition, comparable with those of supervised learning methods.

Topics

3d object recognition cad models gan generative adversarial networks reference image shape descriptors supervised machine learning

Keywords

3D generative-adversarial modeling probabilistic latent space object shape generation shape synthesis shape retrieval volumetric autoencoder variational autoencoder adversarial networks multi-view CNNs CAD models

1 Introduction
2 Related Work
3 Models
3.1 3D Generative Adversarial Network (3D-GAN)
3.2 3D-VAE-GAN
4 Evaluation
4.1 3D Object Generation
4.2 3D Object Classification
4.3 Single Image 3D Reconstruction
5 Analyzing Learned Representations
5.1 The Generative Representation
5.2 The Discriminative Representation
6 Conclusion
References
A.1 Network Structure
A.2 3D Shape Classification
A.3 3D-VAE-GAN Training

Knowls

Knowl 1 — 3D-GAN Architecture for Volumetric Shape Synthesis
model/method
The 3D Generative Adversarial Network (3D-GAN) synthesizes 3D volumetric objects from a probabilistic latent distribution by coupling a volumetric convolutional generator $G$ with a volumetric convolutional discriminator $D$ .

Generator ( $G$ ):
- Input: A 200-dimensional latent noise vector $z \in [0, 1]^{200}$ , where each coordinate is sampled independently from a uniform distribution $\text{Uniform}[0, 1]$ .
- Architecture: Five volumetric transposed convolutional layers with channel counts $\{512, 256, 128, 64, 1\}$ , kernel sizes $4 \times 4 \times 4$ , and strides $\{1, 2, 2, 2, 2\}$ .
- Normalization and Activations: Each intermediate layer uses batch normalization followed by a ReLU activation function. The final convolutional layer uses a Sigmoid activation to produce voxel occupancy probabilities in $[0, 1]$ .
- Output: A 3D voxel grid $G(z) \in [0, 1]^{64 \times 64 \times 64}$ .
Discriminator ( $D$ ):
- Input: A 3D voxel representation $x \in [0, 1]^{64 \times 64 \times 64}$ .
- Architecture: Five volumetric convolutional layers with channel counts $\{64, 128, 256, 512, 1\}$ , kernel sizes $4 \times 4 \times 4$ , and strides $\{2, 2, 2, 2, 1\}$ .
- Normalization and Activations: Intermediate layers apply batch normalization and Leaky ReLU activations with negative slope $\alpha = 0.2$ . The final layer applies a Sigmoid activation.
- Output: A scalar probability $D(x) \in [0, 1]$ indicating the likelihood that $x$ is an authentic 3D shape from the training distribution rather than a generated sample.
Neither $G$ nor $D$ employs spatial pooling or fully connected linear layers.
Knowl 2 — 3D-GAN Adversarial Objective and Adaptive Training Procedure
algorithm
The 3D-GAN is optimized using the binary cross-entropy adversarial objective:

$\mathcal{L}_{\text{3D-GAN}} = \log D(x) + \log\bigl(1 - D(G(z))\bigr)$

where $x \in [0, 1]^{64 \times 64 \times 64}$ denotes a ground-truth voxel object and $z \sim \text{Uniform}[0, 1]^{200}$ is an i.i.d. random vector.

To address the issue where the discriminator learns faster than the generator in 3D voxel spaces (causing vanishing gradient signals for $G$ ), 3D-GAN employs an adaptive training rule: in each batch, discriminator parameters are updated only if discriminator classification accuracy on the previous batch does not exceed $80\%$ .
Input: Training set of voxelized 3D shapes $X$ , learning rates $\eta_G = 0.0025$ , $\eta_D = 10^{-5}$ , batch size $B = 100$ , accuracy threshold $\tau = 0.8$
Initialize parameters $\theta_G$ and $\theta_D$
Initialize previous discriminator accuracy $acc_D = 0$
for each epoch do
    for each batch $\{x_1, \dots, x_B\} \subset X$ do
        Sample latent vectors $\{z_1, \dots, z_B\}$ where $z_i \sim \text{Uniform}[0, 1]^{200}$
        Generate voxel shapes $\hat{x}_i = G(z_i)$ for $i = 1, \dots, B$
        Compute current batch classification accuracy of $D$
        if $acc_D \le \tau$ then
            Update $\theta_D$ with ADAM step on $\nabla_{\theta_D} \frac{1}{B} \sum_{i=1}^B \left( \log D(x_i) + \log(1 - D(\hat{x}_i)) \right)$
        end if
         $acc_D \leftarrow$ current batch classification accuracy of $D$
        Update $\theta_G$ with ADAM step on $\nabla_{\theta_G} \frac{1}{B} \sum_{i=1}^B \log(1 - D(G(z_i)))$
    end for
end for
Both networks use the ADAM optimizer with momentum parameter $\beta = 0.5$ .
Knowl 3 — 3D-VAE-GAN Architecture for Single-Image 3D Reconstruction
model/method
The 3D-VAE-GAN extends 3D-GAN to infer 3D shapes directly from single 2D images by integrating an image encoder $E$ with the 3D-GAN generator $G$ (acting as a decoder) and discriminator $D$ .

Image Encoder ( $E$ ):
- Input: A single RGB image $y \in \mathbb{R}^{3 \times 256 \times 256}$ .
- Architecture: Five 2D convolutional layers with kernel sizes $\{11 \times 11, 5 \times 5, 5 \times 5, 5 \times 5, 8 \times 8\}$ , strides $\{4, 2, 2, 2, 1\}$ , and channel dimensions $\{64, 128, 256, 512, 400\}$ .
- Intermediate operations: Batch normalization and ReLU activations are applied between successive convolutional layers.
- Probabilistic output: The 400-dimensional output vector is partitioned into a 200-dimensional predicted mean vector $E_{\text{mean}}(y)$ and a 200-dimensional predicted diagonal variance vector $E_{\text{var}}(y)$ , parametrizing the variational posterior distribution $q(z \mid y) = \mathcal{N}\bigl(E_{\text{mean}}(y), \text{diag}(E_{\text{var}}(y))\bigr)$ .
3D Decoder ( $G$ ) and Discriminator ( $D$ ):
- A 200-dimensional vector $z$ sampled from $q(z \mid y)$ is decoded by $G$ through five volumetric transposed convolutional layers into a $64 \times 64 \times 64$ voxel volume $G(z)$ .
- The 3D discriminator $D$ evaluates whether the synthesized voxel grid aligns with the distribution of realistic 3D shapes.
Knowl 4 — 3D-VAE-GAN Loss Formulation and Training Algorithm
algorithm
The 3D-VAE-GAN objective combines an adversarial loss $\mathcal{L}_{\text{3D-GAN}}$ , a Kullback-Leibler divergence regularization loss $\mathcal{L}_{\text{KL}}$ , and a voxel reconstruction loss $\mathcal{L}_{\text{recon}}$ :

$\mathcal{L} = \mathcal{L}_{\text{3D-GAN}} + \alpha_1 \mathcal{L}_{\text{KL}} + \alpha_2 \mathcal{L}_{\text{recon}}$

where:

$\mathcal{L}_{\text{3D-GAN}} = \log D(x) + \log\bigl(1 - D(G(z))\bigr)$

$\mathcal{L}_{\text{KL}} = D_{\text{KL}}\bigl(q(z \mid y) \parallel p(z)\bigr) = D_{\text{KL}}\bigl(\mathcal{N}(E_{\text{mean}}(y), \text{diag}(E_{\text{var}}(y))) \parallel \mathcal{N}(0, I)\bigr)$

$\mathcal{L}_{\text{recon}} = \|G(E(y)) - x\|_2$

with prior $p(z) = \mathcal{N}(0, I)$ and loss hyperparameter weights $\alpha_1 = 5$ and $\alpha_2 = 10^{-4}$ .
Input: Paired training samples $\{(x_i, y_i)\}_{i=1}^N$ where $x_i \in [0, 1]^{64 \times 64 \times 64}$ and $y_i \in \mathbb{R}^{3 \times 256 \times 256}$ , prior $p(z) = \mathcal{N}(0, I)$
Initialize encoder $E$ , generator $G$ , and discriminator $D$
for each training iteration $t$ do
    Sample $z_t \sim \mathcal{N}(0, I)$
    Select training pair $(x_i, y_i)$
    Update $D$ by minimizing $\mathcal{L}_D = -\log D(x_i) - \log\bigl(1 - D(G(z_t))\bigr)$
    Compute $(E_{\text{mean}}(y_i), E_{\text{var}}(y_i)) = E(y_i)$
    Sample $z_{\text{pred}} \sim \mathcal{N}(E_{\text{mean}}(y_i), \text{diag}(E_{\text{var}}(y_i)))$
    Update $E$ by minimizing $\mathcal{L}_E = D_{\text{KL}}\bigl(\mathcal{N}(E_{\text{mean}}(y_i), \text{diag}(E_{\text{var}}(y_i))) \parallel \mathcal{N}(0, I)\bigr) + \|G(z_{\text{pred}}) - x_i\|_2$
    Update $G$ by minimizing $\mathcal{L}_G = -\log\bigl(D(G(z_t))\bigr) + \|G(z_{\text{pred}}) - x_i\|_2$
end for
Training pairs are constructed by rendering 3D shapes across 72 camera poses (24 azimuths and 3 elevations) against background images from the SUN database.
Knowl 5 — Discriminator Feature Representation for Unsupervised 3D Object Classification
model/method
The internal representations learned by the 3D-GAN discriminator without class supervision serve as general 3D shape descriptors for downstream classification tasks.

Feature Extraction Pipeline:
1. A single 3D-GAN is trained without supervision on seven major shape categories from the ShapeNet dataset: chairs, sofas, tables, boats, airplanes, rifles, and cars.
2. For any candidate 3D input volume $x \in [0, 1]^{64 \times 64 \times 64}$ , intermediate feature activations are extracted from the second, third, and fourth volumetric convolutional layers of the trained discriminator $D$ .
3. Volumetric max pooling is applied to these three layer outputs using kernel sizes $8 \times 8 \times 8$ , $4 \times 4 \times 4$ , and $2 \times 2 \times 2$ , respectively.
4. The pooled activations are flattened and concatenated into a single 7,168-dimensional feature vector.
Classifier Configuration:
- A linear Support Vector Machine (SVM) is fitted to the 7,168-dimensional descriptors in a one-versus-all classification scheme.
- Training uses $L_2$ regularization, balanced class weighting, and intercept scaling.
- The regularization penalty hyperparameter is set to $C = 0.07$ for ModelNet40 evaluation and $C = 0.01$ for ModelNet10 evaluation.

Knowl 6 — ModelNet Benchmark Classification Performance

data/table

Classification accuracy (per-class) on the ModelNet40 and ModelNet10 benchmark datasets demonstrates the discriminative capacity of representations learned by the 3D-GAN discriminator without supervision, outperforming other unsupervised representations and matching supervised 3D voxel baselines.

Supervision	Pretraining	Method	ModelNet40	ModelNet10
Category labels	ImageNet	MVCNN	90.1%	–
Category labels	ImageNet	MVCNN-MultiRes	91.4%	–
Category labels	None	3D ShapeNets	77.3%	83.5%
Category labels	None	DeepPano	77.6%	85.5%
Category labels	None	VoxNet	83.0%	92.0%
Category labels	None	ORION	–	93.8%
Unsupervised	None	SPH	68.2%	79.8%
Unsupervised	None	LFD	75.5%	79.9%
Unsupervised	None	T-L Network	74.4%	–
Unsupervised	None	VConv-DAE	75.5%	80.5%
Unsupervised	None	3D-GAN (ours)	83.3%	91.0%

When evaluated under limited training data regimes on ModelNet40, 3D-GAN trained with approximately 25 labeled objects per class matches the accuracy of competing unsupervised methods (e.g., VConv-DAE) trained with 80 or more objects per class.

Knowl 7 — Single-Image 3D Voxel Reconstruction Performance on the IKEA Dataset

data/table

Single-image 3D shape reconstruction evaluated on 1,039 in-the-wild objects cropped from 759 IKEA dataset images across six furniture classes. Predictions are evaluated at a voxel resolution of $20 \times 20 \times 20$ using average precision (AP), allowing alignment over permutations, flips, and translational shifts up to $10\%$ .

Method	Bed	Bookcase	Chair	Desk	Sofa	Table	Mean
AlexNet-fc8	29.5	17.3	20.4	19.7	38.8	16.0	23.6
AlexNet-conv4	38.2	26.6	31.4	26.6	69.3	19.1	35.2
T-L Network	56.3	30.2	32.9	25.8	71.7	23.3	40.0
3D-VAE-GAN (jointly trained)	49.1	31.9	42.6	34.8	79.8	33.1	45.2
3D-VAE-GAN (separately trained)	63.2	46.3	47.2	40.7	78.8	42.3	53.1

Both the unified (jointly trained) and category-specific (separately trained) 3D-VAE-GAN models outperform prior direct prediction and autoencoder-based 3D reconstruction models across all evaluated object categories.

Knowl 8 — Latent Space Vector Arithmetic and Interpolation in 3D-GAN
empirical result
The 200-dimensional latent space $z$ discovered by 3D-GAN possesses linear semantic structure:
- Shape Vector Arithmetic: Adding difference vectors between latent representations of specific attributes transfers those semantic properties to unrelated 3D objects without needing explicit 3D mesh templates. For example, computing an armrest offset vector $z_{\text{arm}} = z_{\text{armchair}} - z_{\text{chair}}$ and adding it to an armless chair latent code produces an armchair with armrests; adding a multi-layer offset vector to a standard table vector synthesizes a layered table.
- Smooth Shape Interpolation: Linear interpolation $z(t) = (1-t)z_a + t z_b$ for $t \in [0, 1]$ between two latent vectors generates continuous, topologically coherent 3D shapes across intermediate steps, functioning both within a single category (e.g., morphing between chair styles) and across different object categories (e.g., morphing from a chair to a table).
- Single Dimension Specialization: Individual dimensions of the latent vector $z$ isolate continuous geometric properties of the synthesized shapes, such as the thickness of surfaces or the width of supporting structures.
Knowl 9 — Discriminator Neuron Specialization and Semantic Part Localization
empirical result
Neurons in the penultimate convolutional layer of the unsupervised 3D-GAN discriminator develop category and semantic part selectivity:
- Global Shape Selectivity: Iterating through training objects to identify those yielding maximal activation values for a specific neuron shows that individual neurons fire selectively for consistent overall 3D structural shapes within an object class.
- Part-Level Gradient Localization: Applying 3D guided backpropagation to visualize input voxel gradients that contribute most strongly to a neuron's activation reveals that individual units localize specific functional parts (such as chair legs, armrests, chair backs, or table surfaces) consistently across diverse 3D shape variations.

Coverage note — No substantial contributed material was omitted; all primary architectural designs, loss formulations, training algorithms, evaluation benchmark results on ModelNet and IKEA, and latent space / discriminator representation analyses are covered.

References

1.Aayush Bansal, Bryan Russell, and Abhinav Gupta. Marr revisited: 2d-3d alignment via surface normal prediction. In CVPR, 2016.
2.Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In SIGGRAPH, 1999.
3.Wayne E Carlson. An algorithm and data structure for 3d object synthesis using surface patch intersections. In SIGGRAPH, 1982.
4.Angel X Chang, Thomas Funkhouser, Leonidas Guibas, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.
5.Siddhartha Chaudhuri, Evangelos Kalogerakis, Leonidas Guibas, and Vladlen Koltun. Probabilistic reasoning for assembly-based 3d modeling. ACM TOG, 30(4):35, 2011.
6.Ding-Yun Chen, Xiao-Pei Tian, Yu-Te Shen, and Ming Ouhyoung. On visual similarity based 3d model retrieval. CGF, 2003.
7.Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In ECCV, 2016.
8.Emily L Denton, Soumith Chintala, and Rob Fergus. Deep generative image models using a laplacian pyramid of adversarial networks. In NIPS, 2015.
9.Alexey Dosovitskiy, Jost Tobias Springenberg, and Thomas Brox. Learning to generate chairs with convolutional neural networks. In CVPR, 2015.
10.Rohit Girdhar, David F Fouhey, Mikel Rodriguez, and Abhinav Gupta. Learning a predictable and generative vector representation for objects. In ECCV, 2016.
11.Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
12.Haibin Huang, Evangelos Kalogerakis, and Benjamin Marlin. Analysis and synthesis of 3d shape families via deep-learned generative models of surfaces. CGF, 34(5):25–38, 2015.
13.Daniel Jiwoong Im, Chris Dongjoo Kim, Hui Jiang, and Roland Memisevic. Generating images with recurrent adversarial networks. arXiv preprint arXiv:1602.05110, 2016.
14.Evangelos Kalogerakis, Siddhartha Chaudhuri, Daphne Koller, and Vladlen Koltun. A probabilistic model for component-based shape synthesis. ACM TOG, 31(4):55, 2012.
15.Abhishek Kar, Shubham Tulsiani, Joao Carreira, and Jitendra Malik. Category-specific object reconstruction from a single image. In CVPR, 2015.
16.Michael Kazhdan, Thomas Funkhouser, and Szymon Rusinkiewicz. Rotation invariant spherical harmonic representation of 3 d shape descriptors. In SGP, 2003.
17.Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
18.Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
19.Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. In ICML, 2016.
20.Chuan Li and Michael Wand. Precomputed real-time texture synthesis with markovian generative adversarial networks. arXiv preprint arXiv:1604.04382, 2016.
21.Yangyan Li, Hao Su, Charles Ruizhongtai Qi, Noa Fish, Daniel Cohen-Or, and Leonidas J Guibas. Joint embeddings of shapes and images via cnn image purification. ACM TOG, 34(6):234, 2015.
22.Joseph J. Lim, Hamed Pirsiavash, and Antonio Torralba. Parsing ikea objects: Fine pose estimation. In ICCV, 2013.
23.Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural network acoustic models. In ICML, 2013.
24.Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In IROS, 2015.
25.Charles R Qi, Hao Su, Matthias Niessner, Angela Dai, Mengyuan Yan, and Leonidas J Guibas. Volumetric and multi-view cnns for object classification on 3d data. In CVPR, 2016.
26.Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016.
27.Danilo Jimenez Rezende, SM Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess. Unsupervised learning of 3d structure from images. In NIPS, 2016.
28.Nima Sedaghat, Mohammadreza Zolfaghari, and Thomas Brox. Orientation-boosted voxel nets for 3d object recognition. arXiv preprint arXiv:1604.03351, 2016.
29.Abhishek Sharma, Oliver Grau, and Mario Fritz. Vconv-dae: Deep volumetric shape learning without object labels. arXiv preprint arXiv:1604.03755, 2016.
30.Baoguang Shi, Song Bai, Zhichao Zhou, and Xiang Bai. Deeppano: Deep panoramic representation for 3-d shape recognition. IEEE SPL, 22(12):2339–2343, 2015.
31.Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. In ICLR Workshop, 2015.
32.Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view convolutional neural networks for 3d shape recognition. In ICCV, 2015a.
33.Hao Su, Charles R Qi, Yangyan Li, and Leonidas Guibas. Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views. In ICCV, 2015b.
34.Johan WH Tangelder and Remco C Veltkamp. A survey of content based 3d shape retrieval methods. Multimedia tools and applications, 39(3):441–471, 2008.
35.Oliver Van Kaick, Hao Zhang, Ghassan Hamarneh, and Daniel Cohen-Or. A survey on shape correspondence. CGF, 2011.
36.Xiaolong Wang and Abhinav Gupta. Generative image modeling using style and structure adversarial networks. In ECCV, 2016.
37.Jiajun Wu, Tianfan Xue, Joseph J Lim, Yuandong Tian, Joshua B Tenenbaum, Antonio Torralba, and William T Freeman. Single image 3d interpreter network. In ECCV, 2016.
38.Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In CVPR, 2015.
39.Yu Xiang, Wongun Choi, Yuanqing Lin, and Silvio Savarese. Data-driven 3d voxel patterns for object category recognition. In CVPR, 2015.
40.Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
41.Tianfan Xue, Jianzhuang Liu, and Xiaoou Tang. Example-based 3d object reconstruction from line drawings. In CVPR, 2012.
42.Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. In NIPS, 2016.
43.Jun-Yan Zhu, Philipp Krahenbuhl, Eli Shechtman, and Alexei A Efros. Generative visual manipulation on the natural image manifold. In ECCV, 2016.

Citation

MLA

Wu, J., et al. “Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling”. arXiv, 2016, http://arxiv.org/abs/1610.07584v2.

APA

Wu, J., Zhang, C., Xue, T., Freeman, W. T., & Tenenbaum, J. B. (2016). Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling. arXiv. http://arxiv.org/abs/1610.07584v2

Chicago

Wu, J., C. Zhang, T. Xue, W. T. Freeman, and J. B. Tenenbaum. 2016. “Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling”. arXiv. http://arxiv.org/abs/1610.07584v2.

Harvard

Wu, J. et al. (2016) “Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1610.07584v2.

Vancouver

1. Wu J, Zhang C, Xue T, Freeman WT, Tenenbaum JB (2016) Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling. arXiv

BibTeX

@article{wu2016learning,
  title = {Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling},
  author = {Wu, Jiajun and Zhang, Chengkai and Xue, Tianfan and Freeman, William T. and Tenenbaum, Joshua B.},
  year = {2016},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1610.07584v2},
  eprint = {1610.07584}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Published with permission

Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — 3D-GAN Architecture for Volumetric Shape Synthesis

Knowl 2 — 3D-GAN Adversarial Objective and Adaptive Training Procedure

Knowl 3 — 3D-VAE-GAN Architecture for Single-Image 3D Reconstruction

Knowl 4 — 3D-VAE-GAN Loss Formulation and Training Algorithm

Knowl 5 — Discriminator Feature Representation for Unsupervised 3D Object Classification