Learning Representations and Generative Models for 3D Point Clouds

Panos AchlioptasOlga DiamantiIoannis MitliagkasLeonidas Guibas

article2017ICML1,705 citations

Introduces a foundational autoencoder and generative modeling framework for 3D point clouds that achieves high reconstruction quality, enables direct latent space shape manipulation, and establishes standard metrics for evaluating geometric generation quality.

Listen

Three-dimensional (3D) data representations are vital across computer vision, robotics, medicine, and virtual reality. While point clouds captured by sensors like LiDAR are compact and efficient, directly manipulating them or building generative statistical models is challenging due to their unordered, irregular structure. Traditional generative approaches like raw Generative Adversarial Networks (GANs) frequently suffer from severe training instability and mode collapse, and the field lacks standardized evaluation metrics to measure sample fidelity and diversity.

The article designs and evaluates deep learning architectures for representation learning and generative modeling specifically for 3D point clouds. It develops an AutoEncoder (AE) to learn compact latent representations, introduces multiple generative models operating either directly on raw points or within the learned latent space, and establishes quantitative metrics to reliably assess generative quality.

The authors implemented deep autoencoder networks using permutation-invariant operations, evaluating both Earth Mover’s Distance (EMD) and Chamfer Distance (CD) as reconstruction objectives. Generative approaches were tested using large-scale 3D computer-aided design repositories (such as ShapeNet) and human motion datasets (such as D-FAUST). The researchers evaluated raw point cloud GANs (r-GANs), latent-space GANs (l-GANs), latent Wasserstein GANs (l-WGANs), and latent Gaussian Mixture Models (GMMs). They introduced three quantitative evaluation measures: Jensen-Shannon Divergence (JSD), Coverage (COV), and Minimum Matching Distance (MMD).

The evaluation revealed several key findings: First, decoupling the generative process by fitting a 32-component GMM with full covariance in the latent space of an EMD-trained autoencoder achieved the best overall performance, matching the visual fidelity of perfect training baselines and delivering a coverage rate of 67.4% on held-out test data. Second, GANs trained in the latent space significantly outperformed raw point cloud GANs, which exhibited low coverage (19.0%) and high distributional divergence. Third, representations learned by the autoencoder enabled successful shape completions, linear latent morphing, semantic part editing, and state-of-the-art classification on standard benchmarks (achieving 95.4% accuracy on ModelNet10). Finally, the widely used Chamfer distance was shown to be blind to certain structural flaws, such as point clustering, making EMD-based metrics far more reliable for evaluating visual fidelity.

These findings demonstrate that complex 3D generative modeling is substantially more stable, computationally efficient, and reliable when separated into a two-step framework: learning a compact geometric embedding first and fitting simpler probabilistic models second. This approach mitigates the risk of training instability, reduces development costs, and eliminates the need for hand-crafted parametric 3D models.

For engineering and product teams building 3D synthesis or reconstruction pipelines, the authors recommend adopting decoupled latent-space architectures utilizing EMD reconstruction objectives rather than training end-to-end GANs on raw point clouds. GMMs or l-WGANs should be prioritized for synthetic data generation and shape interpolation. Future efforts should focus on refining raw-point architectures and expanding multi-class frameworks to better preserve high-frequency stylistic details.

The primary limitations of this approach include difficulty reconstructing fine, high-frequency details (such as small perforations) and potential shape distortion when processing rare or atypical geometries. Additionally, the evaluation metrics require pre-aligned shapes. Despite these boundary constraints, the quantitative results across diverse benchmarks provide high confidence in the robustness and efficiency of latent-space 3D modeling.

Mila Stanford University Université de Montréal autoencoders gaussian mixture model gaussian mixtures 3D point clouds generative models representation learning

arXiv: 1707.02392 optas/latent_3d_points

Paper: PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation, Charles R. Qi et al. (2017). PointNet establishes the fundamental architecture for extracting permutation-invariant global representations directly from unordered 3D point clouds, which forms the basis of the autoencoder encoder design in the source paper.
Paper: A Point Set Generation Network for 3D Object Reconstruction from a Single Image, Haoqiang Fan et al. (2017). This work introduces deep neural network generation directly on unordered 3D point sets using permutation-invariant Chamfer and Earth Mover's distances as reconstruction objectives.
Paper: Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling, Jiajun Wu et al. (2016). This paper pioneered 3D generative adversarial modeling for synthesizing shapes, providing the foundational conceptual paradigm that the source extends from volumetric grids to point clouds.
Paper: Adversarial Autoencoders, Alireza Makhzani et al. (2015). It provides the foundational framework for latent-space generative modeling and shape interpolation by regularizing autoencoder representations with adversarial training.
Paper: 3D ShapeNets: A deep representation for volumetric shapes, Zhirong Wu et al. (2014). This seminal work introduced deep representation learning and probabilistic modeling for 3D geometric shapes alongside the standard ModelNet benchmark.
Paper: Tutorial on Variational Autoencoders, Carl Doersch (2016). It details the core principles of continuous latent variable generative modeling and autoencoding required to understand latent space shape generation.
Paper: Generative Adversarial Networks, Ian J. Goodfellow et al. (2014). It establishes the underlying generative adversarial network formulation evaluated and adapted for raw and latent 3D point cloud synthesis.

Paper: Learning Implicit Fields for Generative Shape Modeling, Zhiqin Chen et al. (2018). This work builds on point cloud and generative shape modeling by using implicit field decoders to overcome the resolution and discrete point limitations of point set autoencoders.
Paper: Occupancy Networks: Learning 3D Reconstruction in Function Space, Lars Mescheder et al. (2018). Occupancy Networks advance beyond discrete point cloud generation by learning continuous 3D shape functions conditioned on latent vectors for high-resolution shape representation and completion.
Paper: DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation, Jeong Joon Park et al. (2019). DeepSDF extends continuous latent-space shape modeling from point clouds to continuous signed distance functions for high-fidelity shape representation and completion.
Paper: Dynamic Graph CNN for Learning on Point Clouds, Yue Wang et al. (2018). Dynamic Graph CNN enhances point cloud feature learning beyond global pooling networks by incorporating dynamic local neighborhood graphs for superior representation quality.
Paper: Deep Learning for 3D Point Clouds: A Survey, Yulan Guo et al. (2019). This comprehensive survey provides an overview of subsequent advances in deep representation learning and generative architectures operating directly on raw 3D point clouds.
Paper: PointConv: Deep Convolutional Networks on 3D Point Clouds, Wenxuan Wu et al. (2018). PointConv generalizes convolution operations to non-uniform 3D point clouds, offering a more expressive alternative to PointNet-style encoders for representation learning.
Paper: PCT: Point cloud transformer, Meng-Hao Guo et al. (2020). PCT incorporates self-attention mechanisms to learn richer contextual geometric representations directly from unordered point clouds.
Paper: Efficient Geometry-aware 3D Generative Adversarial Networks, Eric R. Chan et al. (2022). EG3D scales 3D generative modeling to high-fidelity, geometry-aware implicit representations capable of multi-view consistent synthesis.

Cover for Learning Representations and Generative Models for 3D Point Clouds

Abstract

Three-dimensional geometric data offer an excellent domain for studying representation learning and generative modeling. In this paper, we look at geometric data represented as point clouds. We introduce a deep AutoEncoder (AE) network with state-of-the-art reconstruction quality and generalization ability. The learned representations outperform existing methods on 3D recognition tasks and enable shape editing via simple algebraic manipulations, such as semantic part editing, shape analogies and shape interpolation, as well as shape completion. We perform a thorough study of different generative models including GANs operating on the raw point clouds, significantly improved GANs trained in the fixed latent space of our AEs, and Gaussian Mixture Models (GMMs). To quantitatively evaluate generative models we introduce measures of sample fidelity and diversity based on matchings between sets of point clouds. Interestingly, our evaluation of generalization, fidelity and diversity reveals that GMMs trained in the latent space of our AEs yield the best results overall.

1 Introduction
2 Background
2.1 Point clouds
2.2 Fundamental building blocks
3 Evaluation Metrics for Generative Models
4 Models for Representation and Generation
4.1 Learning representations of 3D point clouds
4.2 Generative models for Point Clouds
5 Experimental Evaluation
5.1 Representational power of the AE
5.2 Evaluating the generative models
6 Related Work
7 Conclusion
References
A AE Details
A.1 AE used for SVM-based experiments
A.2 All other AEs
A.3 AE regularization
B Applications of the Latent Space Representation
C Autoencoding Human Forms
D Shape Completions
D.1 Architecture
D.2 Evaluation
E SVM Parameters for Auto-encoder Evaluation
F r-GAN Details
G l-GAN Details
H Model Selection of GANs
I Voxel AE Details
J Memorization Baseline
K More Comparisons with Wu et al.
L Limitations

Knowls

Knowl 1 — Point Cloud AutoEncoder Architecture (AE-EMD and AE-CD)
model/method
The 3D point cloud AutoEncoder (AE) compresses an input point cloud $S ⊂ \mathbb{R}^3$ of resolution $N = 2048$ ( $2048 \times 3$ matrix) into a low-dimensional bottleneck representation $z \in \mathbb{R}^k$ (default $k = 128$ ) and reconstructs an output point cloud $\hat{S} \in \mathbb{R}^{2048 \times 3}$ .
- Encoder: Employs five 1D convolutional layers with kernel size 1 and stride 1, operating on each 3D point independently. The filter dimensions across layers are $64, 128, 128, 256, k$ . Each convolutional layer is followed by a Rectified Linear Unit (ReLU) and a batch normalization layer. A permutation-invariant feature-wise maximum pooling operation across all $N$ points aggregates the local point activations into the single $k$ -dimensional latent vector $z$ .
- Decoder: A Multi-Layer Perceptron (MLP) consisting of three fully connected (FC) layers with dimensions $256, 256, 2048 \times 3$ . The first two FC layers use ReLU activations, while the final layer outputs the coordinates of the 2048 reconstructed points.
- Losses: Optimization is performed using permutation-invariant structural losses: the Earth Mover's Distance (EMD) or the Chamfer Distance (CD), yielding AE-EMD and AE-CD models respectively. Training uses the Adam optimizer with an initial learning rate of $0.0005$ , $\beta_1 = 0.9$ , and batch size 50.
Knowl 2 — Latent Space Generative Models for Point Clouds (l-GAN, l-WGAN, and Latent GMM)
model/method
Rather than generating raw 3D point clouds end-to-end, generative models can be trained within the fixed latent space $\mathbb{R}^k$ ( $k=128$ ) of a pre-trained point cloud AutoEncoder (AE):
1. Latent GAN (l-GAN): An adversarial network operating directly on latent vectors. The generator is an MLP with two FC-ReLU layers of sizes $\{128, 128\}$ mapping a 128-dimensional spherical Gaussian vector $z \sim \mathcal{N}(0, 0.2^2 I)$ to a latent code $\hat{x} \in \mathbb{R}^k$ . The discriminator is an MLP with two FC-ReLU layers of sizes $\{256, 512\}$ followed by a 1-unit sigmoid FC layer. Training uses the standard non-saturating GAN objective with Adam (initial learning rate $0.0001$ , $\beta_1 = 0.5$ , batch size 50).
2. Latent Wasserstein GAN (l-WGAN): Uses the same MLP generator and critic architecture as l-GAN but optimizes the Wasserstein distance with gradient penalty ( $̂\lambda = 10$ , with 5 critic update steps per generator step).
3. Latent Gaussian Mixture Model (latent GMM): A parametric GMM with $K$ components ( $K=32$ or $K=40$ ) and full covariance matrices is fitted to the training set latent codes using the Expectation-Maximization (EM) algorithm.
To synthesize novel point clouds, a latent code is sampled from the generator (l-GAN, l-WGAN, or GMM) and mapped through the pre-trained AE decoder to generate $2048 \times 3$ point sets.
Knowl 3 — Set-Matching Evaluation Metrics for Generative Point Cloud Models
definition
Quantitative evaluation of a generated point cloud set $A = \{S_a\}_{a=1}^{|A|}$ relative to a reference test set $B = \{S_b\}_{b=1}^{|B|}$ , where each point cloud $S \subset \mathbb{R}^3$ has $N$ points, is defined via the following point-set distances and set-level metrics:

Point-Set Distances:
- Earth Mover's Distance (EMD) between equal-sized point sets $S_1, S_2 \subset \mathbb{R}^3$ : $d_{\mathrm{EMD}}(S_1, S_2) = \min_{\phi: S_1 \to S_2} \sum_{x \in S_1} \|x - \phi(x)\|_2$ where $\phi$ is a bijection between $S_1$ and $S_2$ .
- Chamfer (pseudo-)Distance (CD): $d_{\mathrm{CD}}(S_1, S_2) = \sum_{x \in S_1} \min_{y \in S_2} \|x - y\|_2^2 + \sum_{y \in S_2} \min_{x \in S_1} \|x - y\|_2^2$
Set-Level Metrics:
1. Jensen-Shannon Divergence (JSD): Measures marginal spatial similarity across a canonical $28^3$ voxel grid in the ambient bounding volume. Point counts per voxel across all shapes in $A$ and $B$ define empirical distributions $P_A$ and $P_B$ : $\mathrm{JSD}(P_A \parallel P_B) = \frac{1}{2} D_{\mathrm{KL}}(P_A \parallel M) + \frac{1}{2} D_{\mathrm{KL}}(P_B \parallel M), \quad M = \frac{1}{2}(P_A + P_B)$ where $D_{\mathrm{KL}}$ denotes Kullback-Leibler divergence.
2. Minimum Matching Distance (MMD): Evaluates sample fidelity by averaging nearest-neighbor distances from each reference cloud to the synthetic set: $\mathrm{MMD}(A, B) = \frac{1}{|B|} \sum_{S_b \in B} \min_{S_a \in A} d(S_a, S_b)$ reported as MMD-CD or MMD-EMD depending on the metric $d$ .
3. Coverage (COV): Evaluates sample diversity as the fraction of reference point clouds matched to at least one synthetic point cloud: $\mathrm{COV}(A, B) = \frac{\left| \{ \arg\min_{S_b \in B} d(S_a, S_b) : S_a \in A \} \right|}{|B|}$ reported as COV-CD or COV-EMD in percentage.

Knowl 4 — Comparison of Generative Models on Point Clouds

data/table

Five generative models were trained on point clouds of the ShapeNet chair category and evaluated on the held-out test split. Models were selected at the training epoch or parameter setting minimizing JSD on the validation split, and evaluated by drawing synthetic sets of size $3\times$ the test set (averaged over three random runs):

Model	Latent Space	JSD ( $\downarrow$ )	MMD-CD ( $\downarrow$ )	MMD-EMD ( $\downarrow$ )	COV-EMD (%, $\uparrow$ )	COV-CD (%, $\uparrow$ )
Memorization Baseline	Train Data	0.017	0.0018	0.063	78.6	79.4
Raw GAN (r-GAN)	None (Raw Points)	0.176	0.0020	0.123	19.0	52.3
Latent GAN	AE-CD	0.048	0.0020	0.079	32.2	59.4
Latent GAN	AE-EMD	0.030	0.0023	0.069	57.1	59.3
Latent WGAN	AE-EMD	0.022	0.0019	0.066	66.9	67.6
Latent GMM (32-comp, Full)	AE-EMD	0.020	0.0018	0.065	67.4	68.9

Key empirical findings:

Training generative models in the AutoEncoder latent space significantly outperforms end-to-end raw point cloud GANs (r-GAN) in both fidelity and coverage.
Latent representations optimized with EMD loss achieve much higher sample diversity and fidelity than those trained with CD loss.
Standard latent GANs experience mode collapse mid-training, dropping coverage to under $1\%$ . Latent WGAN and a 32-component full-covariance latent GMM resist mode collapse and reach fidelity and coverage close to the memorization baseline.

Knowl 5 — Latent Space Vector Arithmetic for Semantic Editing, Analogies, and Interpolation
model/method
The latent space learned by a multi-class point cloud AutoEncoder (trained across 55 shape categories with EMD loss) exhibits smooth linear geometric structure amenable to vector arithmetic:
1. Semantic Part Editing: Let $\mathcal{A}$ and $\mathcal{B}$ be shape sub-populations where objects in $\mathcal{A}$ possess a semantic feature (e.g., chairs with armrests, or mugs with handles) and objects in $\mathcal{B}$ lack it. The average feature direction vector is computed as: $v = \frac{1}{|\mathcal{B}|} \sum_{B \in \mathcal{B}} z_B - \frac{1}{|\mathcal{A}|} \sum_{A \in \mathcal{A}} z_A$ For any shape $A \in \mathcal{A}$ with code $z_A$ , the modified code $z_{A'} = z_A + v$ decoded by the AE decoder produces a modified shape $A'$ removing or modifying the target part.
2. Latent Shape Interpolation: Linear interpolation between latent codes $z_1, z_2$ of two shapes via $z(\alpha) = (1 - \alpha) z_1 + \alpha z_2$ for $\alpha \in [0, 1]$ yields smooth morphing sequences that transition topologically and structurally across different objects, categories, or human motion poses.
3. Shape Analogies: Given an exemplar transformation $A \to A'$ and a query base shape $B$ , the analogous target shape $B'$ is obtained by computing $z = z_B + (z_{A'} - z_A)$ and finding the nearest neighbor shape code in the latent space.

Knowl 6 — ModelNet Classification via Unsupervised Point Cloud AutoEncoder Embeddings

data/table

An AutoEncoder with a 512-dimensional bottleneck was trained unsupervised on 57,000 shapes from 55 ShapeNet categories with random rotations along the gravity axis. The fixed 512-dimensional bottleneck activation vectors were then used to train a linear Support Vector Machine (SVM) classifier with $\ell_2$ -regularization and balanced class weights on the ModelNet10 and ModelNet40 benchmarks:

Method	ModelNet10 Accuracy (%)	ModelNet40 Accuracy (%)
Spherical Harmonics (SPH)	79.8	68.2
Light Field Descriptors (LFD)	79.9	75.5
T-L-Network	—	74.4
VConv-DAE	80.5	75.5
3D-GAN (Wu et al., 2016)	91.0	83.3
Ours AE-EMD (Linear SVM on 512-dim code)	95.4	84.0
Ours AE-CD (Linear SVM on 512-dim code)	95.4	84.5

The 512-dimensional point cloud AutoEncoder representations outperform prior unsupervised volumetric and view-based methods on both benchmarks, including 3D-GAN which extracts 7168-dimensional multi-layer features.

Knowl 7 — Raw Point Cloud Generative Adversarial Network (r-GAN)
model/method
The raw point cloud GAN (r-GAN) directly synthesizes $2048 \times 3$ point clouds without a pre-trained autoencoder:
- Discriminator: Consists of five 1D convolutional layers with kernel size 1, stride 1, and filter depths $\{64, 128, 256, 256, 512\}$ , interleaved with LeakyReLU activations (leak rate 0.2, no batch normalization). A feature-wise max-pooling layer aggregates point features into a global shape descriptor. This descriptor is processed by two fully connected LeakyReLU layers of sizes $\{128, 64\}$ leading to a single-unit sigmoid output neuron.
- Generator: Maps a 128-dimensional spherical Gaussian noise vector $z \sim \mathcal{N}(0, 0.2^2 I)$ to an output of size $2048 \times 3$ via five FC-ReLU layers with hidden dimensions $\{64, 128, 512, 1024, 2048 \times 3\}$ .
- Training: Optimized with Adam (initial learning rate $0.0001, \beta_1 = 0.5$ , batch size 50) using the non-saturating binary cross-entropy GAN objective.

Knowl 8 — Point-based Latent GMM vs. Volumetric 3D-GAN Generative Performance

data/table

A point-based 32-component full-covariance GMM trained in the latent space of an AE-EMD was compared against the volumetric 3D-GAN (Wu et al., 2016) across four ShapeNet categories. For 3D-GAN, $64^3$ voxel grids were converted to meshes using marching cubes (isovalue 0.1), filtered to a single connected component, and uniformly sampled to 2048 points. Evaluation was conducted on the held-out test splits:

Class	Fidelity: MMD-EMD ( $\downarrow$ )		Coverage: COV-EMD (%, $\uparrow$ )
	3D-GAN (Wu et al., 2016)	Ours (Point GMM)	3D-GAN (Wu et al., 2016)	Ours (Point GMM)
car	0.059	0.041	28.6	65.3
rifle	0.051	0.045	69.0	74.8
sofa	0.077	0.055	52.5	66.6
table	0.103	0.061	18.3	71.1

The point-based latent GMM achieves up to $4\times$ higher coverage and nearly $2\times$ lower MMD-EMD compared to 3D-GAN, despite 3D-GAN having access to all class models during training whereas the point GMM was evaluated strictly out-of-sample on the test split.

Knowl 9 — Abstractor-Predictor Architecture for Point Cloud Shape Completion
model/method
Point cloud shape completion maps a partial, occluded point cloud of resolution $N_{\mathrm{in}} = 2048$ to a dense complete shape of resolution $N_{\mathrm{out}} = 4096$ using an Abstractor-Predictor (AP) neural network:
- Architecture: The abstractor consists of three per-point 1D convolutional layers with filter sizes $[64, 128, 1024]$ without batch normalization, followed by a feature-wise max-pool producing a 1024-dimensional latent embedding. The predictor consists of an FC-ReLU layer with 1024 neurons and a final linear FC layer with $4096 \times 3$ output units.
- Optimization: Trained for up to 100 epochs using the Adam optimizer (initial learning rate 0.0005, batch size 50) using either EMD or CD loss between predicted and ground-truth full shapes.
- Empirical Characteristics: Evaluating at distance threshold $\rho = 0.02$ reveals that CD training yields higher completion accuracy (predicted points within $\rho$ of ground truth: Airplane 96.9%, Chair 86.5%, Table 87.6%) but lower completion coverage (ground-truth points within $\rho$ of prediction: Airplane 96.6%, Chair 77.5%, Table 75.2%) than EMD training (Coverage: Airplane 96.8%, Chair 82.6%, Table 83.0%). This reflects the greedy nature of CD, which concentrates points close to existing partial input points rather than dispersing them across missing regions.
Knowl 10 — Chamfer Distance Blindness to Point Hedging vs. Earth Mover's Distance
theoretical result
The Chamfer Distance (CD) and Earth Mover's Distance (EMD) exhibit fundamentally different sensitivity when evaluating generative point cloud fidelity:
1. Chamfer Metric Blindness: CD is vulnerable to "point hedging," wherein a generator clusters points heavily in high-density regions (such as chair seat centers) and places only a few sparse points elsewhere. In the CD formula, $d_{\mathrm{CD}}(S_1, S_2) = \sum_{x \in S_1} \min_{y \in S_2} \|x - y\|_2^2 + \sum_{y \in S_2} \min_{x \in S_1} \|x - y\|_2^2$ the first summand remains small because every point in $S_1$ has an immediate neighbor in the over-dense region of $S_2$ , while the second summand is only moderately penalized by the sparse points. Consequently, CD can score visually poor or mode-collapsed point clouds as high-fidelity matches.
2. EMD Metric Sensitivity: Because EMD requires a strict bijection $\phi: S_1 \to S_2$ , extra points clustered in one region must be transported over large Euclidean distances to match points in empty regions of the reference shape. As a result, EMD and EMD-based metrics (MMD-EMD, COV-EMD) heavily penalize point clustering and correlate much better with visual quality and mode coverage.
Knowl 11 — Limitations of Point Cloud AutoEncoders and Generative Models
limitation
The point cloud representation and generative architectures have several documented limitations:
1. Loss of High-Frequency Geometric Details: Point cloud AutoEncoders can miss intricate high-frequency geometry (such as decorative perforations or specific hole patterns in chair backs), altering the fine style of the shape.
2. Degradation on Rare Geometries: Shapes with atypical or rare structures within an object category are often reconstructed poorly by the AE.
3. Raw GAN Instability on Complex Objects: Training GANs directly on raw 3D coordinates (r-GAN) is unstable and fails to form coherent surface points on complex categories like cars, generating noisy and scattered geometries.
4. Style Transfer Limitation in Shape Completion: While the Abstractor-Predictor network reliably recovers global missing shape components, it may fail to preserve fine stylistic traits present in the partial input.

Coverage note — No substantial contributed material was omitted; all key model architectures, evaluation metrics, empirical results, and analytical findings are represented.

References

1.Arora, S. and Zhang, Y. Do gans actually learn the distribution? an empirical study. CoRR, abs/1706.08224, 2017.
2.Bogo, F., Romero, J., Pons-Moll, G., and Black, M. J. Dynamic FAUST: Registering human bodies in motion. In IEEE CVPR, 2017.
3.Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A. M., Jozefowicz, R., and Bengio, S. Generating sentences from a continuous space. CoRR, abs/1511.06349, 2015.
4.Brock, A., Lim, T., Ritchie, J. M., and Weston, N. Generative and discriminative voxel modeling with convolutional neural networks. CoRR, abs/1608.04236, 2016.
5.Bruna, J., Zaremba, W., Szlam, A., and LeCun, Y. Spectral networks and locally connected networks on graphs. CoRR, abs/1312.6203, 2013.
6.Chang, A. X., Funkhouser, T. A., Guibas, L. J., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L., and Yu, F. Shapenet: An information-rich 3d model repository. CoRR, abs/1512.03012, 2015.
7.Che, T., Li, Y., Jacob, A. P., Bengio, Y., and Li, W. Mode regularized generative adversarial networks. CoRR, abs/1612.02136, 2016.
8.Chen, D.-Y., Tian, X.-P., Shen, Y.-T., and Ouhyoung, M. On Visual Similarity Based 3D Model Retrieval. Computer Graphics Forum, 2003.
9.Dai, A., Qi, C. R., and Nießner, M. Shape completion using 3d-encoder-predictor cnns and shape synthesis. http://arxiv.org/abs/1612.00101, 2016.
10.Defferrard, M., Bresson, X., and Vandergheynst, P. Convolutional neural networks on graphs with fast localized spectral filtering. In NIPS, 2016.
11.Dempster, A. P., Laird, N. M., and Rubin, D. B. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society, 39(1), 1977.
12.Dilokthanakul, N., Mediano, P. A., Garnelo, M., Lee, M. C., Salimbeni, H., Arulkumaran, K., and Shanahan, M. Deep unsupervised clustering with gaussian mixture variational autoencoders. CoRR, abs/1611.02648, 2016.
13.Fan, H., Su, H., and Guibas, L. J. A point set generation network for 3d object reconstruction from a single image. CoRR, abs/1612.00603, 2016.
14.Girdhar, R., Fouhey, D. F., Rodriguez, M., and Gupta, A. Learning a predictable and generative vector representation for objects. In ECCV, 2016.
15.Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In NIPS, 2014.
16.Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C. Improved training of wasserstein gans. CoRR, abs/1704.00028, 2017.
17.Hegde, V. and Zadeh, R. Fusionnet: 3d object classification using multiple data representations. CoRR, abs/1607.05695, 2016.
18.Henaff, M., Bruna, J., and LeCun, Y. Deep convolutional networks on graph-structured data. CoRR, abs/1506.05163, 2015.
19.Huang, R., Achlioptas, P., Guibas, L., and Ovsjanikov, M. Latent space representation for shape analysis and learning. http://arxiv.org/abs/1806.03967, 2018.
20.Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
21.Jakob, W. Mitsuba renderer, 2010. http://www.mitsuba-renderer.org.
22.Kalogerakis, E., Averkiou, M., Maji, S., and Chaudhuri, S. 3d shape segmentation with projective convolutional networks. CoRR, abs/1612.02808, 2016.
23.Kazhdan, M., Funkhouser, T., and Rusinkiewicz, S. Rotation invariant spherical harmonic representation of 3d shape descriptors. In ACM SGP, 2003.
24.Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
25.Kingma, D. P. and Welling, M. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013.
26.Kingma, D. P., Salimans, T., and Welling, M. Improving variational inference with inverse autoregressive flow. CoRR, abs/1606.04934, 2016.
27.Kullback, S. and Leibler, R. A. On information and sufficiency. Annals of Mathematical Statistics, 1951.
28.Lewiner, T., Lopes, H., Vieira, A. W., and Tavares, G. Efficient implementation of marching cubes’ cases with topological guarantees. Journal of Graphics Tools, 2003.
29.Maas, A. L., Hannun, A. Y., and Ng, A. Y. Rectifier nonlinearities improve neural network acoustic models. In ICML, 2013.
30.Maimaitimin, M., Watanabe, K., and Maeyama, S. Stacked convolutional auto-encoders for surface recognition based on 3d point cloud data. Artificial Life and Robotics, 2017.
31.Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., and Frey, B. Adversarial autoencoders. CoRR, abs/1511.05644, 2015.
32.Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J. Distributed representations of words and phrases and their compositionality. CoRR, abs/1310.4546, 2013.
33.Nair, V. and Hinton, G. E. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
34.Qi, C. R., Su, H., Mo, K., and Guibas, L. J. Pointnet: Deep learning on point sets for 3d classification and segmentation. CoRR, abs/1612.00593, 2016a.
35.Qi, C. R., Su, H., Niener, M., Dai, A., Yan, M., and Guibas, L. J. Volumetric and multi-view cnns for object classification on 3d data. In IEEE CVPR, 2016b.
36.Qi, C. R., Yi, L., Su, H., and Guibas, L. J. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. CoRR, 2017.
37.Radford, A., Metz, L., and Chintala, S. CoRR, abs/1511.06434.
38.Rubner, Y., Tomasi, C., and Guibas, L. J. The earth mover’s distance as a metric for image retrieval. IJCV, 2000.
39.Rumelhart, D. E., Hinton, G. E., and Williams, R. J. Learning representations by back-propagating errors. Cognitive modeling, 5, 1988.
40.Rustamov, R. M., Ovsjanikov, M., Azencot, O., Ben-Chen, M., Chazal, F., and Guibas, L. Map-based exploration of intrinsic shape differences and variability. ACM Trans. Graph., 32(4), July 2013.
41.Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. In NIPS, 2016.
42.Sharma, A., Grau, O., and Fritz, M. Vconv-dae: Deep volumetric shape learning without object labels. In ECCV Workshop, 2016.
43.Snderby, C. K., Raiko, T., Maale, L., Snderby, S. K., and Winther, O. How to train deep variational autoencoders and probabilistic ladder networks. CoRR, abs/1602.02282, 2016.
44.Su, H., Maji, S., Kalogerakis, E., and Learned-Miller, E. G. Multi-view convolutional neural networks for 3d shape recognition. In 2015 IEEE ICCV, 2015.
45.Sung, M., Kim, V. G., Angst, R., and Guibas, L. J. Data-driven structural priors for shape completion. ACM Transactions on Graphics (TOG), 34(6):175, 2015.
46.Tasse, F. P. and Dodgson, N. Shape2vec: Semantic-based descriptors for 3d shapes, sketches and images. ACM Trans. Graph., 2016.
47.Tatarchenko, M., Dosovitskiy, A., and Brox, T. Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs. CoRR, abs/1703.09438, 2017.
48.Wang, Y., Xie, Z., Xu, K., Dou, Y., and Lei, Y. An efficient and effective convolutional auto-encoder extreme learning machine network for 3d feature learning. Neurocomputing, 174, 2016.
49.Wei, L., Huang, Q., Ceylan, D., Vouga, E., and Li, H. Dense human body correspondences using convolutional networks. In IEEE CVPR, 2016.
50.Wu, J., Zhang, C., Xue, T., Freeman, B., and Tenenbaum, J. Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. In Lee, D. D., Sugiyama, M., Luxburg, U. V., Guyon, I., and Garnett, R. (eds.), NIPS. 2016.
51.Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., and Xiao, J. 3d shapenets: A deep representation for volumetric shapes. In IEEE CVPR, 2015.
52.Yi, L., Kim, V. G., Ceylan, D., Shen, I., Yan, M., Su, H., Lu, C., Huang, Q., Sheffer, A., and Guibas, L. J. A scalable active framework for region annotation in 3d shape collections. ACM Trans. Graph., 35(6), 2016a.
53.Yi, L., Su, H., Guo, X., and Guibas, L. J. Syncspeccnn: Synchronized spectral CNN for 3d shape segmentation. CoRR, abs/1612.00606, 2016b.
54.Zhu, Z., Wang, X., Bai, S., Yao, C., and Bai, X. Deep learning representation using autoencoder for 3d shape retrieval. Neurocomputing, 2016.

Citation

MLA

Achlioptas, P., et al. “Learning Representations and Generative Models for 3D Point Clouds”. 35th International Conference on Machine Learning (ICML), 2018, 2017, http://arxiv.org/abs/1707.02392v3.

APA

Achlioptas, P., Diamanti, O., Mitliagkas, I., & Guibas, L. (2017). Learning Representations and Generative Models for 3D Point Clouds. 35th International Conference on Machine Learning (ICML), 2018. http://arxiv.org/abs/1707.02392v3

Chicago

Achlioptas, P., O. Diamanti, I. Mitliagkas, and L. Guibas. 2017. “Learning Representations and Generative Models for 3D Point Clouds”. 35th International Conference on Machine Learning (ICML), 2018. http://arxiv.org/abs/1707.02392v3.

Harvard

Achlioptas, P. et al. (2017) “Learning Representations and Generative Models for 3D Point Clouds”, 35th International Conference on Machine Learning (ICML), 2018 [Preprint]. Available at: http://arxiv.org/abs/1707.02392v3.

Vancouver

1. Achlioptas P, Diamanti O, Mitliagkas I, Guibas L (2017) Learning Representations and Generative Models for 3D Point Clouds. 35th International Conference on Machine Learning (ICML), 2018

BibTeX

@article{achlioptas2017learning,
  title = {Learning Representations and Generative Models for 3D Point Clouds},
  author = {Achlioptas, Panos and Diamanti, Olga and Mitliagkas, Ioannis and Guibas, Leonidas},
  year = {2017},
  journal = {35th International Conference on Machine Learning (ICML), 2018},
  url = {http://arxiv.org/abs/1707.02392v3},
  eprint = {1707.02392}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Learning Representations and Generative Models for 3D Point Clouds

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Point Cloud AutoEncoder Architecture (AE-EMD and AE-CD)

Knowl 2 — Latent Space Generative Models for Point Clouds (l-GAN, l-WGAN, and Latent GMM)

Knowl 3 — Set-Matching Evaluation Metrics for Generative Point Cloud Models

Knowl 4 — Comparison of Generative Models on Point Clouds

Knowl 5 — Latent Space Vector Arithmetic for Semantic Editing, Analogies, and Interpolation

Knowl 6 — ModelNet Classification via Unsupervised Point Cloud AutoEncoder Embeddings

Knowl 7 — Raw Point Cloud Generative Adversarial Network (r-GAN)

Knowl 8 — Point-based Latent GMM vs. Volumetric 3D-GAN Generative Performance

Knowl 9 — Abstractor-Predictor Architecture for Point Cloud Shape Completion

Knowl 10 — Chamfer Distance Blindness to Point Hedging vs. Earth Mover's Distance

Knowl 11 — Limitations of Point Cloud AutoEncoders and Generative Models

References

Citation

BibTeX

Source Code

Access the Paper