Back

Learning 3D Representations from 2D Pre-Trained Models via Image-to-Point Masked Autoencoders

Renrui ZhangLiuhui WangYu QiaoPeng GaoHongsheng Li

article2023CVPR190 citations

Proposes I2P-MAE, a self-supervised framework that transfers rich visual knowledge from off-the-shelf 2D pre-trained models to 3D point cloud learning through semantically guided masking and multi-view feature reconstruction.

Centre for Perceptual and Interactive IntelligencePeking UniversityShanghai Artificial Intelligence LaboratoryThe Chinese University of Hong Kongautoencodersencoder-decodermulti-views
arXiv: 2212.06785ZrrSkywalker/I2P-MAE
Cover for Learning 3D Representations from 2D Pre-Trained Models via Image-to-Point Masked Autoencoders

Abstract

Pre-training by numerous image data has become de-facto for robust 2D representations. In contrast, due to the expensive data processing, a paucity of 3D datasets severely hinders the learning for high-quality 3D features. In this paper, we propose an alternative to obtain superior 3D representations from 2D pre-trained models via Image-to-Point Masked Autoencoders, named as I2P-MAE. By self-supervised pre-training, we leverage the well learned 2D knowledge to guide 3D masked autoencoding, which reconstructs the masked point tokens with an encoder-decoder architecture. Specifically, we first utilize off-the-shelf 2D models to extract the multi-view visual features of the input point cloud, and then conduct two types of image-to-point learning schemes. For one, we introduce a 2D-guided masking strategy that maintains semantically important point tokens to be visible. Compared to random masking, the network can better concentrate on significant 3D structures with key spatial cues. For another, we enforce these visible tokens to reconstruct multi-view 2D features after the decoder. This enables the network to effectively inherit high-level 2D semantics for discriminative 3D modeling. Aided by our image-to-point pre-training, the frozen I2P-MAE, without any fine-tuning, achieves 93.4% accuracy for linear SVM on ModelNet40, competitive to existing fully trained methods. By further fine-tuning on on ScanObjectNN's hardest split, I2P-MAE attains the state-of-the-art 90.11% accuracy, +3.68% to the second-best, demonstrating superior transferable capacity. Code is available at https://github.com/ZrrSkywalker/I2P-MAE.

Topics

autoencodersencoder-decodermulti-viewspoint cloudsemanticssvmtwo dimensionalvisual featureauto encodersroot mean square errors

Table of Contents

  • 1. Introduction
  • 2. Related Work
  • 3. Method
  • 3.1. Basic 3D Architecture
  • 3.2. 2D Pre-trained Representations
  • 4. Experiments
  • 4.1. Image-to-Point Pre-training
  • 4.2. Downstream Tasks
  • 4.3. Ablation Study
  • 5. Visualization
  • 6. Conclusion
  • References

Knowls

  1. Knowl 1 — I2P-MAE Architecture and Pre-training Pipeline

    model/method

    Image-to-Point Masked Autoencoders (I2P-MAE) is a self-supervised pre-training framework that transfers representations from pre-trained 2D vision models into 3D point cloud masked autoencoders.

    Given an input point cloud P∈RN×3P \in \mathbb{R}^{N \times 3}, Furthest Point Sampling (FPS) downsamples the NN points to MM center points, denoted as PT∈RM×3P^T \in \mathbb{R}^{M \times 3}. For each downsampled center, a kk-Nearest Neighbors (kk-NN) search finds kk neighboring points, which are mapped into MM point tokens T∈RM×CT \in \mathbb{R}^{M \times C} via a mini-PointNet, where CC is the token feature dimension.

    A 2D-guided masking strategy divides the MM tokens into MvisM_{\text{vis}} visible tokens Tvis∈RMvis×CT_{\text{vis}} \in \mathbb{R}^{M_{\text{vis}} \times C} and MmaskM_{\text{mask}} masked tokens, where M=Mvis+MmaskM = M_{\text{vis}} + M_{\text{mask}}. The visible tokens TvisT_{\text{vis}} are processed by a hierarchical multi-stage transformer encoder. The encoded visible features TviseT_{\text{vis}}^e are concatenated with shared learnable mask tokens Tmask∈RMmask×CT_{\text{mask}} \in \mathbb{R}^{M_{\text{mask}} \times C} and passed to a lightweight multi-stage transformer decoder, producing decoded tokens {Tvisd,Tmaskd}\{T_{\text{vis}}^d, T_{\text{mask}}^d\}.

    Two parallel reconstruction heads perform multi-task decoding:

    1. A 3D coordinate head H3D(⋅)H_{\text{3D}}(\cdot) predicts the 3D coordinates of masked point patches Pmask∈RMmask×k×3P_{\text{mask}} \in \mathbb{R}^{M_{\text{mask}} \times k \times 3} from TmaskdT_{\text{mask}}^d.
    2. A 2D semantic head H2D(⋅)H_{\text{2D}}(\cdot) reconstructs projected multi-view 2D visual features Fvis3D∈RMvis×3CF_{\text{vis}}^{\text{3D}} \in \mathbb{R}^{M_{\text{vis}} \times 3C} from TvisdT_{\text{vis}}^d.

  2. Knowl 2 — Multi-View Orthogonal Depth Projection and 2D Representation Extraction

    model/method

    To extract 2D representations without costly rendering or learnable prompting, I2P-MAE projects the input point cloud P∈RN×3P \in \mathbb{R}^{N \times 3} onto three orthogonal 2D planes aligned with the Cartesian axes (x,y,zx, y, z). For each axis, the corresponding coordinate is omitted and treated as the depth pixel value, while the remaining two coordinates are floored to index 2D pixel coordinates on a discrete image grid. The single-channel depth value is repeated across three channels to form three multi-view depth maps {Ii}i=13∈RH0×W0×3\{I_i\}_{i=1}^3 \in \mathbb{R}^{H_0 \times W_0 \times 3}.

    A frozen pre-trained 2D vision backbone (such as a Vision Transformer pre-trained via CLIP) processes {Ii}i=13\{I_i\}_{i=1}^3 to extract two modalities of 2D guidance:

    1. Multi-view 2D visual feature maps {Fi2D}i=13\{F_i^{\text{2D}}\}_{i=1}^3, where each Fi2D∈RH×W×CF_i^{\text{2D}} \in \mathbb{R}^{H \times W \times C}.
    2. Multi-view 2D attention maps {Si2D}i=13\{S_i^{\text{2D}}\}_{i=1}^3, where each Si2D∈RH×W×1S_i^{\text{2D}} \in \mathbb{R}^{H \times W \times 1}. For Vision Transformers, these are the self-attention weights between the class token and patch tokens at the final layer; for ResNet models, they are obtained via pixel-wise channel max-pooling.

  3. Knowl 3 — 2D-Guided Masking via Spatial Attention Clouds

    equation

    Rather than masking point tokens uniformly at random, I2P-MAE leverages 2D attention maps to prioritize preserving semantically informative 3D structural regions.

    Let {Si2D}i=13∈RH×W×1\{S_i^{\text{2D}}\}_{i=1}^3 \in \mathbb{R}^{H \times W \times 1} denote the multi-view 2D attention maps and PT∈RM×3P^T \in \mathbb{R}^{M \times 3} denote the downsampled point patch centers. The 2D-to-3D back-projection operator I2P⁡(Si2D,PT)\operatorname{I2P}(S_i^{\text{2D}}, P^T) maps 3D point coordinates to their corresponding 2D spatial pixel coordinates on the ii-th view's attention map. The multi-view scores are averaged and normalized across all MM tokens via a softmax function to generate a 3D spatial attention cloud S3D∈RM×1S^{\text{3D}} \in \mathbb{R}^{M \times 1}:

    S3D=Softmax⁡(13∑i=13I2P⁡(Si2D,PT))S^{\text{3D}} = \operatorname{Softmax}\left(\frac{1}{3}\sum_{i=1}^3 \operatorname{I2P}(S_i^{\text{2D}}, P^T)\right)

    Each scalar entry in S3DS^{\text{3D}} represents the sampling probability for the corresponding point token to be retained as a visible token TvisT_{\text{vis}}. Tokens located in semantically critical regions receive higher visible probabilities, ensuring the encoder focuses on structural landmarks.

  4. Knowl 4 — 2D-Semantic Feature Reconstruction and Joint Pre-training Objective

    equation

    To transfer high-level semantic knowledge into 3D representations, I2P-MAE tasks the decoded visible tokens Tvisd∈RMvis×CT_{\text{vis}}^d \in \mathbb{R}^{M_{\text{vis}} \times C} with reconstructing multi-view 2D visual features.

    Target 2D representations Fvis3D∈RMvis×3CF_{\text{vis}}^{\text{3D}} \in \mathbb{R}^{M_{\text{vis}} \times 3C} are constructed by indexing into the multi-view 2D feature maps {Fi2D}i=13\{F_i^{\text{2D}}\}_{i=1}^3 using the visible point patch center coordinates PvisT∈RMvis×3P_{\text{vis}}^T \in \mathbb{R}^{M_{\text{vis}} \times 3} and concatenating across all 3 views:

    Fvis3D=Concat⁡{I2P⁡(Fi2D,PvisT)}i=13F_{\text{vis}}^{\text{3D}} = \operatorname{Concat}\Big\{\operatorname{I2P}(F_i^{\text{2D}}, P_{\text{vis}}^T)\Big\}_{i=1}^3

    The 2D semantic reconstruction loss L2D\mathcal{L}_{\text{2D}} is the mean squared error (ℓ2\ell_2 loss) between a linear projection head H2D(Tvisd)H_{\text{2D}}(T_{\text{vis}}^d) and the target Fvis3DF_{\text{vis}}^{\text{3D}}:

    L2D=1Mvis(H2D(Tvisd)−Fvis3D)2\mathcal{L}_{\text{2D}} = \frac{1}{M_{\text{vis}}}\left(H_{\text{2D}}(T_{\text{vis}}^d) - F_{\text{vis}}^{\text{3D}}\right)^2

    The 3D coordinate reconstruction loss L3D\mathcal{L}_{\text{3D}} is the Chamfer distance between the predicted points from the linear head H3D(Tmaskd)H_{\text{3D}}(T_{\text{mask}}^d) and the ground-truth masked patch coordinates Pmask∈RMmask×k×3P_{\text{mask}} \in \mathbb{R}^{M_{\text{mask}} \times k \times 3}:

    L3D=1MmaskkChamfer⁡(H3D(Tmaskd),Pmask)\mathcal{L}_{\text{3D}} = \frac{1}{M_{\text{mask}} k}\operatorname{Chamfer}\left(H_{\text{3D}}(T_{\text{mask}}^d), P_{\text{mask}}\right)

    The total pre-training loss is:

    LI2P=L3D+L2D\mathcal{L}_{\text{I2P}} = \mathcal{L}_{\text{3D}} + \mathcal{L}_{\text{2D}}

  5. Knowl 5 — Experimental Pre-training Setup for I2P-MAE

    experimental setup

    I2P-MAE is pre-trained on ShapeNet, which contains 57,448 synthetic CAD point clouds spanning 55 object categories.

    Hyperparameters and structural details include:

    • Point Cloud Sampling: N=2048N = 2048 input points per cloud, downsampled via FPS to M=512M = 512 centers, with k=16k = 16 nearest neighbors per patch.
    • Architecture: Hierarchical transformer backbone with a 3-stage encoder (5 blocks per stage) and a 2-stage decoder (1 block per stage). Feature channel dimension C=384C = 384.
    • Masking: Masking ratio of 80% (Mmask=410M_{\text{mask}} = 410, Mvis=102M_{\text{vis}} = 102).
    • 2D Teacher: Frozen CLIP pre-trained ViT-Base. Multi-view depth maps are rendered at 224×224224 \times 224 pixels, outputting visual feature maps of spatial size H×W=14×14H \times W = 14 \times 14.
    • Optimization: AdamW optimizer, initial learning rate 10−310^{-3}, weight decay 5×10−25 \times 10^{-2}, batch size 64, trained for 300 epochs with a cosine learning rate scheduler and a 10-epoch linear warm-up.

  6. Knowl 6 — Linear SVM Classification on Synthetic and Real-World Point Clouds

    data/table

    To evaluate the quality and transferability of representations learned during self-supervised pre-training without fine-tuning the backbone weights, a linear Support Vector Machine (SVM) classifier is trained on fixed encoder representations on synthetic ModelNet40 and real-world ScanObjectNN (OBJ-BG split).

    Method ModelNet40 (%) OBJ-BG (%)
    SO-Net 87.3 -
    FoldingNet 88.4 -
    VIP-GAN 90.2 -
    DGCNN + Jigsaw 90.6 59.5
    DGCNN + OcCo 90.7 78.3
    DGCNN + CrossPoint 91.2 81.7
    Transformer + OcCo 89.6 -
    Point-BERT 87.4 -
    Point-MAE 91.0 77.7
    3D-OAE 92.3 -
    Point-M2AE 92.9 84.1
    I2P-MAE 93.4 87.1
    Improvement +0.5 +3.0

    I2P-MAE achieves 93.4% accuracy on ModelNet40 and 87.1% on ScanObjectNN OBJ-BG, outperforming Point-M2AE by +0.5% and +3.0% respectively. Without fine-tuning, the linear SVM accuracy of I2P-MAE exceeds the fully fine-tuned accuracies of several supervised and self-supervised baseline networks.

  7. Knowl 7 — Fine-Tuning on Real-World 3D Classification via ScanObjectNN

    data/table

    The pre-trained I2P-MAE encoder is fine-tuned on the real-world ScanObjectNN dataset, which contains 11,416 training and 2,882 test point clouds scanned from indoor scenes containing background clutter and occlusions. Evaluation is conducted across three official benchmark splits: OBJ-BG, OBJ-ONLY, and the hardest PB-T50-RS.

    Method OBJ-BG (%) OBJ-ONLY (%) PB-T50-RS (%)
    PointNet 73.3 79.2 68.0
    PointNet++ 82.3 84.3 77.9
    PointCNN 86.1 85.5 78.5
    MVTN - - 82.8
    PointMLP - - 85.2
    Transformer 79.86 80.55 77.24
    Point-BERT (Pre-trained) 87.43 88.12 83.07
    3D-OAE (Pre-trained) 89.16 88.64 83.17
    MaskPoint (Pre-trained) 89.30 88.10 84.30
    Point-MAE (Pre-trained) 90.02 88.29 85.18
    MAE3D (Pre-trained) - - 86.20
    Point-M2AE (Pre-trained) 91.22 88.81 86.43
    I2P-MAE (Pre-trained) 94.15 91.57 90.11
    Improvement vs. Second-Best +2.93 +2.76 +3.68

    I2P-MAE outperforms all prior methods across all splits, achieving 94.15% (+2.93%), 91.57% (+2.76%), and 90.11% (+3.68%) accuracy over Point-M2AE, making it the first model to reach over 90% accuracy on the PB-T50-RS split.

  8. Knowl 8 — Fine-Tuning on Synthetic 3D Classification and ShapeNetPart Segmentation

    data/table

    I2P-MAE is fine-tuned and evaluated on synthetic point cloud classification using ModelNet40 (9,843 training and 2,468 testing shapes across 40 CAD categories, evaluated with 1k1\text{k} points) and 3D part segmentation on ShapeNetPart (14,007 training and 2,874 validation shapes across 16 categories and 50 parts).

    ModelNet40 Classification Accuracy:

    Method No Voting (%) Voting (%)
    PointNet 89.2 -
    PointNet++ 90.7 -
    PointCNN 92.2 -
    Point Transformer - 93.7
    Transformer 91.4 -
    Point-BERT (Pre-trained) 92.7 93.2
    3D-OAE (Pre-trained) - 93.4
    Point-MAE (Pre-trained) 93.2 93.8
    Point-M2AE (Pre-trained) 93.4 94.0
    I2P-MAE (Linear SVM, No fine-tuning) 93.4 -
    I2P-MAE (Fine-tuned) 93.7 94.1

    ShapeNetPart Segmentation Performance:

    Method mIoUC\text{mIoU}_C (%) mIoUI\text{mIoU}_I (%)
    PointNet 80.39 83.70
    PointNet++ 81.85 85.10
    DGCNN 82.33 85.20
    PointMLP 84.60 86.10
    Transformer 83.42 85.10
    Point-BERT (Pre-trained) 84.11 85.60
    MaskPoint (Pre-trained) 84.40 86.00
    Point-MAE (Pre-trained) - 86.10
    Point-M2AE (Pre-trained) 84.86 86.51
    I2P-MAE (Pre-trained) 85.15 86.76

    Fine-tuned I2P-MAE attains 93.7% accuracy (94.1% with offline multi-view voting) on ModelNet40 and sets new state-of-the-art results on ShapeNetPart with 85.15% category mean IoU (mIoUC\text{mIoU}_C) and 86.76% instance mean IoU (mIoUI\text{mIoU}_I).

  9. Knowl 9 — Data Efficiency in Self-Supervised 3D Pre-training under Limited Data

    data/table

    To evaluate pre-training data efficiency when 3D data is scarce, I2P-MAE, Point-MAE, and Point-M2AE are pre-trained on varying sub-sampled proportions (20%, 40%, 60%, 80%, 100%) of the ShapeNet dataset and evaluated via linear SVM classification accuracy (%) on ModelNet40.

    Method 20% 40% 60% 80% 100%
    Point-MAE 89.4 90.2 90.3 90.5 91.0
    Point-M2AE 90.8 92.0 92.1 92.4 92.9
    I2P-MAE 92.1 92.7 93.1 93.1 93.4
    Improvement vs Point-M2AE +1.3 +0.7 +1.0 +0.7 +0.5

    I2P-MAE trained on only 20% of ShapeNet reaches 92.1% accuracy, exceeding Point-MAE pre-trained on 100% of ShapeNet (91.0%). Furthermore, I2P-MAE pre-trained on 60% of ShapeNet achieves 93.1% accuracy, outperforming Point-M2AE pre-trained on 100% of the data (92.9%), demonstrating that 2D prior knowledge effectively compensates for 3D data scarcity.

  10. Knowl 10 — Ablation Studies on 2D-Guided Masking and Semantic Reconstruction Targets

    data/table

    Ablations on ModelNet40 and ScanObjectNN (OBJ-BG) via linear SVM accuracy (%) isolate the contributions of the 2D-guided masking strategy and the assignment of 2D vs. 3D reconstruction targets.

    Masking Strategy Ablation:

    2D-guided Visible Selection Mask Ratio ModelNet40 (%) OBJ-BG (%)
    ✓ Important 0.8 93.4 87.1
    - Random 0.8 93.0 86.2
    ✓ Unimportant 0.8 92.8 84.5
    ✓ Important 0.7 92.5 83.8
    ✓ Important 0.9 92.8 85.6

    Selecting semantically critical tokens as visible via 2D attention improves accuracy by +0.4% on ModelNet40 and +0.9% on OBJ-BG over uniform random masking. Inverting selection to retain unimportant tokens drops performance significantly (92.8% / 84.5%).

    Reconstruction Target Assignment Ablation:

    Targeted Tokens Target Modality ModelNet40 (%) OBJ-BG (%)
    Masked Visible 3D 2D
    ✓ ✓ Masked Visible 93.4 87.1
    ✓ - Masked - 92.9 84.7
    - ✓ - Visible 91.9 80.2
    ✓ - - Masked 91.2 77.8
    ✓ - Masked Masked 92.6 84.9

    Reconstructing 2D semantics on visible tokens combined with 3D coordinates on masked tokens achieves the best accuracy (+0.5% and +2.4% over 3D-only reconstruction). Reconstructing both 3D coordinates and 2D semantics exclusively on masked tokens degrades performance (92.6% / 84.9%) due to 2D-3D target conflicts and forcing semantic reconstruction onto less critical masked regions.

Coverage note — No substantial contributed material was omitted; the knowls encompass the core I2P-MAE architecture, orthogonal projection mechanics, loss formulations, experimental configurations, linear probe classifications, fine-tuned classification and segmentation benchmarks, data scarcity studies, and detailed ablation analyses.

References

  1. 1.Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo. Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. arXiv preprint arXiv:2203.00680, 2022.
  2. 2.Anonymous. Rethinking network design and local geometry in point cloud: A simple residual MLP framework. In Submitted to The Tenth International Conference on Learning Representations, 2022. under review.
  3. 3.Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint arXiv:2202.03555, 2022.
  4. 4.Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021.
  5. 5.Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650–9660, 2021.
  6. 6.Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.
  7. 7.Anthony Chen, Kevin Zhang, Renrui Zhang, Zihan Wang, Yuheng Lu, Yandong Guo, and Shanghang Zhang. Pimae: Point cloud and image interactive masked autoencoders for 3d object detection. arXiv preprint arXiv:2303.08129, 2023.
  8. 8.Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2017.
  9. 9.Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  10. 10.Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750–15758, 2021.
  11. 11.Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  12. 12.Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11162–11173, 2021.
  13. 13.Runpei Dong, Zekun Qi, Linfeng Zhang, Junbo Zhang, Jianjian Sun, Zheng Ge, Li Yi, and Kaisheng Ma. Autoencoders as cross-modal teachers: Can pretrained 2d image transformers help 3d representation learning? arXiv preprint arXiv:2212.08320, 2022.
  14. 14.Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  15. 15.Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 605–613, 2017.
  16. 16.Kexue Fu, Peng Gao, ShaoLei Liu, Renrui Zhang, Yu Qiao, and Manning Wang. Pos-bert: Point cloud one-stage bert pre-training. arXiv preprint arXiv:2204.00989, 2022.
  17. 17.Kexue Fu, Peng Gao, Renrui Zhang, Hongsheng Li, Yu Qiao, and Manning Wang. Distillation with contrast is all you need for self-supervised point cloud representation learning. arXiv preprint arXiv:2202.04241, 2022.
  18. 18.Peng Gao, Teli Ma, Hongsheng Li, Jifeng Dai, and Yu Qiao. Convmae: Masked convolution meets masked autoencoders. arXiv preprint arXiv:2205.03892, 2022.
  19. 19.Peng Gao, Renrui Zhang, Rongyao Fang, Ziyi Lin, Hongyang Li, Hongsheng Li, and Qiao Yu. Mimic before reconstruct: Enhancing masked autoencoders with feature mimicking. arXiv preprint arXiv:2303.05475, 2023.
  20. 20.Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, and Jia Deng. Revisiting point cloud shape classification with a simple and effective baseline. arXiv preprint arXiv:2106.05304, 2021.
  21. 21.Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271–21284, 2020.
  22. 22.Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pct: Point cloud transformer. Computational Visual Media, 7(2):187–199, 2021.
  23. 23.Ziyu Guo, Xianzhi Li, and Pheng Ann Heng. Joint-mae: 2d-3d joint masked autoencoders for 3d point cloud pre-training. arXiv preprint arXiv:2302.14007, 2023.
  24. 24.Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, and Bin Cui. Calip: Zero-shot enhancement of clip with parameter-free attention. arXiv preprint arXiv:2209.14169, 2022.
  25. 25.Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. Mvtn: Multi-view transformation network for 3d shape recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1–11, 2021.
  26. 26.Zhizhong Han, Mingyang Shang, Yu-Shen Liu, and Matthias Zwicker. View inter-prediction gan: Unsupervised representation learning for 3d shapes by learning global shape memories to support local view predictions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8376–8384, 2019.
  27. 27.Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377, 2021.
  28. 28.Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
  29. 29.Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  30. 30.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  31. 31.Lang Huang, Shan You, Mingkai Zheng, Fei Wang, Chen Qian, and Toshihiko Yamasaki. Green hierarchical vision transformer for masked image modeling. arXiv preprint arXiv:2205.13515, 2022.
  32. 32.Siyuan Huang, Yichen Xie, Song-Chun Zhu, and Yixin Zhu. Spatio-temporal self-supervised representation learning for 3d point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6535–6545, 2021.
  33. 33.Jincen Jiang, Xuequan Lu, Lizhi Zhao, Richard Dazeley, and Meili Wang. Masked autoencoders in 3d point cloud representation learning. arXiv preprint arXiv:2207.01545, 2022.
  34. 34.Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  35. 35.Jiaxin Li, Ben M Chen, and Gim Hee Lee. So-net: Self-organizing network for point cloud analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9397–9406, 2018.
  36. 36.Ruihui Li, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. Pu-gan: a point cloud upsampling adversarial network. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7203–7212, 2019.
  37. 37.Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointcnn: Convolution on x-transformed points. Advances in neural information processing systems, 31:820–830, 2018.
  38. 38.Zhenyu Li, Zehui Chen, Ang Li, Liangji Fang, Qinhong Jiang, Xianming Liu, Junjun Jiang, Bolei Zhou, and Hang Zhao. Simipu: Simple 2d image and 3d point cloud unsupervised pre-training for spatial-aware visual representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1500–1508, 2022.
  39. 39.Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  40. 40.Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014.
  41. 41.Haotian Liu, Mu Cai, and Yong Jae Lee. Masked discrimination for self-supervised learning on point clouds. arXiv preprint arXiv:2203.11183, 2022.
  42. 42.Jihao Liu, Xin Huang, Yu Liu, and Hongsheng Li. Mixmim: Mixed and masked image modeling for efficient visual representation learning. arXiv preprint arXiv:2205.13137, 2022.
  43. 43.Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan. Relation-shape convolutional neural network for point cloud analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8895–8904, 2019.
  44. 44.Yueh-Cheng Liu, Yu-Kai Huang, Hung-Yueh Chiang, Hung-Ting Su, Zhe-Yu Liu, Chin-Tang Chen, Ching-Yu Tseng, and Winston H Hsu. Learning from 2d: Contrastive pixel-to-point knowledge transfer for 3d pretraining. arXiv preprint arXiv:2104.04687, 2021.
  45. 45.Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.
  46. 46.Zhengzhe Liu, Xiaojuan Qi, and Chi-Wing Fu. 3d-to-2d distillation for indoor scene parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4464–4474, 2021.
  47. 47.Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. arXiv preprint arXiv:2203.06604, 2022.
  48. 48.Bui Tuong Phong. Illumination for computer generated pictures. Communications of the ACM, 18(6):311–317, 1975.
  49. 49.Omid Poursaeed, Tianxing Jiang, Han Qiao, Nayun Xu, and Vladimir G Kim. Self-supervised learning of point clouds via orientation estimation. In 2020 International Conference on 3D Vision (3DV), pages 1018–1028. IEEE, 2020.
  50. 50.Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  51. 51.Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint arXiv:1706.02413, 2017.
  52. 52.Guocheng Qian, Xingdi Zhang, Abdullah Hamdi, and Bernard Ghanem. Pix4point: Image pretrained transformers for 3d point cloud understanding. arXiv preprint arXiv:2208.12259, 2022.
  53. 53.Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  54. 54.Yongming Rao, Jiwen Lu, and Jie Zhou. Global-local bidirectional reasoning for unsupervised representation learning of 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5376–5385, 2020.
  55. 55.Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
  56. 56.Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211–252, 2015.
  57. 57.Jonathan Sauder and Bjarne Sievers. Self-supervised deep learning on point clouds by reconstructing space. Advances in Neural Information Processing Systems, 32, 2019.
  58. 58.Jonathan Sauder and Bjarne Sievers. Self-supervised deep learning on point clouds by reconstructing space. Advances in Neural Information Processing Systems, 32, 2019.
  59. 59.Corentin Sautier, Gilles Puy, Spyros Gidaris, Alexandre Boulch, Andrei Bursuc, and Renaud Marlet. Image-to-lidar self-supervised distillation for autonomous driving data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9891–9901, 2022.
  60. 60.Jong-Chyi Su, Matheus Gadelha, Rui Wang, and Subhransu Maji. A deeper look at 3d shape classifiers. In The European Conference on Computer Vision (ECCV) Workshops, 2018.
  61. 61.Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64–73, 2016.
  62. 62.Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9627–9636, 2019.
  63. 63.Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1588–1597, 2019.
  64. 64.Hanchen Wang, Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt J Kusner. Unsupervised point cloud pre-training via occlusion completion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9782–9792, 2021.
  65. 65.Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog), 38(5):1–12, 2019.
  66. 66.Ziyi Wang, Xumin Yu, Yongming Rao, Jie Zhou, and Jiwen Lu. P2p: Tuning pre-trained image models for point cloud analysis with point-to-pixel prompting. arXiv preprint arXiv:2208.02812, 2022.
  67. 67.Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912–1920, 2015.
  68. 68.Tiange Xiang, Chaoyi Zhang, Yang Song, Jianhui Yu, and Weidong Cai. Walk in the cloud: Learning curves for point clouds shape analysis. arXiv preprint arXiv:2105.01288, 2021.
  69. 69.Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In European conference on computer vision, pages 574–591. Springer, 2020.
  70. 70.Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. arXiv preprint arXiv:2111.09886, 2021.
  71. 71.Chenfeng Xu, Shijia Yang, Bohan Zhai, Bichen Wu, Xiangyu Yue, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Image2point: 3d point-cloud understanding with pretrained 2d convnets. arXiv preprint arXiv:2106.04180, 2021.
  72. 72.Mutian Xu, Runyu Ding, Hengshuang Zhao, and Xiaojuan Qi. Paconv: Position adaptive convolution with dynamic kernel assembling on point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3173–3182, 2021.
  73. 73.Xu Yan, Jiantao Gao, Chaoda Zheng, Chao Zheng, Ruimao Zhang, Shuguang Cui, and Zhen Li. 2dpass: 2d priors assisted semantic segmentation on lidar point clouds. In European Conference on Computer Vision, pages 677–695. Springer, 2022.
  74. 74.Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 206–215, 2018.
  75. 75.Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Transactions on Graphics (ToG), 35(6):1–12, 2016.
  76. 76.Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. arXiv preprint arXiv:2111.14819, 2021.
  77. 77.Haokui Zhang, Ying Li, Yenan Jiang, Peng Wang, Qiang Shen, and Chunhua Shen. Hyperspectral classification based on lightweight 3-d-cnn with transfer learning. IEEE Transactions on Geoscience and Remote Sensing, 57(8):5813–5828, 2019.
  78. 78.Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li. Point-m2ae: Multi-scale masked autoencoders for hierarchical point cloud pre-training. arXiv preprint arXiv:2205.14401, 2022.
  79. 79.Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. arXiv preprint arXiv:2112.02413, 2021.
  80. 80.Renrui Zhang, Liuhui Wang, Ziyu Guo, and Jianbo Shi. Nearest neighbors meet deep neural networks for point cloud analysis. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1246–1255, 2023.
  81. 81.Renrui Zhang, Liuhui Wang, Yali Wang, Peng Gao, Hongsheng Li, and Jianbo Shi. Parameter is not all you need: Starting from non-parametric networks for 3d point cloud analysis. arXiv preprint arXiv:2303.08134, 2023.
  82. 82.Renrui Zhang, Ziyao Zeng, Ziyu Guo, Xinben Gao, Kexue Fu, and Jianbo Shi. Dspoint: Dual-scale point cloud recognition with high-frequency fusion. arXiv preprint arXiv:2111.10332, 2021.
  83. 83.Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3d features on any point-cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10252–10263, 2021.
  84. 84.Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16259–16268, 2021.
  85. 85.Junsheng Zhou, Xin Wen, Yu-Shen Liu, Yi Fang, and Zhizhong Han. Self-supervised point cloud representation learning with occlusion auto-encoder. arXiv preprint arXiv:2203.14084, 2022.
  86. 86.Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyao Zeng, Shanghang Zhang, and Peng Gao. Pointclip v2: Adapting clip for powerful 3d open-world learning. arXiv preprint arXiv:2211.11682, 2022.

Citation

MLA
Zhang, R., et al. “Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoders”. arXiv, 2022, http://arxiv.org/abs/2212.06785v1.
APA
Zhang, R., Wang, L., Qiao, Y., Gao, P., & Li, H. (2022). Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoders. arXiv. http://arxiv.org/abs/2212.06785v1
Chicago
Zhang, R., L. Wang, Y. Qiao, P. Gao, and H. Li. 2022. “Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoders”. arXiv. http://arxiv.org/abs/2212.06785v1.
Harvard
Zhang, R. et al. (2022) “Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoders”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2212.06785v1.
Vancouver
1. Zhang R, Wang L, Qiao Y, Gao P, Li H (2022) Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoders. arXiv

BibTeX

@article{zhang2022learning,
  title = {Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoders},
  author = {Zhang, Renrui and Wang, Liuhui and Qiao, Yu and Gao, Peng and Li, Hongsheng},
  year = {2022},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2212.06785v1},
  eprint = {2212.06785}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: IEEE