Back

Point Transformer

Nico EngelVasileios BelagiannisKlaus Dietmayer

article2020IEEE Access3,235 citations

Proposes a deep transformer architecture that combines local-global attention with a learned score-based sorting module to generate permutation-invariant feature representations directly from unordered 3D point clouds.

Listen

Three-dimensional point clouds generated by sensors such as lidar are critical for applications like autonomous driving, robotics, and computer vision. However, processing these point sets presents a core technical challenge: they are inherently unstructured, varying in size, and unordered. Existing deep learning approaches typically force points into rigid voxel grids or 2D image projections—which discards fine geometric detail and demands heavy memory—or compress points into a single fixed-length vector via symmetric pooling operations. This pooling bottleneck restricts the model's capacity to retain rich spatial structures and shape context.

The article introduces and evaluates Point Transformer, a deep neural network designed to operate directly on unordered 3D point sets using attention mechanisms. The primary objective is to replace traditional set pooling with an architecture that preserves permutation invariance, captures geometric dependencies, and directly relates local point features with the global object shape.

To achieve this, the architecture splits processing into two distinct branches: a global feature generation stream and a novel local feature selection module named SortNet. SortNet uses self-attention and a feed-forward layer to score points based on learned importance, selecting the top candidates and aggregating their local neighborhoods into an ordered feature list. The network then employs a local-global cross-attention mechanism to relate global representations to these sorted local features, producing an ordered, permutation-invariant output suitable for standard vision tasks. The approach was evaluated on standard 3D benchmarks, specifically ModelNet40 for object classification and ShapeNet for part segmentation.

The findings demonstrate that Point Transformer delivers competitive, state-of-the-art results across both core tasks. On ModelNet40 object classification, the model achieved an accuracy of 92.8%, outperforming other attention-based architectures and matching leading point-convolution methods. On ShapeNet part segmentation, it attained an 85.9% mean Intersection-over-Union. In ablation studies, SortNet's learned top-k scoring achieved 83.4% classification accuracy compared to 74.8% for furthest point sampling and 60.1% for random sampling, confirming that the module actively learns key geometric regions. Furthermore, the network exhibited robust rotation stability, maintaining 92.3% accuracy under unlearned random rotations where baseline models dropped significantly, while achieving faster inference times (110 ms) than several competitors due to parallelized multi-head attention.

These results demonstrate that attention-based ordering can bypass the information bottleneck of set pooling without requiring manual kernel tuning or rigid spatial discretizations. The modularity of SortNet—adding only about 10,000 parameters and 1.25 ms of inference overhead—means it can be integrated into broader perception pipelines with negligible computational penalty, improving spatial awareness and robustness in safety-critical perception systems.

Engineering teams and decision-makers should consider evaluating SortNet and local-global attention mechanisms in current 3D computer vision workflows, particularly where rotational variation or fine local geometry poses challenges. The primary technical limitation highlighted is that multi-head attention computational complexity scales quadratically with the total number of input points. Future development should focus on testing linear-complexity attention approximations to scale the architecture efficiently to denser, full-scale point clouds.

Institute of Measurement, Control and MicrotechnologyUlm Universityattention mechanismclassificationcomputer visionPointPoint TransformerAttention
arXiv: 2011.00931engelnico/point-transformer
Cover for Point Transformer

Abstract

In this work, we present Point Transformer, a deep neural network that operates directly on unordered and unstructured point sets. We design Point Transformer to extract local and global features and relate both representations by introducing the local-global attention mechanism, which aims to capture spatial point relations and shape information. For that purpose, we propose SortNet, as part of the Point Transformer, which induces input permutation invariance by selecting points based on a learned score. The output of Point Transformer is a sorted and permutation invariant feature list that can directly be incorporated into common computer vision applications. We evaluate our approach on standard classification and part segmentation benchmarks to demonstrate competitive results compared to the prior work. Code is publicly available at: this https URL

Topics

attention mechanismclassificationcomputer visioncomputer vision applicationsencoder-decoderglobal featureshape information

Keywords

PointPoint TransformerAttentionPointsLocalInputFeatureSetSortNetPoint Cloud

Table of Contents

  • I Introduction
  • II Related Work
  • II-A Point set processing
  • II-B Attention
  • III Fundamentals
  • III-A Attention
  • III-B Transformer
  • IV Point Transformer
  • IV-A SortNet
  • IV-B Global Feature Generation
  • IV-C Local-Global Attention
  • IV-D Complete Model
  • V Experiments
  • V-A Point cloud classification
  • V-B Point cloud part segmentation
  • V-C Network complexity
  • V-D Hyperparameter Study
  • V-E Point Transformer Design Analysis
  • VI Conclusion and Future Work
  • References

Knowls

  1. Knowl 1 — SortNet Module for Permutation-Invariant Point Cloud Keypoint Selection

    model/method

    SortNet is a neural network module designed to select an ordered subset of keypoints and extract local geometric features from an unordered 3D point cloud P={pi∈RD}i=1N\mathcal{P} = \{p_i \in \mathbb{R}^D\}_{i=1}^N (where D=3D=3 for coordinate positions, or D=6D=6 when point normals are included).

    SortNet operates as follows:

    1. The input points are mapped to a latent feature representation P=[p1latent,…,pNlatent]T∈RN×dm\mathbf{P} = [p_1^{\text{latent}}, \dots, p_N^{\text{latent}}]^T \in \mathbb{R}^{N \times d_m} using a shared row-wise feed-forward network (rFF).

    2. Self multi-head attention Aself(P)A^{\text{self}}(\mathbf{P}) is applied to capture pairwise spatial and geometric relations between all input points.

    3. A row-wise feed-forward layer reduces the feature dimension of each point to 1, producing a learned scalar score si∈Rs_i \in \mathbb{R} for each point pip_i.

    4. A top-KK selection operator selects the K≤NK \le N points with the highest scalar scores and sorts them in descending score order: s1≥s2≥⋯≥sKs_1 \ge s_2 \ge \dots \ge s_K, yielding an ordered keypoint set: Q={qj=(pij,sij)∣j=1,…,K}\mathcal{Q} = \{q_j = (p_i^j, s_i^j) \mid j = 1, \dots, K\}

    5. For each selected keypoint pijp_i^j, a ball query search finds all points in P\mathcal{P} within Euclidean radius rr. These neighborhood points are aggregated into a local feature vector gj∈Rdm−1−Dg^j \in \mathbb{R}^{d_m - 1 - D}.

    6. The final local feature vector for keypoint jj is created by concatenating the coordinates, the learned score, and the aggregated neighborhood feature: fij=pij⊕sij⊕gj∈Rdmf_i^j = p_i^j \oplus s_i^j \oplus g^j \in \mathbb{R}^{d_m}

    7. The output of one SortNet module mm is an ordered feature set FmL={fij∣j=1,…,K}\mathcal{F}_m^L = \{f_i^j \mid j = 1, \dots, K\}. To capture geometric patterns across diverse feature subspaces, MM independent SortNet modules operate in parallel with separate weights. Their outputs are concatenated into a single ordered local feature matrix: FL=F1L∪⋯∪FML∈R(K⋅M)×dm\mathcal{F}^L = \mathcal{F}_1^L \cup \dots \cup \mathcal{F}_M^L \in \mathbb{R}^{(K \cdot M) \times d_m}

  2. Knowl 2 — Local-Global Attention for Fusing Local Keypoint Features with Global Point Cloud Shape

    model/method

    Local-Global Attention (ALGA^{LG}) is a transformer-based attention mechanism that relates an ordered set of local keypoint features FL∈R(K⋅M)×dm\mathcal{F}^L \in \mathbb{R}^{(K \cdot M) \times d_m} extracted by MM SortNet modules to an unordered global feature set FG∈RN′×dm\mathcal{F}^G \in \mathbb{R}^{N' \times d_m} extracted from the entire point cloud via set abstraction.

    Local-Global Attention is defined as: ALG:=Across(Aself(FL),Aself(FG))A^{LG} := A^{\text{cross}}(A^{\text{self}}(\mathbf{F}^L), A^{\text{self}}(\mathbf{F}^G)) where FL\mathbf{F}^L and FG\mathbf{F}^G are the matrix representations of FL\mathcal{F}^L and FG\mathcal{F}^G, respectively. First, self multi-head attention AselfA^{\text{self}} is applied independently within FL\mathbf{F}^L and FG\mathbf{F}^G. Next, cross multi-head attention AcrossA^{\text{cross}} treats the transformed local features as queries (Nqueries=K⋅MN_{\text{queries}} = K \cdot M) and the transformed global features as keys and values (Nkeys=N′N_{\text{keys}} = N').

    The final row-wise feed-forward layer of ALGA^{LG} reduces the channel dimension from dmd_m to dm′<dmd'_m < d_m, yielding an output matrix of size (K⋅M)×dm′(K \cdot M) \times d'_m: ALG:R(K⋅M)×dm×RN′×dm→R(K⋅M)×dm′A^{LG} : \mathbb{R}^{(K \cdot M) \times d_m} \times \mathbb{R}^{N' \times d_m} \to \mathbb{R}^{(K \cdot M) \times d'_m}

    Because the query sequence follows the deterministic score-ordered sequence established by SortNet, the output of ALGA^{LG} is an ordered and permutation-invariant representation of fixed length K⋅MK \cdot M. This bypasses traditional symmetric set pooling operations (such as max pooling), preserving fine-grained geometric relations between local keypoint neighborhoods and the global object shape.

  3. Knowl 3 — Multi-Head Attention Formulations for Set-Structured Point Cloud Representations

    equation

    The multi-head attention mechanism adapted for point set processing is defined through pairwise dot-product query-key matching and projection matrices.

    Given query matrix Q∈RN×dkQ \in \mathbb{R}^{N \times d_k}, key matrix K∈RNk×dkK \in \mathbb{R}^{N_k \times d_k}, and value matrix V∈RNk×dvV \in \mathbb{R}^{N_k \times d_v}, the scaled dot-product attention function is: A(Q,K,V)=softmax(QKTdk)VA(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V

    Multi-head attention with hh parallel attention heads is defined by: Multihead(Q,K,V)=(head1⊕⋯⊕headh)WO\text{Multihead}(Q, K, V) = (\text{head}_1 \oplus \dots \oplus \text{head}_h) W^O where headi=A(QWiQ,KWiK,VWiV)\text{head}_i = A(Q W_i^Q, K W_i^K, V W_i^V), ⊕\oplus denotes horizontal matrix concatenation, and the learnable projection matrices are WiQ∈Rdm×dkW_i^Q \in \mathbb{R}^{d_m \times d_k}, WiK∈Rdm×dkW_i^K \in \mathbb{R}^{d_m \times d_k}, WiV∈Rdm×dvW_i^V \in \mathbb{R}^{d_m \times d_v}, and WO∈Rhdv×dmW^O \in \mathbb{R}^{h d_v \times d_m}, with head dimensions dk=dv=dm/hd_k = d_v = d_m / h.

    A multi-head attention transformer block AMH(X,Y)A^{\text{MH}}(X, Y) mapping query input X∈RN×dmX \in \mathbb{R}^{N \times d_m} and context input Y∈RNk×dmY \in \mathbb{R}^{N_k \times d_m} to RN×dm\mathbb{R}^{N \times d_m} is defined as: S=LayerNorm(X+Multihead(X,Y,Y))S = \text{LayerNorm}(X + \text{Multihead}(X, Y, Y)) AMH(X,Y)=LayerNorm(S+rFF(S))A^{\text{MH}}(X, Y) = \text{LayerNorm}(S + \text{rFF}(S)) where rFF\text{rFF} is a row-wise feed-forward network applied to each row independently.

    For point sets, self-attention and cross-attention are defined respectively as: Aself(P):=AMH(P,P)A^{\text{self}}(P) := A^{\text{MH}}(P, P) Across(P,Q):=AMH(P,Q)A^{\text{cross}}(P, Q) := A^{\text{MH}}(P, Q)

  4. Knowl 4 — Point Transformer Dual-Branch Architecture for Direct Point Cloud Processing

    model/method

    Point Transformer is a deep feedforward architecture that operates directly on irregular, unordered point clouds P={pi∈RD}i=1N\mathcal{P} = \{p_i \in \mathbb{R}^D\}_{i=1}^N by decoupling feature extraction into two parallel branches:

    1. Local Feature Branch (SortNet): The input points are projected to latent dimension dmd_m and processed via self-attention. MM separate SortNet modules each score all NN points, select the top-KK points according to their learned scores, and aggregate local Euclidean ball neighborhoods (rr) to produce an ordered local feature representation FL∈R(K⋅M)×dm\mathcal{F}^L \in \mathbb{R}^{(K \cdot M) \times d_m}.

    2. Global Feature Branch (Set Abstraction): The input point cloud is subsampled to N′<NN' < N points using Furthest Point Sampling (FPS). A multi-scale grouping (MSG) set abstraction module extracts multi-scale spatial neighborhood features around each sampled point to form an unordered global feature matrix FG∈RN′×dm\mathcal{F}^G \in \mathbb{R}^{N' \times d_m}.

    3. Local-Global Fusion: Local-Global Attention (ALGA^{LG}) attends the global features FG\mathcal{F}^G against each local query feature in FL\mathcal{F}^L, producing an ordered, permutation-invariant latent feature matrix of shape (K⋅M)×dm′(K \cdot M) \times d'_m.

    Because the output order is fixed by SortNet's score rankings, the output representation is fully permutation invariant with respect to input point ordering and can be directly consumed by downstream classification or segmentation heads.

  5. Knowl 5 — Classification and Part Segmentation Heads in Point Transformer

    model/method

    Point Transformer uses task-specific heads to map the output of the Local-Global Attention module ALG∈R(K⋅M)×dm′A^{LG} \in \mathbb{R}^{(K \cdot M) \times d'_m} to class predictions:

    • Shape Classification Head: The sorted output matrix of ALGA^{LG} is flattened into a 1D vector of dimension (M⋅K⋅dm′)(M \cdot K \cdot d'_m). A multi-layer row-wise feed-forward network (e.g., with hidden layer dimensions 4096, 1024, 512, 128, and CC) projects this vector to CC class logits, followed by a softmax function to generate global shape category probabilities.

    • Part Segmentation Head: Points in the global feature branch are projected to latent dimension dm′′d''_m and processed with self-attention to generate per-point representations of size N×dm′′N \times d''_m. An additional cross multi-head attention layer uses these per-point representations as queries and the output of ALGA^{LG} as keys and values, producing an N×dm′N \times d'_m feature matrix that relates each individual point in P\mathcal{P} to the aggregated global shape representation. A row-wise feed-forward network then projects each point feature to CC part classes, producing per-point classification probabilities via softmax.

  6. Knowl 6 — Performance Comparison on ModelNet40 Classification and ShapeNet Part Segmentation

    data/table

    Point Transformer was evaluated on ModelNet40 3D shape classification (40 classes, 9,843 train / 2,468 test samples, N=1024N=1024 points with normal vectors) and ShapeNet part segmentation (16 object categories, 50 part labels, 13,998 train / 2,874 test samples, N=1024N=1024 points with normal vectors). Point Transformer outperforms all attention-based and set-pooling baselines on ModelNet40 (92.8% accuracy) and ShapeNet (85.9% mean IoU), while achieving accuracy on par with point convolution methods.

    Method ModelNet40 Accuracy (%) ShapeNet mIoU (%)
    PointNet 89.2 83.7
    PointNet++ 91.9 85.1
    ShapeContextNet 89.8 84.6
    Deep Sets 90.3 -
    Point2Sequence 92.6 85.2
    Set Transformer 90.4 -
    PAT 91.7 -
    Tao et al. 87.5 75.2
    Point Transformer 92.8 85.9
    KPConv 92.9 86.2
    PointCNN 92.2 86.1
    SpiderCNN 90.5 85.3

  7. Knowl 7 — Ablation Study on SortNet Selection and Global Feature Generation Schemes

    data/table

    An ablation study on ModelNet40 classification evaluated the influence of different point selection mechanisms inside SortNet (using a standalone SortNet model with M=10M=10 and K=12K=12, selecting 120 points from N=1024N=1024) and different global feature generation strategies in the full Point Transformer architecture.

    SortNet Architecture Variant Accuracy (%)
    SortNet with learnable score 83.4
    SortNet with furthest point sampling (FPS) 74.8
    SortNet with random points 60.1
    Global Feature Generation Variant Accuracy (%)
    No sampling 91.9
    Furthest point sampling (FPS, N′=128N' = 128) 92.3
    Set Abstraction (MSG, N′=128N' = 128) 92.8

    The SortNet ablation shows that selecting keypoints via learned importance scores improves accuracy by 8.6% over FPS and 23.3% over random selection, verifying that SortNet focuses on salient shape regions. The global branch ablation indicates that aggregating local features via Set Abstraction with multi-scale grouping before cross-attention improves classification performance over uniform point sampling or raw unsampled point sets.

  8. Knowl 8 — Impact of SortNet Module Count and Top-K Point Selections on Classification Accuracy

    data/table

    A grid search on ModelNet40 evaluated classification accuracy across varying numbers of SortNet modules (M∈{4,8,16,32}M \in \{4, 8, 16, 32\}) and numbers of top-KK selected keypoints per module (K∈{8,16,32,64,96}K \in \{8, 16, 32, 64, 96\}).

    MM 4 4 4 4 8 8 8 8 8 16 16 16 16 32 32 32
    KK 16 32 64 96 8 16 32 64 96 8 16 32 64 8 16 32
    Accuracy (%) 91.7 92.3 92.8 91.7 90.2 90.5 91.9 92.4 92.0 91.2 91.6 92.0 91.7 90.8 91.3 91.1

    The optimal configuration on ModelNet40 was found at M=4M=4 and K=64K=64, achieving 92.8% classification accuracy. Configurations with moderate numbers of SortNets (M=4M=4 or M=8M=8) and higher keypoint counts (K=32K=32 or K=64K=64) consistently outperformed configurations with many small SortNets (e.g., M=32,K=8M=32, K=8 achieving 90.8%).

  9. Knowl 9 — Model Complexity, Parameter Footprint, and Inference Latency of Point Transformer

    data/table

    Network complexity and inference time were benchmarked on a single Nvidia GeForce 1080Ti GPU.

    Method Learnable Parameters Network Size Inference Time
    Point Transformer 13.5 M 51 MB 110 ms
    SortNet (single module) 10 k 0.04 MB 1.25 ms
    PAT - 5.8 MB 88 ms
    KPConv 15 M - 210 ms
    PointNet++ 2.1 M 24 MB 160 ms
    Point2Seq 2 M - -

    While Point Transformer contains 13.5M parameters due to multi-head attention projections, its inference latency (110 ms) is faster than PointNet++ (160 ms) and KPConv (210 ms) because multi-head attention operations are parallelized across heads. Individual SortNet modules are lightweight, adding only 10,000 parameters and 1.25 ms latency each. The upper bound computational complexity of Point Transformer is governed by the multi-head attention layer and scales quadratically with input points: O(N2⋅dm)\mathcal{O}(N^2 \cdot d_m).

  10. Knowl 10 — Robustness of Point Transformer to Unseen 3D Coordinate Rotations

    empirical result

    The robustness of Point Transformer against arbitrary spatial rotations was evaluated by applying random 3D rotations to the ModelNet40 test point clouds without training the network with rotational data augmentation.

    Under unseen test rotations:

    • Point Transformer classification accuracy dropped by only 0.5 percentage points, from 92.8% (unrotated) to 92.3% (rotated).
    • Under identical test conditions, PointNet++ accuracy dropped by 3.3 percentage points, from 91.9% to 88.6%.

    Qualitative inspection of the learned top-KK selections shows that SortNet consistently focuses on the same local geometric structures (e.g., legs of chairs/tables, corners of beds/monitors) regardless of object orientation in metric space.

Coverage note — Detailed training hyperparameter listings from Table 5 (such as exact layer-by-layer rFF dimension tuples) were omitted in favor of the core architectural parameters and empirical grid search results.

References

  1. 1.Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12697–12705, 2019.
  2. 2.Nico Engel, Stefan Hoermann, Philipp Henzler, and Klaus Dietmayer. Deep object tracking on dynamic occupancy grid maps using rnns. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pages 3852–3858. IEEE, 2018.
  3. 3.Nico Engel, Stefan Hoermann, Markus Horn, Vasileios Belagiannis, and Klaus Dietmayer. Deeplocalization: Landmark-based self-localization with deep neural networks. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 926–933. IEEE, 2019.
  4. 4.Martin Simon, Kai Fischer, Stefan Milz, Christian Tobias Witt, and Horst-Michael Gross. Stickypillars: Robust feature matching on point clouds using graph neural networks. arXiv preprint arXiv:2002.03983, 2020.
  5. 5.Markus Horn, Nico Engel, Vasileios Belagiannis, Michael Buchholz, and Klaus Dietmayer. Deepclr: Correspondence-less architecture for deep end-to-end point cloud registration. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pages 1–7. IEEE, 2020.
  6. 6.Julian Wiederer, Arij Bouazizi, Ulrich Kressel, and Vasileios Belagiannis. Traffic control gesture recognition for autonomous vehicles. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10676–10683, 2020.
  7. 7.Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in neural information processing systems, pages 5099–5108, 2017.
  8. 8.Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In Proceedings of the IEEE International Conference on Computer Vision, pages 9277–9286, 2019.
  9. 9.Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 922–928. IEEE, 2015.
  10. 10.Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912–1920, 2015.
  11. 11.Charles R Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas J Guibas. Volumetric and multi-view cnns for object classification on 3d data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656, 2016.
  12. 12.Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision, pages 945–953, 2015.
  13. 13.Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 652–660, 2017.
  14. 14.Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in neural information processing systems, pages 3391–3401, 2017.
  15. 15.Edward Wagstaff, Fabian Fuchs, Martin Engelcke, Ingmar Posner, and Michael A. Osborne. On the limitations of representing functions on sets. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, volume 97 of Proceedings of Machine Learning Research, pages 6487–6494. PMLR, 2019.
  16. 16.Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, 2015.
  17. 17.Xinhai Liu, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker. Point2sequence: Learning the shape representation of 3d point clouds with an attention-based sequence to sequence network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8778–8785, 2019.
  18. 18.Jiaxin Li, Ben M Chen, and Gim Hee Lee. So-net: Self-organizing network for point cloud analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9397–9406, 2018.
  19. 19.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.
  20. 20.Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariant neural networks. In International Conference on Machine Learning, pages 3744–3753, 2019.
  21. 21.Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4490–4499, 2018.
  22. 22.Dominic Zeng Wang and Ingmar Posner. Voting for voting in online point cloud object detection. In Robotics: Science and Systems, volume 1, pages 10–15607, 2015.
  23. 23.Yangyan Li, Soeren Pirk, Hao Su, Charles R Qi, and Leonidas J Guibas. Fpnn: Field probing neural networks for 3d data. In Advances in Neural Information Processing Systems, pages 307–315, 2016.
  24. 24.Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representations at high resolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3577–3586, 2017.
  25. 25.Baoguang Shi, Song Bai, Zhichao Zhou, and Xiang Bai. Deeppano: Deep panoramic representation for 3-d shape recognition. IEEE Signal Processing Letters, 22(12):2339–2343, 2015.
  26. 26.Asako Kanezaki, Yasuyuki Matsushita, and Yoshifumi Nishida. Rotationnet: Joint object categorization and pose estimation using multiviews from unsupervised viewpoints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5010–5019, 2018.
  27. 27.Jong-Chyi Su, Matheus Gadelha, Rui Wang, and Subhransu Maji. A deeper look at 3d shape classifiers. In Proceedings of the European Conference on Computer Vision (ECCV), pages 0–0, 2018.
  28. 28.Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas. Frustum pointnets for 3d object detection from rgb-d data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 918–927, 2018.
  29. 29.Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE International Conference on Computer Vision, pages 6411–6420, 2019.
  30. 30.Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. Spidercnn: Deep learning on point sets with parameterized convolutional filters. In Proceedings of the European Conference on Computer Vision (ECCV), pages 87–102, 2018.
  31. 31.Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointcnn: Convolution on x-transformed points. In Advances in neural information processing systems, pages 820–830, 2018.
  32. 32.Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun. Deep learning for 3d point clouds: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
  33. 33.Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025, 2015.
  34. 34.Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence to sequence for sets. In 4th International Conference on Learning Representations, ICLR 2016, 2016.
  35. 35.Saining Xie, Sainan Liu, Zeyu Chen, and Zhuowen Tu. Attentional shapecontextnet for point cloud recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4606–4615, 2018.
  36. 36.Jiancheng Yang, Qiang Zhang, Bingbing Ni, Linguo Li, Jinxian Liu, Mengdie Zhou, and Qi Tian. Modeling point clouds with self-attention and gumbel subset sampling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3323–3332, 2019.
  37. 37.Zhiyong Tao, Yixin Zhu, Tong Wei, and Sen Lin. Multi-head attentional point cloud classification and segmentation using strictly rotation-invariant representations. IEEE Access, 9:71133–71144, 2021.
  38. 38.Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  39. 39.Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog), 38(5):1–12, 2019.
  40. 40.Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
  41. 41.Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.03265, 2019.
  42. 42.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015.
  43. 43.Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  44. 44.Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020.
  45. 45.Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self-attention. arXiv preprint arXiv:2102.03902, 2021.

Citation

MLA
Engel, N., et al. “Point Transformer”. IEEE Access, vol. 9, 2021, pp. 134826–40, https://doi.org/10.1109/ACCESS.2021.3116304.
APA
Engel, N., Belagiannis, V., & Dietmayer, K. (2021). Point Transformer. IEEE Access, 9, 134826–134840. https://doi.org/10.1109/ACCESS.2021.3116304
Chicago
Engel, N., V. Belagiannis, and K. Dietmayer. 2021. “Point Transformer”. IEEE Access 9: 134826–40. https://doi.org/10.1109/ACCESS.2021.3116304.
Harvard
Engel, N., Belagiannis, V. and Dietmayer, K. (2021) “Point Transformer”, IEEE Access, 9, pp. 134826–134840. Available at: https://doi.org/10.1109/ACCESS.2021.3116304.
Vancouver
1. Engel N, Belagiannis V, Dietmayer K (2021) Point Transformer. IEEE Access 9:134826–134840

BibTeX

@article{Engel_2021, title={Point Transformer}, volume={9}, ISSN={2169-3536}, url={http://dx.doi.org/10.1109/ACCESS.2021.3116304}, DOI={10.1109/access.2021.3116304}, journal={IEEE Access}, publisher={Institute of Electrical and Electronics Engineers (IEEE)}, author={Engel, Nico and Belagiannis, Vasileios and Dietmayer, Klaus}, year={2021}, pages={134826–134840} }
Metadata:Crossref

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF