Back

Frustum PointNets for 3D Object Detection from RGB-D Data

Charles R. QiWei LiuChenxia WuHao SuLeonidas J. Guibas

article2018CVPR2,567 citations

Proposes a real-time 3D object detection framework that projects 2D region proposals into 3D viewing frustums to directly process raw point clouds with PointNet, achieving state-of-the-art accuracy on KITTI and SUN RGB-D benchmarks.

Listen

The article addresses the challenge of accurate and efficient 3D object detection from RGB-D data for applications such as autonomous driving and augmented reality. Prior approaches often project point clouds into 2D images or 3D voxels, which can obscure natural geometric patterns and increase computational cost for large scenes.

The work evaluates a new pipeline that lifts mature 2D object detections into 3D viewing frustums and then applies PointNet variants directly on the resulting raw point clouds to perform instance segmentation followed by amodal oriented bounding-box regression.

Experiments were conducted on the KITTI outdoor LiDAR benchmark and the SUN RGB-D indoor depth benchmark using standard train/val/test splits and established 3D IoU metrics. The method achieved 8 percentage-point gains in 3D average precision for cars on KITTI while running at 5 frames per second, and 6–9 percentage-point gains in mean average precision on SUN RGB-D with one to three orders of magnitude faster inference than previous state-of-the-art systems.

These results demonstrate that operating natively in 3D point space after 2D-guided frustum cropping yields both higher accuracy under occlusion and sparsity and real-time performance, directly lowering risk and latency for downstream planning and perception modules.

Next steps supported by the article include combining RGB-view and bird’s-eye-view proposals via 3D non-maximum suppression, adding image features to handle extremely sparse distant objects, and extending the single-object-per-frustum assumption to multiple instances.

The main limitations are dependence on the quality of the upstream 2D detector, reduced robustness when multiple same-category objects occupy one frustum, and the need for denser data or auxiliary cues for objects with fewer than five points; readers should therefore treat the reported gains as conditional on reliable 2D proposals and moderate point density.

Nuro, Inc.Stanford UniversityUniversity of California, San Diegodeep learningobject localizationpoint cloudFrustum PointNet3D Object DetectionRGB-D
arXiv: 1711.08488charlesq34/frustum-pointnets
Cover for Frustum PointNets for 3D Object Detection from RGB-D Data

Abstract

In this work, we study 3D object detection from RGB-D data in both indoor and outdoor scenes. While previous methods focus on images or 3D voxels, often obscuring natural 3D patterns and invariances of 3D data, we directly operate on raw point clouds by popping up RGB-D scans. However, a key challenge of this approach is how to efficiently localize objects in point clouds of large-scale scenes (region proposal). Instead of solely relying on 3D proposals, our method leverages both mature 2D object detectors and advanced 3D deep learning for object localization, achieving efficiency as well as high recall for even small objects. Benefited from learning directly in raw point clouds, our method is also able to precisely estimate 3D bounding boxes even under strong occlusion or with very sparse points. Evaluated on KITTI and SUN RGB-D 3D detection benchmarks, our method outperforms the state of the art by remarkable margins while having real-time capability.

Topics

deep learningobject localizationpoint clouddetection algorithmtwo dimensional

Keywords

Frustum PointNet3D Object DetectionRGB-DPoint CloudInstance SegmentationAmodal 3D Bounding BoxT-NetKITTISUN-RGBDMulti-Task Loss

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 Problem Definition
  • 4 3D Detection with Frustum PointNets
  • 4.1 Frustum Proposal
  • 4.2 3D Instance Segmentation
  • 4.3 Amodal 3D Box Estimation
  • 4.4 Training with Multi-task Losses
  • 5 Experiments
  • 5.1 Comparing with state-of-the-art Methods
  • 5.2 Architecture Design Analysis
  • 5.3 Qualitative Results and Discussion
  • References
  • A Overview
  • B Details on Frustum PointNets (Sec 4.2, 4.3)
  • B.1 Network Architectures
  • B.2 Data Augmentation and Training
  • C Details on RGB Detector (Sec 4.1)
  • D Bird’s Eye View PointNets (Sec 5.3)
  • E More Experiments (Sec 5.2)
  • E.1 Effects of PointNet Architectures
  • E.2 Effects of Training Data Size
  • E.3 Runtime and Model Size
  • F Visualizations for SUN-RGBD (Sec 5.1)

Knowls

  1. Knowl 1 — Frustum PointNets 3D Object Detection Pipeline

    model/method

    Frustum PointNets is a 3D object detection framework that estimates oriented, amodal 3D bounding boxes from RGB-D data (such as LiDAR point clouds or indoor depth maps paired with RGB images) by sequentially processing data across three modules:

    1. Frustum Proposal Generation: A 2D convolutional object detector (e.g., Feature Pyramid Network) detects objects in the RGB image. Given known camera projection parameters, each 2D bounding box is extruded into a 3D viewing frustum with near and far clipping bounds determined by sensor range. Points within the frustum are collected to form a frustum point cloud, which is then canonicalized to a center view by rotating the frustum axis to be orthogonal to the image plane.
    2. 3D Instance Segmentation: A deep network (based on PointNet or PointNet++) processes the normalized frustum point cloud. To resolve spatial clutter and foreground/background occluders within the frustum, the network performs binary classification on every point, segmenting the points that belong to the specific object of interest. The network is conditioned on the semantic category predicted by the 2D detector via a concatenated one-hot class vector.
    3. Amodal 3D Box Estimation: Segmented object points are shifted to their centroid, passed through a light-weight spatial translation network (T-Net) that predicts the residual offset to the true amodal 3D object center, and then fed to an amodal 3D box estimation PointNet that regresses the amodal oriented bounding box parameters (center, size dimensions, and heading angle).

  2. Knowl 2 — Coordinate Normalization Pipeline for Frustum Point Clouds

    model/method

    To achieve rotation and translation invariance and simplify feature learning directly from irregular 3D point sets, Frustum PointNets applies a sequence of four canonical coordinate transformations:

    1. Camera Coordinate Frame: The default coordinate system where points are represented in metric 3D camera coordinates (X,Y,Z)(X, Y, Z) with the depth axis ZZ pointing forward and YY pointing downward.
    2. Frustum Coordinate Frame: The 3D frustum generated from a 2D bounding box is rotated toward a canonical center view such that the central viewing axis of the frustum is orthogonal to the image plane. This eliminates viewpoint variations across different directions in the camera field of view.
    3. 3D Mask Coordinate Frame: After 3D instance segmentation isolates the points belonging to the target object, the coordinates of these segmented points are centralized by subtracting their local centroid (Cmask)(C_{\text{mask}}). The point cloud is deliberately left unscaled so that true physical size dimensions are preserved.
    4. 3D Object Coordinate Frame: A learned regression network (T-Net) predicts the translation residual (ΔCt-net)(\Delta C_{\text{t-net}}) from the mask centroid to the true amodal object center. The origin is translated to this predicted amodal center, forming a canonical frame in which the bounding box regression PointNet predicts size and orientation.

  3. Knowl 3 — Multi-Task Loss and Corner Regularization Loss for 3D Box Estimation

    equation

    Frustum PointNets optimizes the 3D instance segmentation network, the T-Net, and the amodal box estimation network simultaneously using a multi-task loss:

    Lmulti-task=Lseg+λ(Lc1-reg+Lc2-reg+Lh-cls+Lh-reg+Ls-cls+Ls-reg+γLcorner)L_{\text{multi-task}} = L_{\text{seg}} + \lambda \left( L_{c1\text{-reg}} + L_{c2\text{-reg}} + L_{h\text{-cls}} + L_{h\text{-reg}} + L_{s\text{-cls}} + L_{s\text{-reg}} + \gamma L_{\text{corner}} \right)

    where:

    • LsegL_{\text{seg}} is the cross-entropy classification loss for 3D point instance segmentation.
    • Lc1-regL_{c1\text{-reg}} is the smooth-l1l_1 (Huber) regression loss for the T-Net translation prediction.
    • Lc2-regL_{c2\text{-reg}} is the smooth-l1l_1 regression loss for the box estimation network's residual center prediction.
    • Lh-clsL_{h\text{-cls}} (softmax cross-entropy) and Lh-regL_{h\text{-reg}} (smooth-l1l_1) are heading angle classification and residual regression losses over NHN_H predefined angle bins.
    • Ls-clsL_{s\text{-cls}} (softmax cross-entropy) and Ls-regL_{s\text{-reg}} (smooth-l1l_1) are 3D box size classification and residual regression losses over NSN_S template shape classes.
    • λ\lambda and γ\gamma are loss weighting hyperparameters.

    To address the issue where separate losses for center, size, and heading fail to optimize overall 3D bounding box IoU, the model incorporates a joint regularizer called the corner loss:

    Lcorner=∑i=1NS∑j=1NHδijmin⁡{∑k=18∥Pijk−Pk∗∥,∑k=18∥Pijk−Pk∗∗∥}L_{\text{corner}} = \sum_{i=1}^{N_S} \sum_{j=1}^{N_H} \delta_{ij} \min \left\{ \sum_{k=1}^8 \| P_{ij}^k - P_k^* \|, \sum_{k=1}^8 \| P_{ij}^k - P_k^{**} \| \right\}

    where PijkP_{ij}^k denotes the kk-th corner coordinate (k∈{1,…,8}k \in \{1, \dots, 8\}) of the predicted box constructed from size template ii and heading bin jj, translated to the predicted center. Pk∗P_k^* denotes the kk-th ground-truth corner, Pk∗∗P_k^{**} denotes the corresponding corner of the 180∘180^\circ flipped ground-truth box (to prevent penalizing correctly oriented boxes with opposite heading labels), and δij\delta_{ij} is an indicator mask that equals 11 if ii and jj match the ground-truth size and heading categories, and 00 otherwise.

  4. Knowl 4 — Amodal 3D Bounding Box Parameterization and Residual Center Estimation

    model/method

    An amodal 3D bounding box is parameterized by its 3D center (cx,cy,cz)(c_x, c_y, c_z), physical dimensions (height hh, width ww, length ll), and heading angle θ\theta around the up-axis.

    1. Box Center Estimation: Box center localization uses a two-stage residual formulation. The final predicted center CpredC_{\text{pred}} in camera coordinates is computed as:

    Cpred=Cmask+ΔCt-net+ΔCbox-netC_{\text{pred}} = C_{\text{mask}} + \Delta C_{\text{t-net}} + \Delta C_{\text{box-net}}

    where CmaskC_{\text{mask}} is the spatial centroid of the segmented object points, ΔCt-net∈R3\Delta C_{\text{t-net}} \in \mathbb{R}^3 is the offset predicted by the spatial transformation network (T-Net), and ΔCbox-net∈R3\Delta C_{\text{box-net}} \in \mathbb{R}^3 is the residual offset estimated by the amodal box estimation network.

    1. Box Size and Heading Estimation: Size and heading estimation use a hybrid classification and regression formulation. The network outputs classification logits over NSN_S predefined size category templates (with 3×NS3 \times N_S dimension residuals for Δh,Δw,Δl\Delta h, \Delta w, \Delta l) and NHN_H discrete heading angle bins (with NHN_H angular residuals Δθ\Delta \theta). The total output dimension of the box estimation network is 3+4NS+2NH3 + 4 N_S + 2 N_H parameters.

  5. Knowl 5 — Semantic Conditioning in 3D Instance Segmentation

    model/method

    To segment the 3D points belonging to an object within a cluttered frustum containing ground, vegetation, and other occluding or background objects, the 3D instance segmentation PointNet conditions its point-level geometric features on the semantic class predicted by the 2D detector.

    The input point cloud (n×cn \times c array, where points contain XYZ coordinates and optional intensity channels) is mapped through shared Multi-Layer Perceptrons (MLPs) and max-pooled into a global feature vector. The kk-dimensional one-hot vector representing the 2D detection class (e.g., car, pedestrian, cyclist) is concatenated with the global feature vector and per-point intermediate features. This semantic prior guides the segmentation layers to apply geometry filters matching the expected object category (e.g., human-shaped vs. vehicle-shaped geometry).

  6. Knowl 6 — 3D Object Detection and Bird's Eye View Performance on KITTI Benchmark

    data/table

    The performance of Frustum PointNets (v1 with PointNet backbone and v2 with PointNet++ backbone) was evaluated on the official KITTI test set against prior 2D-projection and 3D voxel methods. The evaluation uses Average Precision (AP) with 3D IoU thresholds of 70%70\% for cars and 50%50\% for pedestrians and cyclists under Easy, Moderate, and Hard difficulty categories.

    Method Cars (3D AP) Pedestrians (3D AP) Cyclists (3D AP)
    Easy Moderate Hard Easy Moderate Hard Easy Moderate Hard
    DoBEM 7.42 6.95 13.45 - - - - - -
    MV3D 71.09 62.35 55.12 - - - - - -
    Ours (v1) 80.62 64.70 56.07 50.88 41.55 38.04 69.36 53.50 52.88
    Ours (v2) 81.20 70.39 62.19 51.21 44.89 40.23 71.96 56.77 50.39
    Method Cars (BEV AP) Pedestrians (BEV AP) Cyclists (BEV AP)
    Easy Moderate Hard Easy Moderate Hard Easy Moderate Hard
    DoBEM 36.49 36.95 38.10 - - - - - -
    3D FCN 69.94 62.54 55.94 - - - - - -
    MV3D 86.02 76.90 68.49 - - - - - -
    Ours (v1) 87.28 77.09 67.90 55.26 47.56 42.57 73.42 59.87 52.88
    Ours (v2) 88.70 84.00 75.33 58.09 50.22 47.20 75.38 61.96 54.68

    Frustum PointNets (v2) achieves a Moderate Car 3D AP of 70.39%70.39\%, outperforming MV3D (62.35%62.35\%) by 8.04%8.04\% without requiring multi-sensor feature fusion during 3D estimation. The PointNet++ (v2) model shows notable improvements over v1 on small, sparse categories like pedestrians and hard cars due to hierarchical multi-scale density grouping.

  7. Knowl 7 — 3D Object Detection Performance on SUN-RGBD Benchmark

    data/table

    Frustum PointNets was evaluated on the SUN-RGBD indoor validation set across 10 indoor furniture and fixture categories using Average Precision (AP) with a 3D IoU threshold of 0.250.25.

    Method bathtub bed bksf chair desk dresser nigtst sofa table toilet mAP Runtime
    DSS 44.2 78.8 11.9 61.2 20.5 6.4 15.4 53.5 50.3 78.9 42.1 19.55s
    COG 58.3 63.7 31.8 62.2 45.2 15.5 27.4 51.0 51.3 70.1 47.6 10–30m
    2D-driven 43.5 64.5 31.4 48.3 27.9 25.9 41.9 50.4 37.0 80.4 45.1 4.15s
    Ours (v1) 43.3 81.1 33.3 64.2 24.7 32.0 58.1 61.1 51.1 90.9 54.0 0.12s

    Frustum PointNets achieves 54.0%54.0\% mAP on SUN-RGBD, surpassing COG (47.6%47.6\%) and 2D-driven (45.1%45.1\%) by 6.4%6.4\% and 8.9%8.9\% mAP respectively, while operating at 0.12 s0.12\text{ s} per frame (10x to 1000x faster than prior volumetric or sliding-window methods). Unlike COG and 2D-driven, Frustum PointNets achieves this without relying on explicit room layout context modeling.

  8. Knowl 8 — Ablation Analysis of Point Cloud Normalizations and 3D Segmentation

    empirical result

    Ablation experiments conducted on the KITTI car dataset (using ground-truth 2D proposals and evaluated at 3D bounding box IoU≥0.7\text{IoU} \ge 0.7) validate the critical design choices of Frustum PointNets:

    1. 2D vs. 3D Instance Segmentation: Directly regressing 3D boxes from 2D RGB-D image patches using a VGG ConvNet achieves only 18.3%18.3\% accuracy (27.4%27.4\% when masked with a 2D FCN mask). In contrast, performing 3D instance segmentation directly on point clouds using PointNet achieves 74.3%74.3\% accuracy. Bypassing 3D segmentation entirely on the frustum point cloud drops accuracy to 33.5%33.5\%, and using a 2D image mask to crop depth points achieves 61.6%61.6\%.
    2. Coordinate Normalizations: Progressively introducing canonical transformations yields compounding gains:
      • Raw frustum points (no transformations): 12.5%12.5\% accuracy.
      • Adding frustum rotation: 48.1%48.1\% accuracy.
      • Adding mask centroid subtraction (centralization): 71.5%71.5\% accuracy.
      • Adding learned T-Net center alignment: 74.3%74.3\% accuracy.
    3. Regression Loss Formulations: Predicting 3D boxes with pure smooth-l1l_1 regression yields 62.9%62.9\% accuracy. Adding the hybrid classification-regression scheme with normalized bin residuals increases accuracy to 72.2%72.2\%, and adding the joint multi-corner loss achieves the peak accuracy of 74.3%74.3\%.

  9. Knowl 9 — Comparison of PointNet v1 and PointNet++ v2 Backbones, Model Size, and Latency

    empirical result

    Frustum PointNets supports both PointNet (v1) and PointNet++ (v2) network backbones for 3D instance segmentation and amodal bounding box estimation:

    • Architecture and Density Robustness: The v2 model employs Set Abstraction (SA) with Multi-Scale Grouping (MSG) and Feature Propagation (FP) layers. MSG groups neighboring points across multiple radii (e.g., r=[0.2,0.4,0.8]r = [0.2, 0.4, 0.8] m and r=[0.4,0.8,1.6]r = [0.4, 0.8, 1.6] m), providing robustness to the sharp density fall-off of LiDAR point clouds at far distances.
    • Detection Accuracy: On the KITTI car validation set (IoU 0.7), using v2 for both segmentation and box estimation achieves 77.1%77.1\% 3D box accuracy (91.0%91.0\% segmentation accuracy), compared to 74.3%74.3\% 3D box accuracy (90.6%90.6\% segmentation accuracy) with v1.
    • Inference Latency and Memory: On an NVIDIA GTX 1080 GPU processing 32 region proposals per frame:
      • v1 Model: Total latency is 88 ms88\text{ ms} (60 ms60\text{ ms} for frustum proposal generation, 18 ms18\text{ ms} for 3D segmentation, 10 ms10\text{ ms} for box estimation). PointNet model size is 19 MB19\text{ MB} (47 MB47\text{ MB} total with 2D CNN).
      • v2 Model: Total latency is 167 ms167\text{ ms} (60 ms60\text{ ms} for frustum proposal, 88 ms88\text{ ms} for 3D segmentation, 19 ms19\text{ ms} for box estimation). PointNet++ model size is 22 MB22\text{ MB} (50 MB50\text{ MB} total with 2D CNN).

  10. Knowl 10 — Fusion of RGB Frustum PointNets and LiDAR Bird's Eye View Proposals

    model/method

    To detect objects that are severely occluded or poorly lit in RGB camera views, the Frustum PointNet framework can be extended to process 3D cuboids proposed from LiDAR Bird's Eye View (BV):

    1. Bird's Eye View Region Proposal: Projected LiDAR points are discretized onto a 600×600600 \times 600 2D bird's eye view grid (0.1 m0.1\text{ m} resolution over a 60 m×60 m60\text{ m} \times 60\text{ m} area). For each cell, 9 channels are extracted (maximum point intensity, point density, and height of the highest point across 7 vertical height slices). A Faster R-CNN with a VGG-16 backbone generates axis-aligned 2D bounding boxes in BEV, which define 3D cuboid proposals.
    2. Bird's Eye View PointNets (BV PointNets): PointNets are applied to point clouds extracted from the 3D cuboids to perform 3D segmentation and amodal box regression in the same manner as frustum PointNets.
    3. 3D Proposal Fusion: 3D bounding box predictions from the RGB Frustum PointNet and the BV PointNet are merged using 3D Non-Maximum Suppression (NMS) with an IoU threshold of 0.80.8, applying a confidence weight of 0.50.5 to boxes from the BV network.

    On the KITTI car validation set, combining RGB Frustum and BV proposals improves Hard car 3D AP from 63.65%63.65\% to 67.47%67.47\% (+3.82%+3.82\% AP), demonstrating effective recovery of occluded instances missed by the 2D image detector.

  11. Knowl 11 — Failure Modes and Limitations of Frustum PointNets

    limitation

    Frustum PointNets exhibits three primary failure modes:

    1. Point Cloud Sparsity at Extreme Distances: When distant objects contain extremely few LiDAR points (e.g., fewer than 5 points), the network struggle to reliably infer 3D bounding box dimensions and heading orientation from geometry alone.
    2. Multiple Same-Category Instances in a Single Frustum: Because the segmentation PointNet assumes a single target object of interest per frustum proposal, when multiple instances of the same category appear closely grouped (e.g., pedestrians standing side-by-side in the same viewing frustum), the network can output mixed segmentation masks and inaccurate boxes.
    3. 2D Proposal Recall Bottleneck: Because 3D frustums are strictly derived from 2D image bounding box proposals, objects that fail to be detected by the 2D detector (e.g., under extreme low light, severe truncation, or heavy visual occlusion) are completely missed in the 3D pipeline unless supplemented by complementary 3D proposals (such as LiDAR bird's eye view proposals).

Coverage note — None was omitted; all contributed models, loss formulations, coordinate transformations, quantitative benchmark results on KITTI and SUN-RGBD, ablation studies, runtime analyses, BEV extensions, and failure modes are fully covered.

References

  1. 1.Kitti 3d object detection benchmark leader board. http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d. Accessed: 2017-11-14 12PM. 2
  2. 2.Kitti bird's eye view object detection benchmark leader board. http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=bev. Accessed: 2017-11-14 12PM. 2
  3. 3.M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016. 14
  4. 4.X. Chen, K. Kundu, Z. Zhang, H. Ma, S. Fidler, and R. Urtasun. Monocular 3d object detection for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2147–2156, 2016. 2, 6, 11
  5. 5.X. Chen, K. Kundu, Y. Zhu, A. G. Berneshawi, H. Ma, S. Fidler, and R. Urtasun. 3d object proposals for accurate object class detection. In Advances in Neural Information Processing Systems, pages 424–432, 2015. 6
  6. 6.X. Chen, H. Ma, J. Wan, B. Li, and T. Xia. Multi-view 3d object detection network for autonomous driving. In IEEE CVPR, 2017. 2, 5, 6, 11, 12, 13
  7. 7.Z. Deng and L. J. Latecki. Amodal detection of 3d objects: Inferring 3d bounding boxes from 2d ones in rgb-depth images. In Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, 2017. 2
  8. 8.M. Engelcke, D. Rao, D. Z. Wang, C. H. Tong, and I. Posner. Vote3deep: Fast object detection in 3d point clouds using efficient convolutional neural networks. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pages 1355–1361. IEEE, 2017. 1, 2
  9. 9.C.-Y. Fu, W. Liu, A. Ranga, A. Tyagi, and A. C. Berg. Dssd: Deconvolutional single shot detector. arXiv preprint arXiv:1701.06659, 2017. 12
  10. 10.A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):1231–1237, 2013. 5
  11. 11.A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012. 5
  12. 12.R. Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015. 12
  13. 13.R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 580–587. IEEE, 2014. 1
  14. 14.K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. arXiv preprint arXiv:1703.06870, 2017. 1, 3, 7
  15. 15.M. Jaderberg, K. Simonyan, A. Zisserman, et al. Spatial transformer networks. In NIPS 2015. 4
  16. 16.J. Lahoud and B. Ghanem. 2d-driven 3d object detection in rgb-d images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4622–4630, 2017. 2, 7
  17. 17.B. Li. 3d fully convolutional network for vehicle detection in point cloud. arXiv preprint arXiv:1611.08069, 2016. 2, 5, 6
  18. 18.B. Li, T. Zhang, and T. Xia. Vehicle detection from 3d lidar using fully convolutional network. arXiv preprint arXiv:1608.07916, 2016. 2, 13
  19. 19.Y. Li, S. Pirk, H. Su, C. R. Qi, and L. J. Guibas. Fpnn: Field probing neural networks for 3d data. arXiv preprint arXiv:1605.06240, 2016. 2
  20. 20.T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. arXiv preprint arXiv:1612.03144, 2016. 3, 12
  21. 21.T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. arXiv preprint arXiv:1708.02002, 2017. 12
  22. 22.W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pages 21–37. Springer, 2016. 12
  23. 23.D. Maturana and S. Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In IEEE/RSJ International Conference on Intelligent Robots and Systems, September 2015. 1, 2
  24. 24.A. Mousavian, D. Anguelov, J. Flynn, and J. Kosecka. 3d bounding box estimation using deep learning and geometry. arXiv preprint arXiv:1612.00496, 2016. 2, 5
  25. 25.C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017. 1, 2, 4, 5, 10, 11, 13
  26. 26.C. R. Qi, H. Su, M. Nießner, A. Dai, M. Yan, and L. Guibas. Volumetric and multi-view cnns for object classification on 3d data. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2016. 1, 2
  27. 27.C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint arXiv:1706.02413, 2017. 1, 2, 4, 5, 7, 10, 11, 13, 14
  28. 28.J. Ren, X. Chen, J. Liu, W. Sun, J. Pang, Q. Yan, Y.-W. Tai, and L. Xu. Accurate single stage detector using recurrent rolling convolution. In CVPR, 2017. 13
  29. 29.S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91–99, 2015. 2, 5, 12
  30. 30.Z. Ren and E. B. Sudderth. Three-dimensional object detection and layout prediction using clouds of oriented gradients. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1525–1533, 2016. 2, 7, 12
  31. 31.G. Riegler, A. O. Ulusoys, and A. Geiger. Octnet: Learning deep 3d representations at high resolutions. arXiv preprint arXiv:1611.05009, 2016. 2
  32. 32.K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 7, 12, 13
  33. 33.S. Song, S. P. Lichtenberg, and J. Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 567–576, 2015. 1, 5, 7, 12
  34. 34.S. Song and J. Xiao. Sliding shapes for 3d object detection in depth images. In Computer Vision–ECCV 2014, pages 634–651. Springer, 2014. 2
  35. 35.S. Song and J. Xiao. Deep sliding shapes for amodal 3d object detection in rgb-d images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 808–816, 2016. 2, 7
  36. 36.H. Su, S. Maji, E. Kalogerakis, and E. G. Learned-Miller. Multi-view convolutional neural networks for 3d shape recognition. In Proc. ICCV, 2015. 1
  37. 37.C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting unreasonable effectiveness of data in deep learning era. arXiv preprint arXiv:1707.02968, 1, 2017. 14
  38. 38.D. Z. Wang and I. Posner. Voting for voting in online point cloud object detection. Proceedings of the Robotics: Science and Systems, Rome, Italy, 1317, 2015. 2
  39. 39.P.-S. Wang, Y. Liu, Y.-X. Guo, C.-Y. Sun, and X. Tong. O-cnn: Octree-based convolutional neural networks for 3d shape analysis. ACM Transactions on Graphics (TOG), 36(4):72, 2017. 2
  40. 40.Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1912–1920, 2015. 1, 2
  41. 41.Y. Xiang, W. Choi, Y. Lin, and S. Savarese. Data-driven 3d voxel patterns for object category recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1903–1911, 2015. 2
  42. 42.S.-L. Yu, T. Westfechtel, R. Hamada, K. Ohno, and S. Tadokoro. Vehicle detection and localization on birds eye view elevation images using convolutional neural network. 2017 IEEE International Symposium on Safety, Security and Rescue Robotics (SSRR), 2017. 5, 6

Citation

MLA
Qi, C. R., et al. “Frustum PointNets for 3D Object Detection from RGB-D Data”. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 918–27, https://doi.org/10.1109/CVPR.2018.00102.
APA
Qi, C. R., Liu, W., Wu, C., Su, H., & Guibas, L. J. (2018). Frustum PointNets for 3D Object Detection from RGB-D Data. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 918–927. https://doi.org/10.1109/CVPR.2018.00102
Chicago
Qi, C. R., W. Liu, C. Wu, H. Su, and L. J. Guibas. 2018. “Frustum PointNets for 3D Object Detection from RGB-D Data”. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 918–27. https://doi.org/10.1109/CVPR.2018.00102.
Harvard
Qi, C.R. et al. (2018) “Frustum PointNets for 3D Object Detection from RGB-D Data”, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, pp. 918–927. Available at: https://doi.org/10.1109/CVPR.2018.00102.
Vancouver
1. Qi CR, Liu W, Wu C, Su H, Guibas LJ (2018) Frustum PointNets for 3D Object Detection from RGB-D Data. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, pp 918–927

BibTeX

@inproceedings{Qi_2018, title={Frustum PointNets for 3D Object Detection from RGB-D Data}, url={http://dx.doi.org/10.1109/CVPR.2018.00102}, DOI={10.1109/cvpr.2018.00102}, booktitle={2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition}, publisher={IEEE}, author={Qi, Charles R. and Liu, Wei and Wu, Chenxia and Su, Hao and Guibas, Leonidas J.}, year={2018}, month=June, pages={918–927} }
Metadata:Crossref

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE