Back

PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud

Shaoshuai ShiXiaogang WangHongsheng Li

article2019CVPR2,980 citations

Proposes PointRCNN, a two-stage 3D object detection framework that generates accurate proposals directly from raw point clouds through bottom-up foreground segmentation and canonical box refinement, eliminating the need for voxelization or 2D projections.

Listen

In autonomous driving and robotics, reliable 3D object detection from raw LiDAR point clouds remains difficult because the data are irregular and sparse, while the space of possible object positions and orientations is large. Prior approaches often project clouds into bird’s-eye or frontal views, voxelize them, or rely on 2D image detectors to crop regions, all of which discard geometric detail or inherit errors from the 2D stage.

This paper introduces PointRCNN, a two-stage detector that operates directly on unordered point clouds. The first stage segments the entire scene into foreground and background points and simultaneously produces a modest set of oriented 3D box proposals from the foreground points. The second stage pools the points and learned features inside each proposal, transforms them into a canonical coordinate frame aligned with the proposal, and refines the box coordinates and objectness score by fusing local spatial geometry with the global semantic features already computed in stage one. Both stages employ a bin-based regression loss that classifies coarse location or angle bins and then regresses residuals inside the chosen bin.

On the official KITTI test benchmark the method records the highest published accuracy among single-modality (LiDAR-only) entries at the time of submission: 85.94 % / 75.76 % / 68.32 % average precision for cars at the easy, moderate, and hard difficulty levels, together with leading numbers for cyclists. With only 300 proposals it reaches 98.2 % recall at a 0.5 IoU threshold on the validation set, markedly higher than earlier anchor-based or image-guided proposal generators. Ablation experiments confirm that the canonical transformation, the stage-one segmentation features, and the full bin-based loss each contribute several points of accuracy.

These gains matter because they demonstrate that high-quality 3D proposals can be obtained without image fusion or dense voxel grids, lowering both computational cost and the risk of missing objects visible only in 3D. The resulting detections are therefore more robust for downstream planning and safety systems. The framework still shows a modest gap versus multi-sensor methods on small, sparsely observed classes such as pedestrians, and all reported numbers come from a single urban driving dataset.

Further validation on additional LiDAR collections, together with targeted improvements for distant or occluded objects, would strengthen before large-scale deployment.

The Chinese University of Hong Kongdetection algorithmobject detectionpoint cloudPointRCNNGenerate 3D ProposalSegment Point Cloud
arXiv: 1812.04244sshaoshuai/PointRCNN
  • Paper: PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation, Charles R. Qi et al. (2017). PointNet established the foundational architecture for processing unordered point sets directly with deep learning, which the source method adapts for bottom-up 3D object proposal generation.
  • Paper: PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space, C. Qi et al. (2017). PointNet++ introduced hierarchical feature learning on point clouds to handle local neighborhoods and density variations, serving as a key methodological stepping stone for the source's multi-stage refinement.
  • Paper: VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection, Yin Zhou et al. (2017). VoxelNet pioneered end-to-end 3D object detection from raw LiDAR point clouds, providing the primary problem formulation and benchmark context that the source improves upon with its proposal-and-refinement approach.
  • Paper: Multi-view 3D Object Detection Network for Autonomous Driving, Xiaozhi Chen et al. (2017). Multi-view 3D detection networks established standard two-stage object localization frameworks on the KITTI benchmark, directly preceding the source's point-cloud-only proposal and refinement pipeline.
  • Paper: Fast R-CNN, Ross B. Girshick (2015). Fast R-CNN popularized multi-stage proposal refinement and region-of-interest pooling for object detection, concepts that the source directly translates from 2D image grids into raw 3D point cloud coordinates.
Cover for PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud

Abstract

In this paper, we propose PointRCNN for 3D object detection from raw point cloud. The whole framework is composed of two stages: stage-1 for the bottom-up 3D proposal generation and stage-2 for refining proposals in the canonical coordinates to obtain the final detection results. Instead of generating proposals from RGB image or projecting point cloud to bird's view or voxels as previous methods do, our stage-1 sub-network directly generates a small number of high-quality 3D proposals from point cloud in a bottom-up manner via segmenting the point cloud of the whole scene into foreground points and background. The stage-2 sub-network transforms the pooled points of each proposal to canonical coordinates to learn better local spatial features, which is combined with global semantic features of each point learned in stage-1 for accurate box refinement and confidence prediction. Extensive experiments on the 3D detection benchmark of KITTI dataset show that our proposed architecture outperforms state-of-the-art methods with remarkable margins by using only point cloud as input. The code is available at this https URL.

Topics

detection algorithmobject detectionpoint cloudrgb imagessemantic informationproposed architecturescloud servicesforeground objects

Keywords

PointRCNNGenerate 3D ProposalSegment Point CloudTransform Canonical CoordinateBin‑Based LossLiDARKITTI DatasetPoint CloudApply Non‑Maximum SuppressionStage‑1 Sub‑Network

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 PointRCNN for Point Cloud 3D Detection
  • 3.1 Bottom-up 3D proposal generation via point cloud segmentation
  • 3.2 Point cloud region pooling
  • 3.3 Canonical 3D bounding box refinement
  • 4 Experiments
  • 4.1 Implementation Details
  • 4.2 3D Object Detection on KITTI
  • 4.3 Ablation Study
  • 4.4 Qualitative Results
  • 5 Conclusion
  • References

Knowls

  1. Knowl 1 — PointRCNN Two-Stage 3D Object Detection Architecture

    model/method

    PointRCNN is a two-stage 3D object detection framework that operates directly on raw, irregular LiDAR point clouds without requiring 2D projection, voxelization, or 2D image bounding box priors.

    The framework consists of two sequential stages:

    1. Stage 1 (Bottom-Up 3D Proposal Generation): A point-based backbone network (PointNet++) processes raw scene point clouds to simultaneously segment foreground points from the background and generate oriented 3D bounding box proposals directly from segmented foreground points using a bin-based 3D box regression scheme. This bottom-up strategy restricts the proposal search space without relying on predefined 3D dense anchor grids.
    2. Stage 2 (Canonical 3D Box Refinement): For each generated proposal, points within an enlarged spatial bounding context are pooled and transformed into a canonical coordinate system (centered at the proposal with heading aligned to the local X′X'-axis). The local canonical spatial features are merged with global semantic features and depth cues from Stage 1. A second PointNet++ module then predicts residual coordinate refinements and confidence scores for final detection.

  2. Knowl 2 — Stage-1 Bottom-Up 3D Proposal Generation via Point Cloud Segmentation

    model/method

    Stage 1 of PointRCNN extracts point-wise representations directly from raw LiDAR point clouds to simultaneously predict point-level semantic masks and 3D bounding box proposals in a bottom-up manner.

    Network Backbone

    The backbone uses PointNet++ with multi-scale grouping. It downsamples the input scene (16,384 subsampled points) through four set-abstraction layers to group sizes of 4096, 1024, 256, and 64 points, followed by four feature propagation layers to produce per-point feature vectors f(p)∈RCf^{(p)} \in \mathbb{R}^C.

    Foreground Point Segmentation

    Points located inside annotated 3D bounding boxes are labeled as foreground, while external points are background. Points within a 0.2m boundary margin outside the ground-truth boxes are ignored during training to handle annotation noise. To mitigate the heavy foreground-background class imbalance in outdoor LiDAR scenes, point classification is trained with the focal loss:

    Lfocal(pt)=−αt(1−pt)γlog⁡(pt)L_{\text{focal}}(p_t) = -\alpha_t (1 - p_t)^\gamma \log(p_t)

    where αt=0.25\alpha_t = 0.25, γ=2\gamma = 2, and ptp_t is the model's estimated probability for the correct class (pt=pp_t = p for foreground points and pt=1−pp_t = 1 - p for background points).

    Proposal Generation and Filtering

    A bounding box regression head predicts 3D bounding boxes from each foreground point. Non-Maximum Suppression (NMS) based on oriented Bird's Eye View (BEV) IoU is applied to eliminate redundant proposals. During training, an IoU threshold of 0.85 is used to retain the top 300 proposals for Stage 2. During inference, an IoU threshold of 0.80 is used to keep the top 100 proposals.

  3. Knowl 3 — Bin-Based 3D Bounding Box Localization and Regression Loss

    equation

    A 3D bounding box is parameterized by its center location (x,y,z)(x, y, z), dimensions (h,w,l)(h, w, l), and heading angle θ\theta around the vertical axis. To avoid instability in direct smooth L1L_1 coordinate regression, PointRCNN discretizes horizontal localization (x,zx, z) and heading angle (θ\theta) into discrete bins combined with continuous residual regression.

    Localization Targets

    For a foreground point (x(p),y(p),z(p))(x^{(p)}, y^{(p)}, z^{(p)}) and its corresponding ground-truth object center (x,y,z)(x, y, z), a search range [−S,S][-S, S] along the XX and ZZ axes is discretized into bins of uniform length δ\delta with normalization constant C=δC = \delta:

    binu(p)=⌊u−u(p)+Sδ⌋,u∈{x,z}\text{bin}_u^{(p)} = \left\lfloor \frac{u - u^{(p)} + S}{\delta} \right\rfloor, \quad u \in \{x, z\}

    resu(p)=1C(u−u(p)+S−(binu(p)⋅δ+δ2)),u∈{x,z}\text{res}_u^{(p)} = \frac{1}{C} \left( u - u^{(p)} + S - \left(\text{bin}_u^{(p)} \cdot \delta + \frac{\delta}{2}\right) \right), \quad u \in \{x, z\}

    resy(p)=y−y(p)\text{res}_y^{(p)} = y - y^{(p)}

    Orientation and Dimension Targets

    The orientation range [0,2π)[0, 2\pi) is divided into nn discrete bins. The target bin binθ(p)\text{bin}_\theta^{(p)} and normalized residual resθ(p)\text{res}_\theta^{(p)} are calculated identically to the spatial coordinates. Box dimensions (h,w,l)(h, w, l) are directly regressed as residuals relative to the average dimensions (hˉ,wˉ,lˉ)(\bar{h}, \bar{w}, \bar{l}) of each object class across the training set: resv(p)=v−vˉ\text{res}_v^{(p)} = v - \bar{v} for v∈{h,w,l}v \in \{h, w, l\}.

    Stage-1 Regression Loss

    The total Stage-1 box regression loss over all NposN_{\text{pos}} foreground points is:

    Lreg=1Npos∑p∈pos(Lbin(p)+Lres(p))L_{\text{reg}} = \frac{1}{N_{\text{pos}}} \sum_{p \in \text{pos}} \left( L_{\text{bin}}^{(p)} + L_{\text{res}}^{(p)} \right)

    Lbin(p)=∑u∈{x,z,θ}(Fcls(bin^u(p),binu(p))+Freg(res^u(p),resu(p)))L_{\text{bin}}^{(p)} = \sum_{u \in \{x, z, \theta\}} \left( \mathcal{F}_{\text{cls}}(\widehat{\text{bin}}_u^{(p)}, \text{bin}_u^{(p)}) + \mathcal{F}_{\text{reg}}(\widehat{\text{res}}_u^{(p)}, \text{res}_u^{(p)}) \right)

    Lres(p)=∑v∈{y,h,w,l}Freg(res^v(p),resv(p))L_{\text{res}}^{(p)} = \sum_{v \in \{y, h, w, l\}} \mathcal{F}_{\text{reg}}(\widehat{\text{res}}_v^{(p)}, \text{res}_v^{(p)})

    where Fcls\mathcal{F}_{\text{cls}} is cross-entropy loss and Freg\mathcal{F}_{\text{reg}} is smooth L1L_1 loss.

  4. Knowl 4 — Context-Aware Point Cloud Region Pooling and Canonical Feature Representation

    model/method

    Stage 2 refines 3D proposal parameters by pooling local 3D points and fusing multi-level spatial and semantic features.

    Context-Aware Region Pooling

    For each generated proposal bi=(xi,yi,zi,hi,wi,li,θi)b_i = (x_i, y_i, z_i, h_i, w_i, l_i, \theta_i), its spatial dimensions are expanded by a context margin η\eta to obtain bie=(xi,yi,zi,hi+η,wi+η,li+η,θi)b_i^e = (x_i, y_i, z_i, h_i + \eta, w_i + \eta, l_i + \eta, \theta_i) (with η=1.0 m\eta = 1.0\text{ m}). All point coordinates p=(x(p),y(p),z(p))p = (x^{(p)}, y^{(p)}, z^{(p)}) residing within bieb_i^e are retained. Proposals with zero interior points are discarded.

    Point Feature Construction

    For each pooled point pp, five distinct feature sources are gathered:

    1. Transformed local spatial coordinate p~∈R3\tilde{p} \in \mathbb{R}^3 in the canonical proposal frame.
    2. Laser reflection intensity r(p)∈Rr^{(p)} \in \mathbb{R}.
    3. Predicted foreground segmentation mask probability m(p)∈[0,1]m^{(p)} \in [0, 1] from Stage 1.
    4. Radial distance to the sensor d(p)=(x(p))2+(y(p))2+(z(p))2d^{(p)} = \sqrt{(x^{(p)})^2 + (y^{(p)})^2 + (z^{(p)})^2}, which restores absolute depth information lost during canonical coordinate transformation.
    5. Global semantic feature representation f(p)∈RCf^{(p)} \in \mathbb{R}^C learned by the Stage-1 backbone.

    Refinement Network Architecture

    From the pooled context box, 512 points are randomly sampled. The local feature vector [p~,r(p),m(p),d(p)][\tilde{p}, r^{(p)}, m^{(p)}, d^{(p)}] is projected via fully-connected layers to dimension CC and concatenated with f(p)f^{(p)}. The merged representation passes through three set abstraction layers with single-scale grouping (group sizes 128, 32, 1) to extract a single global feature vector that feeds into the confidence classification head and canonical box refinement head.

  5. Knowl 5 — Canonical Coordinate Transformation and Proposal Refinement Losses

    equation

    To constrain the refinement search space, Stage 2 transforms pooled points and ground-truth boxes into the canonical coordinate system of the proposal bi=(xi,yi,zi,hi,wi,li,θi)b_i = (x_i, y_i, z_i, h_i, w_i, l_i, \theta_i).

    Canonical Transformation

    In the canonical coordinate frame, the proposal origin is (0,0,0)(0, 0, 0), the X′X'-axis points along the proposal heading direction θi\theta_i, the Z′Z'-axis is orthogonal on the horizontal plane, and the Y′Y'-axis remains parallel to the vertical LiDAR axis. The transformed proposal b~i\tilde{b}_i and target ground-truth box b~igt\tilde{b}_i^{\text{gt}} are defined as:

    b~i=(0,0,0,hi,wi,li,0)\tilde{b}_i = (0, 0, 0, h_i, w_i, l_i, 0)

    b~igt=(xigt−xi,yigt−yi,zigt−zi,higt,wigt,ligt,θigt−θi)\tilde{b}_i^{\text{gt}} = (x_i^{\text{gt}} - x_i, y_i^{\text{gt}} - y_i, z_i^{\text{gt}} - z_i, h_i^{\text{gt}}, w_i^{\text{gt}}, l_i^{\text{gt}}, \theta_i^{\text{gt}} - \theta_i)

    Canonical Refinement Target Formulation

    A ground-truth box is assigned to proposal bib_i if their 3D IoU ≥0.55\ge 0.55. Center offsets (Δx,Δy,Δz)(\Delta x, \Delta y, \Delta z) are calculated using bin-based classification and residual regression over a smaller search range SS (S=1.5 m,δ=0.5 mS = 1.5\text{ m}, \delta = 0.5\text{ m}). Assuming residual rotation θigt−θi∈[−π/4,π/4]\theta_i^{\text{gt}} - \theta_i \in [-\pi/4, \pi/4], the interval [−π/4,π/4][-\pi/4, \pi/4] of length π/2\pi/2 is partitioned into bins of angular size ω\omega (with ω=10∘\omega = 10^\circ):

    bini,Δθ=⌊θigt−θi+π4ω⌋\text{bin}_{i, \Delta\theta} = \left\lfloor \frac{\theta_i^{\text{gt}} - \theta_i + \frac{\pi}{4}}{\omega} \right\rfloor

    resi,Δθ=2ω(θigt−θi+π4−(bini,Δθ⋅ω+ω2))\text{res}_{i, \Delta\theta} = \frac{2}{\omega} \left( \theta_i^{\text{gt}} - \theta_i + \frac{\pi}{4} - \left( \text{bin}_{i, \Delta\theta} \cdot \omega + \frac{\omega}{2} \right) \right)

    Dimension residuals (resi,Δh,resi,Δw,resi,Δl)(\text{res}_{i, \Delta h}, \text{res}_{i, \Delta w}, \text{res}_{i, \Delta l}) are regressed relative to dataset class averages.

    Stage-2 Loss Function

    Lrefine=1∥B∥∑i∈BFcls(probi,labeli)+1∥Bpos∥∑i∈Bpos(L~bin(i)+L~res(i))L_{\text{refine}} = \frac{1}{\|B\|} \sum_{i \in B} \mathcal{F}_{\text{cls}}(\text{prob}_i, \text{label}_i) + \frac{1}{\|B_{\text{pos}}\|} \sum_{i \in B_{\text{pos}}} (\tilde{L}_{\text{bin}}^{(i)} + \tilde{L}_{\text{res}}^{(i)})

    where BB is the set of all proposals, probi\text{prob}_i is the predicted confidence score, labeli∈{0,1}\text{label}_i \in \{0, 1\} is 1 if 3D IoU≥0.6\text{3D IoU} \ge 0.6 and 0 if 3D IoU<0.45\text{3D IoU} < 0.45, BposB_{\text{pos}} contains proposals with 3D IoU≥0.55\text{3D IoU} \ge 0.55, Fcls\mathcal{F}_{\text{cls}} is cross-entropy loss, and L~bin(i),L~res(i)\tilde{L}_{\text{bin}}^{(i)}, \tilde{L}_{\text{res}}^{(i)} follow the same form as the Stage-1 regression losses applied to the canonical targets.

  6. Knowl 6 — 3D Proposal Recall Performance on KITTI Validation Set

    data/table

    The Stage-1 bottom-up proposal network achieves high bounding box recall with a small number of proposals on the KITTI validation split (Car class, moderate difficulty):

    RoIs # Recall (IoU=0.5) Recall (IoU=0.7)
    MV3D AVOD Ours Ours
    10 - 86.00 86.66 29.87
    20 - - 91.83 32.55
    30 - - 93.31 32.76
    40 - - 95.55 40.04
    50 - 91.00 96.01 40.28
    100 - - 96.79 74.81
    200 - - 98.03 76.29
    300 91.00 - 98.21 82.29

    With 50 proposals at an IoU threshold of 0.5, PointRCNN reaches 96.01% recall, outperforming multi-sensor AVOD (91.00%) by 5.01% while using only LiDAR point cloud input. With 300 proposals, PointRCNN attains 98.21% recall at IoU=0.5\text{IoU}=0.5 and 82.29% recall at IoU=0.7\text{IoU}=0.7.

  7. Knowl 7 — 3D Detection Benchmark Results on KITTI Test Server

    data/table

    PointRCNN performance on the official KITTI 3D object detection test server evaluated with Average Precision (AP) against published methods:

    Method Modality Car (IoU=0.7) Pedestrian (IoU=0.5) Cyclist (IoU=0.5)
    Easy Mod. Hard Easy Mod. Hard Easy Mod. Hard
    MV3D RGB + LiDAR 71.09 62.35 55.12 - - - - - -
    UberATG-ContFuse RGB + LiDAR 82.54 66.22 64.04 - - - - - -
    AVOD-FPN RGB + LiDAR 81.94 71.88 66.38 50.80 42.81 40.88 64.00 52.18 46.61
    F-PointNet RGB + LiDAR 81.20 70.39 62.19 51.21 44.89 40.23 71.96 56.77 50.39
    VoxelNet LiDAR 77.47 65.11 57.73 39.48 33.69 31.51 61.22 48.36 44.37
    SECOND LiDAR 83.13 73.66 66.20 51.07 42.56 37.29 70.51 53.85 46.90
    PointRCNN (Ours) LiDAR 85.94 75.76 68.32 49.43 41.78 38.63 73.93 59.60 53.59

    Using only LiDAR point cloud data, PointRCNN outperforms all previous methods (including multi-sensor approaches using both RGB images and LiDAR) in Car AP (75.76% moderate) and Cyclist AP (59.60% moderate). On Pedestrian detection, it exceeds prior LiDAR-only methods (VoxelNet: 33.69% moderate; PointRCNN: 41.78% moderate).

  8. Knowl 8 — Ablation of Stage-2 Input Feature Components and Canonical Transformation

    data/table

    Ablation study evaluating the contribution of individual input feature components to the Stage-2 refinement sub-network on the KITTI validation split (Car class, without GT-AUG):

    Canonical Trans. (CT) RPN Features (f(p)f^{(p)}) Distance (d(p)d^{(p)}) Seg. Mask (m(p)m^{(p)}) APEasy\text{AP}_{\text{Easy}} APModerate\text{AP}_{\text{Moderate}} APHard\text{AP}_{\text{Hard}}
    ×\times ✓ ✓ ✓ 7.64 13.68 13.94
    ✓ ×\times ✓ ✓ 84.75 74.96 74.29
    ✓ ✓ ×\times ✓ 87.34 76.79 75.46
    ✓ ✓ ✓ ×\times 86.25 76.64 75.86
    ✓ ✓ ✓ ✓ 88.45 77.67 76.30

    Removing the canonical transformation causes performance to collapse (moderate AP drops from 77.67% to 13.68%), showing that canonical alignment is essential for learning local spatial features. Excluding the Stage-1 semantic features f(p)f^{(p)} causes a 2.71% drop in moderate AP. Sensor distance d(p)d^{(p)} and segmentation mask m(p)m^{(p)} provide additional performance improvements.

  9. Knowl 9 — Ablation of Context Margin Width in Region Pooling

    data/table

    Effect of the spatial context enlargement margin η\eta in Stage-2 point cloud region pooling on the KITTI validation split (Car class, without GT-AUG):

    Context Width η\eta APEasy\text{AP}_{\text{Easy}} APModerate\text{AP}_{\text{Moderate}} APHard\text{AP}_{\text{Hard}}
    No context (0.0m) 86.65 75.68 68.92
    0.5m 87.87 77.12 75.61
    0.8m 88.27 77.40 76.07
    1.0m 88.45 77.67 76.30
    1.5m 86.82 76.87 75.88
    2.0m 86.47 76.61 75.53

    Setting η=1.0 m\eta = 1.0\text{ m} yields optimal detection performance. Omitting contextual points (no context) causes a substantial decrease on hard examples (from 76.30% to 68.92% AP), where occluded or distant objects rely on surrounding context. Margins larger than 1.0m degrade precision due to the inclusion of surrounding clutter and points from neighboring objects.

  10. Knowl 10 — Comparative Effectiveness of Bin-Based 3D Bounding Box Regression Loss

    empirical result

    Empirical comparison of five bounding box regression loss formulations for Stage-1 3D proposal generation demonstrates the superiority of full bin-based regression:

    1. Residual-based loss (RB-loss): Direct smooth L1L_1 regression of center and angle offsets.
    2. Residual-cos-based loss (RCB-loss): Smooth L1L_1 regression with heading residual encoded as (cos⁡(Δθ),sin⁡(Δθ))(\cos(\Delta\theta), \sin(\Delta\theta)) to remove angle ambiguity.
    3. Corner loss (CN-loss): Direct regression of 8 box corner locations.
    4. Partial-bin-based loss (PBB-loss): Bin-based classification and residual regression applied only to orientation θ\theta and center x,zx, z.
    5. Full bin-based loss (BB-loss, PointRCNN): Uniform spatial bin discretization and residual regression along XX and ZZ axes combined with angular binning.

    Evaluation on proposal recall curves shows that the full bin-based loss achieves the fastest convergence and highest final proposal recall across 200 epochs at both IoU=0.5\text{IoU}=0.5 and IoU=0.7\text{IoU}=0.7. In contrast, RB-loss and CN-loss converge slowly and plateau at lower recall levels, especially at the strict IoU=0.7\text{IoU}=0.7 threshold.

Coverage note — None was omitted; all key contributions including framework architecture, stage-1 proposal segmentation and bin-based regression, stage-2 canonical refinement, pooling formulation, loss definitions, benchmark comparisons, and full ablation studies are represented.

References

  1. 1.Florian Chabot, Mohamed Chaouch, Jaonary Rabarisoa, Celine Teuliere, and Thierry Chateau. Deep manta: A coarse-to-fine many-task network for joint 2d and 3d vehicle analysis from monocular image. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit.(CVPR), pages 2040–2049, 2017.
  2. 2.Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, and Raquel Urtasun. Monocular 3d object detection for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2147–2156, 2016.
  3. 3.Xiaozhi Chen, Kaustav Kundu, Yukun Zhu, Andrew G Berneshawi, Huimin Ma, Sanja Fidler, and Raquel Urtasun. 3d object proposals for accurate object class detection. In Advances in Neural Information Processing Systems, pages 424–432, 2015.
  4. 4.Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1907–1915, 2017.
  5. 5.Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pages 1635–1643, 2015.
  6. 6.Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware semantic segmentation via multi-task network cascades. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3150–3158, 2016.
  7. 7.Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
  8. 8.Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015.
  9. 9.Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. CVPR, 2018.
  10. 10.Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Computer Vision (ICCV), 2017 IEEE International Conference on, pages 2980–2988. IEEE, 2017.
  11. 11.Jan Hosang, Rodrigo Benenson, Piotr Dollár, and Bernt Schiele. What makes for effective detection proposals? IEEE transactions on pattern analysis and machine intelligence, 38(4):814–830, 2016.
  12. 12.Qiangui Huang, Weiyue Wang, and Ulrich Neumann. Recurrent slice networks for 3d segmentation of point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2626–2635, 2018.
  13. 13.Mingyang Jiang, Yiran Wu, and Cewu Lu. Pointsift: A siftlike network module for 3d point cloud semantic segmentation. CoRR, abs/1807.00652, 2018.
  14. 14.Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven Lake Waslander. Joint 3d proposal generation and object detection from view aggregation. CoRR, abs/1712.02294, 2017.
  15. 15.Buyu Li, Wanli Ouyang, Lu Sheng, Xingyu Zeng, and Xiaogang Wang. Gs3d: An efficient 3d object detection framework for autonomous driving. 2019.
  16. 16.Hongyang Li, Bo Dai, Shaoshuai Shi, Wanli Ouyang, and Xiaogang Wang. Feature Intertwiner for Object Detection. In ICLR, 2019.
  17. 17.Ming Liang, Bin Yang, Shenlong Wang, and Raquel Urtasun. Deep continuous fusion for multi-sensor 3d object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pages 641–656, 2018.
  18. 18.Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2117–2125, 2017.
  19. 19.Tsung-Yi Lin, Priyal Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. IEEE transactions on pattern analysis and machine intelligence, 2018.
  20. 20.Shu Liu, Jiaya Jia, Sanja Fidler, and Raquel Urtasun. Sgn: Sequential grouping networks for instance segmentation. In The IEEE International Conference on Computer Vision (ICCV), 2017.
  21. 21.Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pages 21–37. Springer, 2016.
  22. 22.Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on, pages 922–928. IEEE, 2015.
  23. 23.Roozbeh Mottaghi, Yu Xiang, and Silvio Savarese. A coarse-to-fine model for 3d pose estimation and sub-category recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 418–426, 2015.
  24. 24.Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Košecká. 3d bounding box estimation using deep learning and geometry. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, pages 5632–5640. IEEE, 2017.
  25. 25.Charles Ruizhongtai Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J. Guibas. Frustum pointnets for 3d object detection from RGB-D data. CoRR, abs/1711.08488, 2017.
  26. 26.Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 1(2):4, 2017.
  27. 27.Charles R Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas J Guibas. Volumetric and multi-view cnns for object classification on 3d data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656, 2016.
  28. 28.Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems, pages 5099–5108, 2017.
  29. 29.Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016.
  30. 30.Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7263–7271, 2017.
  31. 31.Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. CoRR, abs/1804.02767, 2018.
  32. 32.Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91–99, 2015.
  33. 33.Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representations at high resolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, volume 3, 2017.
  34. 34.Shuran Song and Jianxiong Xiao. Deep sliding shapes for amodal 3d object detection in rgb-d images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 808–816, 2016.
  35. 35.Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, pages 190–198. IEEE, 2017.
  36. 36.Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision, pages 945–953, 2015.
  37. 37.Hao Su, Fan Wang, Eric Yi, and Leonidas J Guibas. 3d-assisted feature synthesis for novel views of an object. In Proceedings of the IEEE International Conference on Computer Vision, pages 2677–2685, 2015.
  38. 38.Bin Xu and Zhenzhong Chen. Multi-level fusion based 3d object detection from monocular images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2345–2353, 2018.
  39. 39.Danfei Xu, Dragomir Anguelov, and Ashesh Jain. Pointfusion: Deep sensor fusion for 3d bounding box estimation. CoRR, abs/1711.10871, 2017.
  40. 40.Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018.
  41. 41.Bin Yang, Ming Liang, and Raquel Urtasun. Hdnet: Exploiting hd maps for 3d object detection. In Conference on Robot Learning, pages 146–155, 2018.
  42. 42.Bin Yang, Wenjie Luo, and Raquel Urtasun. Pixor: Real-time 3d object detection from point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7652–7660, 2018.
  43. 43.Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. CoRR, abs/1711.06396, 2017.
  44. 44.Menglong Zhu, Konstantinos G Derpanis, Yinfei Yang, Samarth Brahmbhatt, Mabel Zhang, Cody Phillips, Matthieu Lecce, and Kostas Daniilidis. Single image 3d object detection and pose estimation for grasping. In Robotics and Automation (ICRA), 2014 IEEE International Conference on, pages 3936–3943. IEEE, 2014.

Citation

MLA
Shi, S., et al. “PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud”. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 770–79, https://doi.org/10.1109/CVPR.2019.00086.
APA
Shi, S., Wang, X., & Li, H. (2019). PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 770–779. https://doi.org/10.1109/CVPR.2019.00086
Chicago
Shi, S., X. Wang, and H. Li. 2019. “PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud”. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 770–79. https://doi.org/10.1109/CVPR.2019.00086.
Harvard
Shi, S., Wang, X. and Li, H. (2019) “PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud”, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp. 770–779. Available at: https://doi.org/10.1109/CVPR.2019.00086.
Vancouver
1. Shi S, Wang X, Li H (2019) PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 770–779

BibTeX

@inproceedings{Shi_2019, title={PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud}, url={http://dx.doi.org/10.1109/CVPR.2019.00086}, DOI={10.1109/cvpr.2019.00086}, booktitle={2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, publisher={IEEE}, author={Shi, Shaoshuai and Wang, Xiaogang and Li, Hongsheng}, year={2019}, month=June, pages={770–779} }
Metadata:Crossref

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE