PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection

Shaoshuai ShiChaoxu GuoLi JiangZhe WangJianping ShiXiaogang WangHongsheng Li

article2019CVPR2,345 citations

Proposes PV-RCNN, a high-performance 3D object detection architecture that combines the computational efficiency of 3D voxel convolutional networks with the flexible feature learning of PointNet-based set abstraction to achieve state-of-the-art accuracy on the KITTI and Waymo Open datasets.

Listen

Autonomous driving and robotic perception rely heavily on LiDAR sensors to accurately identify surrounding obstacles in three dimensions. However, existing 3D object detection architectures face a core trade-off: grid-based methods divide irregular point clouds into structured voxels to achieve high processing speed but lose spatial detail, while point-based methods retain precise coordinates but suffer from high computational costs. Finding an architecture that combines computational efficiency with fine-grained spatial accuracy has become a critical engineering challenge for real-world automated mobility.

The article introduces and evaluates PV-RCNN, a novel two-stage 3D object detection framework designed to integrate the speed and proposal quality of 3D voxel convolutional networks with the accurate spatial context of point-based feature learning. To demonstrate the system's effectiveness and generalization capability, the authors performed extensive empirical evaluations and ablation studies using two major industry-standard benchmarks: the KITTI dataset and the large-scale Waymo Open Dataset.

The high-level approach employs a two-step feature integration pipeline that controls memory overhead while maximizing contextual awareness. First, a voxel-based network processes the raw point cloud to generate high-quality candidate object proposals and multi-scale scene features. Instead of attempting a memory-heavy direct pooling across thousands of sparse voxels, the network summarizes the entire 3D scene into a compact set of representative keypoints sampled across the environment. Second, the system pools features from these keypoints into a 3D grid within each candidate bounding box across multiple search radii, weighting foreground objects to refine final object classification and spatial boundaries.

The evaluation produced four key findings: First, PV-RCNN achieved top-ranked performance on the KITTI benchmark, outperforming existing LiDAR-only and multi-modal methods by 1.58% to 1.73% mean Average Precision across easy, moderate, and hard car detection categories. Second, on the Waymo Open Dataset, the system outperformed previous state-of-the-art methods by an overall margin of 7.37% in 3D vehicle detection accuracy, with performance gains reaching up to 9.19% for mid-range targets (30 to 50 meters). Third, the backbone network achieved an 85.5% object proposal recall rate on KITTI, substantially higher than the 74.8% to 76.8% achieved by leading point-based baselines. Fourth, ablation analyses confirmed that combining voxel-to-keypoint encoding with keypoint-to-grid RoI pooling provided superior localization accuracy while keeping memory usage manageable.

These findings indicate that autonomous systems do not need to choose between the operational speed of voxel models and the precision of raw point models. By providing significantly higher detection accuracy—particularly for distant or sparsely observed obstacles—the framework directly enhances vehicle safety and situational awareness without relying on expensive camera-LiDAR fusion pipelines. Engineering teams deploying 3D perception stacks should consider adopting hybrid point-voxel architectures as a primary baseline and evaluate integrating keypoint-based pooling into their existing region-proposal workflows.

Confidence in these findings is supported by rigorous cross-dataset validation on large-scale public benchmarks and detailed ablation studies. However, practical deployment decisions should account for boundary conditions: performance is evaluated exclusively on standard autonomous driving point-cloud formats and voxel resolutions, and real-time latency trade-offs on specialized vehicle edge hardware were not deeply explored in the article. Teams should run targeted on-vehicle pilot tests to benchmark latency and hardware resource utilization prior to production deployment.

Institute of Automation, Chinese Academy of Sciences SenseTime The Chinese University of Hong Kong context information convolutional neural networks point cloud Feature Detection Point

arXiv: 1912.13192 open-mmlab/OpenPCDet

Paper: PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud, Shaoshuai Shi et al. (2019). PointRCNN directly provides the foundational two-stage 3D proposal generation and point cloud feature pooling architecture that PV-RCNN modifies and builds upon.
Paper: PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation, Charles R. Qi et al. (2017). PointNet establishes the core set abstraction and permutation-invariant point processing techniques that PV-RCNN adapts for keypoint feature learning.
Paper: PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space, C. Qi et al. (2017). PointNet++ introduces hierarchical feature learning on point sets, which is essential for understanding the multi-scale receptive fields used in PV-RCNN.

Paper: Center-based 3D Object Detection and Tracking, Tianwei Yin et al. (2020). CenterPoint extends the principles of 3D object detection established in PV-RCNN by introducing an efficient center-based representation and tracking framework.
Paper: KPConv: Flexible and Deformable Convolution for Point Clouds, Hugues Thomas et al. (2019). KPConv builds upon point-cloud processing methods like PV-RCNN by introducing flexible and deformable convolutions that operate directly on continuous 3D spaces.

Abstract

We present a novel and high-performance 3D object detection framework, named PointVoxel-RCNN (PV-RCNN), for accurate 3D object detection from point clouds. Our proposed method deeply integrates both 3D voxel Convolutional Neural Network (CNN) and PointNet-based set abstraction to learn more discriminative point cloud features. It takes advantages of efficient learning and high-quality proposals of the 3D voxel CNN and the flexible receptive fields of the PointNet-based networks. Specifically, the proposed framework summarizes the 3D scene with a 3D voxel CNN into a small set of keypoints via a novel voxel set abstraction module to save follow-up computations and also to encode representative scene features. Given the high-quality 3D proposals generated by the voxel CNN, the RoI-grid pooling is proposed to abstract proposal-specific features from the keypoints to the RoI-grid points via keypoint set abstraction with multiple receptive fields. Compared with conventional pooling operations, the RoI-grid feature points encode much richer context information for accurately estimating object confidences and locations. Extensive experiments on both the KITTI dataset and the Waymo Open dataset show that our proposed PV-RCNN surpasses state-of-the-art 3D detection methods with remarkable margins by using only point clouds. Code is available at this https URL.

1 Introduction
2 Related Work
3 PV-RCNN for Point Cloud Object Detection
3.1 3D Voxel CNN for Efficient Feature Encoding and Proposal Generation
3.2 Voxel-to-keypoint Scene Encoding via Voxel Set Abstraction
3.3 Keypoint-to-grid RoI Feature Abstraction for Proposal Refinement
3.4 Training losses
4 Experiments
4.1 Experimental Setup
4.2 3D Detection on the KITTI Dataset
4.3 3D Detection on the Waymo Open Dataset
4.4 Ablation Studies
5 Conclusion
References

Knowls

Knowl 1 — PV-RCNN Framework Architecture
model/method
PointVoxel-RCNN (PV-RCNN) is a two-stage 3D object detection framework designed for LiDAR point clouds that combines 3D voxel Convolutional Neural Networks (CNNs) with PointNet-based set abstraction.

The framework operates in three main stages:
1. 3D Voxel CNN Backbone and Proposal Generation: The input point cloud is voxelized and processed by a 3D sparse convolutional network with $1\times$ , $2\times$ , $4\times$ , and $8\times$ downsampling to extract multi-scale sparse 3D feature volumes. The $8\times$ downsampled 3D feature volume is stacked along the vertical ( $Z$ ) axis to form a 2D bird's-eye view (BEV) feature map. An anchor-based 3D Region Proposal Network (RPN) operates on this BEV map to generate high-quality 3D bounding box proposals.
2. Voxel-to-Keypoint Scene Encoding: A small set of $n$ keypoints is selected from the input point cloud via Furthest Point Sampling (FPS). A Voxel Set Abstraction (VSA) module pools multi-scale features from all levels of the 3D voxel CNN, raw point features, and BEV map features into these keypoints. A Predicted Keypoint Weighting (PKW) module re-weights the keypoint features based on predicted foreground probabilities.
3. Keypoint-to-Grid RoI Feature Abstraction and Refinement: For each 3D proposal (RoI), a uniform $6 \times 6 \times 6$ grid of RoI-grid points is sampled within the proposed box. Multi-scale keypoint features are aggregated into each grid point using keypoint set abstraction across multiple search radii. The RoI-grid features are vectorized and passed through a multi-layer perceptron (MLP) to output refined 3D box parameters and an IoU-guided confidence score.
Knowl 2 — Voxel Set Abstraction and Extended Keypoint Feature Encoding
model/method
The Voxel Set Abstraction (VSA) module summarizes the multi-scale 3D voxel CNN feature volumes into a small set of $n$ keypoints $\mathcal{K} = \{p_1, \dots, p_n\} \subset \mathbb{R}^3$ sampled via Furthest Point Sampling (FPS) from the input point cloud $\mathcal{P}$ .

Let $\mathcal{F}^{(l_k)} = \{f_1^{(l_k)}, \dots, f_{N_k}^{(l_k)}\}$ be the set of voxel-wise feature vectors at the $k$ -th level of the 3D voxel CNN, and $\mathcal{V}^{(l_k)} = \{v_1^{(l_k)}, \dots, v_{N_k}^{(l_k)}\} \subset \mathbb{R}^3$ be their 3D coordinates calculated from voxel indices and voxel dimensions, where $N_k$ is the number of non-empty voxels at level $k$ . For each keypoint $p_i$ , neighboring non-empty voxels within radius $r_k$ are gathered:

$S_i^{(l_k)} = \left\{ \begin{bmatrix} f_j^{(l_k)} \\ v_j^{(l_k)} - p_i \end{bmatrix} \;\middle|\; \|v_j^{(l_k)} - p_i\|_2 < r_k, \; v_j^{(l_k)} \in \mathcal{V}^{(l_k)}, \; f_j^{(l_k)} \in \mathcal{F}^{(l_k)} \right\}$

where $v_j^{(l_k)} - p_i$ represents the local relative coordinates. The keypoint feature vector at level $k$ is computed by a PointNet block:

$f_i^{(pv_k)} = \max\left( \mathcal{G}\left( \mathcal{M}\left( S_i^{(l_k)} \right) \right) \right)$

where $\mathcal{M}(\cdot)$ randomly samples at most $T_k$ voxels to bound computation, $\mathcal{G}(\cdot)$ is a multi-layer perceptron (MLP) encoding voxel features and relative offsets, and $\max(\cdot)$ denotes channel-wise max-pooling. Multi-scale contextual information is captured by using multiple radii $r_k$ at each level. Features from all four voxel CNN levels are concatenated:

$f_i^{(pv)} = \left[ f_i^{(pv_1)}, f_i^{(pv_2)}, f_i^{(pv_3)}, f_i^{(pv_4)} \right]$

In the Extended VSA module, this representation is further concatenated with raw point-cloud features $f_i^{(raw)}$ (aggregated using the same PointNet grouping directly on raw points) and 2D BEV feature map features $f_i^{(bev)}$ (obtained by projecting $p_i$ onto the 2D BEV coordinate system and applying bilinear interpolation):

$f_i^{(p)} = \left[ f_i^{(pv)}, f_i^{(raw)}, f_i^{(bev)} \right]$
Knowl 3 — Predicted Keypoint Weighting Module
model/method
Because keypoints sampled via Furthest Point Sampling (FPS) are distributed across the entire scene, many keypoints lie in background regions. The Predicted Keypoint Weighting (PKW) module re-weights keypoint features using foreground point segmentation supervision to emphasize keypoints belonging to foreground objects during proposal refinement.

Point segmentation labels are generated directly from ground-truth 3D bounding boxes by assigning a foreground label (1) to keypoints located inside any ground-truth bounding box and background (0) otherwise.

For each keypoint $p_i$ with feature vector $f_i^{(p)}$ , a three-layer MLP $\mathcal{A}(\cdot)$ with a sigmoid output layer predicts a foreground confidence score in $[0, 1]$ . The weighted feature vector $\tilde{f}_i^{(p)}$ is computed as:

$\tilde{f}_i^{(p)} = \mathcal{A}\left(f_i^{(p)}\right) \cdot f_i^{(p)}$

The PKW module is trained using focal loss to address class imbalance between foreground and background keypoints:

$L_{seg} = -\alpha_t (1 - p_t)^\gamma \log(p_t)$

where $p_t$ is the predicted probability corresponding to the ground-truth binary label.
Knowl 4 — RoI-Grid Pooling via Keypoint Set Abstraction
model/method
Given a 3D Region of Interest (RoI) proposal, the RoI-grid pooling module aggregates features from the weighted keypoint set $\tilde{\mathcal{F}} = \{\tilde{f}_1^{(p)}, \dots, \tilde{f}_n^{(p)}\}$ with coordinates $\mathcal{K} = \{p_1, \dots, p_n\}$ into the proposal volume.

A regular $6 \times 6 \times 6$ grid of 216 points $\mathcal{G} = \{g_1, \dots, g_{216}\} \subset \mathbb{R}^3$ is sampled uniformly within the 3D bounding box proposal. For each grid point $g_i$ , neighboring keypoints within a search radius $\tilde{r}$ are identified:

$\tilde{\Psi}_i = \left\{ \begin{bmatrix} \tilde{f}_j^{(p)} \\ p_j - g_i \end{bmatrix} \;\middle|\; \|p_j - g_i\|_2 < \tilde{r}, \; p_j \in \mathcal{K}, \; \tilde{f}_j^{(p)} \in \tilde{\mathcal{F}} \right\}$

where $p_j - g_i$ denotes the relative coordinate offset. A PointNet block aggregates these keypoint features into a grid point feature:

$\tilde{f}_i^{(g)} = \max\left( \mathcal{G}\left( \mathcal{M}\left( \tilde{\Psi}_i \right) \right) \right)$

where $\mathcal{M}(\cdot)$ subsamples a fixed maximum number of neighboring keypoints, $\mathcal{G}(\cdot)$ is an MLP, and $\max(\cdot)$ denotes channel-wise max-pooling.

Multiple radii $\tilde{r}$ are evaluated at each grid point and their aggregated features are concatenated to capture multi-scale context beyond the RoI boundary. Finally, the features of all 216 grid points are vectorized and mapped through a two-layer MLP with 256 output units to form the proposal's global feature representation.
Knowl 5 — 3D Proposal Refinement and IoU-Guided Confidence Prediction
model/method
The proposal refinement network processes the 256-dimensional feature vector of each RoI using a two-layer MLP with two parallel output branches: 3D bounding box regression and quality-aware confidence prediction.
1. Box Refinement: The regression branch predicts 3D bounding box residual parameters relative to the proposal: center offsets $(\Delta x, \Delta y, \Delta z)$ , size residuals $(\Delta l, \Delta h, \Delta w)$ , and orientation residual $\Delta \theta$ , optimized using smooth- $L_1$ loss.
2. IoU-Guided Confidence Prediction: Instead of binary classification labels, the confidence branch is trained using continuous 3D Intersection-over-Union (IoU) targets. For the $k$ -th 3D RoI, let $\text{IoU}_k$ be its 3D overlap with the assigned ground-truth box. The normalized target $y_k \in [0, 1]$ is defined as:
$y_k = \min\left(1, \max\left(0, 2\,\text{IoU}_k - 0.5\right)\right)$

The confidence branch is trained to minimize binary cross-entropy loss relative to the predicted confidence $\tilde{y}_k$ :

$L_{iou} = -y_k \log(\tilde{y}_k) - (1 - y_k) \log(1 - \tilde{y}_k)$
Knowl 6 — Multi-Task Training Loss Formulation
equation
The PV-RCNN detector is trained end-to-end with an equally weighted multi-task loss consisting of the Region Proposal Network loss $L_{rpn}$ , the keypoint segmentation loss $L_{seg}$ , and the proposal refinement loss $L_{rcnn}$ :

$L_{total} = L_{rpn} + L_{seg} + L_{rcnn}$

where:
1. Region Proposal Loss:
$L_{rpn} = L_{cls} + \beta \sum_{r \in \{x, y, z, l, h, w, \theta\}} L_{\text{smooth-}L1}\left(\Delta \hat{r}_a, \Delta r_a\right)$

where $L_{cls}$ is the anchor classification focal loss, $\Delta \hat{r}_a$ is the predicted anchor regression residual, $\Delta r_a$ is the ground-truth anchor regression target, and $\beta$ is a loss balance factor.
1. Keypoint Segmentation Loss: $L_{seg}$ is the focal loss computed on binary foreground/background classification for all sampled keypoints.
2. Proposal Refinement Loss:
$L_{rcnn} = L_{iou} + \sum_{r \in \{x, y, z, l, h, w, \theta\}} L_{\text{smooth-}L1}\left(\Delta \hat{r}_p, \Delta r_p\right)$

where $L_{iou} = -y_k \log(\tilde{y}_k) - (1 - y_k) \log(1 - \tilde{y}_k)$ is the binary cross-entropy loss against the continuous target $y_k = \min(1, \max(0, 2\,\text{IoU}_k - 0.5))$ , and $\Delta \hat{r}_p$ , $\Delta r_p$ are predicted residuals and regression targets for the proposal bounding box parameters.
Knowl 7 — Experimental Setup and Implementation Details
experimental setup
The PV-RCNN network is evaluated on the KITTI 3D Detection dataset and the Waymo Open Dataset.

Network Hyperparameters:
- 3D Voxel CNN Backbone: 4 levels with feature dimensions 16, 32, 64, 64. The neighboring search radii $r_k$ for the VSA module at each level are $(0.4\,\text{m}, 0.8\,\text{m})$ , $(0.8\,\text{m}, 1.2\,\text{m})$ , $(1.2\,\text{m}, 2.4\,\text{m})$ , and $(2.4\,\text{m}, 4.8\,\text{m})$ . Raw point set abstraction radii are $(0.4\,\text{m}, 0.8\,\text{m})$ .
- Keypoint Sampling: $n = 2048$ keypoints for KITTI and $n = 4096$ keypoints for Waymo, sampled via Furthest Point Sampling (FPS).
- RoI-Grid Pooling: $6 \times 6 \times 6 = 216$ grid points per 3D proposal, with keypoint search radii $\tilde{r} \in (0.8\,\text{m}, 1.6\,\text{m})$ .
Coordinate Ranges and Voxelization:
- KITTI: Detection range is $[0, 70.4]\,\text{m}$ ( $X$ ), $[-40, 40]\,\text{m}$ ( $Y$ ), $[-3, 1]\,\text{m}$ ( $Z$ ) with voxel size $(0.05\,\text{m}, 0.05\,\text{m}, 0.1\,\text{m})$ .
- Waymo: Detection range is $[-75.2, 75.2]\,\text{m}$ ( $X, Y$ ), $[-2, 4]\,\text{m}$ ( $Z$ ) with voxel size $(0.1\,\text{m}, 0.1\,\text{m}, 0.15\,\text{m})$ .
Training and Inference Details:
- Optimization: Trained from scratch using ADAM optimizer with cosine annealing learning rate decay. KITTI training uses batch size 24, learning rate 0.01, 80 epochs on 8 GPUs (~5 hours). Waymo training uses batch size 64, learning rate 0.01, 30 epochs on 32 GPUs.
- Proposal Sampling: 128 sampled proposals per scene with a 1:1 positive-to-negative ratio (positive if 3D $\text{IoU} \ge 0.55$ ).
- Data Augmentation: Random flipping along $X$ -axis, global scaling in $[0.95, 1.05]$ , global rotation around $Z$ -axis in $[-\pi/4, \pi/4]$ , and ground-truth bounding box pasting.
- Inference: Top-100 proposals from RPN after NMS (threshold 0.7) are refined; final predictions are filtered with NMS threshold 0.01. For Waymo multi-class detection, 500 proposals are retained.

Knowl 8 — 3D Object Detection Performance on KITTI Benchmark

data/table

Performance of PV-RCNN on the official KITTI test set evaluated with the 40 recall positions ( $R_{40}$ ) metric across Easy, Moderate, and Hard difficulty levels for Car (IoU threshold 0.7) and Cyclist (IoU threshold 0.5).

Method	Modality	Car 3D mAP (%)			Car BEV mAP (%)			Cyclist 3D mAP (%)			Cyclist BEV mAP (%)
Method	Modality	Easy	Mod.	Hard	Easy	Mod.	Hard	Easy	Mod.	Hard	Easy	Mod.	Hard
MV3D	RGB + LiDAR	74.97	63.63	54.00	86.62	78.93	69.80	-	-	-	-	-	-
ContFuse	RGB + LiDAR	83.68	68.78	61.67	94.07	85.35	75.88	-	-	-	-	-	-
AVOD-FPN	RGB + LiDAR	83.07	71.76	65.73	90.99	84.82	79.62	63.76	50.55	44.93	69.39	57.12	51.09
F-PointNet	RGB + LiDAR	82.19	69.79	60.59	91.17	84.67	74.77	72.27	56.12	49.01	77.26	61.37	53.78
UberATG-MMF	RGB + LiDAR	88.40	77.43	70.22	93.67	88.21	81.99	-	-	-	-	-	-
SECOND	LiDAR only	83.34	72.55	65.82	89.39	83.77	78.59	71.33	52.08	45.83	76.50	56.05	49.45
PointPillars	LiDAR only	82.58	74.31	68.99	90.07	86.56	82.81	77.10	58.65	51.92	79.90	62.73	55.58
PointRCNN	LiDAR only	86.96	75.64	70.70	92.13	87.39	82.72	74.96	58.82	52.53	82.56	67.24	60.28
3D IoU Loss	LiDAR only	86.16	76.50	71.39	91.36	86.22	81.20	-	-	-	-	-	-
Fast Point R-CNN	LiDAR only	85.29	77.40	70.24	90.87	87.84	80.52	-	-	-	-	-	-
STD	LiDAR only	87.95	79.71	75.09	94.74	89.19	86.42	78.69	61.59	55.30	81.36	67.23	59.35
Patches	LiDAR only	88.67	77.20	71.82	92.72	88.39	83.19	-	-	-	-	-	-
Part- $A^2$ -Net	LiDAR only	87.81	78.49	73.51	91.70	87.79	84.61	-	-	-	-	-	-
PV-RCNN (Ours)	LiDAR only	90.25	81.43	76.82	94.98	90.65	86.14	78.60	63.71	57.65	82.49	68.89	62.41
Improvement	-	+1.58	+1.72	+1.73	+0.24	+1.46	-0.28	-0.06	+2.12	+2.35	-0.07	+1.65	+2.13

On the KITTI validation split evaluated with 11 recall positions ( $R_{11}$ ), PV-RCNN achieves $83.90\%$ 3D mAP on Moderate Car, outperforming STD ( $79.80\%$ ), Fast Point R-CNN ( $79.00\%$ ), and PointRCNN ( $78.63\%$ ). On the $R_{40}$ metric, PV-RCNN reaches $92.57\%$ (Easy), $84.83\%$ (Moderate), and $82.69\%$ (Hard) 3D mAP for Car. Furthermore, PV-RCNN achieves $85.5\%$ RPN proposal recall at $\text{IoU}=0.7$ on Moderate Car, outperforming PointRCNN ( $74.8\%$ ) and STD ( $76.8\%$ ).

Knowl 9 — 3D Object Detection Performance on Waymo Open Dataset

data/table

Evaluation of PV-RCNN on the 202 validation sequences of the Waymo Open Dataset for Vehicle detection under version 1.0 (IoU threshold 0.7) and for multi-class detection (Vehicle, Pedestrian, Cyclist) under version 1.2 across LEVEL 1 (objects with $\ge 5$ LiDAR points) and LEVEL 2 (objects with $\ge 1$ LiDAR point).

Vehicle Detection Results on Waymo Open Dataset v1.0:

Difficulty	Method	3D mAP (%)				3D mAPH (%)
Difficulty	Method	Overall	0–30m	30–50m	50m–Inf	Overall	0–30m	30–50m	50m–Inf
LEVEL 1	PointPillar	56.62	81.01	51.75	27.94	-	-	-	-
	MVF	62.93	86.30	60.02	36.02	-	-	-	-
	PV-RCNN	70.30	91.92	69.21	42.17	69.69	91.34	68.53	41.31
	Improvement	+7.37	+5.62	+9.19	+6.15	-	-	-	-
LEVEL 2	PV-RCNN	65.36	91.58	65.13	36.46	64.79	91.00	64.49	35.70

For LEVEL 1 bird's-eye view (BEV) evaluation on Waymo v1.0, PV-RCNN achieves $82.96\%$ overall BEV mAP ( $+2.56\%$ improvement over MVF) and $82.06\%$ BEV mAPH.

Multi-Category Detection Results on Waymo Open Dataset v1.2 (single model predicting 500 proposals):

Method

Vehicle (L1)

Vehicle (L2)

Pedestrian (L1)

Pedestrian (L2)

Cyclist (L1)

Cyclist (L2)

mAP

mAPH

mAP

mAPH

mAP

mAPH

mAP

mAPH

mAP

mAPH

mAP

mAPH

SECOND

72.27

71.69

63.85

63.33

68.70

58.18

60.72

51.31

60.62

59.28

58.34

57.05

PV-RCNN

77.51

76.89

68.98

68.41

75.01

65.65

66.04

57.61

67.81

66.35

65.39

63.98

Knowl 10 — Ablation Analysis of Architectural Components and Features

data/table

Ablation studies on the KITTI validation split for the Car class isolate the impact of the scene encoding framework, Voxel Set Abstraction (VSA) feature components, Predicted Keypoint Weighting (PKW), RoI-grid pooling, and IoU-guided scoring.

Effect of Keypoint Scene Encoding vs. Direct Voxel Pooling:

RPN with 3D Voxel CNN	Keypoints Encoding	RoI-grid Pooling	Easy mAP (%)	Mod. mAP (%)	Hard mAP (%)
✓			90.46	80.87	77.30
✓		(Pool from Encoder)	91.88	82.86	80.52
✓	✓	✓	92.57	84.83	82.69

Directly pooling features from the 3D voxel CNN feature volumes without the intermediate keypoint encoding ("Pool from Encoder") achieves $82.86\%$ Moderate 3D mAP, whereas the keypoint encoding achieves $84.83\%$ with substantially lower GPU memory usage.

Effect of Keypoint Feature Components in Extended VSA:

$f^{(pv1)}$	$f^{(pv2)}$	$f^{(pv3)}$	$f^{(pv4)}$	$f^{(bev)}$	$f^{(raw)}$	Moderate 3D mAP (%)
					✓	81.98
			✓			83.32
		✓				83.17
		✓	✓			84.54
		✓	✓	✓		84.69
✓		✓	✓	✓		84.72
✓	✓	✓	✓	✓		84.75
✓	✓	✓	✓	✓	✓	84.83

Deep features from levels 3 and 4 ( $f^{(pv3)}, f^{(pv4)}$ ) and BEV map features ( $f^{(bev)}$ ) provide the largest performance gains over raw point features alone, while adding shallow features ( $f^{(pv1)}, f^{(pv2)}$ ) and raw point coordinates provides further fine-grained improvements.

Effect of PKW, Pooling Method, and Scoring Strategy:

PKW	RoI Pooling Method	Confidence Prediction	Easy mAP (%)	Mod. mAP (%)	Hard mAP (%)
No	RoI-grid Pooling	IoU-guided scoring	92.09	82.95	81.93
Yes	RoI-aware Pooling	IoU-guided scoring	92.54	82.97	80.30
Yes	RoI-grid Pooling	Classification	91.71	82.50	81.41
Yes	RoI-grid Pooling	IoU-guided scoring	92.57	84.83	82.69

Removing the PKW module drops Moderate mAP by $1.88\%$ . Replacing RoI-grid pooling with RoI-aware pooling drops Moderate mAP by $1.86\%$ . Replacing IoU-guided confidence prediction with standard binary classification targets drops Moderate mAP by $2.33\%$ .

Coverage note — No substantial contributed material was omitted; qualitative visualizations and standard background details on sparse convolution and PointNet were excluded as non-contributed or supplementary.

References

1.Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
2.Yilun Chen, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Fast point r-cnn. In Proceedings of the IEEE international conference on computer vision (ICCV), 2019.
3.Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3075–3084, 2019.
4.Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
5.Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. CVPR, 2018.
6.Benjamin Graham and Laurens van der Maaten. Submanifold sparse convolutional networks. CoRR, abs/1706.01307, 2017.
7.Qiangui Huang, Weiyue Wang, and Ulrich Neumann. Recurrent slice networks for 3d segmentation of point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2626–2635, 2018.
8.Maximilian Jaritz, Jiayuan Gu, and Hao Su. Multi-view pointnet for 3d scene understanding. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 0–0, 2019.
9.Borui Jiang, Ruixuan Luo, Jiayuan Mao, Tete Xiao, and Yuning Jiang. Acquisition of localization confidence for accurate object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pages 784–799, 2018.
10.KITTI leader board of 3D object detection benchmark. http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d, Accessed on 2019-11-15.
11.Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven Waslander. Joint 3d proposal generation and object detection from view aggregation. IROS, 2018.
12.Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. CVPR, 2019.
13.Johannes Lehner, Andreas Mitterecker, Thomas Adler, Markus Hofmarcher, Bernhard Nessler, and Sepp Hochreiter. Patch refinement - localized 3d object detection. CoRR, abs/1910.04093, 2019.
14.Buyu Li, Wanli Ouyang, Lu Sheng, Xingyu Zeng, and Xiaogang Wang. Gs3d: An efficient 3d object detection framework for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1019–1028, 2019.
15.Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointcnn: Convolution on x-transformed points. In Advances in Neural Information Processing Systems, pages 820–830, 2018.
16.Ming Liang*, Bin Yang*, Yun Chen, Rui Hu, and Raquel Urtasun. Multi-task multi-sensor fusion for 3d object detection. In CVPR, 2019.
17.Ming Liang, Bin Yang, Shenlong Wang, and Raquel Urtasun. Deep continuous fusion for multi-sensor 3d object detection. In ECCV, 2018.
18.Tsung-Yi Lin, Priyal Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. ´ IEEE transactions on pattern analysis and machine intelligence, 2018.
19.Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel CNN for efficient 3d deep learning. CoRR, abs/1907.03739, 2019.
20.Jiquan Ngiam, Benjamin Caine, Wei Han, Brandon Yang, Yuning Chai, Pei Sun, Yin Zhou, Xi Yi, Ouais Alsharif, Patrick Nguyen, Zhifeng Chen, Jonathon Shlens, and Vijay Vasudevan. Starnet: Targeted computation for object detection in point clouds. CoRR, abs/1908.11069, 2019.
21.Charles R. Qi, Or Litany, Kaiming He, and Leonidas J. Guibas. Deep hough voting for 3d object detection in point clouds. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
22.Charles R. Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J. Guibas. Frustum pointnets for 3d object detection from rgb-d data. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
23.Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 652–660, 2017.
24.Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems, pages 5099–5108, 2017.
25.Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrcnn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–779, 2019.
26.Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
27.Shuran Song and Jianxiong Xiao. Deep sliding shapes for amodal 3d object detection in rgb-d images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 808–816, 2016.
28.Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, and Jan Kautz. Splatnet: Sparse lattice networks for point cloud processing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2530–2539, 2018.
29.Hugues Thomas, Charles R. Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, Francois Goulette, and Leonidas J. Guibas. Kpconv: Flexible and deformable convolution for point clouds. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
30.Bei Wang, Jianping An, and Jiayan Cao. Voxel-fpn: multiscale voxel feature aggregation in 3d object detection from point clouds. CoRR, abs/1907.05286, 2019.
31.Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (TOG), 38(5):146, 2019.
32.Zhixin Wang and Kui Jia. Frustum convnet: Sliding frustums to aggregate local point-wise features for amodal 3d object detection. In IROS. IEEE, 2019.
33.Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9621–9630, 2019.
34.Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018.
35.Bin Yang, Ming Liang, and Raquel Urtasun. Hdnet: Exploiting hd maps for 3d object detection. In 2nd Conference on Robot Learning (CoRL), 2018.
36.Bin Yang, Wenjie Luo, and Raquel Urtasun. Pixor: Realtime 3d object detection from point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7652–7660, 2018.
37.Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. STD: sparse-to-dense 3d object detector for point cloud. ICCV, 2019.
38.Hengshuang Zhao, Li Jiang, Chi-Wing Fu, and Jiaya Jia. Pointweb: Enhancing local neighborhood features for point cloud processing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5565–5573, 2019.
39.Dingfu Zhou, Jin Fang, Xibin Song, Chenye Guan, Junbo Yin, Yuchao Dai, and Ruigang Yang. Iou loss for 2d/3d object detection. In International Conference on 3D Vision (3DV). IEEE, 2019.
40.Yin Zhou, Pei Sun, Yu Zhang, Dragomir Anguelov, Jiyang Gao, Tom Ouyang, James Guo, Jiquan Ngiam, and Vijay Vasudevan. End-to-end multi-view fusion for 3d object detection in lidar point clouds. CoRR, abs/1910.06528, 2019.
41.Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4490–4499, 2018.
42.Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. CoRR, abs/1908.09492, 2019.

Citation

MLA

Shi, S., et al. “PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection”. arXiv, 2019, http://arxiv.org/abs/1912.13192v2.

APA

Shi, S., Guo, C., Jiang, L., Wang, Z., Shi, J., Wang, X., & Li, H. (2019). PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection. arXiv. http://arxiv.org/abs/1912.13192v2

Chicago

Shi, S., C. Guo, L. Jiang, et al. 2019. “PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection”. arXiv. http://arxiv.org/abs/1912.13192v2.

Harvard

Shi, S. et al. (2019) “PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1912.13192v2.

Vancouver

1. Shi S, Guo C, Jiang L, Wang Z, Shi J, Wang X, Li H (2019) PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection. arXiv

BibTeX

@article{shi2019rcnn,
  title = {PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection},
  author = {Shi, Shaoshuai and Guo, Chaoxu and Jiang, Li and Wang, Zhe and Shi, Jianping and Wang, Xiaogang and Li, Hongsheng},
  year = {2019},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1912.13192v2},
  eprint = {1912.13192}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE

PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — PV-RCNN Framework Architecture

Knowl 2 — Voxel Set Abstraction and Extended Keypoint Feature Encoding

Knowl 3 — Predicted Keypoint Weighting Module

Knowl 4 — RoI-Grid Pooling via Keypoint Set Abstraction

Knowl 5 — 3D Proposal Refinement and IoU-Guided Confidence Prediction

Knowl 6 — Multi-Task Training Loss Formulation

Knowl 7 — Experimental Setup and Implementation Details

Knowl 8 — 3D Object Detection Performance on KITTI Benchmark

Knowl 9 — 3D Object Detection Performance on Waymo Open Dataset

Knowl 10 — Ablation Analysis of Architectural Components and Features

References

Citation

BibTeX

Source Code

Access the Paper