Uni3D: A Unified Baseline for Multi-Dataset 3D Object Detection

Bo ZhangJiakang YuanBotian ShiTao ChenYikang LiYu Qiao

article2023CVPR53 citations

Presents Uni3D, a unified framework that overcomes sensor discrepancies and taxonomy variations across diverse LiDAR datasets to train multi-dataset 3D object detectors that outperform models trained on single datasets with minimal parameter overhead.

Fudan University Shanghai Artificial Intelligence Laboratory 3d object 3d objects detection algorithm

arXiv: 2303.06880 PJLab-ADG/3DTrans

Cover for Uni3D: A Unified Baseline for Multi-Dataset 3D Object Detection

Abstract

Current 3D object detection models follow a single dataset-specific training and testing paradigm, which often faces a serious detection accuracy drop when they are directly deployed in another dataset. In this paper, we study the task of training a unified 3D detector from multiple datasets. We observe that this appears to be a challenging task, which is mainly due to that these datasets present substantial data-level differences and taxonomy-level variations caused by different LiDAR types and data acquisition standards. Inspired by such observation, we present a Uni3D which leverages a simple data-level correction operation and a designed semantic-level coupling-and-recoupling module to alleviate the unavoidable data-level and taxonomy-level differences, respectively. Our method is simple and easily combined with many 3D object detection baselines such as PV-RCNN and Voxel-RCNN, enabling them to effectively learn from multiple off-the-shelf 3D datasets to obtain more discriminative and generalizable representations. Experiments are conducted on many dataset consolidation settings. Their results demonstrate that Uni3D exceeds a series of individual detectors trained on a single dataset, with a 1.04× parameter increase over a selected baseline detector. We expect this work will inspire the research of 3D generalization since it will push the limits of perceptual performance. Our code is available at: https://github.com/PJLab-ADG/3DTrans.

Topics

3d object 3d objects detection algorithm detection rates object detection semantics taxonomy lidar systems

1. Introduction
2. Related Works
2.1. LiDAR-based General 3D Object Detection
2.2. Joint Training on Multiple Datasets
3. The Proposed Method
3.1. Preliminary
3.2. When Single-dataset 3D Detectors Meet Multi-ple Datasets
3.3. Uni3D: A Unified 3D Multi-dataset Object Detection Baseline
4. Experiments
4.1. Experimental Setup.
4.2. Design of Comparison Baselines
4.3. Results of Multi-Dataset 3D Object Detection
4.4. Further Analyses
5. Conclusion
Acknowledgement
References

Knowls

Knowl 1 — Uni3D Framework for Multi-Dataset 3D Object Detection
model/method
Uni3D is a unified multi-domain fusion framework designed to train a single 3D LiDAR object detector on multiple distinct datasets simultaneously. It addresses two primary cross-dataset interference sources: data-level differences (inconsistent point cloud ranges, sensor beam counts, and sensor mounting heights) and taxonomy-level differences (conflicting class names and annotation granularities across autonomous driving datasets).

The framework operates across four interconnected processing stages:
1. Data-Level Spatial Pre-Processing: Normalizes raw input point clouds across datasets to a unified bounding range and applies coordinate-origin alignment (C.A.) along the vertical ( $Z$ ) axis to reconcile sensor height discrepancies.
2. Statistics-Level Alignment (S.A.) in Shared Backbones: Replaces standard batch normalization layers in shared 3D sparse convolutional and 2D Bird-Eye-View (BEV) backbones with dataset-specific standardization followed by shared affine transformation parameters.
3. Semantic-Level Feature Coupling-and-Recoupling (C.R.) Module: Fuses multi-dataset BEV feature maps using foreground-aware spatial attention and dataset-level attention masks, then recalibrates and redistributes the shared representations back to dataset-specific streams via Squeeze-and-Excitation networks.
4. Dataset-Specific Detection Heads: Attaches dedicated task heads to output predictions conforming to each individual dataset's class taxonomy while sharing the upstream feature extraction backbones.
Uni3D is detector-agnostic and can be integrated into standard 3D LiDAR detectors (such as Voxel-RCNN and PV-RCNN) with approximately a $1.04\times$ parameter increase.
Knowl 2 — Statistics-Level Alignment for Multi-Dataset 3D Backbone Normalization
model/method
In multi-dataset 3D object detection training, standard shared batch normalization fails because mini-batches containing point clouds from different LiDAR configurations present disparate channel-wise feature distributions. Statistics-Level Alignment (S.A.) decouples distribution normalization across datasets while retaining shared parameter capacity across domains.

Let $x_j^t$ denote the input feature activation for channel $j$ at a specific network layer originating from dataset $t \in \{1, \dots, N\}$ . S.A. first standardizes the activation using dataset-specific mean $\mu_j^t$ and variance $(\sigma_j^t)^2$ :

$\hat{x}_j^t = \frac{x_j^t - \mu_j^t}{\sqrt{(\sigma_j^t)^2 + \xi}}$

where $\xi > 0$ is a small constant ensuring numerical stability.

To restore expressive representational capacity while enforcing inter-dataset feature space alignment, shared affine transformation parameters $\gamma_j$ (scale) and $\beta_j$ (shift) are applied to the zero-mean, unit-variance standardized activations across all datasets:

$\hat{y}_j^t = \gamma_j \hat{x}_j^t + \beta_j$

Because first- and second-order statistics are aligned prior to the affine transform, domain-invariant features are projected into a common representation space in both 3D and 2D shared backbones.
Knowl 3 — Semantic-Level Feature Coupling-and-Recoupling Module
model/method
The Semantic-Level Feature Coupling-and-Recoupling (C.R.) module enables knowledge sharing and feature reusability across disparate datasets at the Bird-Eye-View (BEV) feature level.

Let $f_{bev}^i, f_{bev}^j \in \mathbb{R}^{C \times H \times W}$ denote BEV feature maps extracted from the 2D backbone for samples from dataset $i$ and dataset $j$ , where $C$ is the channel dimension and $H, W$ are spatial height and width.

Feature Coupling: Features from $N$ datasets are concatenated along the channel dimension:

$f_{cat}^{bev} = [f_{bev}^1, f_{bev}^2, \dots, f_{bev}^N] \in \mathbb{R}^{NC \times H \times W}$

A shared BEV representation $f_{shared}^{bev}$ is extracted by combining foreground-aware spatial attention and dataset-level attention masks:

$f_{shared}^{bev} = \left[ M_{shared} \odot \Phi_d(\text{Conv}(f_{cat}^{bev})) \right] f_{cat}^{bev}$

where $\odot$ denotes element-wise multiplication. The foreground-aware spatial attention map $M_{shared} \in \mathbb{R}^{1 \times H \times W}$ is computed via channel-wise maximum pooling: $M_{shared} = \phi_p(f_{cat}^{bev})$ . The dataset-level attention mask $\Phi_d(\cdot)$ applies a Multi-Layer Perceptron (MLP) followed by an $N$ -class channel-wise softmax to re-scale the concatenated features into dataset-agnostic representations.

Feature Recoupling: The shared representation $f_{shared}^{bev}$ is fused back into dataset-specific feature streams via residual Squeeze-and-Excitation ( $SE$ ) blocks:

$\hat{f}_{bev}^k = SE_k(f_{shared}^{bev}) + f_{bev}^k \quad \text{for } k \in \{1, \dots, N\}$

where $SE_k$ recalibrates channel-wise dependencies specifically for the $k$ -th dataset before feeding into the detection head.
Knowl 4 — BEV Feature Copy Inference Strategy for Multi-Dataset 3D Detectors
model/method
During standard single-dataset deployment or evaluation, test point clouds arrive from only a single target sensor/dataset at a time. This introduces a structural mismatch with the multi-branch Semantic-Level Feature Coupling-and-Recoupling (C.R.) module, which expects concurrent inputs from $N$ datasets during training.

To eliminate runtime dependencies on real data from auxiliary datasets, the framework uses a BEV feature copy inference strategy:
1. Given a single input frame from dataset $i$ , its extracted BEV feature map $f_{bev}^i$ is duplicated across all $N$ input channels of the coupling module, setting $f_{bev}^k = f_{bev}^i$ for all $k \in \{1, \dots, N\}$ .
2. The coupling-and-recoupling module processes these duplicated features to produce recalibrated feature map $\hat{f}_{bev}^i$ .
3. $\hat{f}_{bev}^i$ is routed directly to dataset-specific detection head $H_i$ to generate 3D bounding box predictions.
This mechanism allows the unified detector to perform standalone single-frame inference on any target domain without requiring paired or synthetic auxiliary frames at test time.
Knowl 5 — Dataset-Specific Detection Heads and Multi-Domain Fusion Objective
model/method
To handle taxonomy discrepancies across autonomous driving benchmarks (such as Waymo's unified Vehicle class versus nuScenes' separate Car, Truck, Van, and Bus classes), Uni3D employs separate, dataset-specific detection heads $H_k$ for each domain $k \in \{1, \dots, N\}$ .

Each head $H_k$ receives the recoupled feature map $\hat{f}_{bev}^k$ and produces class scores and 3D bounding box regressions tailored to domain $k$ 's taxonomy.

The overall training loss $\mathcal{L}_{det}^{overall}$ is formulated as the sum of dataset-specific detection losses:

$\mathcal{L}_{det}^{overall} = \sum_{k=1}^N \mathcal{L}_{det}^k(H_k(\hat{f}_{bev}^k))$

where $\mathcal{L}_{det}^k$ denotes the standard multi-task detection loss for dataset $k$ , comprising region proposal network (RPN) classification/regression loss, region-of-interest (RoI) refinement loss, and keypoint auxiliary prediction loss when using keypoint-based backbones (e.g., PV-RCNN).
Knowl 6 — Impact of Point Range Discrepancies and Sensor Height Variations on Multi-Dataset 3D Object Detection
empirical result
LiDAR sensors across autonomous driving datasets have substantially different point cloud ranges, vertical fields of view (VFOV), and sensor mounting heights. Directly training a single detector on merged raw data leads to severe feature interference and performance degradation.

Key dataset differences:
- Waymo: 64 beams, VFOV $[-18.0^\circ, 2.0^\circ]$ , default range $X, Y \in [-75.2, 75.2]\,\text{m}, Z \in [-2.0, 4.0]\,\text{m}$ .
- KITTI: 64 beams, VFOV $[-23.6^\circ, 3.2^\circ]$ , default range $X \in [0.0, 70.4]\,\text{m}, Y \in [-40.0, 40.0]\,\text{m}, Z \in [-3.0, 1.0]\,\text{m}$ .
- nuScenes: 32 beams, VFOV $[-30.0^\circ, 10.0^\circ]$ , default range $X, Y \in [-51.2, 51.2]\,\text{m}, Z \in [-5.0, 3.0]\,\text{m}$ .
When Voxel-RCNN is trained jointly on multi-dataset pairs without point range alignment, detection accuracy drops drastically:
- Joint training on Waymo + KITTI without range alignment yields Waymo Vehicle $\text{AP}_{3\text{D}} / \text{APH}_{3\text{D}}$ of $26.93 / 26.56\%$ . Aligning point ranges to $[-75.2, 75.2]\,\text{m}$ for $X, Y$ and $[-2.0, 4.0]\,\text{m}$ for $Z$ restores performance to $74.83 / 74.33\%$ .
- Joint training on nuScenes + KITTI without range alignment yields nuScenes Car $\text{AP}_{\text{BEV}} / \text{AP}_{3\text{D}}$ of $21.32 / 15.35\%$ . Aligning point ranges improves performance to $59.25 / 41.51\%$ .
In addition, coordinate-origin alignment (shifting the vertical origin by $+1.6\,\text{m}$ for KITTI and $+1.8\,\text{m}$ for nuScenes) aligns ground heights across platforms, further mitigating cross-dataset sensor placement differences.

Knowl 7 — Multi-Dataset Joint Training Performance on Waymo and nuScenes Benchmarks

data/table

Joint training experiments on Waymo and nuScenes demonstrate that naive direct merging (D.M.) causes severe degradation compared to models trained solely on individual datasets, whereas Uni3D achieves competitive or superior performance on both datasets simultaneously.

Evaluation on Waymo uses LEVEL_1 3D Average Precision (AP) and Average Precision re-weighted by Heading (APH) at IoU thresholds of 0.7 for Vehicle and 0.5 for Pedestrian/Cyclist. Evaluation on nuScenes uses $\text{AP}_{\text{BEV}}$ and $\text{AP}_{3\text{D}}$ over 40 recall positions.

Method	Tested on Waymo (AP/APH)			Tested on nuScenes ( $\text{AP}_{\text{BEV}}$ / $\text{AP}_{3\text{D}}$ )
Method	Vehicle	Pedestrian	Cyclist	Car	Pedestrian	Cyclist
Voxel-RCNN Backbone
Only Waymo	75.08 / 74.60	75.17 / 68.76	65.28 / 64.33	34.10 / 17.31	2.99 / 1.69	0.05 / 0.01
Only nuScenes	36.77 / 36.50	4.64 / 3.18	2.49 / 2.45	53.63 / 39.05	22.47 / 17.85	10.86 / 9.70
Direct Merging (D.M.)	66.67 / 66.23	60.36 / 54.08	52.03 / 51.25	51.40 / 31.68	15.04 / 9.99	5.40 / 3.87
+ C.A.	69.40 / 68.86	63.43 / 56.49	52.83 / 51.93	51.39 / 29.04	16.24 / 10.96	4.55 / 3.13
+ C.A. + S.A.	75.16 / 74.67	74.83 / 68.07	64.68 / 63.73	58.41 / 40.84	26.52 / 20.98	9.19 / 7.65
+ C.R.	74.56 / 74.05	74.29 / 67.04	63.14 / 62.21	59.10 / 42.25	29.86 / 23.76	14.46 / 12.73
Uni3D (C.A.+S.A.+C.R.)	75.26 / 74.77	75.46 / 68.75	65.02 / 64.12	60.18 / 42.23	30.08 / 24.37	14.60 / 12.32
PV-RCNN Backbone
Only Waymo	74.97 / 74.46	73.41 / 66.57	64.58 / 63.49	32.99 / 17.55	3.34 / 1.94	0.02 / 0.01
Only nuScenes	41.01 / 40.58	4.57 / 2.96	0.98 / 0.95	57.78 / 41.10	24.52 / 18.56	10.24 / 8.25
Direct Merging (D.M.)	66.22 / 65.75	55.41 / 49.29	56.50 / 55.48	48.67 / 30.43	12.66 / 8.12	1.67 / 1.04
Uni3D (C.A.+S.A.+C.R.)	75.54 / 74.90	74.12 / 66.90	63.28 / 62.12	60.77 / 42.66	27.44 / 21.85	13.50 / 11.87

Uni3D matches or outperforms individual single-dataset models across categories, improving nuScenes Car detection to $60.18 / 42.23\%$ (Voxel-RCNN) and $60.77 / 42.66\%$ (PV-RCNN) while preserving top-tier Waymo Vehicle performance ( $75.26 / 74.77\%$ and $75.54 / 74.90\%$ ).

Knowl 8 — Multi-Dataset Consolidation Performance Across KITTI, nuScenes, and Waymo

data/table

Uni3D consolidates models across pair-wise combinations (KITTI + nuScenes, KITTI + Waymo) and full three-dataset joint training (Waymo + KITTI + nuScenes). Results on KITTI report moderate-case Car $\text{AP}_{\text{BEV}} / \text{AP}_{3\text{D}}$ at IoU 0.7, and Pedestrian/Cyclist at IoU 0.5.

Consolidation	Method	Tested on KITTI ( $\text{AP}_{\text{BEV}}$ / $\text{AP}_{3\text{D}}$ )			Tested on nuScenes ( $\text{AP}_{\text{BEV}}$ / $\text{AP}_{3\text{D}}$ )
Consolidation	Method	Car	Pedestrian	Cyclist	Car	Pedestrian	Cyclist
KITTI+nuScenes	Voxel-RCNN (Single)	89.34 / 80.91	59.67 / 56.88	61.10 / 60.49	53.63 / 39.05	22.47 / 17.85	10.86 / 9.70
	Voxel-RCNN (Uni3D)	90.09 / 83.10	62.99 / 58.30	70.20 / 68.10	59.25 / 41.51	29.12 / 23.18	15.16 / 13.16
	PV-RCNN (Single)	89.41 / 83.15	59.09 / 54.73	62.25 / 61.71	57.78 / 41.10	24.52 / 18.56	10.24 / 8.25
	PV-RCNN (Uni3D)	89.77 / 85.49	60.03 / 55.58	69.03 / 66.10	59.08 / 41.67	25.27 / 19.26	12.26 / 10.83
KITTI + Waymo Consolidation (Voxel-RCNN Backbone)
KITTI+Waymo	Single Baseline	89.34 / 80.91	59.67 / 56.88	61.10 / 60.49	74.35 / 73.85	74.80 / 68.39	64.87 / 63.95
KITTI+Waymo	Uni3D	90.03 / 82.39	62.51 / 57.01	69.52 / 66.30	74.83 / 74.34	74.79 / 68.24	66.83 / 65.82

For three-dataset joint consolidation (KITTI (K) + nuScenes (N) + Waymo (W)) using Voxel-RCNN evaluated on Car/Vehicle 3D AP:

Single detector on KITTI: $80.91\%$ ; Single on nuScenes: $39.05\%$ ; Single on Waymo: $36.50\%$ (average $36.34\%$ across all three).
Unified model trained with Uni3D on K+N+W achieves $83.41\%$ on KITTI, $42.30\%$ on nuScenes, and $74.97\%$ on Waymo, reaching an average Car/Vehicle 3D AP of $66.89\%$ across all three benchmarks simultaneously.

Knowl 9 — Few-Shot Data Efficiency and Training Scalability of Uni3D
empirical result
When expanding 3D object detectors to new sensor domains where annotated data is scarce, Uni3D enables high data efficiency by joint training on the full source dataset (KITTI) alongside low-data fractions (100%, 10%, 5%, 1%) of the target dataset (nuScenes).

Key results using Voxel-RCNN on nuScenes validation:
- Single detector trained on 10% nuScenes: Car $\text{AP}_{\text{BEV}} / \text{AP}_{3\text{D}} = 45.42 / 31.09\%$ , Pedestrian $= 10.39 / 7.16\%$ , Cyclist $= 1.55 / 0.89\%$ .
- Uni3D trained on full KITTI + 10% nuScenes: Car $\text{AP}_{\text{BEV}} / \text{AP}_{3\text{D}} = 52.08 / 34.40\%$ , Pedestrian $= 20.40 / 15.60\%$ , Cyclist $= 8.42 / 7.40\%$ .
- Single detector trained on 1% nuScenes: Collapses to $0.00 / 0.00\%$ across all classes due to extreme overfitting.
- Uni3D trained on full KITTI + 1% nuScenes: Retains Car $\text{AP}_{\text{BEV}} / \text{AP}_{3\text{D}} = 44.74 / 28.28\%$ , Pedestrian $= 15.94 / 11.11\%$ , and Cyclist $= 1.28 / 0.99\%$ .
Similar improvements hold for PV-RCNN (e.g., reaching $41.09 / 25.38\%$ Car AP on 1% nuScenes data). This demonstrates that Uni3D effectively transfers domain-agnostic geometric knowledge to prevent severe over-fitting under limited target supervision.
Knowl 10 — Zero-Shot Cross-Dataset Transfer and Unsupervised Domain Adaptation with Uni3D
empirical result
Pre-training 3D detectors across multiple datasets using Uni3D substantially enhances zero-shot generalizability on unseen target domains and improves downstream Unsupervised Domain Adaptation (UDA).

Evaluated on KITTI Car detection ( $ext{AP}_{\text{BEV}} / \text{AP}_{3\text{D}}$ at IoU 0.7):
- Zero-Shot Direct Transfer (source-only models evaluated directly on KITTI without fine-tuning):
  - PV-RCNN trained solely on Waymo: $61.18 / 22.01\%$
  - PV-RCNN trained solely on nuScenes: $68.15 / 37.17\%$
  - PV-RCNN trained jointly on Waymo + nuScenes via Uni3D: $73.51 / 39.71\%$
- Unsupervised Domain Adaptation (ST3D):
  - ST3D adapting from single-source Waymo to KITTI: $86.65 / 76.86\%$
  - ST3D adapting from single-source nuScenes to KITTI: $84.29 / 72.94\%$
  - ST3D adapting from the Uni3D multi-source pre-trained model (Waymo + nuScenes) to KITTI: $\mathbf{88.25} / \mathbf{77.01}\%$
Multi-dataset unified pre-training produces feature representations that are more robust to domain variations, leading to higher baseline transfer and superior pseudo-label self-training convergence in downstream target domains.
Knowl 11 — Sensitivity of Class-Shared Coordinate-Origin Alignment to Rare Categories
limitation
In multi-dataset consolidation, Coordinate-Origin Alignment (C.A.) shifts the vertical origin $Z$ by a fixed, global parameter per dataset (e.g., $+1.6\,\text{m}$ for KITTI and $+1.8\,\text{m}$ for nuScenes) to align sensor heights with Waymo. While effective for vehicles, sharing a single coordinate-origin shift parameter across all classes can adversely affect detection accuracy for low-frequency and small-scale categories such as Pedestrian and Cyclist.

In KITTI + nuScenes consolidation with Voxel-RCNN, applying raw coordinate-origin alignment without statistics-level alignment causes detection performance on KITTI Pedestrian and Cyclist to drop compared to training without origin alignment. This occurs because global vertical shifts distort the height distribution of minority classes that have limited training samples, indicating that uniform shift parameters are suboptimal across heterogeneous object sizes.

Coverage note — None was omitted. All contributed components of the Uni3D framework, normalization mechanisms, inference procedures, empirical multi-dataset evaluation results (2-dataset and 3-dataset consolidations), few-shot/zero-shot analyses, and identified limitations have been faithfully converted into knowls.

References

1.Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020. 1, 2, 3, 4, 5, 7
2.Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1907–1915, 2017. 1, 2
3.Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang. Dynamic head: Unifying object detection heads with attentions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7373–7382, 2021. 2
4.Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1201–1209, 2021. 1, 2, 3, 4, 5, 6, 7, 8
5.Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3354–3361. IEEE, 2012. 1, 2, 3, 5, 7
6.Rui Gong, Dengxin Dai, Yuhua Chen, Wen Li, and Luc Van Gool. mdalu: Multi-source domain adaptation and label unification with partial datasets. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8876–8885, 2021. 2
7.Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018. 5
8.Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448–456. PMLR, 2015. 4
9.John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun. Mseg: A composite dataset for multi-domain semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2879–2888, 2020. 2
10.Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12697–12705, 2019. 1, 2, 3
11.Zhipeng Luo, Zhongang Cai, Changqing Zhou, Gongjie Zhang, Haiyu Zhao, Shuai Yi, Shijian Lu, Hongsheng Li, Shanghang Zhang, and Ziwei Liu. Unsupervised domain adaptive 3d detection with multi-level consistency. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8866–8875, 2021. 1
12.Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas. Frustum pointnets for 3d object detection from rgb-d data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 918–927, 2018. 2
13.Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017. 2
14.Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017. 2
15.Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015. 2
16.Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-rcnn: Pointvoxel feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10529–10538, 2020. 1, 2, 3, 4, 5, 6, 7, 8
17.Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-rcnn++: Point-voxel feature set abstraction with local vector representation for 3d object detection. arXiv preprint arXiv:2102.00463, 2021. 1, 3
18.Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrcnn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 770–779, 2019. 2
19.Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. IEEE transactions on pattern analysis and machine intelligence, 43(8):2647–2664, 2020. 2, 3
20.Vishwanath A Sindagi, Yin Zhou, and Oncel Tuzel. Mvxnet: Multimodal voxelnet for 3d object detection. In 2019 International Conference on Robotics and Automation (ICRA), pages 7276–7282. IEEE, 2019. 2
21.Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2446–2454, 2020. 1, 2, 3, 4, 5, 7
22.OpenPCDet Development Team. Openpcdet: An open-source toolbox for 3d object detection from point clouds. https://github.com/open-mmlab/OpenPCDet, 2020. 5
23.Xudong Wang, Zhaowei Cai, Dashan Gao, and Nuno Vasconcelos. Towards universal object detection by domain attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7289–7298, 2019. 2
24.Yi Wei, Zibu Wei, Yongming Rao, Jiaxin Li, Jie Zhou, and Jiwen Lu. Lidar distillation: Bridging the beam-induced domain gap for 3d object detection. arXiv preprint arXiv:2203.14956, 2022. 1, 7
25.Qiangeng Xu, Yin Zhou, Weiyue Wang, Charles R Qi, and Dragomir Anguelov. Spg: Unsupervised domain adaptation for 3d object detection via semantic point generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15446–15456, 2021. 1
26.Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018. 2, 3
27.Bin Yang, Wenjie Luo, and Raquel Urtasun. Pixor: Realtime 3d object detection from point clouds. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7652–7660, 2018. 2
28.Jihan Yang, Shaoshuai Shi, Zhe Wang, Hongsheng Li, and Xiaojuan Qi. St3d: Self-training for unsupervised domain adaptation on 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10368–10378, 2021. 1, 3, 6, 7, 8
29.Jihan Yang, Shaoshuai Shi, Zhe Wang, Hongsheng Li, and Xiaojuan Qi. St3d++: Denoised self-training for unsupervised domain adaptation on 3d object detection. IEEE Transactions on Pattern Analysis & Machine Intelligence, (01):1–17, 2022. 1, 3, 6, 7
30.Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Std: Sparse-to-dense 3d object detector for point cloud. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1951–1960, 2019. 2
31.Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11784–11793, 2021. 1
32.Yifan Zhang, Qingyong Hu, Guoquan Xu, Yanxin Ma, Jianwei Wan, and Yulan Guo. Not all points are equal: Learning highly efficient point-based detectors for 3d lidar point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18953–18962, 2022. 2
33.Xiangyun Zhao, Samuel Schulter, Gaurav Sharma, Yi-Hsuan Tsai, Manmohan Chandraker, and Ying Wu. Object detection with a unified label space from multiple datasets. In European Conference on Computer Vision, pages 178–193. Springer, 2020. 2
34.Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6881–6890, 2021. 2
35.Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi-dataset detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7571–7580, 2022. 2
36.Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4490–4499, 2018. 1, 2
37.Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020. 2

Citation

MLA

Zhang, B., et al. “Uni3D: A Unified Baseline for Multi-dataset 3D Object Detection”. arXiv, 2023, http://arxiv.org/abs/2303.06880v2.

APA

Zhang, B., Yuan, J., Shi, B., Chen, T., Li, Y., & Qiao, Y. (2023). Uni3D: A Unified Baseline for Multi-dataset 3D Object Detection. arXiv. http://arxiv.org/abs/2303.06880v2

Chicago

Zhang, B., J. Yuan, B. Shi, T. Chen, Y. Li, and Y. Qiao. 2023. “Uni3D: A Unified Baseline for Multi-dataset 3D Object Detection”. arXiv. http://arxiv.org/abs/2303.06880v2.

Harvard

Zhang, B. et al. (2023) “Uni3D: A Unified Baseline for Multi-dataset 3D Object Detection”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2303.06880v2.

Vancouver

1. Zhang B, Yuan J, Shi B, Chen T, Li Y, Qiao Y (2023) Uni3D: A Unified Baseline for Multi-dataset 3D Object Detection. arXiv

BibTeX

@article{zhang2023uni3d,
  title = {Uni3D: A Unified Baseline for Multi-dataset 3D Object Detection},
  author = {Zhang, Bo and Yuan, Jiakang and Shi, Botian and Chen, Tao and Li, Yikang and Qiao, Yu},
  year = {2023},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2303.06880v2},
  eprint = {2303.06880}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE

Uni3D: A Unified Baseline for Multi-Dataset 3D Object Detection

Abstract

Topics

Table of Contents

Knowls

Knowl 1 — Uni3D Framework for Multi-Dataset 3D Object Detection

Knowl 2 — Statistics-Level Alignment for Multi-Dataset 3D Backbone Normalization

Knowl 3 — Semantic-Level Feature Coupling-and-Recoupling Module

Knowl 4 — BEV Feature Copy Inference Strategy for Multi-Dataset 3D Detectors

Knowl 5 — Dataset-Specific Detection Heads and Multi-Domain Fusion Objective

Knowl 6 — Impact of Point Range Discrepancies and Sensor Height Variations on Multi-Dataset 3D Object Detection