Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D

article2020ECCV1,758 citations

Proposes an end-to-end architecture that lifts multi-view camera images into 3D frustums and splats them into a bird's-eye-view grid, establishing a foundational approach for multi-camera 3D perception and motion planning without relying on lidar.

Listen

Autonomous driving systems rely on converting visual inputs from multiple surrounding cameras into a unified, top-down coordinate frame—known as a bird's-eye view—to safely plan vehicle trajectories. Traditional methods process camera feeds separately and merge detections post hoc, which prevents end-to-end learning across sensors and limits system-wide optimization. Developing a unified architecture that learns to fuse multi-camera data directly into a planning-ready representation without relying on expensive depth hardware is a critical step toward safer, more scalable autonomous navigation.

The article demonstrates an end-to-end deep learning framework, named Lift-Splat-Shoot, that maps 2D images from an arbitrary camera configuration directly into a 3D bird's-eye-view grid for semantic scene understanding and motion planning. The core objective is to evaluate whether predicting per-pixel depth distributions across multi-camera rigs enables accurate environmental mapping, sensor-fusion learning, and interpretable trajectory selection entirely without active depth sensors like lidar.

The evaluated method operates in three distinct stages. First, each camera image is "lifted" into a 3D feature representation by estimating discrete depth probability distributions at each pixel. Second, these 3D feature frustums are "splatted" onto a common ground-plane grid using an optimized pooling technique that eliminates spatial padding and speeds up training by a factor of two. Finally, candidate trajectories are evaluated or "shot" across the generated top-down cost map to identify the optimal path. The authors validated the architecture using large-scale autonomous driving benchmarks, specifically the nuScenes and Lyft Level 5 datasets, encompassing thousands of multi-camera urban driving scenes.

Evaluation against competitive camera-based baselines revealed several key findings. First, the proposed framework outperformed all camera-based alternatives across all benchmark tasks; for instance, car segmentation on the nuScenes benchmark achieved an intersection-over-union score of 32.06%, compared to 22.78% for standard networks and 29.72% for orthographic feature methods. Second, the architecture showed strong zero-shot transfer capability, demonstrating higher performance when evaluating on entirely unseen camera configurations, including transferring directly from nuScenes to Lyft datasets without model retraining. Third, intentional camera dropout during training increased real-world fault tolerance, enabling the network to maintain robust performance even when physical cameras failed or suffered from calibration errors. Fourth, in motion planning evaluations, the camera-only approach generated sensible behaviors, such as slowing down at crosswalks and following road contours, achieving a 15.52% top-5 trajectory matching accuracy despite operating without explicit speed inputs.

These findings indicate that end-to-end multi-view representation learning provides substantial architectural advantages over disconnected single-camera pipelines. By implicitly unprojecting images into 3D, the model allows downstream vehicle motion planners to train directly against raw camera feeds. This unified approach reduces sensor-rig rigidity, mitigates risks associated with camera calibration drift, and allows developers to utilize cost maps learned from human driving logs, which are significantly cheaper to scale than manually annotated 3D bounding boxes.

Organizations developing autonomous perception and planning stacks should consider adopting differentiable multi-view bird's-eye-view architectures to reduce dependency on rigid sensor geometry and high-cost active sensors. Teams should implement sensor dropout and extrinsic calibration perturbation during training to bake fault tolerance directly into perception models. Furthermore, trajectory selection frameworks can leverage learned cost maps to streamline planning validation while preserving system interpretability.

While the model sets a new standard for camera-only bird's-eye-view segmentation, some limitations remain. The camera-only model still trails behind lidar-based oracle systems, which achieved a 40.26% car segmentation score on nuScenes and higher motion-planning accuracy, particularly in low-light conditions and at longer distances. Because the current implementation analyzes individual static time steps, extending the model to incorporate multi-frame video sequences over time is the recommended next step to close the performance gap with lidar.

NVIDIA University of Toronto Vector Institute object segmentation self-driving auto semantics Lift-Splat architecture arbitrary camera rigs monocular depth prediction

arXiv: 2008.05711 nv-tlabs/lift-splat-shoot

Paper: Multi-view 3D Object Detection Network for Autonomous Driving, Xiaozhi Chen et al. (2017). MV3D pioneered multi-view 3D object detection fusing bird's-eye-view geometric representations with camera images, establishing the core paradigm of BEV-based autonomous perception that Lift, Splat, Shoot builds upon.
Paper: Frustum PointNets for 3D Object Detection from RGB-D Data, Charles R. Qi et al. (2018). Frustum PointNets introduced the concept of projecting 2D camera detections into 3D viewing frustums to bridge 2D visual information with 3D space, which directly motivates the "lift" stage of the frustum-based architecture.
Paper: Depth Map Prediction from a Single Image using a Multi-Scale Deep Network, David Eigen et al. (2014). This foundational work establishes depth distribution prediction from monocular images, which provides the underlying mechanics for estimating per-pixel categorical depth distributions in the lift operation.
Paper: 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction, Christopher B. Choy et al. (2016). 3D-R2N2 established early mechanisms for mapping multi-view 2D images directly into unified 3D voxel representations without explicit 3D correspondence matching.
Paper: Fully convolutional networks for semantic segmentation, Jonathan Long et al. (2015). Fully Convolutional Networks introduced end-to-end dense pixel prediction architectures that form the backbone image feature extractors and downstream rasterized BEV segmentation heads.

Paper: BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers, Zhiqi Li et al. (2022). BEVFormer extends camera-based BEV perception from the geometry-guided lift-splat paradigm to spatiotemporal transformer attention across multiple cameras and temporal frames.
Paper: VGGT: Visual Geometry Grounded Transformer, Jianyuan Wang et al. (2025). VGGT generalizes multi-camera 3D visual geometry by learning direct, feed-forward 3D reconstructions and camera parameterizations across multi-view image sets using foundational transformer architectures.

Cover for Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D

Abstract

The goal of perception for autonomous vehicles is to extract semantic representations from multiple sensors and fuse these representations into a single "bird's-eye-view" coordinate frame for consumption by motion planning. We propose a new end-to-end architecture that directly extracts a bird's-eye-view representation of a scene given image data from an arbitrary number of cameras. The core idea behind our approach is to "lift" each image individually into a frustum of features for each camera, then "splat" all frustums into a rasterized bird's-eye-view grid. By training on the entire camera rig, we provide evidence that our model is able to learn not only how to represent images but how to fuse predictions from all cameras into a single cohesive representation of the scene while being robust to calibration error. On standard bird's-eye-view tasks such as object segmentation and map segmentation, our model outperforms all baselines and prior work. In pursuit of the goal of learning dense representations for motion planning, we show that the representations inferred by our model enable interpretable end-to-end motion planning by "shooting" template trajectories into a bird's-eye-view cost map output by our network. We benchmark our approach against models that use oracle depth from lidar. Project page with code: this https URL .

Topics

object segmentation self-driving auto semantics sensors viewpoint stereo cameras

Keywords

Lift-Splat architecture arbitrary camera rigs monocular depth prediction BEV frame Orthographic Feature Transform Pyramid Occupancy Networks autonomous driving sensor fusion

1 Introduction
2 Related Work
2.1 Monocular Object Detection
2.2 Inference in the Bird’s-Eye-View Frame
3 Method
3.1 Lift: Latent Depth Distribution
3.2 Splat: Pillar Pooling
3.3 Shoot: Motion Planning
4 Implementation
4.1 Architecture Details
4.2 Frustum Pooling Cumulative Sum Trick
5 Experiments and Results
5.1 Description of Baselines
5.2 Segmentation
5.3 Robustness
5.4 Zero-Shot Camera Rig Transfer
5.5 Benchmarking Against Oracle Depth
5.6 Motion Planning
6 Conclusion
References

Knowls

Knowl 1 — Latent Depth Distribution for 2D-to-3D Image Lifting
equation
The "Lift" step transforms a 2D camera image into a 3D frustum-shaped point cloud of contextual features without requiring ground-truth depth supervision.

Let $X \in \mathbb{R}^{3 \times H \times W}$ be an input image, $I \in \mathbb{R}^{3 \times 3}$ its intrinsic calibration matrix, and $E \in \mathbb{R}^{3 \times 4}$ its extrinsic calibration matrix. The depth dimension is discretized into a predefined set of $|D|$ discrete depth bins $D = \{d_0 + \Delta, d_0 + 2\Delta, \dots, d_0 + |D|\Delta\}$ . For each pixel $p$ at image coordinates $(h, w)$ , a convolutional neural network predicts a context feature vector $c \in \mathbb{R}^C$ and a discrete probability distribution over depth $\alpha \in \Delta^{|D|-1}$ , where $\Delta^{|D|-1}$ is the probability simplex.

The feature $c_d \in \mathbb{R}^C$ associated with the 3D point $p_d = (h, w, d)$ along the optical ray at depth $d \in D$ is given by the outer product scaling:

$c_d = \alpha_d c$

where $\alpha_d$ is the predicted probability that the surface at pixel $(h, w)$ lies at depth $d$ .

Each point $p_d$ is mapped to reference coordinates $(x, y, z)$ in the ego frame using $I$ and $E$ . If $\alpha$ is a one-hot vector, this operation collapses to a single depth per pixel as in pseudo-LiDAR. If $\alpha$ is a uniform distribution, the context $c$ is spread uniformly along the entire ray as in orthographic feature transform (OFT). By predicting $\alpha$ continuously, the network dynamically allocates context along spatial rays according to depth confidence.
Knowl 2 — Frustum Pooling for Bird's-Eye-View Splatting
algorithm
Frustum Pooling aggregates the 3D frustum feature point clouds generated from $n$ arbitrary camera views into a uniform Bird's-Eye-View (BEV) pillar grid of dimensions $C \times X \times Y$ in the ego coordinate frame without using zero-padding.
Input: Point features $C_{all} \in \mathbb{R}^{N \times C}$ from $n$ camera frustums where $N = n \cdot H \cdot W \cdot |D|$
Input: 3D point coordinates $P_{all} \in \mathbb{R}^{N \times 3}$ in the ego frame
Input: Grid bounds $[x_{min}, x_{max}] \times [y_{min}, y_{max}]$ and cell resolutions $(\Delta_x, \Delta_y)$
Output: Rasterized BEV pseudo-image tensor $Y_{bev} \in \mathbb{R}^{C \times X \times Y}$
for each point $i = 1$ to $N$ do
    Compute discrete pillar coordinates $(u_i, v_i) = (\lfloor (x_i - x_{min})/\Delta_x \rfloor, \lfloor (y_i - y_{min})/\Delta_y \rfloor)$
    if $0 \le u_i < X$ and $0 \le v_i < Y$ then
        Assign 1D pillar bin index $b_i = u_i \cdot Y + v_i$
    else
        Mark point $i$ as invalid / out-of-bounds
    end if
end for
Filter out invalid points, retaining $M$ valid points
Sort valid features $C_{valid}$ by their bin indices $b$
Compute cumulative sum of sorted features along the point dimension: $S_k = \sum_{j=1}^k C_{valid}[j]$ for $k = 1, \dots, M$
for each unique bin index $b$ with boundary indices $[start_b, end_b]$ in the sorted array do
     $Y_{bev}[b] = S_{end_b} - S_{start_b - 1}$ (with $S_0 = 0$ )
end for
return $Y_{bev}$
By computing pillar sums via cumulative sums of sorted bin segments rather than padding each pillar to a maximum capacity, memory usage is drastically reduced. An explicit analytic gradient is implemented for the cumulative sum and boundary subtraction operations, providing a $2\times$ speedup during backpropagation compared to automatic differentiation.
Knowl 3 — Trajectory Distribution and Cost-Map Scoring for Motion Planning
equation
Motion planning is formulated as categorical classification over a pre-extracted dictionary of $K$ fixed template trajectories $\mathcal{T} = \{\tau_1, \tau_2, \dots, \tau_K\}$ , where each template is a sequence of 2D coordinates over time $\tau_i = \{(x_j, y_j, t_j)\}_{j=1}^T$ .

Given sensor observations $o$ , the network outputs a spatial Bird's-Eye-View (BEV) cost map $c_o(x, y) \in \mathbb{R}$ . The probability distribution over trajectory templates $p(\tau_i \mid o)$ is defined as a Boltzmann distribution parameterized by the accumulated spatial cost along each trajectory:

$p(\tau_i \mid o) = \frac{\exp\left(-\sum_{(x_j, y_j) \in \tau_i} c_o(x_j, y_j)\right)}{\sum_{\tau \in \mathcal{T}} \exp\left(-\sum_{(x_j, y_j) \in \tau} c_o(x_j, y_j)\right)}$

During training, given a ground-truth expert ego trajectory $\tau^*$ , the closest template in $\mathcal{T}$ under the $L_2$ norm is selected as the target class label, and the cost map is optimized end-to-end using standard cross-entropy loss without requiring a hard-margin loss. During inference, the trajectory is selected via $\arg\max_{\tau_i \in \mathcal{T}} p(\tau_i \mid o)$ .
Knowl 4 — Lift-Splat-Shoot Network Architecture and Hyperparameters
experimental setup
The Lift-Splat-Shoot architecture connects an image-plane feature extractor to a Bird's-Eye-View (BEV) processing network via frustum pooling:
- Per-Image Feature Extractor: Backbone is an EfficientNet-B0 pretrained on ImageNet. Input camera images are cropped and resized to $128 \times 352$ .
- Depth Discretization: Depth range spans from $4.0\,\text{m}$ to $45.0\,\text{m}$ spaced at $\Delta = 1.0\,\text{m}$ , yielding $|D| = 41$ discrete depth bins.
- BEV Coordinate Grid: The BEV space spans $x \in [-50\,\text{m}, 50\,\text{m}]$ and $y \in [-50\,\text{m}, 50\,\text{m}]$ at $0.5\,\text{m} \times 0.5\,\text{m}$ resolution, creating a $200 \times 200$ grid ( $X=200, Y=200$ ).
- BEV Convolutional Network: Takes the sum-pooled $C \times 200 \times 200$ pillar tensor, applies a $7 \times 7$ convolution with stride 2 followed by BatchNorm and ReLU, passes features through the first 3 meta-layers of ResNet-18 producing multi-scale features $(x_1, x_2, x_3)$ , upsamples $x_3$ by $4\times$ , concatenates with $x_1$ , applies a ResNet block, and upsamples by $2\times$ to return to the $200 \times 200$ resolution.
- Model Capacity & Inference Speed: The entire network contains 14.3M trainable parameters and executes a forward pass at 35 Hz on an NVIDIA Titan V GPU.
- Training Optimization: Trained using the Adam optimizer with a learning rate of $10^{-3}$ , weight decay of $10^{-7}$ , for 300,000 steps using PyTorch.

Knowl 5 — BEV Semantic Segmentation Performance on nuScenes and Lyft

data/table

The Lift-Splat model was evaluated on multi-view Bird's-Eye-View (BEV) object and map segmentation tasks using the nuScenes and Lyft Level 5 datasets. Ground-truth targets were generated by projecting 3D bounding boxes and map layers into the BEV plane. Baselines include standard CNN concatenation, Frozen ImageNet Encoders, Orthographic Feature Transform (OFT), Pyramid Occupancy Networks (PON), and FISHING Net.

	nuScenes (IoU %)		Lyft (IoU %)
Model	Car	Vehicles	Car	Vehicles
CNN	22.78	24.25	30.71	31.91
Frozen Encoder	25.51	26.83	35.28	32.42
OFT	29.72	30.05	39.48	40.43
Lift-Splat (Us)	32.06	32.07	43.09	44.64
PON	24.7	-	-	-
FISHING Net	-	30.0	-	56.0

Lift-Splat outperforms all baseline architectures across both object segmentation and map segmentation tasks, achieving a 2.34% Car IoU gain on nuScenes and a 3.61% Car IoU gain on Lyft over the strongest baseline (OFT).

Knowl 6 — Robustness to Sensor Dropout and Extrinsic Calibration Noise
empirical result
Training Lift-Splat with intentional sensor noise and sensor dropout provides significant robustness benefits:
- Sensor Dropout Regularization: Training the network with one randomly dropped camera per batch sample improves generalization. When tested with all 6 cameras available, the model trained with 1 dropped camera achieves higher Car segmentation IoU than the model trained on the full 6 cameras without dropout, indicating that camera dropout encourages the network to learn meaningful cross-camera feature correlations.
- Extrinsic Noise Resilience: Adding noise to camera extrinsic calibration matrices during training makes the model more robust to test-time extrinsic miscalibration, maintaining stable performance across high noise levels.
- Camera Sensitivity: Testing car segmentation IoU while systematically removing individual cameras on nuScenes reveals that missing the backward-facing camera causes the steepest drop in performance, consistent with its wider field-of-view coverage relative to the other cameras.

Knowl 7 — Zero-Shot Camera Rig Transfer Evaluation

data/table

Because Lift-Splat explicitly conditions on camera intrinsic and extrinsic calibration matrices during frustum splatting, it can perform inference on camera configurations unseen during training without fine-tuning.

First, a model trained on only 4 of the 6 nuScenes cameras was evaluated when additional cameras ( $1_{fl}$ = front-left, $1_{bl}$ = back-left) were introduced at test time:

Second, the model trained exclusively on nuScenes (with its 6-camera rig) was evaluated zero-shot on the Lyft dataset, which has an entirely different camera setup:

Lift-Splat outperforms OFT by +5.10% Car IoU and CNN by +14.35% Car IoU in zero-shot cross-rig transfer.

Knowl 8 — Benchmarking Lift-Splat Against LiDAR Oracle Depth

data/table

Lift-Splat was benchmarked against the PointPillars architecture trained with oracle 3D depth from LiDAR point clouds (single scan and aggregated >1 scans).

	nuScenes (IoU %)				Lyft (IoU %)
Model	Drivable Area	Lane Boundary	Car	Vehicle	Car	Vehicle
Oracle Depth (1 scan)	74.91	25.12	40.26	44.48	74.96	76.16
Oracle Depth ( $>$ 1 scan)	76.96	26.80	45.36	49.51	75.42	76.49
Lift-Splat (Camera only)	70.81	19.58	32.06	32.07	43.09	44.64

Camera-only Lift-Splat approaches LiDAR performance on planar map tasks (70.81% vs 74.91% on Drivable Area) but exhibits a larger gap on 3D object segmentation. Analysis across environmental variables indicates that Lift-Splat suffers severe degradation in nighttime scenes compared to LiDAR and exhibits a roughly linear decrease in IoU as object distance increases.

Knowl 9 — End-to-End Trajectory Classification Accuracy in Motion Planning

data/table

End-to-end motion planning was evaluated by predicting ego trajectory distributions over $K=1000$ template trajectories (5-second duration at 0.25-second increments) on nuScenes. Accuracy is measured as top- $k$ classification accuracy matching the ground-truth trajectory's nearest neighbor template.

While Lift-Splat produces qualitatively plausible planning behaviors—such as adhering to road boundaries and decelerating at crosswalks and behind braking vehicles—it lags behind LiDAR-based planners in template classification accuracy.

Knowl 10 — Temporal Limitations of Single-Timestep Camera-Based Representation
limitation
The Lift-Splat architecture operates strictly on a single time step of multi-camera images. As a result, the model lacks explicit access to ego-vehicle or surrounding vehicle velocities and temporal dynamics. This contributes to a substantial performance gap between single-frame camera models and multi-scan LiDAR models, particularly in nighttime conditions and motion planning accuracy. Closing this gap requires extending the architecture to multi-timestep video sequences.

Coverage note — None was omitted; all primary contributions, including the lift-splat-shoot architecture, latent depth unprojection, frustum pooling algorithm, trajectory shooting planner, experimental evaluations on segmentation, robustness, zero-shot transfer, and LiDAR comparisons, are covered.

References

1.Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: A deep convolutional encoder-decoder architecture for image segmentation. CoRR abs/1511.00561 (2015), http://arxiv.org/abs/1511.00561
2.Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., Beijbom, O.: nuscenes: A multimodal dataset for autonomous driving. CoRR abs/1903.11027 (2019), http://arxiv.org/abs/1903.11027
3.Chang, M.F., Ramanan, D., Hays, J., Lambert, J., Sangkloy, P., Singh, J., Bak, S., Hartnett, A., Wang, D., Carr, P., et al.: Argoverse: 3d tracking and forecasting with rich maps. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (Jun 2019)
4.Chen, X., Kundu, K., Zhang, Z., Ma, H., Fidler, S., Urtasun, R.: Monocular 3d object detection for autonomous driving. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2147–2156 (06 2016)
5.Ghiasi, G., Lin, T., Le, Q.V.: Dropblock: A regularization method for convolutional networks. CoRR abs/1810.12890 (2018), http://arxiv.org/abs/1810.12890
6.Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016), http://www.deeplearningbook.org
7.He, K., Gkioxari, G., Doll´ar, P., Girshick, R.B.: Mask R-CNN. CoRR abs/1703.06870 (2017), http://arxiv.org/abs/1703.06870
8.He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015), http://arxiv.org/abs/1512.03385
9.Hendy, N., Sloan, C., Tian, F., Duan, P., Charchut, N., Xie, Y., Wang, C., Philbin, J.: Fishing net: Future inference of semantic heatmaps in grids (2020)
10.Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR abs/1502.03167 (2015), http://arxiv.org/abs/1502.03167
11.Kayhan, O.S., Gemert, J.C.v.: On translation invariance in cnns: Convolutional layers can exploit absolute spatial location. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020)
12.Kehl, W., Manhardt, F., Tombari, F., Ilic, S., Navab, N.: SSD-6D: making rgb-based 3d detection and 6d pose estimation great again. CoRR abs/1711.10006 (2017)
13.Kesten, R., Usman, M., Houston, J., Pandya, T., Nadhamuni, K., Ferreira, A., Yuan, M., Low, B., Jain, A., Ondruska, P., Omari, S., Shah, S., Kulkarni, A., Kazakova, A., Tao, C., Platinsky, L., Jiang, W., Shet, V.: Lyft level 5 av dataset 2019. urlhttps://level5.lyft.com/dataset/ (2019)
14.Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. CoRR abs/1412.6980 (2014)
15.Kirillov, A., He, K., Girshick, R.B., Rother, C., Doll´ar, P.: Panoptic segmentation. CoRR abs/1801.00868 (2018), http://arxiv.org/abs/1801.00868
16.Krizhevsky, A.: Learning multiple layers of features from tiny images (2009)
17.Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25, pp. 1097–1105. Curran Associates, Inc. (2012), http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
18.Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., Beijbom, O.: Pointpillars: Fast encoders for object detection from point clouds. CoRR abs/1812.05784 (2018)
19.Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. In: Proceedings of the IEEE. pp. 2278–2324 (1998)
20.Lombardi, S., Simon, T., Saragih, J., Schwartz, G., Lehrmann, A., Sheikh, Y.: Neural volumes. ACM Transactions on Graphics 38(4), 114 (Jul 2019). https://doi.org/10.1145/3306346.3323020, http://dx.doi.org/10.1145/3306346.3323020
21.Mani, K., Daga, S., Garg, S., Shankar, N.S., Jatavallabhula, K.M., Krishna, K.M.: Monolayout: Amodal scene layout from a single image. ArXiv abs/2002.08394 (2020)
22.Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: ICML (2010)
23.Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., K¨opf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. In: NeurIPS (2019)
24.Philion, J.: Fastdraw: Addressing the long tail of lane detection by adapting a sequential prediction network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)
25.Philion, J., Kar, A., Fidler, S.: Learning to evaluate perception models using planner-centric metrics. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020)
26.Poirson, P., Ammirato, P., Fu, C., Liu, W., Kosecka, J., Berg, A.C.: Fast single shot detection and pose estimation. CoRR abs/1609.05590 (2016)
27.Qin, Z., Wang, J., Lu, Y.: Monogrnet: A geometric reasoning network for monocular 3d object localization. Proceedings of the AAAI Conference on Artificial Intelligence 33, 8851–8858 (07 2019). https://doi.org/10.1609/aaai.v33i01.33018851
28.Roddick, T., Cipolla, R.: Predicting semantic map representations from images using pyramid occupancy networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020)
29.Roddick, T., Kendall, A., Cipolla, R.: Orthographic feature transform for monocular 3d object detection. CoRR abs/1811.08188 (2018)
30.Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: Imagenet large scale visual recognition challenge (2014)
31.Simonelli, A., Bul`o, S.R., Porzi, L., L´opez-Antequera, M., Kontschieder, P.: Disentangling monocular 3d object detection. CoRR abs/1905.12365 (2019)
32.Srinivasan, P.P., Mildenhall, B., Tancik, M., Barron, J.T., Tucker, R., Snavely, N.: Lighthouse: Predicting lighting volumes for spatially-coherent illumination (2020)
33.Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15, 1929–1958 (2014)
34.Su, H., Jampani, V., Sun, D., Maji, S., Kalogerakis, E., Yang, M., Kautz, J.: Splatnet: Sparse lattice networks for point cloud processing. CoRR abs/1802.08275 (2018), http://arxiv.org/abs/1802.08275
35.Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., Vasudevan, V., Han, W., Ngiam, J., Zhao, H., Timofeev, A., Ettinger, S., Krivokon, M., Gao, A., Joshi, A., Zhang, Y., Shlens, J., Chen, Z., Anguelov, D.: Scalability in perception for autonomous driving: Waymo open dataset (2019)
36.Takikawa, T., Acuna, D., Jampani, V., Fidler, S.: Gated-scnn: Gated shape cnns for semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (October 2019)
37.Tan, M., Le, Q.V.: Efficientnet: Rethinking model scaling for convolutional neural networks. CoRR abs/1905.11946 (2019), http://arxiv.org/abs/1905.11946
38.Tucker, R., Snavely, N.: Single-view view synthesis with multiplane images (2020)
39.Wang, Y., Chao, W., Garg, D., Hariharan, B., Campbell, M., Weinberger, K.Q.: Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. CoRR abs/1812.07179 (2018)
40.You, Y., Wang, Y., Chao, W., Garg, D., Pleiss, G., Hariharan, B., Campbell, M., Weinberger, K.Q.: Pseudo-lidar++: Accurate depth for 3d object detection in autonomous driving. CoRR abs/1906.06310 (2019)
41.Zeng, W., Luo, W., Suo, S., Sadat, A., Yang, B., Casas, S., Urtasun, R.: End-to-end interpretable neural motion planner. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 8652–8661 (2019)

Citation

MLA

Philion, J., and S. Fidler. “Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D”. arXiv, 2020, http://arxiv.org/abs/2008.05711v1.

APA

Philion, J., & Fidler, S. (2020). Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D. arXiv. http://arxiv.org/abs/2008.05711v1

Chicago

Philion, J., and S. Fidler. 2020. “Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D”. arXiv. http://arxiv.org/abs/2008.05711v1.

Harvard

Philion, J. and Fidler, S. (2020) “Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2008.05711v1.

Vancouver

1. Philion J, Fidler S (2020) Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D. arXiv

BibTeX

@article{philion2020lift,
  title = {Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D},
  author = {Philion, Jonah and Fidler, Sanja},
  year = {2020},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2008.05711v1},
  eprint = {2008.05711}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Model	Drivable Area (IoU %)	Lane Boundary (IoU %)
CNN	68.96	16.51
Frozen Encoder	61.62	16.95
OFT	71.69	18.07
Lift-Splat (Us)	72.94	19.96
PON	60.4	-

Evaluated Camera Set	Car IoU (%)
4 Cameras	26.53
4 + 1 front-left ( $1_{fl}$ )	27.35
4 + 1 back-left ( $1_{bl}$ )	27.27
4 + 1 front-left + 1 back-left	27.94

Model	Lyft Car IoU (%)	Lyft Vehicle IoU (%)
CNN	7.00	8.06
Frozen Encoder	15.08	15.82
OFT	16.25	16.27
Lift-Splat (Us)	21.35	22.59

Model	Top 5 (%)	Top 10 (%)	Top 20 (%)
LiDAR (1 scan)	19.27	28.88	41.93
LiDAR (10 scans)	24.99	35.39	49.84
Lift-Splat (Camera only)	15.52	19.94	27.99

Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Latent Depth Distribution for 2D-to-3D Image Lifting

Knowl 2 — Frustum Pooling for Bird's-Eye-View Splatting

Knowl 3 — Trajectory Distribution and Cost-Map Scoring for Motion Planning

Knowl 4 — Lift-Splat-Shoot Network Architecture and Hyperparameters

Knowl 5 — BEV Semantic Segmentation Performance on nuScenes and Lyft

Knowl 6 — Robustness to Sensor Dropout and Extrinsic Calibration Noise

Knowl 7 — Zero-Shot Camera Rig Transfer Evaluation

Knowl 8 — Benchmarking Lift-Splat Against LiDAR Oracle Depth

Knowl 9 — End-to-End Trajectory Classification Accuracy in Motion Planning

Knowl 10 — Temporal Limitations of Single-Timestep Camera-Based Representation

References

Citation

BibTeX

Source Code

Access the Paper