4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks

Christopher ChoyJunYoung GwakSilvio Savarese

article2019CVPR2,486 citations

Proposes generalized sparse convolutions for high-dimensional neural networks, enabling direct 4D spatio-temporal processing of 3D video streams and outperforming conventional frame-by-frame perception models in both accuracy and speed.

Listen

The article addresses the challenge of processing 3D videos, such as sequences of depth camera images or LIDAR scans, which are increasingly common in robotics and augmented reality but are often handled inefficiently one frame at a time using 2D methods or separate 3D algorithms. This frame-by-frame approach struggles with consistency over time and scales poorly to high-dimensional data due to memory and computation demands.

The work set out to develop networks that directly handle 4D spatio-temporal data by treating time as an additional dimension alongside 3D space. The authors created a library called the Minkowski Engine to support sparse tensor operations and generalized convolutions that work efficiently on mostly empty high-dimensional spaces.

They built 4D convolutional networks using custom hybrid kernels that combine cubic shapes for space with cross shapes for time, along with a 7D conditional random field to enforce consistency across space, time, and color. Experiments ran on established 3D benchmarks like ScanNet and Stanford 3D Indoor Spaces, plus new 4D datasets derived from Synthia and RueMonge sequences, with tests including added noise to simulate real sensor conditions.

The networks achieved strong results, reaching 67.9 percent mean intersection over union on ScanNet and outperforming prior 2D and hybrid methods by substantial margins. On 4D data, the approach proved more robust to noise than 3D counterparts, delivered comparable or better accuracy with only modest added parameters, and sometimes ran faster when processing entire sequences in batch mode.

These outcomes indicate that direct 4D processing can improve accuracy and reliability for perception tasks while cutting unnecessary computation on empty space. For decision makers in robotics or AR, this suggests potential gains in system performance and reduced hardware requirements.

Next steps include scaling the method to larger real-world video collections and integrating it into end-to-end robotic pipelines. The main limitations are that results saturate quickly on small datasets and that the approach relies on sparse data assumptions that may not hold in dense scenes; confidence remains high on the reported benchmarks but warrants caution when extrapolating to untested environments.

Stanford University 3d video ar application conditional random field Minkowski Engine Generalized Sparse Convolution 4D Spatio‑Temporal ConvNet

arXiv: 1904.08755 StanfordVL/MinkowskiEngine

Paper: VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection, Yin Zhou et al. (2017). Learn how VoxelNet pioneered end-to-end learning on volumetric voxel grids from sparse 3D point clouds, which forms the foundational sparse spatial representation generalized by Minkowski Engine.
Paper: PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation, Charles R. Qi et al. (2017). Understand the original deep learning principles for irregular point cloud processing that motivated the development of higher-dimensional sparse convolutional architectures.
Paper: Learning Spatiotemporal Features with 3D Convolutional Networks, Du Tran et al. (2015). Explore standard 3D spatiotemporal convolutions on 2D video sequences (2D space plus time) before seeing them generalized to full 4D Minkowski convolutions (3D space plus time).
Paper: A Closer Look at Spatiotemporal Convolutions for Action Recognition, Du Tran et al. (2017). Examine how factorized spatial and temporal convolutional kernels operate efficiently in lower dimensions, directly inspiring the hybrid spatial-temporal kernels used in 4D ConvNets.
Paper: Non-local Neural Networks, Xiaolong Wang et al. (2018). Understand how non-local operations capture long-range dependencies across space and time to complement localized spatiotemporal convolutions.
Paper: 3D ShapeNets: A deep representation for volumetric shapes, Zhirong Wu et al. (2014). Discover the early volumetric grid representations for 3D shapes that established 3D deep learning benchmarks prior to sparse high-dimensional formulations.

Paper: PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection, Shaoshuai Shi et al. (2019). Explore how point-voxel hybrid abstractions combine sparse voxel convolutions with point-based features for high-performance 3D object detection.
Paper: D-NeRF: neural radiance fields for dynamic scenes, Albert Pumarola et al. (2021). See how dynamic spatiotemporal 3D scenes can be represented and rendered continuously across time using neural radiance fields rather than sparse discrete convolutions.
Paper: Point Transformer, Nico Engel et al. (2020). Examine how self-attention and transformer architectures process unstructured 3D geometric point data directly as an alternative to sparse voxel convolutions.
Paper: PCT: Point cloud transformer, Meng-Hao Guo et al. (2020). Investigate how offset-attention mechanisms capture local and global geometric relationships in irregular 3D point clouds without explicit grid discretization.
Paper: ViViT: A Video Vision Transformer, Anurag Arnab et al. (2021). Learn how pure transformer models factorize spatiotemporal attention across space and time for high-dimensional video understanding.
Paper: Video Swin Transformer, Ze Liu et al. (2021). Discover how 3D spatiotemporal windowing is adapted into vision transformers to process dynamic temporal sequences efficiently.

Cover for 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks

Abstract

In many robotics and VR/AR applications, 3D-videos are readily-available sources of input (a continuous sequence of depth images, or LIDAR scans). However, those 3D-videos are processed frame-by-frame either through 2D convnets or 3D perception algorithms. In this work, we propose 4-dimensional convolutional neural networks for spatio-temporal perception that can directly process such 3D-videos using high-dimensional convolutions. For this, we adopt sparse tensors and propose the generalized sparse convolution that encompasses all discrete convolutions. To implement the generalized sparse convolution, we create an open-source auto-differentiation library for sparse tensors that provides extensive functions for high-dimensional convolutional neural networks. We create 4D spatio-temporal convolutional neural networks using the library and validate them on various 3D semantic segmentation benchmarks and proposed 4D datasets for 3D-video perception. To overcome challenges in the 4D space, we propose the hybrid kernel, a special case of the generalized sparse convolution, and the trilateral-stationary conditional random field that enforces spatio-temporal consistency in the 7D space-time-chroma space. Experimentally, we show that convolutional neural networks with only generalized 3D sparse convolutions can outperform 2D or 2D-3D hybrid methods by a large margin. Also, we show that on 3D-videos, 4D spatio-temporal convolutional neural networks are robust to noise, outperform 3D convolutional neural networks and are faster than the 3D counterpart in some cases.

Topics

3d video ar application conditional random field convolutional neural networks libraries robotics semantics sparse approximations image classification

Keywords

Minkowski Engine Generalized Sparse Convolution 4D Spatio‑Temporal ConvNet Hybrid Kernel Trilateral Stationary CRF Sparse Tensor 3D Video Perception ScanNet Benchmark TS‑CRF High‑Dimensional Convolution

1 Introduction
2 Related Work
3 Sparse Tensor and Convolution
3.1 Generalized Sparse Convolution
4 Minkowski Engine
4.1 Sparse Tensor Quantization
4.2 Generalized Sparse Convolution
4.3 Max Pooling
4.4 Global / Average Pooling, Sum Pooling
4.5 Non-spatial Functions
5 Minkowski Convolutional Neural Networks
5.1 Tesseract Kernel and Hybrid Kernel
5.2 Residual Minkowski Networks
6 Trilateral Stationary-CRF
6.1 Definition
6.2 Variational Inference
6.3 Learning with 7D Sparse Convolution
7 Experiments
7.1 Implementation
7.2 Training and Evaluation
7.3 Datasets
7.4 Results and Analysis
8 Conclusion
9 Acknowledgements
References

Knowls

Knowl 1 — Generalized Sparse Convolution
definition
The generalized sparse convolution extends discrete convolutions to arbitrary input and output coordinate sets in arbitrary dimensions with user-defined kernel offset structures. Let $\mathbf{u} \in \mathbb{R}^D$ denote a $D$ -dimensional coordinate, and let $x_{\mathbf{u}}^{\text{in}} \in \mathbb{R}^{N_{\text{in}}}$ denote an $N_{\text{in}}$ -dimensional input feature vector located at coordinate $\mathbf{u}$ . Let $\mathcal{C}_{\text{in}} \subset \mathbb{R}^D$ and $\mathcal{C}_{\text{out}} \subset \mathbb{R}^D$ be predefined sets of input and output sparse tensor coordinates. Let $\mathcal{N}^D$ define an arbitrary set of discrete spatial offsets defining the kernel geometry, and let $W_{\mathbf{i}} \in \mathbb{R}^{N_{\text{out}} \times N_{\text{in}}}$ denote the convolution weight matrix associated with spatial offset $\mathbf{i} \in \mathcal{N}^D$ .

The generalized sparse convolution computes the output feature vector $x_{\mathbf{u}}^{\text{out}} \in \mathbb{R}^{N_{\text{out}}}$ at coordinate $\mathbf{u} \in \mathcal{C}_{\text{out}}$ as:

$x_{\mathbf{u}}^{\text{out}} = \sum_{\mathbf{i} \in \mathcal{N}^D(\mathbf{u}, \mathcal{C}_{\text{in}})} W_{\mathbf{i}} x_{\mathbf{u} + \mathbf{i}}^{\text{in}} \quad \text{for } \mathbf{u} \in \mathcal{C}_{\text{out}}$

where $\mathcal{N}^D(\mathbf{u}, \mathcal{C}_{\text{in}}) = \{\mathbf{i} \mid \mathbf{u} + \mathbf{i} \in \mathcal{C}_{\text{in}}, \mathbf{i} \in \mathcal{N}^D\}$ is the active set of kernel offsets from center $\mathbf{u}$ that land on an existing input coordinate.

This formulation generalizes standard convolution paradigms:
1. Conventional dense convolution is recovered when $\mathcal{C}_{\text{in}} = \mathcal{C}_{\text{out}} = \mathbb{Z}^D$ and $\mathcal{N}^D = \mathcal{V}^D(K)$ , where $\mathcal{V}^D(K)$ is the set of offsets within a $D$ -dimensional hypercube of side length $K$ .
2. Strided dense convolution is recovered when $\mathcal{C}_{\text{in}}$ and $\mathcal{C}_{\text{out}}$ are regular grids with output grid spacing set as an integer multiple of input spacing, and $\mathcal{N}^D = \mathcal{V}^D(K)$ .
3. Sparse submanifold convolution is recovered when $\mathcal{C}_{\text{out}} = \mathcal{C}_{\text{in}}$ and $\mathcal{N}^D = \mathcal{V}^D(K)$ .
4. Dilated and non-hypercubic convolutions are formed by custom selections of $\mathcal{N}^D$ .
Knowl 2 — Sparse Tensor Representation in Minkowski Engine
definition
A sparse tensor in $D$ spatial dimensions is represented in coordinate list (COO) format by a coordinate matrix $C$ augmented with batch indices and an associated feature matrix $F$ . For a batch of sparse data containing $N$ total non-empty points across all batch items, the representations are defined as:

$C = \begin{bmatrix} x_1 & y_1 & z_1 & t_1 & b_1 \\ \vdots & \vdots & \vdots & \vdots & \vdots \\ x_N & y_N & z_N & t_N & b_N \end{bmatrix} \in \mathbb{R}^{N \times (D + 1)}, \quad F = \begin{bmatrix} \mathbf{f}_1^T \\ \vdots \\ \mathbf{f}_N^T \end{bmatrix} \in \mathbb{R}^{N \times N_f}$

where $(x_i, y_i, z_i, t_i)$ are the spatio-temporal coordinates of the $i$ -th point, $b_i \in \mathbb{Z}_{\ge 0}$ is its integer batch index (which disambiguates points in different batch elements that share spatial coordinates), and $\mathbf{f}_i \in \mathbb{R}^{N_f}$ is its $N_f$ -dimensional feature vector.
Knowl 3 — Hybrid Spatio-Temporal Convolution Kernel
model/method
In 4D spatio-temporal convolutions ( $D = 4$ : 3D space + 1D time), using a standard 4D hypercubic (tesseract) kernel of side length $K$ scales the number of parameters by $K^4$ , which causes severe over-parameterization, overfitting, and high computational and memory cost. The physical surface area of 3D data grows quadratically with spatial resolution but linearly with time.

To match this asymmetry, the hybrid kernel combines:
1. A 3D cubic kernel $\mathcal{V}^3(K)$ along the three spatial dimensions to capture dense local 3D geometric structures.
2. A 1D cross-shaped kernel along the temporal dimension to connect the same spatial coordinate across adjacent time steps without convolving across spatial-temporal cross-terms.
The hybrid kernel offset set $\mathcal{N}^4_{\text{hybrid}}$ is defined by the union of spatial hypercube offsets at time offset $\tau = 0$ and pure temporal offsets along the time axis at spatial offset $(0, 0, 0)$ :

$\mathcal{N}^4_{\text{hybrid}} = \{(\delta_x, \delta_y, \delta_z, 0) \mid |\delta_x|, |\delta_y|, |\delta_z| \le \lfloor K_s / 2 \rfloor\} \cup \{(0, 0, 0, \tau) \mid |\tau| \le \lfloor K_t / 2 \rfloor\}$

For kernel sizes $K_s = 3$ and $K_t = 3$ , a full 4D tesseract kernel requires $3^4 = 81$ weight matrices per channel pair, whereas the hybrid kernel requires only $3^3 + 3 - 1 = 29$ weight matrices, reducing kernel parameters by $64.2\%$ .
Knowl 4 — Generalized Sparse Convolution Computation via Kernel Maps
algorithm
Computing a generalized sparse convolution requires mapping which sparse input elements affect which sparse output elements for each relative offset in the kernel set $\mathcal{N}^D$ . The mapping is stored in kernel maps $\mathcal{M} = \{(I_{\mathbf{i}}, O_{\mathbf{i}})\}_{\mathbf{i} \in \mathcal{N}^D}$ , where $I_{\mathbf{i}}$ and $O_{\mathbf{i}}$ are equal-length arrays of integer indices into the input feature tensor $F^{\text{in}}$ and output feature tensor $F^{\text{out}}$ such that for each index pair $(I_{\mathbf{i}}[n], O_{\mathbf{i}}[n])$ , the input coordinate at $I_{\mathbf{i}}[n]$ shifted by $\mathbf{i}$ equals the output coordinate at $O_{\mathbf{i}}[n]$ .
Input: Kernel weights $W = \{W_{\mathbf{i}}\}_{\mathbf{i} \in \mathcal{N}^D}$ , input features $F^{\text{in}} \in \mathbb{R}^{N_{\text{in\_pts}} \times N_{\text{in}}}$ , output placeholder $F^{\text{out}} \in \mathbb{R}^{N_{\text{out\_pts}} \times N_{\text{out}}}$ , kernel map $\mathcal{M} = \{(I_{\mathbf{i}}, O_{\mathbf{i}})\}_{\mathbf{i} \in \mathcal{N}^D}$
Output: Populated output features $F^{\text{out}}$
$F^{\text{out}} \leftarrow 0$
for each $(W_{\mathbf{i}}, (I_{\mathbf{i}}, O_{\mathbf{i}})) \in (W, \mathcal{M})$ do
     $F_{\text{gathered}} \leftarrow [F^{\text{in}}[I_{\mathbf{i}}[1]], F^{\text{in}}[I_{\mathbf{i}}[2]], \dots, F^{\text{in}}[I_{\mathbf{i}}[n]]]$
     $F_{\text{tmp}} \leftarrow F_{\text{gathered}} W_{\mathbf{i}}^T$
    for $k \leftarrow 1$ to length( $O_{\mathbf{i}}$ ) do
         $F^{\text{out}}[O_{\mathbf{i}}[k]] \leftarrow F^{\text{out}}[O_{\mathbf{i}}[k]] + F_{\text{tmp}}[k]$
    end for
end for
return $F^{\text{out}}$
Knowl 5 — GPU Sparse Tensor Quantization with Collision Handling
algorithm
Sparse tensor quantization converts unorganized, continuous $D$ -dimensional point cloud coordinates into unique discrete voxel coordinates, assigns features, and resolves multi-label collisions within the same voxel cell during supervised training by marking colliding voxels with IGNORE_LABEL.
Input: Raw coordinates $C_p \in \mathbb{R}^{N \times D}$ , raw features $F_p \in \mathbb{R}^{N \times N_f}$ , target ground-truth labels $l \in \mathbb{Z}_+^N$ , voxel step size $v_l$
Output: Quantized unique coordinates $C_p'$ , features $F_p'$ , labels $l'''$
$C_p' \leftarrow \lfloor C_p / v_l \rfloor$
$k \leftarrow \text{hash}(C_p')$
$i \leftarrow [0, 1, \dots, N - 1]$
$((i', l'), k') \leftarrow \text{SortByKey}((i, l), \text{key}=k)$
$(i'', (k'', l'')) \leftarrow \text{UniqueByKey}(i', \text{key}=(k', l'))$
$(l''', i''') \leftarrow \text{ReduceByKey}((l'', i''), \text{key}=k'', \text{fn}=f)$
return $C_p'[i''', :], F_p[i''', :], l'''$
The reduction function $f((l_x, i_x), (l_y, i_y))$ evaluates pairs of key-value tuples sharing the same spatial hash key $k''$ . If $l_x \neq l_y$ (indicating multiple conflicting semantic labels mapped into the same discrete voxel), $f$ returns (IGNORE_LABEL, ix), which excludes the voxel from computing training loss.
Knowl 6 — Trilateral Stationary Conditional Random Field (TS-CRF)
model/method
To enforce spatio-temporal and appearance consistency across network predictions on 3D-video sequences, a Conditional Random Field is formulated over a 7-dimensional space-time-chroma space consisting of 3D spatial world coordinates $(x, y, z)$ , 1D time $(t)$ , and 3D color $(r, g, b)$ .

Let $\mathbf{x}_i$ denote the $i$ -th node in the 7D space. The joint distribution of label assignments $X$ over all nodes is defined as:

$P(X) = \frac{1}{Z} \exp \left( \sum_i \left( \phi_u(\mathbf{x}_i) + \sum_{\mathbf{x}_j \in \mathcal{N}^7(\mathbf{x}_i)} \phi_p(\mathbf{x}_i, \mathbf{x}_j) \right) \right)$

where $\phi_u(\mathbf{x}_i)$ is the unary potential provided by the upstream network logits, $\phi_p(\mathbf{x}_i, \mathbf{x}_j)$ is the pairwise potential between node $\mathbf{x}_i$ and its 7D neighborhood $\mathcal{N}^7(\mathbf{x}_i)$ , and $Z$ is the partition function.

The pairwise potential is constrained to satisfy the stationarity condition:

$\phi_p(\mathbf{u}, \mathbf{v}) = \phi_p(\mathbf{u} + \boldsymbol{\tau}_{\mathbf{u}}, \mathbf{v} + \boldsymbol{\tau}_{\mathbf{v}}) \quad \text{for } \boldsymbol{\tau}_{\mathbf{u}}, \boldsymbol{\tau}_{\mathbf{v}} \in \mathbb{R}^7$

Spatial coordinates of each node $\mathbf{x}_i$ are transformed into the world coordinate system via camera extrinsics so that static 3D physical points retain identical coordinates over time. Embedding color dimensions directly into the neighborhood definition forms a distance gap between spatially adjacent points that have different colors (such as across object boundaries), preventing semantic label information from leaking across boundaries.
Knowl 7 — Variational Inference of TS-CRF via 7D Sparse Convolution
algorithm
Under the mean-field approximation $Q(X) = \prod_i Q_i(\mathbf{x}_i)$ , the optimal distribution minimizing KL-divergence satisfies the fixed-point update equation:

$Q_i^+(\mathbf{x}_i) = \frac{1}{Z_i} \exp \left( \phi_u(\mathbf{x}_i) + \sum_{\mathbf{x}_j \in \mathcal{N}^7(\mathbf{x}_i)} \sum_{\mathbf{x}_j} \phi_p(\mathbf{x}_i, \mathbf{x}_j) Q_j(\mathbf{x}_j) \right)$

Because $\phi_p$ is stationary, the inner product $\sum_{\mathbf{x}_j \in \mathcal{N}^7(\mathbf{x}_i)} \phi_p(\mathbf{x}_i, \mathbf{x}_j) Q_j(\mathbf{x}_j)$ is mathematically equivalent to a generalized sparse convolution in 7D space with kernel weights $\phi_p$ . The fixed-point iteration is executed as a differentiable recurrent layer:
Input: Unary logit potentials $\phi_u(X)$ , 3D coordinates $C$ , color features $F$ , time coordinates $T$ , iteration count $N_{\text{iter}}$ , kernel weights $\phi_p$
Output: Refined marginal distribution $Q^{N_{\text{iter}}}$
$Q^0(X) \leftarrow \exp(\phi_u(X))$
$C_{\text{crf}} \leftarrow [C, F, T]$
for $n \leftarrow 1$ to $N_{\text{iter}}$ do
$\widetilde{Q}^n \leftarrow \text{SparseConvolution}((C_{\text{crf}}, Q^{n-1}), \text{kernel}=\phi_p)$
$Q^n \leftarrow \text{Softmax}(\phi_u + \widetilde{Q}^n)$
end for
return $Q^{N_{\text{iter}}}$
The entire system is trained end-to-end with loss $L$ by computing gradients through the unrolled recurrent inference steps:

$\frac{\partial L}{\partial \phi_p} = \sum_{n=1}^{N_{\text{iter}}} \frac{\partial L}{\partial Q^n} \frac{\partial Q^n}{\partial \phi_p}, \quad \frac{\partial L}{\partial \phi_u} = \sum_{n=1}^{N_{\text{iter}}} \frac{\partial L}{\partial Q^n} \frac{\partial Q^n}{\partial \phi_u}$

where $\phi_u$ is generated by a 4D Minkowski network and $\phi_p$ is parameterized by a 7D Minkowski network layer.
Knowl 8 — Sparse Tensor Pooling and Output Coordinate Generation
algorithm
Strided downsampling in sparse convolutional networks requires dynamically creating output coordinates $\mathcal{C}_{\text{out}}$ from input coordinates $\mathcal{C}_{\text{in}}$ , and aggregating features mapped to each output coordinate via max or average/sum pooling.

Output coordinate generation with input stride $s_i$ and layer stride $s_l$ :
Input: Input coordinates $C$ , input stride $s_i$ , layer stride $s_l$
Output: Output coordinates $C_{\text{out}}$
if $s_l > 1$ then
     $s \leftarrow s_l \times s_i$
     $C_{\text{out}} \leftarrow \emptyset$
    for each $c_i \in C$ do
         $c \leftarrow \lfloor c_i / s \rfloor \times s$
        if $c \notin C_{\text{out}}$ then
             $C_{\text{out}} \leftarrow C_{\text{out}} \cup \{c\}$
        end if
    end for
    return $C_{\text{out}}$
else
    return $C$
end if
Sparse tensor average/sum pooling using sparse BLAS (cuSPARSE CSR multiplication) with kernel map $(I, O)$ :
Input: Kernel map $M = (I, O)$ , feature matrix $F$ , all-ones vector $\mathbf{1}$
Output: Pooled features $F_{\text{pooled}}$
$S_M \leftarrow \text{coo2csr}(\text{row}=O, \text{col}=I, \text{val}=1)$
$F' \leftarrow \text{cusparse\_csrmm}(S_M, F)$
$N_{\text{counts}} \leftarrow \text{cusparse\_csrmv}(S_M, \mathbf{1})$
return $F' / N_{\text{counts}}$ // For sum pooling, return $F'$ without dividing

Knowl 9 — ScanNet 3D Semantic Segmentation Benchmark Performance

data/table

The 3D semantic segmentation benchmark on ScanNet evaluated models on fully reconstructed 3D indoor scenes using mean Intersection over Union (mIoU). MinkowskiNet architectures process whole rooms fully convolutionally using sparse tensors.

Method	mIoU (%)
ScanNet	30.6
SSC-UNet	30.8
PointNet++	33.9
ScanNet-FTSDF	38.3
SPLATNet	39.3
TangentConv	43.8
SurfaceConv	44.2
3DMV (uses 2D images additionally)	48.4
3DMV-FTSDF (uses 2D images additionally)	50.1
PointNet++SW	52.3
MinkowskiNet42 (5cm voxel)	67.9
SparseConvNet	72.5
MinkowskiNet42 (2cm voxel)	73.4

MinkowskiNet42 at 5cm voxel resolution achieved 67.9% mIoU, outperforming prior 3D and 2D-3D hybrid methods (such as 3DMV at 48.4% and PointNet++SW at 52.3%) by at least 15.6% mIoU. Reducing the voxel size to 2cm increased accuracy to 73.4% mIoU.

Knowl 10 — Synthia 4D Spatio-Temporal Segmentation and Noise Robustness

data/table

Experiments on the Synthia 4D dataset (sequences of back-projected driving RGB-D scans) compare 3D networks, temporal post-processing baselines, 4D networks with tesseract kernels, 4D networks with hybrid kernels, and 4D networks with Trilateral Stationary CRF (TS-CRF), under clean and noisy sensory input conditions.

Clean Synthia 4D benchmark:

Method	mIoU (%)	mAcc (%)
3D MinkNet20	76.24	89.31
3D MinkNet20 + TA (Temporal Averaging)	77.03	89.20
4D Tesseract MinkNet20	75.34	89.27
4D MinkNet20 (Hybrid Kernel)	77.46	88.01
4D MinkNet20 + TS-CRF	78.30	90.23
4D MinkNet32 + TS-CRF	78.67	90.51

Noisy Synthia 4D benchmark (with elastic distortion, Gaussian noise, and chromatic shifts):

Method	Building	Road	Sidewalk	Fence	Veget.	Pole	Car	T. Sign	Pedest.	Lane	T. Light	mIoU
3D MinkNet42	87.95	97.51	78.35	84.31	96.23	94.79	87.37	42.71	66.67	52.67	55.35	76.72
3D MinkNet42 + TA	87.80	97.07	78.50	83.94	96.29	94.76	85.25	43.72	62.05	50.32	54.83	75.87
4D Tesseract MinkNet42	89.96	96.92	81.76	82.84	96.56	96.04	91.20	52.15	51.82	70.39	57.96	78.87
4D MinkNet42 (Hybrid)	88.89	97.72	85.21	84.86	97.33	96.15	92.21	61.79	61.65	55.67	56.74	79.84

The 4D hybrid kernel network outperformed the 4D tesseract kernel (77.46% vs 75.34% on clean data; 79.84% vs 78.87% under noise). Under input noise, 3D temporal averaging degraded performance (76.72% to 75.87%) because averaging noisy coordinate locations blurs features, whereas 4D spatio-temporal convolutions natively extracted spatio-temporal features to achieve 79.84% mIoU.

Knowl 11 — Stanford 3D Indoor Spaces (S3DIS) Benchmark Results

data/table

Performance comparison on the Stanford 3D Indoor Spaces (S3DIS) Area 5 test set (Fold #1):

Method	mIoU (%)	mAcc (%)
PointNet	41.09	48.98
SparseUNet	41.72	64.62
SegCloud	48.92	57.35
TangentConv	52.80	60.70
3D RNN	53.40	71.30
PointCNN	57.26	63.86
SuperpointGraph	58.04	66.50
PCCN	58.27	67.01
MinkowskiNet20	62.60	69.62
MinkowskiNet32	65.35	71.71

MinkowskiNet32 reached 65.35% mIoU and 71.71% mAcc on S3DIS Area 5, outperforming PointCNN (57.26% mIoU) and SuperpointGraph (58.04% mIoU).

Knowl 12 — Computational Runtime Scaling of 3D vs 4D MinkowskiNet

data/table

Runtime measurements (in seconds) required to process 3D-video sequences of varying lengths within a $50\text{m} \times 50\text{m} \times 50\text{m}$ volume across different voxel resolutions ( $0.6\text{m}$ , $0.45\text{m}$ , and $0.3\text{m}$ ):

Voxel Size	0.6m			0.45m			0.3m
Video Length (s)	3D	4D	4D-CRF	3D	4D	4D-CRF	3D	4D	4D-CRF
3	0.18	0.14	0.17	0.25	0.22	0.27	0.43	0.49	0.59
5	0.31	0.23	0.27	0.41	0.39	0.47	0.71	0.94	1.13
7	0.43	0.31	0.38	0.58	0.61	0.74	0.99	1.59	2.02

At larger voxel sizes (0.6m and 0.45m), 4D MinkowskiNet processes entire video sequences faster than applying 3D networks frame-by-frame due to joint spatio-temporal batching and sparsity. Adding the 7D TS-CRF adds minimal computational overhead (parameters added to 4D networks are under 6.4%, and under 0.006% for TS-CRF).

Coverage note — None was omitted; all key theoretical definitions, kernel constructions, algorithms, CRF formulations, and benchmark experiments have been captured.

References

1.Andrew Adams, Jongmin Baek, and Myers Abraham Davis. Fast high-dimensional filtering using the permutohedral lattice. In Computer Graphics Forum, volume 29, pages 753–762. Wiley Online Library, 2010. 2
2.Iro Armeni, Ozan Sener, Amir R. Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, 2016. 2, 7, 8
3.Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018. 3
4.Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In Proceedings of the European Conference on Computer Vision (ECCV), 2016. 2
5.Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017. 1, 2, 7
6.Angela Dai and Matthias Nießner. 3dmv: Joint 3d-multi-view prediction for 3d semantic scene segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), 2018. 2, 7
7.Raghudeep Gadde, Varun Jampani, Renaud Marlet, and Peter Gehler. Efficient 2d and 3d facade segmentation using auto-context. IEEE transactions on pattern analysis and machine intelligence, 2017. 9
8.Benjamin Graham. Spatially-sparse convolutional neural networks. arXiv preprint arXiv:1409.6070, 2014. 1, 3
9.Ben Graham. Sparse 3d convolutional neural networks. British Machine Vision Conference, 2015. 1, 2, 3, 8
10.Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3D semantic segmentation with submanifold sparse convolutional networks. CVPR, 2018. 7
11.P. Hermosilla, T. Ritschel, P-P Vazquez, A. Vinacua, and T. Ropinski. Monte carlo convolution for learning on non-uniformly sampled point clouds. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2018), 2018. 1, 2
12.Daphne Koller and Nir Friedman. Probabilistic Graphical Models: Principles and Techniques - Adaptive Computation and Machine Learning. The MIT Press, 2009. 6
13.Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. In Advances in Neural Information Processing Systems 24, 2011. 6
14.Loic Landrieu and Martin Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. arXiv preprint arXiv:1711.09869, 2017. 2, 8
15.Yangyan Li, Rui Bu, Mingchao Sun, and Baoquan Chen. Pointcnn. arXiv preprint arXiv:1801.07791, 2018. 1, 2, 8
16.Maria Lorenzo-Valdés, Gerardo I Sanchez-Ortiz, Andrew G Elkington, Raad H Mohiaddin, and Daniel Rueckert. Segmentation of 4d cardiac mr images using a probabilistic atlas and the em algorithm. Medical Image Analysis, 8(3):255–265, 2004. 3
17.Andelo Martinovic, Jan Knopp, Hayko Riemenschneider, and Luc Van Gool. 3d all the way: Semantic segmentation of urban scenes from start to end in 3d. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015. 9
18.Tim McInerney and Demetri Terzopoulos. A dynamic finite element surface model for segmentation and tracking in multidimensional medical images with application to cardiac 4d image analysis. Computerized Medical Imaging and Graphics, 19(1):69–83, 1995. 2
19.Nvidia. Thrust: Parallel algorithm library. 3
20.Hao Pan, Shilin Liu, Yang Liu, and Xin Tong. Convolutional neural networks on 3d surfaces using parallel frames. arXiv preprint arXiv:1808.04952, 2018. 1, 2, 7
21.Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017. 7
22.Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. arXiv preprint arXiv:1612.00593, 2016. 1, 2, 8
23.Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems, 2017. 1, 2, 7
24.Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representations at high resolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. 1, 2, 9
25.Hayko Riemenschneider, András Bódis-Szomorú, Julien Weissenberg, and Luc Van Gool. Learning where to classify in multi-view semantic segmentation. In European Conference on Computer Vision. Springer, 2014. 2, 7, 9
26.Hayko Riemenschneider, András Bódis-Szomorú, Julien Weissenberg, and Luc Van Gool. Learning where to classify in multi-view semantic segmentation. In European Conference on Computer Vision, 2014. 9
27.German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M. Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 2, 7
28.Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Vangelis Kalogerakis, Ming-Hsuan Yang, and Jan Kautz. Splatnet: Sparse lattice networks for point cloud processing. arXiv preprint arXiv:1802.08275, 2018. 2, 7, 9
29.Maxim Tatarchenko*, Jaesik Park*, Vladlen Koltun, and Qian-Yi Zhou. Tangent convolutions for dense prediction in 3D. CVPR, 2018. 1, 2, 7, 8
30.Lyne P Tchapmi, Christopher B Choy, Iro Armeni, JunYoung Gwak, and Silvio Savarese. Segcloud: Semantic segmentation of 3d point clouds. International Conference on 3D Vision (3DV), 2017. 2, 8
31.Parker Allen Tew. An investigation of sparse tensor formats for tensor libraries. PhD thesis, Massachusetts Institute of Technology, 2016. 3
32.Xiaoqing Ye, Jiamao Li, Hexiao Huang, Liang Du, and Xiaolin Zhang. 3d recurrent neural networks with context fusion for point cloud semantic segmentation. In The European Conference on Computer Vision (ECCV), September 2018. 8
33.A. Zeng, S. Song, M. Nießner, M. Fisher, J. Xiao, and T. Funkhouser. 3dmatch: Learning the matching of local 3d geometry in range scans. In CVPR, 2017. 2
34.Yu Zhao, Xiang Li, Wei Zhang, Shijie Zhao, Milad Makkie, Mo Zhang, Quanzheng Li, and Tianming Liu. Modeling 4d fmri data via spatio-temporal convolutional neural networks (st-cnn). arXiv preprint arXiv:1805.12564, 2018. 3
35.Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip H. S. Torr. Conditional random fields as recurrent neural networks. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), 2015. 6

Citation

MLA

Choy, C., et al. “4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks”. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 3070–79, https://doi.org/10.1109/CVPR.2019.00319.

APA

Choy, C., Gwak, J., & Savarese, S. (2019). 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 3070–3079. https://doi.org/10.1109/CVPR.2019.00319

Chicago

Choy, C., J. Gwak, and S. Savarese. 2019. “4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks”. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 3070–79. https://doi.org/10.1109/CVPR.2019.00319.

Harvard

Choy, C., Gwak, J. and Savarese, S. (2019) “4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks”, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp. 3070–3079. Available at: https://doi.org/10.1109/CVPR.2019.00319.

Vancouver

1. Choy C, Gwak J, Savarese S (2019) 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 3070–3079

BibTeX

@inproceedings{Choy_2019, title={4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks}, url={http://dx.doi.org/10.1109/CVPR.2019.00319}, DOI={10.1109/cvpr.2019.00319}, booktitle={2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, publisher={IEEE}, author={Choy, Christopher and Gwak, JunYoung and Savarese, Silvio}, year={2019}, month=June, pages={3070–3079} }

Metadata:Crossref

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE

4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Generalized Sparse Convolution

Knowl 2 — Sparse Tensor Representation in Minkowski Engine

Knowl 3 — Hybrid Spatio-Temporal Convolution Kernel

Knowl 4 — Generalized Sparse Convolution Computation via Kernel Maps

Knowl 5 — GPU Sparse Tensor Quantization with Collision Handling

Knowl 6 — Trilateral Stationary Conditional Random Field (TS-CRF)

Knowl 7 — Variational Inference of TS-CRF via 7D Sparse Convolution

Knowl 8 — Sparse Tensor Pooling and Output Coordinate Generation

Knowl 9 — ScanNet 3D Semantic Segmentation Benchmark Performance

Knowl 10 — Synthia 4D Spatio-Temporal Segmentation and Noise Robustness

Knowl 11 — Stanford 3D Indoor Spaces (S3DIS) Benchmark Results

Knowl 12 — Computational Runtime Scaling of 3D vs 4D MinkowskiNet

References

Citation

BibTeX

Source Code

Access the Paper