Back

Neural Sparse Voxel Fields

Lingjie LiuJiatao GuKyaw Zaw LinTat-Seng ChuaChristian Theobalt

article2020NeurIPS1,526 citations

Proposes Neural Sparse Voxel Fields, a hybrid representation that structures implicit functions within a sparse voxel octree to render novel views more than ten times faster than NeRF while improving image quality and enabling scene editing.

geometryhigh resolution imageinferenceNeural Sparse Voxel Fieldsneural scene representationnovel view synthesis
arXiv: 2007.11571
Cover for Neural Sparse Voxel Fields

Abstract

Photo-realistic free-viewpoint rendering of real-world scenes using classical computer graphics techniques is challenging, because it requires the difficult step of capturing detailed appearance and geometry models. Recent studies have demonstrated promising results by learning scene representations that implicitly encode both geometry and appearance without 3D supervision. However, existing approaches in practice often show blurry renderings caused by the limited network capacity or the difficulty in finding accurate intersections of camera rays with the scene geometry. Synthesizing high-resolution imagery from these representations often requires time-consuming optical ray marching. In this work, we introduce Neural Sparse Voxel Fields (NSVF), a new neural scene representation for fast and high-quality free-viewpoint rendering. NSVF defines a set of voxel-bounded implicit fields organized in a sparse voxel octree to model local properties in each cell. We progressively learn the underlying voxel structures with a differentiable ray-marching operation from only a set of posed RGB images. With the sparse voxel octree structure, rendering novel views can be accelerated by skipping the voxels containing no relevant scene content. Our method is typically over 10 times faster than the state-of-the-art (namely, NeRF(Mildenhall et al., 2020)) at inference time while achieving higher quality results. Furthermore, by utilizing an explicit sparse voxel representation, our method can easily be applied to scene editing and scene composition. We also demonstrate several challenging tasks, including multi-scene learning, free-viewpoint rendering of a moving human, and large-scale scene rendering. Code and data are available at our website: this https URL.

Topics

geometryhigh resolution imageinferencergb imagesviewpointsparse approximationsobject appearance

Keywords

Neural Sparse Voxel Fieldsneural scene representationnovel view synthesisneural renderingvolume renderingvoxel octreeray-voxel intersectionNeural Radiance Fields

Table of Contents

  • 1 Introduction
  • 2 Background
  • 2.1 Neural Rendering with Implicit Fields
  • 2.2 Limitations of Existing Methods
  • 3 Neural Sparse Voxel Fields
  • 3.1 Voxel-bounded Implicit Fields
  • 3.2 Volume Rendering
  • 3.3 Learning
  • 4 Experiments
  • 4.1 Experimental Settings
  • 4.2 Results
  • 4.3 Ablation Studies
  • 5 Related Work
  • 6 Conclusion
  • 7 Broader Impact
  • References
  • A Additional Details of the Method
  • A.1 Algorithm
  • A.2 Overall Pipeline
  • B Additional Experimental Settings
  • B.1 Datasets
  • B.2 Implementation Details
  • B.3 Additional Baseline Details
  • C Additional Results
  • C.1 Per-scene breakdown
  • C.2 Additional Examples
  • C.3 Additional Analysis
  • C.4 Details for Experiments on ScanNet
  • C.5 Details for Experiments on Maria Sequence
  • C.6 Procedure for Scene Editing and Composition
  • D Limitations and Future Work

Knowls

  1. Knowl 1 — Neural Sparse Voxel Fields Representation

    model/method

    Neural Sparse Voxel Fields (NSVF) is a hybrid 3D scene representation that couples an explicit sparse voxel octree with local continuous neural implicit fields. The non-empty regions of a 3D scene are bounded by a set of sparse voxels V={V1,…,VK}\mathcal{V} = \{V_1, \dots, V_K\}. Within each voxel ViV_i, the scene geometry and appearance are modeled by a voxel-bounded implicit function:

    Fθ(p,v)=Fθi(gi(p),v)for p∈ViF_\theta(p, v) = F_\theta^i(g_i(p), v) \quad \text{for } p \in V_i

    where p∈R3p \in \mathbb{R}^3 is a spatial query point, v∈S2v \in \mathbb{S}^2 is the camera ray viewing direction, and FθiF_\theta^i is a multi-layer perceptron (MLP) with shared parameters θ\theta across all voxels that predicts the RGB color c∈R3c \in \mathbb{R}^3 and volume density σ∈R+\sigma \in \mathbb{R}^+.

    The spatial feature representation gi(p)g_i(p) at query point pp inside voxel ViV_i is computed from learnable feature vectors stored at the eight vertices p1∗,…,p8∗∈R3p_1^*, \dots, p_8^* \in \mathbb{R}^3 of ViV_i:

    gi(p)=ζ(χ(g~i(p1∗),…,g~i(p8∗)))g_i(p) = \zeta\left(\chi\left(\tilde{g}_i(p_1^*), \dots, \tilde{g}_i(p_8^*)\right)\right)

    where g~i(pk∗)∈Rd\tilde{g}_i(p_k^*) \in \mathbb{R}^d (with default dimension d=32d = 32) is the feature embedding at vertex pk∗p_k^*, χ(⋅)\chi(\cdot) denotes trilinear interpolation across the eight vertex embeddings, and ζ(⋅)\zeta(\cdot) is a frequency positional encoding function with maximum frequency L=6L = 6. Storing local geometry and appearance in distributed voxel vertex embeddings allows the shared MLP FθF_\theta to remain shallow (approximately 0.5M0.5\text{M} parameters) while capturing high-frequency scene details.

  2. Knowl 2 — Differentiable Ray Marching with Sparse Voxel Skipping and Early Termination

    algorithm

    Rendering an image from an NSVF representation is performed by casting rays through non-empty voxels while skipping empty space. For each ray defined by origin p0p_0 and direction vv, ray-voxel intersections are computed using the Axis-Aligned Bounding Box (AABB) intersection test. If the ray intersects k>0k > 0 voxels, stratified samples and voxel boundary entry and exit points are evaluated using numerical volume integration. Accumulation stops early once accumulated transparency drops below a threshold ϵ=0.01\epsilon = 0.01.

    Input: camera origin p0p_0, ray direction vv, step size τ\tau, early termination threshold ϵ\epsilon, sparse voxels V={V1,…,VK}\mathcal{V} = \{V_1, \dots, V_K\}, background color cbgc_{bg}, background max depth zmaxz_{max}, MLP parameters θ\theta
    Output: rendered color CC, expected depth ZZ, transparency AA
    Initialize A←1A \leftarrow 1, C←0C \leftarrow 0, Z←0Z \leftarrow 0
    Compute ray-voxel intersections with kk voxels, yielding sorted entry and exit depths: zt1in,zt1out,…,ztkin,ztkoutz_{t_1}^{in}, z_{t_1}^{out}, \dots, z_{t_k}^{in}, z_{t_k}^{out}, where {t1,…,tk}⊂{1,…,K}\{t_1, \dots, t_k\} \subset \{1, \dots, K\}
    if k>0k > 0 then
        Sample stratified points z1,…,zmz_1, \dots, z_m with step size τ\tau between zt1inz_{t_1}^{in} and ztkoutz_{t_k}^{out}
        Merge and sort stratified samples and voxel boundaries: z~1,…,z~2k+m←sort(z1,…,zm;zt1in,zt1out,…,ztkin,ztkout)\tilde{z}_1, \dots, \tilde{z}_{2k+m} \leftarrow \text{sort}(z_1, \dots, z_m; z_{t_1}^{in}, z_{t_1}^{out}, \dots, z_{t_k}^{in}, z_{t_k}^{out})
        for j←1j \leftarrow 1 to 2k+m−12k + m - 1 do
            z^j←z~j+z~j+12\hat{z}_j \leftarrow \frac{\tilde{z}_j + \tilde{z}_{j+1}}{2}
            Δj←z~j+1−z~j\Delta_j \leftarrow \tilde{z}_{j+1} - \tilde{z}_j
            if A>ϵA > \epsilon and Δj>0\Delta_j > 0 and p(z^j)∈Vip(\hat{z}_j) \in V_i for some i∈{t1,…,tk}i \in \{t_1, \dots, t_k\} then
                α←exp⁡(−σθ(gi(p(z^j)))⋅Δj)\alpha \leftarrow \exp(-\sigma_\theta(g_i(p(\hat{z}_j))) \cdot \Delta_j)
                c←cθ(gi(p(z^j)),v)c \leftarrow c_\theta(g_i(p(\hat{z}_j)), v)
                C←C+A⋅(1−α)⋅cC \leftarrow C + A \cdot (1 - \alpha) \cdot c
                Z←Z+A⋅(1−α)⋅z^jZ \leftarrow Z + A \cdot (1 - \alpha) \cdot \hat{z}_j
                A←A⋅αA \leftarrow A \cdot \alpha
    C←C+A⋅cbgC \leftarrow C + A \cdot c_{bg}
    Z←Z+A⋅zmaxZ \leftarrow Z + A \cdot z_{max}
    return C,Z,AC, Z, A

  3. Knowl 3 — Self-Pruning and Progressive Training Strategy

    model/method

    NSVF progressively refines scene geometry and model capacity from coarse to fine during training across 3 to 4 stages:

    1. Voxel Initialization: The scene is initialized with a uniform bounding grid of voxels enclosing the bounding volume VV, with initial voxel edge length l≈V/10003l \approx \sqrt[3]{V / 1000} (around 1,000 initial voxels). When available, point clouds or visual hulls can be voxelized directly. Initial ray step size is set to τ=l/8\tau = l / 8.

    2. Self-Pruning: Non-essential voxels are identified and pruned periodically during training (every 2,500 iterations) directly using predicted densities without external cues. A voxel Vi∈VV_i \in \mathcal{V} is pruned if:

    min⁡j=1,…,Gexp⁡(−σ(gi(pj)))>γ\min_{j=1,\dots,G} \exp(-\sigma(g_i(p_j))) > \gamma

    where {pj}j=1G\{p_j\}_{j=1}^G are G=163=4096G = 16^3 = 4096 uniformly sampled query points inside ViV_i, σ(gi(pj))\sigma(g_i(p_j)) is the network-predicted volume density at pjp_j, and γ=0.5\gamma = 0.5 is a transparency threshold.

    1. Progressive Subdivision: At preset milestones (e.g., 5k, 25k, and 75k iterations), the voxel edge length ll and step size τ\tau are halved. Each surviving voxel is subdivided into 23=82^3 = 8 sub-voxels. Vertex feature embeddings g~\tilde{g} for newly created vertices are initialized via trilinear interpolation of the eight parent voxel vertex embeddings. This increases capacity to between 10k and 100k active voxels (0.32M–3.2M0.32\text{M}\text{--}3.2\text{M} learnable embedding parameters), allowing the network to capture progressively finer details.

  4. Knowl 4 — NSVF Optimization Objective and Loss Formulations

    equation

    NSVF is trained end-to-end through differentiable volume rendering by minimizing a photometric reconstruction loss combined with a regularizer on ray transmittance over batches of camera rays RR:

    L=∑(p0,v)∈R∥C(p0,v)−C∗(p0,v)∥22+λ⋅Ω(A(p0,v))\mathcal{L} = \sum_{(p_0, v) \in R} \|C(p_0, v) - C^*(p_0, v)\|_2^2 + \lambda \cdot \Omega(A(p_0, v))

    where:

    • p0∈R3p_0 \in \mathbb{R}^3 is the camera position and v∈S2v \in \mathbb{S}^2 is the ray viewing direction.
    • C(p0,v)∈R3C(p_0, v) \in \mathbb{R}^3 is the volume-rendered ray RGB color and C∗(p0,v)∈R3C^*(p_0, v) \in \mathbb{R}^3 is the ground-truth pixel color.
    • A(p0,v)=∏i=1Nα(zi,Δi)A(p_0, v) = \prod_{i=1}^N \alpha(z_i, \Delta_i) is the total ray transmittance.
    • Ω(A)\Omega(A) is a beta-distribution regularizer that penalizes semi-transparent ray terminations to encourage sharp object boundaries, scaled by regularizer weight λ\lambda.
    • Ray batch RR uses a biased sampling strategy that only selects camera rays that intersect at least one active voxel in V\mathcal{V}.

    When ground-truth depth Z∗(p0,v)Z^*(p_0, v) is available (e.g., in RGB-D indoor scene reconstruction), an additional depth loss is optimized:

    Ldepth=∑(p0,v)∈R∣Z(p0,v)−Z∗(p0,v)∣1\mathcal{L}_{\text{depth}} = \sum_{(p_0, v) \in R} |Z(p_0, v) - Z^*(p_0, v)|_1

    where Z(p0,v)Z(p_0, v) is the expected ray termination depth.

  5. Knowl 5 — Novel View Synthesis Performance on Synthetic and Real Benchmarks

    empirical result

    NSVF was evaluated against Scene Representation Networks (SRN), Neural Volumes (NV), and Neural Radiance Fields (NeRF) across four view synthesis datasets: Synthetic-NeRF, Synthetic-NSVF, BlendedMVS, and Tanks & Temples. Performance was measured by Peak Signal-to-Noise Ratio (PSNR ↑\uparrow), Structural Similarity Index (SSIM ↑\uparrow), and Learned Perceptual Image Patch Similarity (LPIPS ↓\downarrow).

    Models Synthetic-NeRF Synthetic-NSVF BlendedMVS Tanks and Temples
    PSNR↑\uparrow SSIM↑\uparrow LPIPS↓\downarrow PSNR↑\uparrow SSIM↑\uparrow LPIPS↓\downarrow PSNR↑\uparrow SSIM↑\uparrow LPIPS↓\downarrow PSNR↑\uparrow SSIM↑\uparrow LPIPS↓\downarrow
    SRN 22.26 0.846 0.170 24.33 0.882 0.141 20.51 0.770 0.294 24.10 0.847 0.251
    NV 26.05 0.893 0.160 25.83 0.892 0.124 23.03 0.793 0.243 23.70 0.834 0.260
    NeRF 31.01 0.947 0.081 30.81 0.952 0.043 24.15 0.828 0.192 25.78 0.864 0.198
    NSVF (ϵ=0\epsilon=0) 31.75 0.954 0.048 35.18 0.979 0.015 26.89 0.898 0.114 28.48 0.901 0.155
    NSVF (ϵ=0.01\epsilon=0.01) 31.74 0.953 0.047 35.13 0.979 0.015 26.90 0.898 0.113 28.40 0.900 0.153

    NSVF outperforms all baselines on all four benchmarks. By skipping empty voxels and applying early termination (ϵ=0.01\epsilon = 0.01), NSVF renders 10×–20×10\times\text{--}20\times faster than NeRF at inference time (e.g., rendering an 800×800800 \times 800 image in 1.5–3.81.5\text{--}3.8 seconds on a single Nvidia V100 GPU compared to ∼30\sim 30 seconds for NeRF) without degradation in rendering fidelity.

  6. Knowl 6 — Multi-Scene and Dynamic Scene Modeling with NSVF

    model/method

    NSVF supports multi-scene and dynamic sequence learning under two conditioning schemes:

    1. Multi-Scene Learning with Shared MLPs: When learning multiple distinct objects simultaneously (evaluated across 10 objects: 8 from Synthetic-NeRF and 2 from Synthetic-NSVF), separate voxel embedding tables are allocated per object while sharing a single MLP for density and color prediction. On multi-scene learning, NSVF achieves 30.68 dB30.68\text{ dB} PSNR, 0.9470.947 SSIM, and 0.0430.043 LPIPS, substantially outperforming NeRF conditioned on scene latent codes via a hypernetwork (25.71 dB25.71\text{ dB} PSNR, 0.8910.891 SSIM, 0.1750.175 LPIPS).

    2. Dynamic Sequences via Hypernetworks: For dynamic sequences sharing consistent underlying geometry (such as the 200-frame Maria dataset of a moving human), all frames share a common bounding voxel octree, and a frame-indexed embedding vector is passed to a hypernetwork to predict the MLP weights for each time frame. On the Maria sequence test set, NSVF achieves 38.92 dB38.92\text{ dB} PSNR, 0.9910.991 SSIM, and 0.0100.010 LPIPS, outperforming NeRF (34.19 dB34.19\text{ dB} PSNR, 0.9800.980 SSIM, 0.0260.026 LPIPS).

  7. Knowl 7 — Direct Scene Composition and Spatial Editing via Sparse Voxels

    model/method

    Because NSVF explicitly associates neural fields with localized 3D voxels in a sparse octree, scene editing and multi-object scene composition can be executed directly on the voxel structure without retraining:

    1. Scene Editing: Users can modify learned objects by directly transforming, translating, rotating, cloning, or removing individual sparse voxels (e.g., removing structural components of a model or rearranging parts).

    2. Multi-Object Composition: Voxels representing distinct objects learned individually or from a multi-scene model can be placed together within a single shared 3D coordinate space. During volume rendering, rays traverse the union of the active voxels, evaluating the corresponding voxel-bounded implicit fields with zero additional architectural overhead.

  8. Knowl 8 — Ablations on Feature Representations, Progressive Training, and Early Termination

    empirical result

    Ablation experiments conducted on the synthetic Wineholder scene evaluate the impact of voxel embeddings, positional encoding, early termination thresholds, and progressive training stages:

    Voxel Feature Configuration PSNR↑\uparrow SSIM↑\uparrow LPIPS↓\downarrow
    Full NSVF (Embeddings + Positional Encoding) 32.04 0.965 0.020
    Without Positional Encoding (w/o POS, embedding dim d=416d=416) 30.89 0.954 0.043
    Without Voxel Embeddings (w/o EMB) 27.02 0.931 0.077
    Without Voxel Embeddings and Positional Encoding (w/o POS, EMB) 24.47 0.906 0.118
    • Feature Embeddings vs Positional Encoding: Voxel embeddings provide the primary quality boost (+5.02 dB+5.02\text{ dB} PSNR over raw coordinates). Combining vertex embeddings with positional encoding yields the best results for recovering high-frequency geometric grids.
    • Early Termination Threshold ϵ\epsilon: Setting ϵ=0.01\epsilon = 0.01 achieves 32.04 dB32.04\text{ dB} PSNR, 0.9650.965 SSIM, 0.0200.020 LPIPS, and 2.0 s/frame2.0\text{ s/frame} rendering time, compared to ϵ=0.000\epsilon = 0.000 which yields 31.93 dB31.93\text{ dB} PSNR and takes 4.0 s/frame4.0\text{ s/frame}, halving render time with negligible difference in visual quality.
    • Progressive Training: Progressive training across stages achieves 32.04 dB32.04\text{ dB} PSNR and 0.0200.020 LPIPS, whereas single-stage training directly at the finest voxel resolution reaches only 29.77 dB29.77\text{ dB} PSNR and 0.0330.033 LPIPS while requiring more memory and training time.

  9. Knowl 9 — Stated Limitations of NSVF

    limitation

    The authors identify four main limitations of the Neural Sparse Voxel Fields representation:

    1. Constant Background Model: NSVF assumes a single constant background color cbgc_{bg}, limiting its ability to model real scenes with complex, view-dependent backgrounds without interfering with foreground learning.
    2. Pruning Sensitivity on Thin Structures: Using a fixed density threshold (γ=0.5\gamma = 0.5) for self-pruning can prematurely delete voxels containing very thin structures if density has not accumulated sufficiently prior to pruning.
    3. Black-Box View-Dependent Rendering: Modeling view-dependent color without explicitly enforcing physical rendering equations (radiance conservation) can produce artifacts under sparse training views or complex optical phenomena such as glass refractions.
    4. Requirement of Accurate Camera Poses: The framework relies on known, calibrated camera poses to shoot rays and cannot handle uncalibrated image collections or severe camera registration errors.

Coverage note — None was omitted; all key contributions—including the hybrid representation, rendering algorithm, self-pruning and progressive training, losses, multi-scene/dynamic/editing applications, quantitative benchmarks, ablations, and stated limitations—are fully covered.

References

  1. 1.Kara-Ali Aliev, Dmitry Ulyanov, and Victor Lempitsky. 2019. Neural point-based graphics. arXiv preprint arXiv:1906.08240.
  2. 2.Z. Chen and H. Zhang. 2019. Learning implicit fields for generative shape modeling. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5932–5941.
  3. 3.Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 2017. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE.
  4. 4.SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. 2018. Neural scene representation and rendering. Science, 360(6394):1204–1210.
  5. 5.John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker. 2019. Deepview: View synthesis with learned gradient descent. International Conference on Computer Vision and Pattern Recognition (CVPR).
  6. 6.John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. 2016. Deepstereo: Learning to predict new views from the world’s imagery. In Computer Vision and Pattern Recognition (CVPR).
  7. 7.David Ha, Andrew Dai, and Quoc Le. 2016. Hypernetworks.
  8. 8.Eric Haines. 1989. Essential Ray Tracing Algorithms, page 33–77. Academic Press Ltd., GBR.
  9. 9.Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Brostow. 2018. Deep blending for free-viewpoint image-based rendering. ACM Trans. Graph., 37(6):257:1–257:15.
  10. 10.Yue Jiang, Dantong Ji, Zhizhong Han, and Matthias Zwicker. 2020. Sdfdiff: Differentiable rendering of signed distance fields for 3d shape optimization. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  11. 11.Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim Kehl, and Adrien Gaidon. 2020. Differentiable rendering: A survey. arXiv preprint arXiv:2006.12057.
  12. 12.Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Nießner, Patrick Pérez, Christian Richardt, Michael Zollöfer, and Christian Theobalt. 2018. Deep video portraits. ACM Transactions on Graphics (TOG), 37.
  13. 13.Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. 2017. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics, 36(4).
  14. 14.Samuli Laine and Tero Karras. 2010. Efficient sparse voxel octrees–analysis, extensions, and implementation.
  15. 15.Lingjie Liu, Weipeng Xu, Marc Habermann, Michael Zollhöfer, Florian Bernard, Hyeongwoo Kim, Wenping Wang, and Christian Theobalt. 2020. Neural human video rendering by learning dynamic textures and rendering-to-video translation. IEEE Transactions on Visualization and Computer Graphics, PP:1–1.
  16. 16.Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Hyeongwoo Kim, Florian Bernard, Marc Habermann, Wenping Wang, and Christian Theobalt. 2019a. Neural rendering and reenactment of human actor videos. ACM Transactions on Graphics (TOG).
  17. 17.Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, and Zhaopeng Cui. 2019b. Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. arXiv preprint arXiv:1911.13225.
  18. 18.Shichen Liu, Weikai Chen, Tianye Li, and Hao Li. 2019c. Soft rasterizer: Differentiable rendering for unsupervised single-view mesh reconstruction. arXiv preprint arXiv:1901.05567.
  19. 19.Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. 2019d. Learning to infer implicit surfaces without 3d supervision. In Advances in Neural Information Processing Systems, pages 8295–8306.
  20. 20.Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. 2019. Neural volumes: Learning dynamic renderable volumes from images. ACM Transactions on Graphics (TOG), 38(4):65.
  21. 21.Ricardo Martin Brualla, Peter Lincoln, Adarsh Kowdle, Christoph Rhemann, Dan Goldman, Cem Keskin, Steve Seitz, Shahram Izadi, Sean Fanello, Rohit Pandey, Shuoran Yang, Pavel Pidlypenskyi, Jonathan Taylor, Julien Valentin, Sameh Khamis, Philip Davidson, and Anastasia Tkach. 2018. Lookingood: Enhancing performance capture with real-time neural re-rendering. volume 37.
  22. 22.Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. 2019. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR).
  23. 23.Moustafa Meshry, Dan B. Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, and Ricardo Martin-Brualla. 2019. Neural rerendering in the wild. In Computer Vision and Pattern Recognition (CVPR).
  24. 24.Mateusz Michalkiewicz, Jhony K. Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson. 2019. Implicit surface representations as layers in neural networks. In The IEEE International Conference on Computer Vision (ICCV).
  25. 25.Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. 2019. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 38(4):1–14.
  26. 26.Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2020. Nerf: Representing scenes as neural radiance fields for view synthesis. arXiv preprint arXiv:2003.08934.
  27. 27.Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. 2019. Hologan: Unsupervised learning of 3d representations from natural images. In Proceedings of the IEEE International Conference on Computer Vision, pages 7588–7597.
  28. 28.Thu H Nguyen-Phuoc, Chuan Li, Stephen Balaban, and Yongliang Yang. 2018. Rendernet: A deep convolutional network for differentiable rendering from 3d shapes. In Advances in Neural Information Processing Systems (NIPS).
  29. 29.Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. 2019. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. arXiv preprint arXiv:1912.07372.
  30. 30.Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019. Deepsdf: Learning continuous signed distance functions for shape representation. International Conference on Computer Vision and Pattern Recognition (CVPR).
  31. 31.Songyou Peng, Michael Niemeyer, Lars M. Mescheder, Marc Pollefeys, and Andreas Geiger. 2020. Convolutional occupancy networks. ArXiv, abs/2003.04618.
  32. 32.Steven M Rubin and Turner Whitted. 1980. A 3-dimensional representation for fast rendering of complex scenes. In Proceedings of the 7th annual conference on Computer graphics and interactive techniques, pages 110–116.
  33. 33.Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. 2019. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2304–2314.
  34. 34.Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. 2020. Pifuhd: Multi-level pixelaligned implicit function for high-resolution 3d human digitization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
  35. 35.Harry Shum and Sing Bing Kang. 2000. Review of image-based rendering techniques. In Visual Communications and Image Processing 2000, volume 4067, pages 2 – 13. International Society for Optics and Photonics, SPIE.
  36. 36.Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Niessner, Gordon Wetzstein, and Michael Zollhofer. 2019a. Deepvoxels: Learning persistent 3d feature embeddings. In Computer Vision and Pattern Recognition (CVPR).
  37. 37.Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. 2019b. Scene representation networks: Continuous 3d-structure-aware neural scene representations. In Advances in Neural Information Processing Systems, pages 1119–1130.
  38. 38.Richard Szeliski. 2010. Computer vision: algorithms and applications. Springer Science & Business Media.
  39. 39.A. Tewari, O. Fried, J. Thies, V. Sitzmann, S. Lombardi, K. Sunkavalli, R. Martin-Brualla, T. Simon, J. Saragih, M. Nießner, R. Pandey, S. Fanello, G. Wetzstein, J.-Y. Zhu, C. Theobalt, M. Agrawala, E. Shechtman, D. B Goldman, and M. Zollhöfer. 2020. State of the Art on Neural Rendering. Computer Graphics Forum (EG STAR 2020).
  40. 40.Justus Thies, Michael Zollhöfer, and Matthias Nießner. 2019. Deferred neural rendering: image synthesis using neural textures. ACM Transactions on Graphics, 38.
  41. 41.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
  42. 42.Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. 2016. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. In Advances in neural information processing systems, pages 1696–1704.
  43. 43.Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. 2020. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. Computer Vision and Pattern Recognition (CVPR).
  44. 44.Cha Zhang and Tsuhan Chen. 2004. A survey on image-based rendering—representation, sampling and compression. Signal Processing: Image Communication, 19(1):1–28.
  45. 45.Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR.
  46. 46.Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. 2018. Stereo magnification: Learning view synthesis using multiplane images. In SIGGRAPH.

Citation

MLA
Liu, L., et al. “Neural Sparse Voxel Fields”. Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 15651–63, https://proceedings.neurips.cc/paper_files/paper/2020/file/b4b758962f17808746e9bb832a6fa4b8-Paper.pdf.
APA
Liu, L., Gu, J., Zaw Lin, K., Chua, T.-S., & Theobalt, C. (2020). Neural Sparse Voxel Fields. Advances in Neural Information Processing Systems, 33, 15651–15663. https://proceedings.neurips.cc/paper_files/paper/2020/file/b4b758962f17808746e9bb832a6fa4b8-Paper.pdf
Chicago
Liu, L., J. Gu, K. Zaw Lin, T.-S. Chua, and C. Theobalt. 2020. “Neural Sparse Voxel Fields”. Advances in Neural Information Processing Systems 33: 15651–63. https://proceedings.neurips.cc/paper_files/paper/2020/file/b4b758962f17808746e9bb832a6fa4b8-Paper.pdf.
Harvard
Liu, L. et al. (2020) “Neural Sparse Voxel Fields”, Advances in Neural Information Processing Systems. Curran Associates, Inc., pp. 15651–15663. Available at: https://proceedings.neurips.cc/paper_files/paper/2020/file/b4b758962f17808746e9bb832a6fa4b8-Paper.pdf.
Vancouver
1. Liu L, Gu J, Zaw Lin K, Chua T-S, Theobalt C (2020) Neural Sparse Voxel Fields. In: Advances in Neural Information Processing Systems. Curran Associates, Inc., pp 15651–15663

BibTeX

@inproceedings{liu2020neural,
  title = {Neural Sparse Voxel Fields},
  author = {Liu, Lingjie and Gu, Jiatao and Zaw Lin, Kyaw and Chua, Tat-Seng and Theobalt, Christian},
  year = {2020},
  booktitle = {Advances in Neural Information Processing Systems},
  publisher = {Curran Associates, Inc.},
  volume = {33},
  pages = {15651-15663},
  url = {https://proceedings.neurips.cc/paper_files/paper/2020/file/b4b758962f17808746e9bb832a6fa4b8-Paper.pdf}
}
Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: Authors