Nerfies: Deformable Neural Radiance Fields

High-quality three-dimensional scanning of humans has traditionally required expensive specialized studio laboratories with multiple synchronized cameras and controlled lighting rigs. Standard reconstruction algorithms fail when applied to casual hand-held mobile phone videos because subjects inevitably move, and fine features such as hair, glasses, and jewelry break core assumptions about rigid geometry. The article addresses this accessibility barrier by introducing a method to generate photorealistic, deformable three-dimensional models—termed "nerfies"—from casual mobile phone videos without requiring domain-specific templates or specialized multi-camera hardware.

The main objective of the article is to demonstrate and evaluate a system that reconstructs non-rigidly deforming scenes by extending Neural Radiance Fields (NeRF) with an optimized continuous volumetric deformation field. To achieve this, the authors decompose a dynamic scene into a canonical static three-dimensional template and a per-observation deformation field modeled by a neural network. To prevent distortions and stabilize training, they introduce three core innovations: a rigid motion parameterization, an elastic energy regularization that penalizes deviations from local rigidity, and a coarse-to-fine optimization strategy that gradually introduces high-frequency details. The method was evaluated using monocular RGB image sequences captured on mobile phones, with a dual-phone synchronized rig used to collect ground-truth novel viewpoints for benchmarking.

The core evaluation demonstrates several key findings. First, the proposed approach consistently outperforms baseline methods in perceptual visual quality across both quasi-static and dynamic scenes, achieving substantially better perceptual error scores on unseen viewpoints. Second, the elastic regularization successfully eliminates geometric distortion artifacts in under-constrained capture scenarios, such as when a user sweeps the camera predominantly across one side of their face. Third, the coarse-to-fine optimization scheme proves essential for resolving complex motions; it avoids suboptimal registrations while capturing subtle expressions like smiles alongside large head turns. Finally, the framework generalizes beyond human portraits to dynamic objects and full-body scans without modifying the underlying architecture.

These findings indicate that accessible, consumer-grade hardware can replace specialized laboratory rigs for high-fidelity three-dimensional human modeling, significantly reducing capture costs and technical friction. For product teams and technical leaders, this approach unlocks practical pipelines for virtual avatars, telepresence, and interactive media directly from standard smartphone cameras. Furthermore, the volumetric elastic prior and coarse-to-fine annealing establish an effective blueprint for regularizing under-constrained coordinate-based neural representations.

Organizations seeking to implement this technology should pilot casual video capture workflows while accounting for current computational requirements, which involve intensive training times across multiple high-performance graphics processors. Next development efforts should focus on optimizing training and inference speeds, handling topological changes like opening and closing mouths, and developing robust handling for rapid subject motions and orientation flips.