LumiMotion: Improving Gaussian Relighting with Scene Dynamics

In 3D computer vision and digital content creation, inverse rendering—the task of estimating 3D geometry, material properties, and lighting from video or images—is critical for realistic scene editing, visual effects, and gaming. Existing methods based on point-based 3D reconstructions primarily target static scenes and struggle to separate intrinsic surface colors from cast shadows and lighting conditions. Consequently, static methods frequently "bake" shadows and lighting artifacts directly into surface textures, preventing accurate relighting of scenes under novel illumination.

To address this limitation, the article presents LumiMotion, an inverse rendering framework that uses scene motion as a supervisory signal to disentangle intrinsic material properties from environmental lighting in dynamic scenes without requiring prior knowledge of lighting conditions or object types.

The framework operates in two distinct stages using flat, disc-like 2D Gaussian primitives. In the first stage, the system jointly learns canonical scene geometry and a neural deformation network that tracks object motion and temporal color variations while applying novel regularization constraints to separate static and dynamic regions. In the second stage, geometry and motion parameters are frozen, and the system optimizes material properties—specifically diffuse surface color and roughness—alongside an unknown global environment lighting map using ray tracing. The authors evaluated the approach across real-world multi-view video datasets and a newly constructed synthetic benchmark containing five scenes across four diverse lighting environments in both static and dynamic settings.

The experimental evaluation demonstrated three primary findings. First, LumiMotion improved perceptual image quality for intrinsic material color estimation by 23% and relighting by 15% compared to the best-performing baseline method. Second, the framework achieved at least a two-fold reduction in roughness estimation error relative to existing techniques, recovering cleaner specular and diffuse material properties. Third, baseline static models exhibited severe quality drops (ranging from 16% to nearly 49% in peak signal-to-noise ratio) when evaluated under novel lighting environments due to overfitting, whereas LumiMotion maintained consistent performance across both training and test illuminations.

These results indicate that motion provides vital photometric cues across varying surface orientations and moving shadows, substantially improving the reliability of material and illumination estimation. By operating without category-specific human body templates or controlled lighting rigs, the method broadens the applicability of dynamic neural rendering for practical virtual production, gaming pipelines, and augmented reality.

Practitioners looking to deploy inverse rendering pipelines for relightable asset generation should leverage dynamic captures over static captures when moving shadows and lighting variations are present. To benchmark these tasks systematically, teams should adopt the authors' publicly available dynamic relighting dataset and open-source implementation. Future development should focus on integrating specialized motion supervision, such as optical flow, to handle intricate surface contacts and fine dynamic details, while also addressing pipeline sensitivities to sparse camera setups and camera calibration errors.