Implicit Neural Spatial Representations for Time-dependent PDEs

Honglin ChenRundi WuEitan GrinspunChangxi ZhengPeter Yichen Chen

article2023ICML52 citations

Proposes replacing traditional spatial grids with implicit neural representations evolved through classical time integrators, enabling higher accuracy, lower memory usage, and intrinsic spatial adaptivity for time-dependent physics simulations without requiring pre-generated training data.

Listen

Simulating complex physical systems governed by time-dependent partial differential equations (PDEs), such as fluid flow and solid mechanics, is critical across modern science and engineering. Classical simulation techniques rely on discrete spatial grids, meshes, or point clouds. However, these traditional spatial representations face severe trade-offs: achieving high accuracy requires fine spatial resolutions that cause memory footprints to spike, while managing memory through dynamic remeshing is algorithmically complex and computationally burdensome.

The article demonstrates that replacing classical spatial discretizations with Implicit Neural Spatial Representations (INSR)—while retaining classical, time-proven temporal integration schemes—serves as an accurate, memory-efficient, and inherently adaptive alternative for solving time-dependent physical systems. Unlike data-driven machine learning models that require extensive pre-computed training datasets from existing solvers, this method acts directly as a self-contained PDE solver by optimizing neural network weights sequentially across time steps.

To evaluate this framework, the authors performed controlled numerical experiments across three standard physical PDE benchmarks: 1D wave advection, 2D incompressible fluid dynamics, and non-linear elastodynamics involving high-speed collisions and large deformations. The approach was systematically compared against standard classical solvers (grid-based finite difference, tetrahedral finite element methods, and meshless material point methods) as well as modern physics-informed neural network (PINN) baselines under identical memory constraints and time integrators.

The key findings demonstrate significant performance advantages in spatial accuracy and storage efficiency. First, under identical memory allocations, the INSR framework achieves substantially lower error rates than traditional discrete solvers. For example, in 2D fluid vortex simulations, the proposed method achieved an error rate of 3.35e-4 using roughly 26 kilobytes of memory, whereas a grid-based solver required approximately 12 megabytes (nearly 450 times more memory) to achieve comparable precision. Second, because neural networks inherently allocate capacity to areas of high spatial detail without altering underlying data structures, the framework intrinsically resolves multiscale phenomena and sharp contact boundaries where traditional meshes suffer from artificial dissipation or severe distortion. Third, by seamlessly integrating classical time integrators such as operator splitting and variational formulations, the framework reliably simulates highly discontinuous contact events and turbulent fluid flows where continuous-time neural methods like PINNs consistently fail.

These findings suggest that implicit neural spatial parameterization provides an effective pathway to circumvent the memory bottlenecks and remeshing overheads that constrain large-scale physical simulations. However, this accuracy and memory efficiency come with a substantial computational drawback: the method requires significantly longer wall-clock runtimes due to the repeated optimization of global neural network weights at each time step. For example, a 3D solid dynamic simulation step required approximately 30 minutes with the neural solver compared to less than one minute for standard finite element analysis.

Consequently, decision-makers and technical leaders should not view pure INSR as an immediate drop-in replacement for real-time or time-sensitive engineering pipelines. Instead, the authors recommend pursuing hybrid spatial architectures that combine discrete spatial structures with neural encodings. Such hybrid representations could capture the expressiveness, adaptivity, and low memory usage of neural fields while reducing training overhead from hours to seconds, providing a practical path toward production-ready simulation tools.

While empirical tests show consistent convergence as spatial sampling density increases, the study's primary limitations include soft enforcement of boundary conditions and the absence of formal theoretical proofs regarding long-term numerical stability and convergence. Stakeholders can have high confidence in the demonstrated accuracy and memory benefits for low-dimensional systems, but should exercise caution regarding computational latency when evaluating deployment in time-critical workflows.

Columbia University Massachusetts Institute of Technology University of Toronto continuous time differential equations finite element analysis implicit neural spatial representations time-dependent PDEs neural implicit surfaces

arXiv: 2210.00124 honglin-c/INSR-PDE

Paper: Implicit Neural Representations with Periodic Activation Functions, Vincent Sitzmann et al. (2020). SIREN establishes how periodic-activation implicit neural representations encode continuous fields and their derivatives, the representation foundation for understanding INSR.
Paper: An Energy Approach to the Solution of Partial Differential Equations in Computational Mechanics via Machine Learning: Concepts, Implementation and Applications, Esteban Samaniego et al. (2019). The Deep Energy Method shows how neural networks can serve as continuous trial fields in variational mechanics solvers, a direct methodological precursor to INSR's neural PDE solving.
Paper: The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems, Weinan E et al. (2017). The Deep Ritz Method introduces neural trial functions optimized through variational PDE formulations, preparing readers for the source's use of variational time integration.

No sufficiently relevant recommendations were found.

Cover for Implicit Neural Spatial Representations for Time-dependent PDEs

Abstract

Implicit Neural Spatial Representation (INSR) has emerged as an effective representation of spatially-dependent vector fields. This work explores solving time-dependent PDEs with INSR. Classical PDE solvers introduce both temporal and spatial discretizations. Common spatial discretizations include meshes and meshless point clouds, where each degree-of-freedom corresponds to a location in space. While these explicit spatial correspondences are intuitive to model and understand, these representations are not necessarily optimal for accuracy, memory usage, or adaptivity. Keeping the classical temporal discretization unchanged (e.g., explicit/implicit Euler), we explore INSR as an alternative spatial discretization, where spatial information is implicitly stored in the neural network weights. The network weights then evolve over time via time integration. Our approach does not require any training data generated by existing solvers because our approach is the solver itself. We validate our approach on various PDEs with examples involving large elastic deformations, turbulent fluids, and multi-scale phenomena. While slower to compute than traditional representations, our approach exhibits higher accuracy and lower memory consumption. Whereas classical solvers can dynamically adapt their spatial representation only by resorting to complex remeshing algorithms, our INSR approach is intrinsically adaptive. By tapping into the rich literature of classic time integrators, e.g., operator-splitting schemes, our method enables challenging simulations in contact mechanics and turbulent flows where previous neural-physics approaches struggle. Videos and codes are available on the project page.

1. Introduction
2. Related Works
3. Method: Time Integration on Neural Spatial Representations
3.1. Neural Networks as Spatial Representations
3.2. Time integration
4. Experiments
4.1. Advection Equation
4.2. Incompressible Euler Equations
4.3. Elastodynamic Equation
5. Discussion and Conclusion
Acknowledgements
References
A. Implementation Details
A.1. Optimization
A.2. Advection Equation
A.3. Incompressible Euler Equations
A.4. Elastodynamic Equation
B. Additional Results
B.1. Elastodynamic Equation

Knowls

Knowl 1 — Neural networks provide an adaptive spatial discretization
model/method
At time step $n$ , the spatial field $f^n$ on a domain $\Omega\subseteq\mathbb{R}^m$ is represented as $f^n(x)=f_{\theta^n}(x)$ , where $x\in\Omega$ and $\theta^n$ is the weight vector of a multilayer perceptron. The experiments use SIREN networks, whose sinusoidal activations provide smooth spatial fields; spatial derivatives are computed by automatic differentiation with respect to the input coordinates. The representation is implicit: network weights, rather than values attached to a fixed set of spatial points, store the field. Thus storage for the field depends on the number of weights rather than the number of queried spatial samples. Because weights have global support, the representation can allocate capacity to detail at different spatial locations without changing a mesh, grid, or network architecture.
Knowl 2 — Time stepping is an optimization over the next field’s network weights
model/method
Let $t_n$ be the current time, $\Delta t$ the step size, and $f_{\theta^k}$ the represented field at time $t_k$ . Given the fixed fields from previous steps, the next-step weights are found by minimizing the objective associated with the chosen PDE and time integrator over spatial samples $M\subseteq\Omega$ :
$\theta^{n+1}=\arg\min_{\theta}\sum_{x\in M} I\!\left(\Delta t,\{f_{\theta^k}(x)\}_{k=0}^{n+1},\{\nabla f_{\theta^k}(x)\}_{k=0}^{n+1},\{\nabla^2 f_{\theta^k}(x)\}_{k=0}^{n+1},\ldots\right).$
Here $I$ is the time-integration objective and may involve the field and its spatial derivatives; only the new weights $\theta^{n+1}$ are optimized, while earlier weights remain fixed. The formulation accommodates classical integrators, including Euler methods, variational integrators, and operator splitting. The paper solves these optimization problems with Adam and resamples $M$ at each optimization iteration, treating the samples as a stochastic-gradient mini-batch.
Knowl 3 — Initial and boundary conditions are imposed through optimization penalties
equation
For a spatial domain $\Omega$ with boundary $\partial\Omega$ , the next-step objective can include a soft boundary penalty evaluated at samples $M_b\subseteq\partial\Omega$ :
$\theta^{n+1}=\arg\min_{\theta}\left[\sum_{x\in M\subseteq\Omega} I_x+\lambda\sum_{x_b\in M_b} C\!\left(f_{\theta}(x_b),\nabla f_{\theta}(x_b),\ldots\right)\right].$
Here $I_x$ is the time-integration objective at spatial sample $x$ , $C$ encodes the problem-specific boundary condition, and $\lambda$ weights its penalty. The initial field $\hat f^0$ is fitted by optimizing the initial network weights:
$\theta^0=\arg\min_{\theta}\sum_{x\in M\subseteq\Omega}\left\|f_{\theta}(x)-\hat f^0(x)\right\|_2^2.$
The same stochastic sampling and Adam-based optimization approach is used for these fits. The boundary conditions in the reported method are soft penalties rather than exact constraints.
Knowl 4 — Incompressible Euler flow is advanced by three neural operator-splitting solves
algorithm
For the incompressible Euler equations, the represented velocity is $u$ , pressure is represented by a separate network $p$ , and the experiments set fluid density to $1$ and external force to zero. Each time step performs three sequential optimizations, with the intermediate fields held fixed when they are not being optimized:
1. Advect the velocity using the semi-Lagrangian objective $I_{\mathrm{adv}}=\|u^{n+1}_{\mathrm{adv}}(x)-u^n(x_{\mathrm{back}})\|_2^2$ , where $x_{\mathrm{back}}=x-\Delta t\,u^n(x)$ . Neural-field evaluation directly queries the velocity at the backtracked position, so this step does not require interpolation on a grid.
2. Find the pressure by minimizing $I_{\mathrm{pro}}=\|\nabla^2p^{n+1}(x)-\nabla\cdot u^{n+1}_{\mathrm{adv}}(x)\|_2^2$ , which enforces the pressure-projection equation. The advected velocity field is fixed during this solve.
3. Correct the velocity by minimizing $I_{\mathrm{cor}}=\|u^{n+1}(x)-(u^{n+1}_{\mathrm{adv}}(x)-\nabla p^{n+1}(x))\|_2^2$ .
The velocity correction subtracts the pressure gradient from the advected velocity. For the fluid examples, solid-wall boundary conditions are incorporated as additional soft penalties in the corresponding stages.
Knowl 5 — Elasticity uses a stable Neo-Hookean energy and a variational time integrator
model/method
The solid’s deformation map is $\phi$ , its deformation gradient is $F=\nabla\phi$ , its reference density is $\rho_0$ , and its body force is $b$ . The elastodynamic equation is $\rho_0\ddot\phi=\nabla\cdot P(F)+\rho_0b$ , with first Piola–Kirchhoff stress $P=\partial\Psi/\partial F$ . The paper uses the energy density
$\Psi(F)=\frac{\lambda}{2}\operatorname{tr}^2(\Sigma-I)+\mu\bigl(\det(F)-1\bigr)^2,$
where $\lambda$ and $\mu$ are the first and second Lamé parameters, $\Sigma$ is the diagonal matrix of singular values of $F$ , and $I$ is the identity matrix. For a time step of size $\Delta t$ , the variational integrator minimizes an incremental objective containing kinetic, elastic, and external-force potential terms:
$I=\frac{1}{2}\rho_0\left\|\dot\phi^{n+1}-\dot\phi^n\right\|_2^2+\Psi(\phi^{n+1})-\rho_0 b^T\phi^{n+1},\qquad \dot\phi^{n+1}=\frac{\phi^{n+1}-\phi^n}{\Delta t}.$
The objective is evaluated over spatial samples. Positional and collision constraints can be incorporated as additional energy terms, allowing the same integration framework to handle contact.

Knowl 6 — The advection test shows lower error at matched representation memory

empirical result

The one-dimensional test advects a Gaussian field, initially centered at $x=-1.5$ with standard deviation $0.1$ , on $[-2,2]$ at constant velocity $a=0.25$ . With a midpoint time integrator and 240 time steps, error is mean absolute error against the analytical solution, averaged over time and evaluated at 500 uniformly spaced locations. The neural representation uses two hidden layers of width 20; comparisons match its spatial-storage memory, or match its error as indicated.

At the same memory, the neural representation has substantially lower error than the grid, whose solution diffuses over time. A grid can reach approximately the neural method’s error, but requires substantially more storage. The neural solve is much slower in wall-clock time.

Knowl 7 — Fluid tests demonstrate accuracy on an analytic vortex and a multiscale flow

empirical result

For the two-dimensional Taylor–Green vortex, the velocity is $u(x,y,t)=(\sin x\cos y,-\cos x\sin y)$ on $[0,2\pi]^2$ . With $\Delta t=0.05$ and 100 steps, the paper compares the neural method (three hidden layers of width 32) with a grid-based projection solver. Mean squared velocity error is measured over time against the analytical solution. The grid has resolution 48 and approximately matched representation memory.

The neural solver preserves the stationary analytic solution more accurately than the approximately memory-matched grid. A much larger grid reaches similar error at substantially higher memory cost.

A second fluid test advects two vortices of different scales for 50 steps with $\Delta t=0.05$ . Against a high-resolution grid reference, the neural operator-splitting solver captures the smaller vortex and exhibits the least kinetic-energy dissipation among the reported comparisons, including a grid solver, PINN, temporally subdivided PINN, physics-informed DeepONet, and a neural solver trained on an Euler-residual objective. The latter neural methods do not use the operator-splitting scheme; replacing the split objectives with the residual-based objective in the neural time-stepping method also produces a substantially worse result. These comparisons indicate that the choice of time integration, not only the use of an implicit neural representation, matters for this multiscale flow.

Knowl 8 — The neural representation reduces error in the elastic tension test

data/table

In a two-dimensional elastic tension test, the neural deformation field uses three hidden layers of width 68 and approximately the same storage as the compared FEM mesh. Error is the infinity norm of the $L_2$ distance from a high-resolution FEM reference; runtime is the time to convergence. The neural result has lower error, while the mesh solve is faster. The material point method also appears in the visual comparison but exhibits numerical fracture in this example.

Knowl 9 — Neural spatial fields resolve elastic contact better than matched-memory baselines

empirical result

In a two-dimensional square–circle collision test, the neural representation and FEM use the same variational integrator and collision-handling strategy, with approximately matched spatial-storage memory. The reported error is the maximum overlap distance between the square and circle at the indicated times; smaller overlap indicates better contact resolution.

The neural field conforms more closely to the collision boundary than the matched-memory mesh, which has insufficient resolution for smooth contact. A PINN with the same network structure and memory fails to reproduce the correct contact behavior even after 10,000 Adam iterations; its training loss decreases initially but does not improve with additional iterations. The authors suggest that continuously modeling time makes optimization difficult when collision introduces highly discontinuous dynamics, whereas the incremental potential used by the variational integrator is more stable.

Knowl 10 — Accuracy and adaptivity come with a substantial runtime cost; convergence is empirical
limitation
The neural method is substantially slower than conventional spatial discretizations because optimizing globally supported network weights is more expensive than updating locally supported grid values. For the three-dimensional bunny example, the paper reports about 30 minutes per neural time step, compared with less than one minute for FEM. Thus the accuracy and memory advantages do not imply a speed advantage.

The paper observes empirical convergence as the number of spatial samples used in optimization increases: quasistatic stretching results approach a reference solution as the sample count rises from $5^3$ to $50^3$ . It does not provide a theoretical convergence or stability analysis. The reported boundary treatment is soft; imposing hard boundary conditions directly on the neural representation remains unresolved.

Coverage note — Example-specific optimizer schedules, some implementation details, and supplementary three-dimensional demonstrations are omitted because they support the reported methods and results but do not add a distinct core contribution.

References

1.Alliez, P., Meyer, M., and Desbrun, M. Interactive geometry remeshing. ACM Transactions on Graphics (TOG), 21(3):347–354, 2002.
2.Ascher, U. M. and Petzold, L. R. Computer methods for ordinary differential equations and differential-algebraic equations, volume 61. Siam, 1998.
3.Batty, C., Bertails, F., and Bridson, R. A fast variational framework for accurate solid-fluid coupling. ACM Transactions on Graphics (TOG), 26(3):100–es, 2007.
4.Bouaziz, S., Martin, S., Liu, T., Kavan, L., and Pauly, M. Projective dynamics: Fusing constraint projections for fast simulation. ACM transactions on graphics (TOG), 33(4):1–11, 2014.
5.Brachet, M. E., Meiron, D. I., Orszag, S. A., Nickel, B., Morf, R. H., and Frisch, U. Small-scale structure of the taylor–green vortex. Journal of Fluid Mechanics, 130:411–452, 1983.
6.Bruna, J., Peherstorfer, B., and Vanden-Eijnden, E. Neural galerkin scheme with active learning for high-dimensional evolution equations. arXiv preprint arXiv:2203.01360, 2022.
7.Canuto, C., Hussaini, M. Y., Quarteroni, A., and Zang, T. A. Spectral methods: fundamentals in single domains. Springer Science & Business Media, 2007a.
8.Canuto, C., Hussaini, M. Y., Quarteroni, A., and Zang, T. A. Spectral methods: evolution to complex geometries and applications to fluid dynamics. Springer Science & Business Media, 2007b.
9.Chan, E. R., Lin, C. Z., Chan, M. A., Nagano, K., Pan, B., De Mello, S., Gallo, O., Guibas, L. J., Tremblay, J., Khamis, S., et al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16123–16133, 2022.
10.Chen, P. Y., Chiaramonte, M., Grinspun, E., and Carlberg, K. Model reduction for the material point method via an implicit neural representation of the deformation map. arXiv preprint arXiv:2109.12390, 2021a.
11.Chen, P. Y., Xiang, J., Cho, D. H., Pershing, G., Maia, H. T., Chiaramonte, M., Carlberg, K., and Grinspun, E. CROM: Continuous reduced-order modeling of PDEs using implicit neural representations. arXiv preprint arXiv:2206.02607, 2022.
12.Chen, Y., Liu, S., and Wang, X. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8628–8638, 2021b.
13.Chen, Z. and Zhang, H. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5939–5948, 2019.
14.Chiaramonte, M., Kiener, M., et al. Solving differential equations using neural networks. Machine Learning Project, 1, 2013.
15.Chorin, A. J. Numerical solution of the navier-stokes equations. Mathematics of computation, 22(104):745–762, 1968.
16.Chuang, P.-Y. and Barba, L. A. Experience report of physics-informed neural networks in fluid simulations: pitfalls and frustration. arXiv preprint arXiv:2205.14249, 2022.
17.Courant, R., Isaacson, E., and Rees, M. On the solution of nonlinear hyperbolic differential equations by finite differences. Communications on pure and applied mathematics, 5(3):243–255, 1952.
18.Du, Y. and Zaki, T. A. Evolutional deep neural network. Physical Review E, 104(4):045303, 2021.
19.Du, Y., Collins, K., Tenenbaum, J., and Sitzmann, V. Learning signal-agnostic manifolds of neural fields. Advances in Neural Information Processing Systems, 34:8320–8331, 2021.
20.Dupont, E., Kim, H., Eslami, S., Rezende, D., and Rosenbaum, D. From data to functa: Your data point is a function and you should treat it like one. arXiv preprint arXiv:2201.12204, 2022.
21.Fedkiw, R., Stam, J., and Jensen, H. W. Visual simulation of smoke. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pp. 15–22, 2001.
22.Gast, T. F., Schroeder, C., Stomakhin, A., Jiang, C., and Teran, J. M. Optimization integrator for large time steps. IEEE transactions on visualization and computer graphics, 21(10):1103–1115, 2015.
23.Gonzalez, O. and Stuart, A. M. A first course in continuum mechanics, volume 42. Cambridge University Press, 2008.
24.Hu, Y., Li, T.-M., Anderson, L., Ragan-Kelley, J., and Durand, F. Taichi: a language for high-performance computation on spatially sparse data structures. ACM Transactions on Graphics (TOG), 38(6):1–16, 2019.
25.Hughes, T. J. The finite element method: linear static and dynamic finite element analysis. Courier Corporation, 2012.
26.Jiang, C., Schroeder, C., Teran, J., Stomakhin, A., and Selle, A. The material point method for simulating continuum materials. In ACM SIGGRAPH 2016 Courses, SIGGRAPH ’16. Association for Computing Machinery, 2016.
27.Kane, C., Marsden, J. E., Ortiz, M., and West, M. Variational integrators and the newmark algorithm for conservative and dissipative mechanical systems. International Journal for Numerical Methods in Engineering, 49(10):1295–1325, 2000a.
28.Kane, C., Marsden, J. E., Ortiz, M., and West, M. Variational integrators and the newmark algorithm for conservative and dissipative mechanical systems. International Journal for numerical methods in engineering, 49(10):1295–1325, 2000b.
29.Kharevych, L., Wei, W., Tong, Y., Kanso, E., Marsden, J. E., Schroder, P., and Desbrun, M. Geometric, variational integrators for computer animation. Eurographics Association, 2006.
30.Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
31.Krishnapriyan, A., Gholami, A., Zhe, S., Kirby, R., and Mahoney, M. W. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems, 34:26548–26560, 2021.
32.Lantz, R. Quantitative evaluation of numerical diffusion (truncation error). Society of Petroleum Engineers Journal, 11(03):315–320, 1971.
33.Levin, D. I. Bartels: A lightweight collection of routines for physics simulation, 2020. https://github.com/dilevin/Bartels.
34.Li, M., Ferguson, Z., Schneider, T., Langlois, T. R., Zorin, D., Panozzo, D., Jiang, C., and Kaufman, D. M. Incremental potential contact: intersection-and inversion-free, large-deformation dynamics. ACM Trans. Graph., 39(4):49, 2020a.
35.Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., and Anandkumar, A. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020b.
36.Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., and Anandkumar, A. Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003.03485, 2020c.
37.Lu, L., Jin, P., and Karniadakis, G. E. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03193, 2019.
38.Lu, L., Pestourie, R., Yao, W., Wang, Z., Verdugo, F., and Johnson, S. G. Physics-informed neural networks with hard constraints for inverse design. SIAM Journal on Scientific Computing, 43(6):B1105–B1132, 2021.
39.Martel, J. N., Lindell, D. B., Lin, C. Z., Chan, E. R., Monteiro, M., and Wetzstein, G. Acorn: Adaptive coordinate networks for neural scene representation. arXiv preprint arXiv:2105.02788, 2021.
40.McAdams, A., Zhu, Y., Selle, A., Empey, M., Tamstorf, R., Teran, J., and Sifakis, E. Efficient elasticity for character skinning with contact and collisions. In ACM SIGGRAPH 2011 papers, pp. 1–12. 2011.
41.Mehta, I., Chandraker, M., and Ramamoorthi, R. A level set theory for neural implicit evolution under explicit flows. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part II, pp. 711–729. Springer, 2022.
42.Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., and Geiger, A. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4460–4470, 2019.
43.Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pp. 405–421. Springer, 2020.
44.Mullen, P., Crane, K., Pavlov, D., Tong, Y., and Desbrun, M. Energy-preserving integrators for fluid animation. ACM Transactions on Graphics (TOG), 28(3):1–8, 2009.
45.Muller, M., Chentanez, N., Kim, T.-Y., and Macklin, M. Air meshes for robust collision handling. ACM Transactions on Graphics (TOG), 34(4):1–9, 2015.
46.Muller, T., Evans, A., Schied, C., and Keller, A. Instant neural graphics primitives with a multiresolution hash encoding. arXiv preprint arXiv:2201.05989, 2022.
47.Museth, K. Vdb: High-resolution sparse volumes with dynamic topology. ACM transactions on graphics (TOG), 32(3):1–22, 2013.
48.Narain, R., Samii, A., and O’brien, J. F. Adaptive anisotropic remeshing for cloth simulation. ACM transactions on graphics (TOG), 31(6):1–10, 2012.
49.Pan, S., Brunton, S. L., and Kutz, J. N. Neural implicit flow: a mesh-agnostic dimensionality reduction paradigm of spatio-temporal data. arXiv preprint arXiv:2204.03216, 2022.
50.Park, J. J., Florence, P., Straub, J., Newcombe, R., and Lovegrove, S. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 165–174, 2019.
51.Park, K., Sinha, U., Barron, J. T., Bouaziz, S., Goldman, D. B., Seitz, S. M., and Martin-Brualla, R. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5865–5874, 2021a.
52.Park, K., Sinha, U., Hedman, P., Barron, J. T., Bouaziz, S., Goldman, D. B., Martin-Brualla, R., and Seitz, S. M. Hypernerf: a higher-dimensional representation for topologically varying neural radiance fields. ACM Transactions on Graphics (TOG), 40(6):1–12, 2021b.
53.Pfaff, T., Fortunato, M., Sanchez-Gonzalez, A., and Battaglia, P. W. Learning mesh-based simulation with graph networks. arXiv preprint arXiv:2010.03409, 2020.
54.Raissi, M., Perdikaris, P., and Karniadakis, G. E. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019.
55.Reddy, J. N. Introduction to the Finite Element Method. McGraw-Hill Education, New York, 4th edition edition, 2019.
56.Roache, P. J. Fundamentals of computational fluid dynamics. Hermosa Publishers, 1998.
57.Sadeghirad, A., Brannon, R. M., and Burghardt, J. A convected particle domain interpolation technique to extend applicability of the material point method for problems involving massive deformations. International Journal for numerical methods in Engineering, 86(12):1435–1456, 2011.
58.Sanchez-Gonzalez, A., Godwin, J., Pfaff, T., Ying, R., Leskovec, J., and Battaglia, P. Learning to simulate complex physics with graph networks. In International Conference on Machine Learning, pp. 8459–8468. PMLR, 2020.
59.Schwarz, K., Liao, Y., Niemeyer, M., and Geiger, A. Graf: Generative radiance fields for 3d-aware image synthesis. Advances in Neural Information Processing Systems, 33:20154–20166, 2020.
60.Selle, A., Fedkiw, R., Kim, B., Liu, Y., and Rossignac, J. An unconditionally stable maccormack method. Journal of Scientific Computing, 35(2):350–371, 2008.
61.Setaluri, R., Aanjaneya, M., Bauer, S., and Sifakis, E. Spgrid: A sparse paged grid structure applied to adaptive smoke simulation. ACM Transactions on Graphics (TOG), 33(6):1–12, 2014.
62.Shaham, T. R., Gharbi, M., Zhang, R., Shechtman, E., and Michaeli, T. Spatially-adaptive pixelwise networks for fast image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14882–14891, 2021.
63.Sharp, N. and Jacobson, A. Spelunking the deep: guaranteed queries on general neural implicit surfaces via range analysis. ACM Transactions on Graphics (TOG), 41(4):1–16, 2022.
64.Sitzmann, V., Martel, J., Bergman, A., Lindell, D., and Wetzstein, G. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
65.Smith, B., Goes, F. D., and Kim, T. Stable neo-hookean flesh simulation. ACM Trans. Graph., mar 2018.
66.Sorkine, O. and Alexa, M. As-rigid-as-possible surface modeling. In Symposium on Geometry processing, volume 4, pp. 109–116, 2007.
67.Stam, J. Stable fluids. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pp. 121–128, 1999.
68.Staniforth, A. and Cotˆ e, J. Semi-lagrangian integration schemes for atmospheric models—a review. Monthly weather review, 119(9):2206–2223, 1991.
69.Takikawa, T., Litalien, J., Yin, K., Kreis, K., Loop, C., Nowrouzezahrai, D., Jacobson, A., McGuire, M., and Fidler, S. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11358–11367, 2021.
70.Taylor, G. I. and Green, A. E. Mechanism of the production of small eddies from large ones. Proceedings of the Royal Society of London. Series A-Mathematical and Physical Sciences, 158(895):499–521, 1937.
71.Wandel, N., Weinmann, M., and Klein, R. Learning incompressible fluid dynamics from scratch–towards fast, differentiable fluid models that generalize. arXiv preprint arXiv:2006.08762, 2020.
72.Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., and Wang, W. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Advances in Neural Information Processing Systems, 34:27171–27183, 2021a.
73.Wang, S. and Perdikaris, P. Long-time integration of parametric evolution equations with physics-informed deeponets. arXiv preprint arXiv:2106.05384, 2021.
74.Wang, S., Wang, H., and Perdikaris, P. Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Science advances, 7(40):eabi8605, 2021b.
75.Wessels, H., Weißenfels, C., and Wriggers, P. The neural particle method–an updated lagrangian physics informed neural network for computational fluid dynamics. Computer Methods in Applied Mechanics and Engineering, 368:113127, 2020.
76.Wu, R. and Zheng, C. Learning to generate 3d shapes from a single example. ACM Transactions on Graphics (TOG), 41(6):1–19, 2022.
77.Xian, Z., Tong, X., and Liu, T. A scalable galerkin multigrid method for real-time simulation of deformable objects. ACM Trans. Graph., 38(6), nov 2019.
78.Xie, Y., Takikawa, T., Saito, S., Litany, O., Yan, S., Khan, N., Tombari, F., Tompkin, J., Sitzmann, V., and Sridhar, S. Neural fields in visual computing and beyond. arXiv preprint arXiv:2111.11426, 2021.
79.Yang, G., Belongie, S., Hariharan, B., and Koltun, V. Geometry processing with neural fields. Advances in Neural Information Processing Systems, 34, 2021.
80.Yariv, L., Kasten, Y., Moran, D., Galun, M., Atzmon, M., Ronen, B., and Lipman, Y. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems, 33:2492–2502, 2020.
81.Zehnder, J., Li, Y., Coros, S., and Thomaszewski, B. Ntopo: Mesh-free topology optimization using implicit neural representations. Advances in Neural Information Processing Systems, 34:10368–10381, 2021.

Citation

MLA

Chen, H., et al. “Implicit Neural Spatial Representations for Time-dependent PDEs”. International Conference on Machine Learning, vol. 202, 2023, pp. 5162–77, https://proceedings.mlr.press/v202/chen23af.html.

APA

Chen, H., Wu, R., Grinspun, E., Zheng, C., & Chen, P. Y. (2023). Implicit Neural Spatial Representations for Time-dependent PDEs. International Conference on Machine Learning, 202, 5162–5177. https://proceedings.mlr.press/v202/chen23af.html

Chicago

Chen, H., R. Wu, E. Grinspun, C. Zheng, and P. Y. Chen. 2023. “Implicit Neural Spatial Representations for Time-dependent PDEs”. International Conference on Machine Learning 202: 5162–77. https://proceedings.mlr.press/v202/chen23af.html.

Harvard

Chen, H. et al. (2023) “Implicit Neural Spatial Representations for Time-dependent PDEs”, International Conference on Machine Learning. PMLR, pp. 5162–5177. Available at: https://proceedings.mlr.press/v202/chen23af.html.

Vancouver

1. Chen H, Wu R, Grinspun E, Zheng C, Chen PY (2023) Implicit Neural Spatial Representations for Time-dependent PDEs. In: International Conference on Machine Learning. PMLR, pp 5162–5177

BibTeX

@InProceedings{pmlr-v202-chen23af,
  title = 	 {Implicit Neural Spatial Representations for Time-dependent {PDE}s},
  author =       {Chen, Honglin and Wu, Rundi and Grinspun, Eitan and Zheng, Changxi and Chen, Peter Yichen},
  booktitle = 	 {Proceedings of the 40th International Conference on Machine Learning},
  pages = 	 {5162--5177},
  year = 	 {2023},
  editor = 	 {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan},
  volume = 	 {202},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {23--29 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v202/chen23af/chen23af.pdf},
  url = 	 {https://proceedings.mlr.press/v202/chen23af.html},
  abstract = 	 {Implicit Neural Spatial Representation (INSR) has emerged as an effective representation of spatially-dependent vector fields. This work explores solving time-dependent PDEs with INSR. Classical PDE solvers introduce both temporal and spatial discretizations. Common spatial discretizations include meshes and meshless point clouds, where each degree-of-freedom corresponds to a location in space. While these explicit spatial correspondences are intuitive to model and understand, these representations are not necessarily optimal for accuracy, memory usage, or adaptivity. Keeping the classical temporal discretization unchanged (e.g., explicit/implicit Euler), we explore INSR as an alternative spatial discretization, where spatial information is implicitly stored in the neural network weights. The network weights then evolve over time via time integration. Our approach does not require any training data generated by existing solvers because our approach is the solver itself. We validate our approach on various PDEs with examples involving large elastic deformations, turbulent fluids, and multi-scale phenomena. While slower to compute than traditional representations, our approach exhibits higher accuracy and lower memory consumption. Whereas classical solvers can dynamically adapt their spatial representation only by resorting to complex remeshing algorithms, our INSR approach is intrinsically adaptive. By tapping into the rich literature of classic time integrators, e.g., operator-splitting schemes, our method enables challenging simulations in contact mechanics and turbulent flows where previous neural-physics approaches struggle. Videos and codes are available on the project page: http://www.cs.columbia.edu/cg/INSR-PDE/}
}

Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Method	Error	Time	Memory
Ours	0.0030	5.33h	3.520KB
Grid (same memory)	0.0146	1.13s	3.520KB
Grid (same error)	0.0029	1.80s	27.35KB

Method	Error	Time	Memory
Ours	3.35e-4	14.02h	25.887KB
Grid (same memory)	4.83e-3	2.91s	27.00KB
Grid (same error)	3.24e-4	189.4s	12.00MB

Method	Error	Time	Memory
Ours	8.82e-2	38.33m	56.32KB
Mesh-based FEM	1.99e-1	22.04s	54.00KB

Method	Error at 0.2s	Error at 0.6s	Error at 0.8s	Time	Memory
Ours	1.62e-2	1.04e-2	1.06e-2	23.0m	56.32KB
Mesh-based FEM	3.45e-2	5.47e-2	4.23e-2	98.2s	54.00KB

Implicit Neural Spatial Representations for Time-dependent PDEs

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Neural networks provide an adaptive spatial discretization

Knowl 2 — Time stepping is an optimization over the next field’s network weights

Knowl 3 — Initial and boundary conditions are imposed through optimization penalties

Knowl 4 — Incompressible Euler flow is advanced by three neural operator-splitting solves

Knowl 5 — Elasticity uses a stable Neo-Hookean energy and a variational time integrator

Knowl 6 — The advection test shows lower error at matched representation memory

Knowl 7 — Fluid tests demonstrate accuracy on an analytic vortex and a multiscale flow

Knowl 8 — The neural representation reduces error in the elastic tension test

Knowl 9 — Neural spatial fields resolve elastic contact better than matched-memory baselines

Knowl 10 — Accuracy and adaptivity come with a substantial runtime cost; convergence is empirical

References

Citation

BibTeX

Source Code

Access the Paper