Back

Signal Processing for Implicit Neural Representations

Dejia XuPeihao WangYifan JiangZhiwen FanZhangyang Wang

article2022NeurIPS54 citations

Develops an implicit signal processing framework that uses analytical high-order differential operators to perform continuous convolutions, low-level filtering, and high-level classification directly on coordinate-based neural representations without requiring explicit decoding.

Listen

Implicit Neural Representations (INRs) have emerged as a powerful way to represent continuous multimedia data, such as images, video, and 3D shapes, using multi-layer neural networks. Unlike standard digital representations that store values on discrete pixel or voxel grids, INRs model data as continuous functions. However, editing and processing these neural signals directly has remained an intractable problem because their parameters are difficult to interpret. Current solutions typically decode the neural representations back into discrete grids before editing them, which destroys their continuous nature, consumes significant memory, and limits resolution.

The main objective of the article is to develop and evaluate a mathematical framework and architecture that can directly modify and process implicit neural representations without converting them into discrete grids. The authors introduce the Implicit Neural Signal Processing Network (INSP-Net) along with its convolutional extension (INSP-ConvNet), establishing that continuous signal transformations can be executed analytically via differential operators.

The authors based their approach on the mathematical insight that spatial gradients of neural networks can be calculated in closed form and naturally preserve translation invariance. They proved theoretically that any continuous convolution filter can be uniformly approximated by a linear combination of high-order differential operators. INSP-Net builds computational graphs of an INR's spatial derivatives and merges them through an inception fusion block, where operator weights can be either hand-crafted or trained on data. To validate this framework, the authors conducted experiments across low-level 2D image tasks (edge detection, blurring, deblurring, denoising, and inpainting), 3D geometry smoothing using signed distance functions, and high-level image classification on benchmark datasets.

The experimental and theoretical findings highlight several key capabilities of the proposed system. First, the mathematical proofs confirm that INSP-Net can approximate arbitrary continuous convolution filters while inherently preserving shift invariance. Second, in low-level image processing, INSP-Net performed competitively with or superior to established baselines; for example, in image denoising, it achieved 24.02 dB PSNR compared to 20.51 dB from a dedicated restoration model, and successfully executed text removal and patch inpainting without explicit masks. Third, the framework effectively smoothed complex 3D surfaces by applying continuous filter operations directly to geometric distance fields. Finally, when evaluating high-level vision, INSP-ConvNet classified implicit representations directly, reaching 88.1% accuracy on MNIST and 62.5% on CIFAR-10, slightly outperforming standard convolutional networks operating on explicit pixel grids (87.6% and 59.5%, respectively) and vastly surpassing direct parameter classifiers (which achieved around 10%).

These findings demonstrate that digital signal processing and deep convolutional operations can occur entirely within continuous implicit domains, eliminating the need to rasterize data into fixed resolutions. This capability helps preserve fine details, avoids resolution-dependent memory overhead, and opens the door to end-to-end processing pipelines operating natively on coordinate-based neural representations.

Organizations developing implicit representation pipelines should consider adopting derivative-based operators to perform filtering, transformation, and semantic extraction directly on continuous models. For future development, engineering efforts should focus on optimizing the numerical stability and memory efficiency of computing high-order derivatives, as well as designing more compact parameterizations to scale multi-layer implicit networks to larger, real-time vision workflows.

While the theoretical results guarantee universal approximation of convolutions, they rely on infinite-sequence approximations, and real-world implementations must truncate derivative orders. Additionally, current experiments rely on per-scene network optimization, and high-order automatic differentiation creates memory and stability bottlenecks that limit deeper cascading. Readers can place high confidence in the proof-of-concept demonstrations across standard vision benchmarks, though scaling to production workloads will require further advances in derivative computation efficiency.

University of Texas at Austindigital signal processingimplicit neural representationsimplicit neural signal processingneural fields
arXiv: 2210.08772
Cover for Signal Processing for Implicit Neural Representations

Abstract

Implicit Neural Representations (INRs) encoding continuous multi-media data via multi-layer perceptrons has shown undebatable promise in various computer vision tasks. Despite many successful applications, editing and processing an INR remains intractable as signals are represented by latent parameters of a neural network. Existing works manipulate such continuous representations via processing on their discretized instance, which breaks down the compactness and continuous nature of INR. In this work, we present a pilot study on the question: how to directly modify an INR without explicit decoding? We answer this question by proposing an implicit neural signal processing network, dubbed INSP-Net, via differential operators on INR. Our key insight is that spatial gradients of neural networks can be computed analytically and are invariant to translation, while mathematically we show that any continuous convolution filter can be uniformly approximated by a linear combination of high-order differential operators. With these two knobs, INSP-Net instantiates the signal processing operator as a weighted composition of computational graphs corresponding to the high-order derivatives of INRs, where the weighting parameters can be data-driven learned. Based on our proposed INSP-Net, we further build the first Convolutional Neural Network (CNN) that implicitly runs on INRs, named INSP-ConvNet. Our experiments validate the expressiveness of INSP-Net and INSP-ConvNet in fitting low-level image and geometry processing kernels (e.g. blurring, deblurring, denoising, inpainting, and smoothening) as well as for high-level tasks on implicit fields such as image classification.

Topics

digital signal processing

Keywords

implicit neural representationsimplicit neural signal processingneural fieldssignal processing operatorsfirst-order derivative networkimage processing kernelsview synthesisimage denoisingspatial gradients

Table of Contents

  • 1 Introduction
  • 2 Preliminaries: Implicit Neural Representation
  • 3 Implicit Representation Processing via Differential Operators
  • 3.1 Computational Paradigm
  • 3.2 Theoretical Analysis
  • 3.3 Building CNNs for Implicit Neural Representations
  • 4 Related Work
  • 4.1 Implicit Neural Representation
  • 4.2 Editable Implicit Fields
  • 4.3 PDE Based Image Processing
  • 5 Experiments
  • 5.1 Low-Level Vision for Implicit Neural Images
  • 5.2 Geometry Processing on Signed Distance Function
  • 5.3 Classification on Implicit Neural Representations
  • 6 Conclusion
  • Acknowledgement
  • References

Knowls

  1. Knowl 1 — Implicit neural signal processing operator

    model/method

    INSP-Net processes a differentiable implicit neural representation (INR) directly, without rasterizing or explicitly decoding it. Let Phi:\mathbb{R}^m\to\mathbb{R} be an INR, let x∈Rmx\in\mathbb{R}^m be a spatial or temporal coordinate, and let ∇kΦ(x)\nabla^k\Phi(x) denote the flattened vector of all distinct kkth-order partial derivatives of Φ\Phi at xx. An INSP-Net defines a processed INR Ψ=AΦ\Psi=A\Phi by

    Ψ(x)=AΦ(x)=Π ⁣(Φ(x),∇Φ(x),∇2Φ(x),…,∇KΦ(x)),\Psi(x)=A\Phi(x)=\Pi\!\left(\Phi(x),\nabla\Phi(x),\nabla^2\Phi(x),\ldots,\nabla^K\Phi(x)\right),

    where KK is the highest derivative order and Π:RM→R\Pi:\mathbb{R}^M\to\mathbb{R} is a continuous fusion function. The number of scalar derivative features is

    M=∑k=0K(k+m−1k)=(K+mK).M=\sum_{k=0}^{K}\binom{k+m-1}{k}=\binom{K+m}{K}.

    The fusion function Π\Pi may be hand-designed or represented by a trainable multilayer perceptron. Because the derivatives are evaluated analytically from the INR, INSP-Net produces another continuous INR and preserves the continuous representation during processing.

  2. Knowl 2 — Weight-sharing derivative-network architecture

    model/method

    An INSP-Net is implemented as an inception-like computation graph with one branch for the original INR and additional branches for its first-, second-, and higher-order derivative functions. If the input INR is an MLP, differentiating its computation graph produces derivative subnetworks that reuse the original layer weights and biases; only the activation-related derivative operations change. The branch outputs are concatenated and passed to the fusion function Π\Pi.

    The INR parameters are loaded into the branchy part of INSP-Net and represent the signal being processed. The trainable operator parameters are located in the fusion block, so an operator can be learned end-to-end while the input INR remains fixed. The authors construct the derivative graphs with automatic differentiation in PyTorch and use SIREN networks as the base INR architecture to make high-order derivatives expressive. Processing therefore consists of reorganizing analytically computed derivative graphs rather than modifying or interpreting the stored INR weights.

  3. Knowl 3 — Translation and rotation invariance of INSP-Net

    theoretical result

    Let AA be the INSP-Net operator induced by a continuous fusion function Π\Pi, and let Φ:Rm→R\Phi:\mathbb{R}^m\to\mathbb{R}. For a translation v∈Rmv\in\mathbb{R}^m, define (TvΦ)(x)=Φ(x+v)(T_v\Phi)(x)=\Phi(x+v). For a rotation matrix R∈SO(m)R\in SO(m), define (RΦ)(x)=Φ(Rx)(R\Phi)(x)=\Phi(Rx). The paper proves that the INSP-Net operator satisfies

    A[TvΦ](x)=(AΦ)(x+v)A[T_v\Phi](x)=(A\Phi)(x+v)

    for every translation vv and every continuous fusion function Π\Pi; thus, differential-operator processing is inherently translation-equivariant in the paper's terminology of shift invariance. The paper also proves rotation invariance when the fusion function has the form Π(y)=f(∥y∥22)\Pi(y)=f(\lVert y\rVert_2^2) for a scalar function ff, where y∈RMy\in\mathbb{R}^M is the complete vector of INR and derivative features. General fusion functions do not guarantee rotation invariance, but isotropically pooling the squared directional derivatives through this form provides a sufficient construction.

  4. Knowl 4 — Universal approximation of continuous convolution by derivatives

    theoretical result

    For a real-valued convolution kernel g:Rm→Rg:\mathbb{R}^m\to\mathbb{R}, the paper proves that there is a polynomial pp with real coefficients in the commuting partial-derivative operators such that the differential operator p(∇)p(\nabla) uniformly approximates the continuous convolution operator g∗Φg*\Phi to arbitrary accuracy for real-valued signals Φ\Phi. In other words, for any desired uniform accuracy, a sufficiently high-order linear combination of derivatives can approximate

    p(∇)Φ ≈ g∗Φ.p(\nabla)\Phi\ \approx\ g*\Phi.

    The operator p(∇)p(\nabla) is a special case of INSP-Net in which the fusion function is linear, because each monomial in the polynomial corresponds to a distinct higher-order partial derivative. The paper further states that a neural-network fusion function Πθ\Pi_\theta can approximate the required polynomial, so an INSP-Net can approximate any such continuous convolution. Since Πθ\Pi_\theta may also be nonlinear, the INSP-Net function class is not restricted to linear convolutional filters.

  5. Knowl 5 — INSP-Conv and convolutional networks on INRs

    model/method

    INSP-Conv is the linear specialization of INSP-Net used to build implicit convolutional networks. For an INR Φ:Rm→R\Phi:\mathbb{R}^m\to\mathbb{R} and derivative order KK, it applies a learned linear combination of the original signal and its derivatives:

    INSPConv⁡θ[Φ](x)=θ0Φ(x)+∑k=1KθkT∇kΦ(x),\operatorname{INSPConv}_{\theta}[\Phi](x)=\theta_0\Phi(x)+\sum_{k=1}^{K}\theta_k^{\mathsf T}\nabla^k\Phi(x),

    where θ0∈R\theta_0\in\mathbb{R} and θk∈R(k+m−1k)\theta_k\in\mathbb{R}^{\binom{k+m-1}{k}} match the dimensions of the flattened derivative vectors. This operator is linear and shift-invariant while it is being learned. Repeated INSP-Conv layers interleaved with elementwise nonlinearities form INSP-ConvNet: for LL layers, the initial function is F(0)=ΦF^{(0)}=\Phi, and each layer computes

    F(ℓ)=σℓ ⁣(INSPConv⁡θ(ℓ)[F(ℓ−1)]),ℓ=1,…,L,F^{(\ell)}=\sigma_\ell\!\left(\operatorname{INSPConv}_{\theta^{(\ell)}}[F^{(\ell-1)}]\right),\qquad \ell=1,\ldots,L,

    where each σℓ\sigma_\ell is an elementwise activation. The resulting CNN operates on the INR continuously and analytically, rather than first decoding the INR onto an image or voxel grid.

  6. Knowl 6 — Low-level implicit image-processing experiment

    experimental setup

    For low-level vision, the authors fit natural images from the Set5, Set14, and DIV2K datasets with SIREN-based INRs. The INSP-Net training set contains 90 fitted INRs. The trained operators are evaluated on edge detection, denoising, blurring, deblurring, and inpainting; output INRs are decoded only for visualization and image-quality evaluation.

    For edge detection, the first-order derivative coefficient is set to one and the other coefficients are set to zero, and the resulting implicit edge maps are compared visually with Sobel, Canny, and Prewitt methods. For denoising, additive Gaussian noise is applied to the input images and mean and median filters plus MPRNet are used as comparisons. For blurring, the target is generated by a Gaussian filter and results are compared with 3×33\times3 box and Gaussian filters. For deblurring, Gaussian blur is applied to create inputs and the method is compared with a Wiener filter and MPRNet. For inpainting, the tasks are either filling 30% randomly masked pixels or removing text regions, with mean filtering, median filtering, and LaMa as comparison methods.

  7. Knowl 7 — Low-level image-processing results

    empirical result

    INSP-Net produces visually plausible continuous edge maps, blurred images, denoised images, deblurred images, and inpainted images without processing a decoded pixel grid. The paper reports the following example-level PSNR/SSIM values; the comparison shows that INSP-Net is competitive with both classical and learned explicit-image methods, although its relative advantage depends on the task and example.

    Task and method PSNR/SSIM
    Denoising: INR fitted noisy 20.14/0.60
    Denoising: mean filter 20.09/0.61
    Denoising: MPRNet 20.51/0.66
    Denoising: INSP-Net 24.02/0.76
    Deblurring: INR fitted blurred 23.88/0.77
    Deblurring: Wiener filter 21.72/0.52
    Deblurring: MPRNet 26.73/0.83
    Deblurring: INSP-Net 27.67/0.79
    Inpainting example 1: input 12.60/0.43
    Inpainting example 1: mean filter 17.02/0.53
    Inpainting example 1: median filter 18.99/0.63
    Inpainting example 1: LaMa 26.40/0.88
    Inpainting example 1: INSP-Net 23.07/0.76
    Inpainting example 2: input 26.98/0.96
    Inpainting example 2: mean filter 26.80/0.90
    Inpainting example 2: median filter 26.41/0.88
    Inpainting example 2: LaMa 23.29/0.73
    Inpainting example 2: INSP-Net 33.44/0.95

    The qualitative comparisons also show that simple mean and median filters damage unmasked regions and do not remove text effectively, whereas INSP-Net substantially improves over those filters and performs comparably to LaMa in the reported inpainting examples.

  8. Knowl 8 — Implicit geometry smoothing with signed distance functions

    empirical result

    The authors apply INSP-Net to three-dimensional geometry represented by a signed distance function (SDF). They first fit an SDF INR to point-cloud data using an SDF training loss, train INSP-Net to emulate a Gaussian-like low-pass filter, and then apply the learned operator directly to the SDF INR. Marching cubes is used only afterward to extract meshes for visualization. On the Thai Statue, Armadillo, and Dragon geometries, the processed SDFs produce smoother surfaces and remove high-frequency geometric details, demonstrating that the differential-operator framework applies beyond image-valued INRs.

  9. Knowl 9 — Classification directly on implicit image representations

    data/table

    The paper evaluates a two-layer INSP-ConvNet for image classification on SIREN representations of MNIST images at 28×2828\times28 resolution and CIFAR-10 images at 32×3232\times32 resolution. Each image is first fitted by a SIREN; the classifiers are then optimized for 1000 epochs with AdamW at learning rate 10−410^{-4}. The comparison includes a two-layer depthwise CNN operating on decoded pixels, PCA followed by an SVM operating on vectorized INR parameters, and an MLP classifier operating directly on vectorized INR parameters.

    Dataset Depthwise CNN PCA + SVM MLP classifier INSP-ConvNet
    MNIST 87.6% 11.3% 9.8% 88.1%
    CIFAR-10 59.5% 9.4% 10.1% 62.5%

    INSP-ConvNet reaches 88.1% on MNIST and 62.5% on CIFAR-10, slightly exceeding the decoded-pixel depthwise CNN baselines of 87.6% and 59.5%, respectively. The much lower PCA+SVM and parameter-space MLP accuracies indicate that directly classifying unstructured INR weights is ineffective in these experiments, whereas derivative features provide useful local information while retaining an implicit representation.

  10. Knowl 10 — Limitations of the theoretical and practical framework

    limitation

    The theoretical convolution result guarantees expressiveness through an infinite-sequence approximation perspective; it does not provide a finite, efficient construction for every convolutional filter. The paper identifies the construction of more expressive operators and more effective finite parameterizations for convolution as open problems. In practice, computing high-order derivatives is memory-intensive and numerically unstable, and recursive derivative computation makes deeper INSP-ConvNets difficult to scale. The experiments also rely on per-scene optimization to fit every INR, while scalable INR reconstruction is left outside the paper's scope.

Coverage note — No substantial contributed material was omitted; proof derivations, background, related work, and implementation details that only support the main contributions were excluded.

References

  1. 1.Emilien Dupont, Adam Golinski, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. Coin: Compression with implicit neural representations. arXiv preprint arXiv:2103.03123, 2021.
  2. 2.Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neural representations for videos. Advances in Neural Information Processing Systems, 34, 2021.
  3. 3.Yunfan Zhang, Ties van Rozendaal, Johann Brehmer, Markus Nagel, and Taco Cohen. Implicit neural video compression. arXiv preprint arXiv:2112.11312, 2021.
  4. 4.Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7154–7164, 2019.
  5. 5.Matan Atzmon, Niv Haim, Lior Yariv, Ofer Israelov, Haggai Maron, and Yaron Lipman. Controlling neural level sets. Advances in Neural Information Processing Systems, 32, 2019.
  6. 6.Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5939–5948, 2019.
  7. 7.Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. arXiv preprint arXiv:2002.10099, 2020.
  8. 8.Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4460–4470, 2019.
  9. 9.Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5379–5389, 2019.
  10. 10.Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165–174, 2019.
  11. 11.Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In European Conference on Computer Vision, pages 523–540. Springer, 2020.
  12. 12.Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pages 405–421. Springer, 2020.
  13. 13.Vincent Sitzmann, Semon Rezchikov, William T Freeman, Joshua B Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. arXiv preprint arXiv:2106.02634, 2021.
  14. 14.Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul Srinivasan, and Jonathan T Barron. Nerf in the dark: High dynamic range view synthesis from noisy raw images. arXiv preprint arXiv:2111.13679, 2021.
  15. 15.Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5799–5809, 2021.
  16. 16.Terrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W Taylor, and Joshua M Susskind. Unconstrained scene generation with locally conditioned radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14304–14313, 2021.
  17. 17.Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. arXiv preprint arXiv:2110.08985, 2021.
  18. 18.Zekun Hao, Arun Mallya, Serge Belongie, and Ming-Yu Liu. Gancraft: Unsupervised 3d neural rendering of minecraft worlds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14072–14082, 2021.
  19. 19.Quan Meng, Anpei Chen, Haimin Luo, Minye Wu, Hao Su, Lan Xu, Xuming He, and Jingyi Yu. Gnerf: Gan-based neural radiance field without posed camera. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6351–6361, 2021.
  20. 20.Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11453–11464, 2021.
  21. 21.Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Advances in Neural Information Processing Systems, 33:20154–20166, 2020.
  22. 22.Peng Zhou, Lingxi Xie, Bingbing Ni, and Qi Tian. Cips-3d: A 3d-aware generator of gans based on conditionally-independent pixel synthesis. arXiv preprint arXiv:2110.09788, 2021.
  23. 23.Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, and Bryan Russell. Editing conditional radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5773–5783, 2021.
  24. 24.Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text-and-image driven manipulation of neural radiance fields. arXiv preprint arXiv:2112.05139, 2021.
  25. 25.Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165–174, 2019.
  26. 26.Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5932–5941, 2019.
  27. 27.Matthew Tancik, Pratul P Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. arXiv preprint arXiv:2006.10739, 2020.
  28. 28.Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462–7473, 2020.
  29. 29.Jiequn Han, Arnulf Jentzen, and E Weinan. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34):8505–8510, 2018.
  30. 30.Ellen D Zhong, Tristan Bepler, Bonnie Berger, and Joseph H Davis. Cryodrgn: reconstruction of heterogeneous cryo-em structures using neural networks. Nature Methods, 18(2):176–185, 2021.
  31. 31.Dejia Xu, Yihao Chu, and Qingyan Sun. Moiré pattern removal via attentive fractal network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 472–473, 2020.
  32. 32.Yaqian Xu, Wenqing Zheng, Jingchen Qi, and Qi Li. Blind image blur assessment based on markov-constrained fcm and blur entropy. In 2019 IEEE international conference on image processing (icip), pages 4519–4523. IEEE, 2019.
  33. 33.Peng-Shuai Wang, Xiao-Ming Fu, Yang Liu, Xin Tong, Shi-Lin Liu, and Baining Guo. Rolling guidance normal filter for geometric processing. ACM Transactions on Graphics (TOG), 34(6):1–9, 2015.
  34. 34.David B Lindell, Julien NP Martel, and Gordon Wetzstein. Autoint: Automatic integration for fast neural volume rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14556–14565, 2021.
  35. 35.Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
  36. 36.Jan J Koenderink and Andrea J van Doorn. Image processing done right. In European Conference on Computer Vision, pages 158–172. Springer, 2002.
  37. 37.Hae-Kwang Kim and Jong-Deuk Kim. Region-based shape descriptor invariant to rotation, scale and translation. Signal Processing: Image Communication, 16(1-2):87–93, 2000.
  38. 38.Niloy J Mitra, Mark Pauly, Michael Wand, and Duygu Ceylan. Symmetry in 3d geometry: Extraction and applications. In Computer Graphics Forum, volume 32, pages 1–23. Wiley Online Library, 2013.
  39. 39.Peter J Olver. Applications of Lie groups to differential equations, volume 107. Springer Science & Business Media, 2000.
  40. 40.Risheng Liu, Zhouchen Lin, Wei Zhang, and Zhixun Su. Learning pdes for image restoration via optimal control. In European Conference on Computer Vision, pages 115–128. Springer, 2010.
  41. 41.Bin Dong, Qingtang Jiang, and Zuowei Shen. Image restoration: Wavelet frame shrinkage, nonlinear evolution pdes, and beyond. Multiscale Modeling & Simulation, 15(1):606–660, 2017.
  42. 42.Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data. In International Conference on Machine Learning, pages 3208–3216. PMLR, 2018.
  43. 43.Zichao Long, Yiping Lu, and Bin Dong. Pde-net 2.0: Learning pdes from data with a numeric-symbolic hybrid deep network. Journal of Computational Physics, 399:108925, 2019.
  44. 44.George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 1989.
  45. 45.Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 1991.
  46. 46.Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94:103–114, 2017.
  47. 47.Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison. In-place scene labelling and understanding with implicit scene representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15838–15847, 2021.
  48. 48.Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi SM Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes. arXiv preprint arXiv:2111.13260, 2021.
  49. 49.Brandon Yushan Feng and Amitabh Varshney. Signet: Efficient neural representation for light fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14224–14233, 2021.
  50. 50.Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. Texture fields: Learning texture representations in function space. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4531–4540, 2019.
  51. 51.Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2304–2314, 2019.
  52. 52.Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P Srinivasan, Jonathan T Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2846–2855, 2021.
  53. 53.Jaeho Lee, Jihoon Tack, Namhoon Lee, and Jinwoo Shin. Meta-learning sparse implicit neural representations. Advances in Neural Information Processing Systems, 34:11769–11780, 2021.
  54. 54.Emilien Dupont, Hyunjik Kim, SM Eslami, Danilo Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you should treat it like one. arXiv preprint arXiv:2201.12204, 2022.
  55. 55.James W. Hennessey, Wilmot Li, Bryan Russell, Eli Shechtman, and Niloy J. Mitra. Transferring image-based edits for multi-channel compositing. ACM Transactions on Graphics, 36(6), 2017.
  56. 56.Jerry Liu, Fisher Yu, and Thomas Funkhouser. Interactive 3d modeling with a generative adversarial network. In 2017 International Conference on 3D Vision (3DV), pages 126–134. IEEE, 2017.
  57. 57.Peihao Wang, Zhiwen Fan, Tianlong Chen, and Zhangyang Wang. Neural implicit dictionary learning via mixture-of-expert training. In International Conference on Machine Learning, pages 22613–22624. PMLR, 2022.
  58. 58.Guandao Yang, Serge Belongie, Bharath Hariharan, and Vladlen Koltun. Geometry processing with neural fields. Advances in Neural Information Processing Systems, 34, 2021.
  59. 59.Zhiwen Fan, Yifan Jiang, Peihao Wang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Unified implicit neural stylization. arXiv preprint arXiv:2204.01943, 2022.
  60. 60.Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414–2423, 2016.
  61. 61.Pietro Perona and Jitendra Malik. Scale-space and edge detection using anisotropic diffusion. IEEE Transactions on pattern analysis and machine intelligence, 12(7):629–639, 1990.
  62. 62.Stanley Osher and Leonid I Rudin. Feature-oriented image enhancement using shock filters. SIAM Journal on numerical analysis, 27(4):919–940, 1990.
  63. 63.Xue-Cheng Tai, Stanley Osher, and Randi Holm. Image inpainting using a tv-stokes equation. In Image Processing based on partial differential equations, pages 3–22. Springer, 2007.
  64. 64.Zhouchen Lin, Wei Zhang, and Xiaoou Tang. Designing partial differential equations for image processing by combining differental invariants, 2009.
  65. 65.Lars Hömke, Claudia Frohn-Schauf, Stefan Henn, and Kristian Witsch. Total variation based image registration. In Image processing based on partial differential equations, pages 343–361. Springer, 2007.
  66. 66.Risheng Liu, Junjie Cao, Zhouchen Lin, and Shiguang Shan. Adaptive partial differential equation learning for visual saliency detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3866–3873, 2014.
  67. 67.Adam Rabcewicz. Clg method for optical flow estimation based on gradient constancy assumption. In Image Processing Based on Partial Differential Equations, pages 57–66. Springer, 2007.
  68. 68.Guillermo Sapiro. Geometric partial differential equations and image analysis. Cambridge university press, 2006.
  69. 69.Frédéric Cao. Geometric curve evolution and image processing. Springer Science & Business Media, 2003.
  70. 70.Bart M Haar Romeny. Geometry-driven diffusion in computer vision, volume 1. Springer Science & Business Media, 2013.
  71. 71.Leonid I Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena, 60(1-4):259–268, 1992.
  72. 72.Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th international conference on international conference on machine learning, pages 399–406, 2010.
  73. 73.Jialin Liu and Xiaohan Chen. Alista: Analytic weights are as good as learned weights in lista. In International Conference on Learning Representations (ICLR), 2019.
  74. 74.Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
  75. 75.Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie line Alberi Morel. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In Proceedings of the British Machine Vision Conference, pages 135.1–135.10. BMVA Press, 2012.
  76. 76.Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse-representations. In International conference on curves and surfaces, pages 711–730. Springer, 2010.
  77. 77.Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
  78. 78.I Sobel. An isotropic 3⇥ 3 image gradient operator, presentation at stanford ai project (1968), 2014.
  79. 79.John Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, (6):679–698, 1986.
  80. 80.Judith MS Prewitt et al. Object enhancement and extraction. Picture processing and Psychopictorics, 10(1):15–19, 1970.
  81. 81.Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In CVPR, 2021.
  82. 82.Saeed V Vaseghi. Advanced digital signal processing and noise reduction. John Wiley & Sons, 2008.
  83. 83.Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. arXiv preprint arXiv:2109.07161, 2021.
  84. 84.Brian Curless and Marc Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 303–312, 1996.
  85. 85.Andrew Gardner, Chris Tchou, Tim Hawkins, and Paul Debevec. Linear light source reflectometry. ACM Transactions on Graphics (TOG), 22(3):749–758, 2003.
  86. 86.Venkat Krishnamurthy and Marc Levoy. Fitting smooth surfaces to dense polygon meshes. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 313–324, 1996.
  87. 87.Greg Turk and Marc Levoy. Zippered polygon meshes from range images. In Proceedings of the 21st annual conference on Computer graphics and interactive techniques, pages 311–318, 1994.
  88. 88.Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
  89. 89.Kathrin Schacke. On the kronecker product. Master’s thesis, University of Waterloo, 2004.
  90. 90.François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251–1258, 2017.
  91. 91.Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxim: Multi-axis mlp for image processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5769–5780, 2022.

Citation

MLA
Xu, D., et al. “Signal Processing for Implicit Neural Representations”. Advances in Neural Information Processing Systems, vol. 35, 2022, pp. 13404–18, https://proceedings.neurips.cc/paper_files/paper/2022/file/575c450013d0e99e4b0ecf82bd1afaa4-Paper-Conference.pdf.
APA
Xu, D., Wang, P., Jiang, Y., Fan, Z., & Wang, Z. (2022). Signal Processing for Implicit Neural Representations. Advances in Neural Information Processing Systems, 35, 13404–13418. https://proceedings.neurips.cc/paper_files/paper/2022/file/575c450013d0e99e4b0ecf82bd1afaa4-Paper-Conference.pdf
Chicago
Xu, D., P. Wang, Y. Jiang, Z. Fan, and Z. Wang. 2022. “Signal Processing for Implicit Neural Representations”. Advances in Neural Information Processing Systems 35: 13404–18. https://proceedings.neurips.cc/paper_files/paper/2022/file/575c450013d0e99e4b0ecf82bd1afaa4-Paper-Conference.pdf.
Harvard
Xu, D. et al. (2022) “Signal Processing for Implicit Neural Representations”, Advances in Neural Information Processing Systems. Curran Associates, Inc., pp. 13404–13418. Available at: https://proceedings.neurips.cc/paper_files/paper/2022/file/575c450013d0e99e4b0ecf82bd1afaa4-Paper-Conference.pdf.
Vancouver
1. Xu D, Wang P, Jiang Y, Fan Z, Wang Z (2022) Signal Processing for Implicit Neural Representations. In: Advances in Neural Information Processing Systems. Curran Associates, Inc., pp 13404–13418

BibTeX

@inproceedings{xu2022signal,
  title = {Signal Processing for Implicit Neural Representations},
  author = {Xu, Dejia and Wang, Peihao and Jiang, Yifan and Fan, Zhiwen and Wang, Zhangyang},
  year = {2022},
  booktitle = {Advances in Neural Information Processing Systems},
  publisher = {Curran Associates, Inc.},
  volume = {35},
  pages = {13404-13418},
  url = {https://proceedings.neurips.cc/paper_files/paper/2022/file/575c450013d0e99e4b0ecf82bd1afaa4-Paper-Conference.pdf}
}
Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: Authors