Functional Gradient Descent with Adaptive Representations

From a theoretical point of view, FGD is well known to benefit from good convergence guarantees (cf. e.g. ([1, 2])). However, it is nontrivial to implement in practice. Since the functional gradients reside in the infinite-dimensional space $\mathcal{H}$, they generally cannot be computed exactly nor stored in memory. As a result, to actually implement FGD we must approximate the gradients $\nabla L(f_t)$ via some finite-dimensional representation. Yet using a fixed representation introduces approximation error, preventing convergence to a global minimizer.

Related work FGD is well-established as a theoretical procedure; see, e.g. ([2]) for an overview of the convergence theory for exact FGD. In specific cases, exact FGD is actually feasible to implement; for example, in the specific case of standard regression losses over an RKHS, the gradients are finite-dimensional and can therefore be exactly represented. This is akin to the classic representer theorems for kernel regression ([3, 4]). However, such approaches rely on specific finite-sample structures, do not generalize beyond simple regression settings, and typically scale poorly with sample size. Alternatively, several works approximate functional gradients ([5, 6, 7, 8, 9]), but introduce irreducible approximation error and often do not incorporate this approximation into their theoretical analysis. Our work addresses this by (i) providing a general theory for approximate FGD, and (ii) identifying sufficient conditions that ensure convergence to proper minimizers sans approximation error.

Our work is also related to first-order optimization with inexact gradients ([10, 11, 12]), but differs in that we operate in the infinite-dimensional setting, where many previous methods break down. In particular, most approximation schemes (e.g. ([13, 2, 14])) do not extend to infinite dimensions. Additionally, we consider a more general setting in which gradients lie in a Hilbert space but are approximated in a broader Banach space, beyond the scope of existing theory.

Note that this bound is tight, in the sense that when $\mathcal{H}=\mathcal{B}$, $g_t=\nabla L(f_t)$, and $\eta =1/K$, we recover the usual descent lemma $L(f_t-\eta g_t)\le L(f_t)-\frac{\eta }{2}\Vert \nabla L(f_t){\Vert }_{\mathcal{B}}^2$ (cf. e.g. ([16])).

where ${\mathcal{H}}_K$ is an RKHS with kernel $K$. Here, $\ell$ denotes a pointwise loss such as the mean squared error ($\ell (\hat{y},y)=\frac{1}{2}(\hat{y}-y{)}^2$) or binary cross-entropy ($\ell (\hat{y},y)=-y\log \hat{y}-(1-y)\log (1-\hat{y})$). Such regression problems are commonplace in statistical inference procedures, including conditional independence testing ([17]), asymptotic statistics ([18]), and more.

For this experiment we use the binary classification dataset of ([19]) for the detection of non-coding RNA sequences. We consider two losses $\ell$: mean squared error and binary cross entropy. Figure 2 shows the results of our experiment. We find that our method consistently outperforms a standard MLP neural network as well as FGD with any fixed representation, adapting as necessary to the target function and guaranteeing convergence to the global optimum while achieving a better test loss.

This section shows how our method can be applied to physics-informed machine learning tasks ([20, 21]). As an example, consider the wave equation for $f:\mathbb{R}\times {\mathbb{R}}^2\to \mathbb{R}$,

which models wave propagation phenomena such as sound, mechanical vibrations, and electromagnetic waves ([22, 23, 24, 25]). Although solutions exist under broad conditions on the initial data $h$, explicit closed-form solutions are generally unavailable in realistic settings, requiring numerical approximation methods.

Computing functional gradients in Sobolev spaces is generally challenging ([7]). However, the Fourier transform of the gradient admits a closed-form expression:

We next consider the inverse rendering task of reconstructing a 3D scene from posed images ([26]). This can be framed as a functional optimization problem: given images with known camera parameters, the goal is to learn the scene as a density field and a view-dependent color field,

Given a camera ray $\gamma :[t_{\mathrm{near}},t_{\mathrm{far}}]\to {\mathbb{R}}^3$ with viewing direction ${\gamma }_{\omega }$, where $t_{\mathrm{near}}$ and $t_{\mathrm{far}}$ denote integration bounds, we can render the scene via the volumetric rendering equation ([27]) with distributional antialiasing ([28]), yielding the antialiased rendering operator ${\tilde{R}}_{\sigma ,c}(\gamma )$, which is differentiable w.r.t. $\sigma$ and $c$. Then, given a set of pixels $(Y_{i,j}{)}_{i=1,j=1}^{n,P}$ and corresponding rays $({\gamma }_{i,j}{)}_{i=1,j=1}^{n,P}$ originating from the respective cameras $i=1,\dots ,n$, our goal is to optimize the loss

which is well-defined for $\sigma \in L^2(\Omega )$ and $c\in L^2(\Omega \to {\mathcal{H}}^{\mathbb{S}})$, where $\Omega$ is a compact subset of ${\mathbb{R}}^3$ and ${\mathcal{H}}^{\mathbb{S}}$ is an RKHS with domain over the unit sphere ${\mathbb{S}}^2$. This loss directly measures image reconstruction error and is the standard supervision signal for radiance-field optimization (cf. e.g. ([26, 29])). It is highly nonconvex; nonetheless, it has functional gradients known in closed form, cf. Appendix A.1.4, which we efficiently approximate by the means of uniform 3D grids with spherical harmonics for the color view-dependence.

We evaluate on the Ficus scene from the NeRF-Synthetic dataset ([26]), using the same camera poses and train/test split for all methods. All methods optimize the photometric loss in Equation 10; we compare adaptive FGD with an Adam-trained neural network baseline and fixed-grid FGD variants. As shown in Figure 4, adaptive FGD achieves the lowest test loss and sharpest reconstructions by starting coarse and refining only when needed, whereas fixed-grid FGD plateaus at a resolution-dependent error floor and the neural baseline converges much more slowly.