Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data

Minshuo ChenKaixuan HuangTuo ZhaoMengdi Wang

article2023ICML203 citations

Establishes the first end-to-end theoretical guarantees for diffusion models on low-dimensional subspace data, proving that an encoder-decoder architecture overcomes the curse of ambient dimensionality with sample complexity and convergence rates governed solely by the intrinsic data dimension.

Listen

High-dimensional data in modern applications, such as high-resolution images and audio, frequently exhibit underlying low-dimensional structures due to natural symmetries and patterns. While diffusion models have demonstrated state-of-the-art empirical performance in generating these complex data types, the theoretical understanding of how and why they succeed has lagged behind. Existing theoretical analyses typically assume access to an already well-estimated score function without explaining how neural networks learn it or how the data's intrinsic geometry influences statistical complexity.

The article establishes an integrated theoretical framework that evaluates both score estimation via neural networks and distribution recovery in diffusion models. Specifically, it demonstrates how diffusion models avoid the curse of ambient dimensionality when data reside on an unknown low-dimensional linear subspace.

To conduct this evaluation, the authors mathematically analyzed a standard score-based diffusion model using a variance-preserving Ornstein-Uhlenbeck forward process and a learned reverse process. They designed a specialized score network class featuring an encoder-decoder architecture with shortcut connections, closely mirroring practical architectures like U-Net. The theoretical framework evaluates score function approximation over unbounded domains using a truncation technique, bounds statistical estimation errors using empirical process theory on finite sample sizes, and derives convergence guarantees for the simulated, discretized reverse sampling process.

The analysis reveals several key findings. First, the score function naturally decomposes into an on-support component that captures the underlying data distribution and an orthogonal component that enforces subspace recovery. Second, the proposed neural network architecture accurately approximates the score function, with network size depending primarily on the intrinsic data dimension rather than the ambient dimension. Third, score matching achieves an explicit statistical convergence rate where the estimation error scales with sample size as the intrinsic dimension increases, rather than suffering from ambient dimensionality. Fourth, the generated data distribution converges to the true distribution with strong total variation and Wasserstein distance guarantees, while the variance in orthogonal directions vanishes as the stopping time nears zero.

These results provide a formal mathematical justification for the remarkable empirical efficiency of diffusion models in high-dimensional domains. They show that practitioners do not need separate dimensionality reduction steps, such as principal component analysis, because diffusion models naturally and end-to-end recover low-dimensional structures while estimating data distributions. Furthermore, the findings highlight a practical trade-off regarding the early-stopping time parameter: stopping too early amplifies score estimation errors due to score blowup, whereas stopping too late introduces distributional bias.

For engineering and deploying diffusion models, teams should adopt encoder-decoder architectures with residual connections and enforce Lipschitz regularity during training to maintain stability. Practitioners should also calibrate early-stopping thresholds and discretization step sizes to balance the trade-off between score stability and distribution bias. Organizations should support further research to extend these theoretical guarantees from linear subspaces to non-linear Riemannian manifolds and investigate modern temporal embeddings such as sinusoidal positional encodings.

Confidence in these findings is high for linear subspace settings under standard assumptions, including sub-Gaussian distribution tails and Lipschitz-smooth score components. However, leaders should note that real-world image manifolds frequently contain nonlinear curvatures and complex topological structures that exceed the linear subspace boundaries analyzed in the article.

Georgia Institute of Technology Princeton University diffusion probabilistic models estimation method network architecture diffusion models score approximation distribution recovery

arXiv: 2302.07194

Paper: Denoising Diffusion Probabilistic Models, Jonathan Ho et al. (2020). Its foundational denoising diffusion formulation supplies the reverse-process and score-estimation framework that this paper analyzes theoretically.

No sufficiently relevant recommendations were found.

Cover for Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data

Abstract

Diffusion models achieve state-of-the-art performance in various generation tasks. However, their theoretical foundations fall far behind. This paper studies score approximation, estimation, and distribution recovery of diffusion models, when data are supported on an unknown low-dimensional linear subspace. Our result provides sample complexity bounds for distribution estimation using diffusion models. We show that with a properly chosen neural network architecture, the score function can be both accurately approximated and efficiently estimated. Further, the generated distribution based on the estimated score function captures the data geometric structures and converges to a close vicinity of the data distribution. The convergence rate depends on subspace dimension, implying that diffusion models can circumvent the curse of data ambient dimensionality.

Topics

diffusion probabilistic models estimation method network architecture neural networks rate of convergence

Keywords

diffusion models score approximation distribution recovery low-dimensional data backward SDEs total variation distance L2 approximation

1. Introduction
1.1. Related Work
2. Preliminaries
3. Score Decomposition
4. Score Approximation and Estimation
4.1. Score Approximation
4.2. Score Estimation Theory
5. Distribution Estimation
6. Conclusion and Discussion
Acknowledgements
References
A. Omitted Proofs in Section 3
A.1. Proof of Lemma 1
A.2. Computation in Example 1
B. Omitted Proofs in Section 4
B.1. Proof of Theorem 1
B.2. Proof of Theorem 2
B.3. Conditional Covariance Bound
B.4. Truncation Error
C. Omitted Proofs in Section 5
C.1. Subspace Error and Latent Score Matching Error
C.2. Backward Processes
C.3. Orthogonal Process
C.4. Proof of Theorem 3
D. Omitted Proofs in Section C
D.1. Proof of Lemma 3
D.1.1. Evolution of Score Function
D.1.2. Other Lemmas
D.2. Proof of Lemma 4, Undiscretized Setting
D.3. Proof of Lemma 4, Discretized Setting
D.4. Proof of Lemma 5
E. Helper Lemmas

Knowls

Knowl 1 — Neural score estimation has intrinsic-dimension-dependent sample complexity
theoretical result
Suppose data have the form $X=AZ$ , where $A\in\mathbb{R}^{D\times d}$ has orthonormal columns and the latent variable $Z\in\mathbb{R}^d$ satisfies the paper’s smoothness, tail, and score-Lipschitz assumptions. Train the encoder–decoder score class described below by empirical denoising score matching, and choose its approximation parameter as $\epsilon=n^{-(1-\delta(n))/(d+5)}$ , where $n$ is the sample size and $\delta(n)=d\log\log n/\log n$ . With probability at least $1-1/n$ , the fitted score $\widehat s$ satisfies
$\frac{1}{T-t_0}\int_{t_0}^{T}\|\widehat s(\cdot,t)-\nabla\log p_t(\cdot)\|_{L^2(P_t)}^2\,dt =\widetilde O\!\left(\frac{1}{t_0}\left[n^{-\frac{2-2\delta(n)}{d+5}}+D n^{-\frac{d+3}{d+5}}\right]\log^3 n\right).$
Here $P_t$ is the forward-noised data distribution at time $t$ , $p_t$ its density, $T$ the forward horizon, and $t_0>0$ the early-stopping time. The suppressed factors depend on $\beta$ , $\log D$ , $d$ , $\log t_0$ , and the time-regularity constant $\tau$ of the approximation theorem. In particular, the leading sample-size exponent is governed by intrinsic dimension $d$ , rather than by ambient dimension $D$ .
Knowl 2 — The learned reverse process estimates the latent noised data distribution
theoretical result
Let $P_z$ be the latent data law, let $P_t^{\mathrm{LD}}$ be the law of $e^{-t/2}Z+\sqrt{1-e^{-t}}\,G$ for independent $Z\sim P_z$ and $G\sim\mathcal N(0,I_d)$ , and let $\widehat P_{t_0}^{\mathrm{dis}}$ be the distribution generated by simulating the learned, discretized reverse process from $\mathcal N(0,I_D)$ to time $t_0$ . Under the score-estimation conditions, take $T=\Theta(\log n)$ and assume $\mathrm{KL}(P_z\|\mathcal N(0,I_d))<\infty$ . If the discretization step satisfies $\eta\le (t_0^2/d)n^{-(2-2\delta(n))/(d+5)}$ , then an orthogonal matrix $U\in\mathbb{R}^{d\times d}$ exists such that
$\mathrm{TV}\!\left(P_{t_0}^{\mathrm{LD}},(VU)^\top_{\sharp}\widehat P_{t_0}^{\mathrm{dis}}\right) =\widetilde O\!\left(\frac{1}{\sqrt{c_0t_0}}n^{-\frac{1-\delta(n)}{d+5}}\log^2 n\right).$
Here $V\in\mathbb{R}^{D\times d}$ is the learned encoder–decoder subspace matrix, $(VU)^\top_{\sharp}$ denotes pushforward by $x\mapsto(VU)^\top x$ , $c_0=\lambda_{\min}(\mathbb E[ZZ^\top])$ , and $\delta(n)=d\log\log n/\log n$ . The guarantee compares the generated latent law to the forward-noised latent target at the stopping time, not directly to the clean latent law.
Knowl 3 — Score matching recovers the unknown data subspace
theoretical result
For data $X=AZ$ with orthonormal-column $A\in\mathbb{R}^{D\times d}$ , define $c_0=\lambda_{\min}(\mathbb E[ZZ^\top])>0$ . With the estimated score network trained as in the score-estimation result, choose $T=\Theta(\log n)$ and an admissible early-stopping time $t_0=O(\min\{c_0,1/\beta\})$ . The learned encoder–decoder matrix $V$ then recovers the data subspace in projector distance:
$\|VV^\top-AA^\top\|_F^2 =\widetilde O\!\left(\frac{1}{c_0}n^{-\frac{2-2\delta(n)}{d+5}}\log^{7/2}n\right), \qquad \delta(n)=\frac{d\log\log n}{\log n}.$
This is a guarantee on the column spaces, not on $\|V-A\|_F$ : an orthogonal change of coordinates within the $d$ -dimensional subspace does not change its projector.
Knowl 4 — A low-dimensional encoder–decoder network approximates the score
theoretical result
Assume $X=AZ$ , with orthonormal-column $A\in\mathbb R^{D\times d}$ , and assume the latent density has the sub-Gaussian tail and the on-support score has the $\beta$ -Lipschitz regularity stated in the paper. For any $\epsilon>0$ , a network of the form $s_{V,\theta}(x,t)=h(t)^{-1}Vf_\theta(V^\top x,t)-h(t)^{-1}x$ can be configured so that, for every $t\in[t_0,T]$ ,
$\|\bar s_{V,\theta}(\cdot,t)-\nabla\log p_t(\cdot)\|_{L^2(P_t)} \le \frac{\sqrt{d}+1}{h(t)}\epsilon.$
One valid configuration has depth $L=O(\log(1/\epsilon)+d)$ , output bound $K=O((1+\beta)^d\log^{1/2}(d/(t_0\epsilon)))$ , width
$M=O\!\left((1+\beta)^dT\tau d^{d/2+1}\epsilon^{-(d+1)}\log^{d/2}\!\left(\frac{d}{t_0\epsilon}\right)\right),$
and sparsity
$J=O\!\left((1+\beta)^dT\tau d^{d/2+1}\epsilon^{-(d+1)}\log^{d/2+1}\!\left(\frac{d}{t_0\epsilon}\right)\right).$
The network’s latent-input and time Lipschitz bounds can be taken as $\gamma=10d(1+\beta)$ and $\gamma_t=10\tau$ . Here $h(t)=1-e^{-t}$ , $p_t$ is the density of the forward-noised data, and $\tau=\sup_{t\in[t_0,T],\,\|z\|_\infty\le\sqrt{d\log(d/(t_0\epsilon))}}\|\partial_t[h(t)s_{\parallel}(z,t)]\|_2$ . Thus the approximation complexity is controlled primarily by intrinsic dimension $d$ ; $D$ enters through the ambient encoder and decoder.
Knowl 5 — The score separates into latent and orthogonal components
equation
For $X=AZ$ with $A\in\mathbb R^{D\times d}$ having orthonormal columns, the score of the forward-noised distribution decomposes as
$\nabla\log p_t(x) =A\nabla\log p_t^{\mathrm{LD}}(A^\top x) -\frac{1}{h(t)}(I_D-AA^\top)x.$
Here $p_t^{\mathrm{LD}}(z')=\int \phi_t(z'\mid z)p_z(z)\,dz$ is the latent-space density after forward noise, $\phi_t(\cdot\mid z)$ is the Gaussian density of $\mathcal N(e^{-t/2}z,h(t)I_d)$ , and $h(t)=1-e^{-t}$ . The first term lies in the data subspace and carries information about the latent distribution; the second acts only in the orthogonal complement and does not depend on that distribution. In the reverse-time dynamics, the orthogonal component follows a linear process with drift coefficient $1/2-1/h(T-t)$ , while the on-subspace dynamics use the latent score.
Knowl 6 — The orthogonal part of generated samples vanishes near the data subspace
theoretical result
For the continuous-time reverse process with the paper’s score decomposition, initialized from standard Gaussian noise and run to stopping time $t_0$ , the orthogonal projection of the generated distribution satisfies
$(I_D-VV^\top)_{\sharp}\widehat P_{t_0}^{\mathrm{cont}}=\mathcal N(0,\Sigma), \qquad \Sigma\preceq c t_0 I_D,$
for a constant $c>0$ . Here $V\in\mathbb R^{D\times d}$ is the learned subspace matrix and $\widehat P_{t_0}^{\mathrm{cont}}$ is the continuous-time generated law. Consequently the orthogonal output magnitude tends to zero as $t_0\to0$ , so the generated samples concentrate near the learned data subspace. For the discretized orthogonal process with step size $\eta$ , the stated variance bound is $e^{t_0+\eta}I_D$ when $T>1$ and $t_0+\eta\le1$ .
Knowl 7 — The score network uses a shortcut and a learned low-dimensional bottleneck
model/method
The score class is
$\mathcal S_{\mathrm{NN}}= \left\{s_{V,\theta}(x,t)=\frac{1}{h(t)}Vf_\theta(V^\top x,t)-\frac{1}{h(t)}x: V^\top V=I_d\right\},$
where $x\in\mathbb R^D$ , $V\in\mathbb R^{D\times d}$ , and $f_\theta:\mathbb R^d\times[t_0,T]\to\mathbb R^d$ is a ReLU network. The map $V^\top$ encodes the ambient input into $d$ coordinates, $f_\theta$ processes those coordinates and time, and $V$ decodes back to the ambient space. The term $-x/h(t)$ is a shortcut connection. The network is configured with bounded output, sparse weights, and Lipschitz constraints in latent input and time; score matching learns $V$ rather than requiring the data subspace to be known in advance. The schedule is $h(t)=1-e^{-t}$ .
Knowl 8 — Early stopping trades latent recovery error against score instability
empirical result
The clean latent law $P_z$ and the forward-noised latent target $P_{t_0}^{\mathrm{LD}}$ differ by a smoothing bias bounded in Wasserstein-2 distance as
$W_2(P_{t_0}^{\mathrm{LD}},P_z)=O(\sqrt{dt_0}).$
At the same time, the learned reverse-process total-variation bound for estimating $P_{t_0}^{\mathrm{LD}}$ worsens as $t_0$ decreases, through its factor $1/\sqrt{t_0}$ ; the score itself becomes singular as $t_0\to0$ . Balancing the two reported error scales by taking $t_0=n^{-(1-\delta(n))/(d+5)}$ , with $\delta(n)=d\log\log n/\log n$ , gives respective bounds of order $\widetilde O(n^{-(1-\delta(n))/(2(d+5))}\log^2 n)$ for the latent total-variation estimation term and $\widetilde O(n^{-(1-\delta(n))/(2(d+5))})$ for the Wasserstein smoothing bias. These are bounds in different metrics and are not asserted to combine into a single metric guarantee.
Knowl 9 — Latent-data and score regularity conditions for the guarantees
assumption
The analysis assumes that each ambient data point is $X=AZ$ , where $A\in\mathbb R^{D\times d}$ is unknown with orthonormal columns and $Z\in\mathbb R^d$ has density $p_z$ . For approximation and estimation, $p_z$ is positive and twice continuously differentiable, and there are positive constants $B,C_1,C_2$ such that for $\|z\|_2\ge B$ ,
$p_z(z)\le(2\pi)^{-d/2}C_1\exp\!\left(-\frac{C_2}{2}\|z\|_2^2\right).$
The on-support score $s_{\parallel}(z,t)=\nabla\log p_t^{\mathrm{LD}}(z)$ is assumed $\beta$ -Lipschitz in $z$ for all $t\in[0,T]$ . The results concern linear-subspace support, not arbitrary nonlinear manifolds.
Knowl 10 — Empirical denoising score matching trains the network
model/method
Given independent data $x_1,\ldots,x_n\sim P_{\mathrm{data}}$ , train a score network $s$ by sampling an index $i$ uniformly, a time $t$ uniformly from $[t_0,T]$ , and a noised point $X_t\mid x_i\sim\mathcal N(e^{-t/2}x_i,h(t)I_D)$ , with $h(t)=1-e^{-t}$ . Minimize the empirical denoising loss
$\widehat L(s)=\frac1n\sum_{i=1}^n\frac{1}{T-t_0}\int_{t_0}^T \mathbb E_{X_t\mid x_i}\!\left[ \left\|-\frac{X_t-e^{-t/2}x_i}{h(t)}-s(X_t,t)\right\|_2^2 \right]dt.$
The analytic target $-(X_t-e^{-t/2}x_i)/h(t)$ is the score of the Gaussian transition conditional on $x_i$ . The interval starts at $t_0>0$ to avoid the score blow-up at time zero; the trained network is then used in the discretized reverse process.

Coverage note — The paper’s Gaussian latent example and proof-only technical lemmas are omitted because they illustrate or support the main guarantees rather than adding independent contributed results.

References

1.Anderson, B. D. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313–326, 1982.
2.Barron, A. R. Universal approximation bounds for super-positions of a sigmoidal function. IEEE Trans. Inform. Theory, 39(3):930–945, 1993.
3.Block, A., Mroueh, Y., and Rakhlin, A. Generative modeling with denoising auto-encoders and langevin sampling. arXiv preprint arXiv:2002.00107, 2020.
4.Chen, M., Jiang, H., Liao, W., and Zhao, T. Efficient approximation of deep relu networks for functions on low dimensional manifolds. Advances in neural information processing systems, 32, 2019a.
5.Chen, M., Li, X., and Zhao, T. On generalization bounds of a family of recurrent neural networks. arXiv preprint arXiv:1910.12947, 2019b.
6.Chen, M., Liao, W., Zha, H., and Zhao, T. Statistical guarantees of generative adversarial networks for distribution estimation. arXiv preprint arXiv:2002.03938, 2020.
7.Chen, M., Jiang, H., Liao, W., and Zhao, T. Nonparametric regression on low-dimensional manifolds using deep relu networks: Function approximation and statistical recovery. Information and Inference: A Journal of the IMA, 11(4):1203–1253, 2022a.
8.Chen, S., Chewi, S., Li, J., Li, Y., Salim, A., and Zhang, A. R. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.11215, 2022b.
9.Cybenko, G. Approximation by superpositions of a sigmoidal function. Math. Control Signals Systems, 2(4):303–314, 1989.
10.Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., and Liu, R. Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv:1912.02164, 2019.
11.De Bortoli, V. Convergence of denoising diffusion models under the manifold hypothesis. arXiv preprint arXiv:2208.05314, 2022.
12.De Bortoli, V., Thornton, J., Heng, J., and Doucet, A. Diffusion schrodinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems, 34:17695–17709, 2021.
13.Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and Weinberger, K. (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL https://proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf.
14.Gouk, H., Frank, E., Pfahringer, B., and Cree, M. J. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning, 110(2):393–416, 2021.
15.Guhring, I., Kutyniok, G., and Petersen, P. Error bounds for approximations with deep relu neural networks in w^s,p norms. Anal. Appl., 18(05):803–859, 2020.
16.Haussmann, U. G. and Pardoux, E. Time reversal of diffusions. The Annals of Probability, pp. 1188–1205, 1986.
17.Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
18.Hyvarinen, A. and Dayan, P. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
19.Kim, D., Na, B., Kwon, S. J., Lee, D., Kang, W., and Moon, I.-C. Maximum likelihood training of implicit nonlinear diffusion models. arXiv preprint arXiv:2205.13699, 2022.
20.Le Gall, J.-F. et al. Brownian motion, martingales, and stochastic calculus, volume 274. Springer, 2016.
21.Lee, H., Lu, J., and Tan, Y. Convergence for score-based generative modeling with polynomial complexity. arXiv preprint arXiv:2206.06227, 2022a.
22.Lee, H., Lu, J., and Tan, Y. Convergence of score-based generative modeling for general data distributions. arXiv preprint arXiv:2209.12381, 2022b.
23.Liu, X., Wu, L., Ye, M., and Liu, Q. Let us build bridges: Understanding and extending diffusion generative models. arXiv preprint arXiv:2208.14699, 2022.
24.Nakada, R. and Imaizumi, M. Adaptive approximation and generalization of deep neural network with intrinsic dimensionality. The Journal of Machine Learning Research, 21(1):7018–7055, 2020.
25.Oko, K., Akiyama, S., and Suzuki, T. Diffusion models are minimax optimal distribution estimators. arXiv preprint arXiv:2303.01861, 2023.
26.Pauli, P., Koch, A., Berberich, J., Kohler, P., and Allgower, F. Training robust neural networks using lipschitz bounds. IEEE Control Systems Letters, 6:121–126, 2021.
27.Pidstrigach, J. Score-based generative models detect manifolds. arXiv preprint arXiv:2206.01018, 2022.
28.Pope, P., Zhu, C., Abdelkader, A., Goldblum, M., and Goldstein, T. The intrinsic dimension of images and its impact on learning. arXiv preprint arXiv:2104.08894, 2021.
29.Qi, F. and Mei, J.-Q. Some inequalities of the incomplete gamma and related functions. Zeitschrift fur Analysis und ihre Anwendungen, 18(3):793–799, 1999.
30.Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
31.Rezende, D. and Mohamed, S. Variational inference with normalizing flows. In International conference on machine learning, pp. 1530–1538. PMLR, 2015.
32.Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695, 2022.
33.Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pp. 234–241. Springer, 2015.
34.Roweis, S. T. and Saul, L. K. Nonlinear dimensionality reduction by locally linear embedding. science, 290(5500):2323–2326, 2000.
35.Schmidt-Hieber, J. Nonparametric regression using deep neural networks with relu activation function. arXiv preprint arXiv:1708.06633, 2017.
36.Shen, Z., Yang, H., and Zhang, S. Optimal approximation rate of relu networks in terms of width and depth. Journal de Mathematiques Pures et Appliquées, 157:101–135, 2022.
37.Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2256–2265. PMLR, 2015.
38.Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
39.Song, Y. and Ermon, S. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438–12448, 2020.
40.Song, Y., Garg, S., Shi, J., and Ermon, S. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pp. 574–584. PMLR, 2020a.
41.Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
42.Suzuki, T. Adaptivity of deep relu network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality. arXiv preprint arXiv:1810.08033, 2018.
43.Tenenbaum, J. B., Silva, V. d., and Langford, J. C. A global geometric framework for nonlinear dimensionality reduction. science, 290(5500):2319–2323, 2000.
44.Vahdat, A., Kreis, K., and Kautz, J. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:11287–11302, 2021.
45.Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017.
46.Vershynin, R. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
47.Vincent, P. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661–1674, 2011.
48.Virmaux, A. and Scaman, K. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31, 2018.
49.Wainwright, M. J. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge University Press, 2019.
50.Yarotsky, D. Error bounds for approximations with deep relu networks. Neural Networks, 94:103–114, 2017.

Citation

MLA

Chen, M., et al. “Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data”. International Conference on Machine Learning, vol. 202, 2023, pp. 4672–712, https://proceedings.mlr.press/v202/chen23o.html.

APA

Chen, M., Huang, K., Zhao, T., & Wang, M. (2023). Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data. International Conference on Machine Learning, 202, 4672–4712. https://proceedings.mlr.press/v202/chen23o.html

Chicago

Chen, M., K. Huang, T. Zhao, and M. Wang. 2023. “Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data”. International Conference on Machine Learning 202: 4672–4712. https://proceedings.mlr.press/v202/chen23o.html.

Harvard

Chen, M. et al. (2023) “Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data”, International Conference on Machine Learning. PMLR, pp. 4672–4712. Available at: https://proceedings.mlr.press/v202/chen23o.html.

Vancouver

1. Chen M, Huang K, Zhao T, Wang M (2023) Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data. In: International Conference on Machine Learning. PMLR, pp 4672–4712

BibTeX

@InProceedings{pmlr-v202-chen23o,
  title = 	 {Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data},
  author =       {Chen, Minshuo and Huang, Kaixuan and Zhao, Tuo and Wang, Mengdi},
  booktitle = 	 {Proceedings of the 40th International Conference on Machine Learning},
  pages = 	 {4672--4712},
  year = 	 {2023},
  editor = 	 {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan},
  volume = 	 {202},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {23--29 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v202/chen23o/chen23o.pdf},
  url = 	 {https://proceedings.mlr.press/v202/chen23o.html},
  abstract = 	 {Diffusion models achieve state-of-the-art performance in various generation tasks. However, their theoretical foundations fall far behind. This paper studies score approximation, estimation, and distribution recovery of diffusion models, when data are supported on an unknown low-dimensional linear subspace. Our result provides sample complexity bounds for distribution estimation using diffusion models. We show that with a properly chosen neural network architecture, the score function can be both accurately approximated and efficiently estimated. Further, the generated distribution based on the estimated score function captures the data geometric structures and converges to a close vicinity of the data distribution. The convergence rate depends on subspace dimension, implying that diffusion models can circumvent the curse of data ambient dimensionality.}
}

Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Neural score estimation has intrinsic-dimension-dependent sample complexity

Knowl 2 — The learned reverse process estimates the latent noised data distribution

Knowl 3 — Score matching recovers the unknown data subspace

Knowl 4 — A low-dimensional encoder–decoder network approximates the score

Knowl 5 — The score separates into latent and orthogonal components

Knowl 6 — The orthogonal part of generated samples vanishes near the data subspace

Knowl 7 — The score network uses a shortcut and a learned low-dimensional bottleneck

Knowl 8 — Early stopping trades latent recovery error against score instability

Knowl 9 — Latent-data and score regularity conditions for the guarantees

Knowl 10 — Empirical denoising score matching trains the network

References

Citation

BibTeX

Access the Paper