Improved Techniques for Training Score-Based Generative Models

article2020NeurIPS1,510 citations

Develops practical training methods and theoretical principles that scale score-based generative models up to 256x256 image synthesis, matching the visual quality of state-of-the-art GANs without adversarial training.

Listen

Generating realistic, high-resolution synthetic data is critical for applications in automated design, data synthesis, and anomaly detection. While score-based generative models offer a stable alternative to generative adversarial networks (GANs) by avoiding fragile adversarial training objectives, earlier implementations suffered from training instability and were restricted to low-resolution images (typically 32x32 pixels or lower). The article aims to evaluate the underlying mathematical failure modes of score-based models in high-dimensional spaces and demonstrates a principled framework to scale them to diverse, high-resolution datasets.

To address these limitations, the authors developed a set of five foundational techniques based on mathematical analyses of simplified data distributions and Langevin dynamics sampling. Specifically, they established analytical formulas to set the initial noise scale equal to the maximum pairwise distance between data points, arranged intermediate noise levels in a geometric progression with shared coverage, scaled network outputs inversely by noise intensity to eliminate memory-intensive conditioning, and optimized sampling step sizes under fixed compute budgets. They also implemented an exponential moving average (EMA) of model parameters to stabilize training. These methods were evaluated across standard benchmarks including CIFAR-10, CelebA, LSUN scenes, and FFHQ across image resolutions from 32x32 to 256x256 using standard automated metrics (such as Frechet Inception Distance, or FID) and human perceptual evaluations.

The findings show that this upgraded model (termed NCSNv2) successfully scales score-based generative models to 256x256 resolution, generating sharp, structurally consistent, and diverse images that rival top GANs. Quantitatively, the combined techniques with Tweedie denoising reduced the CIFAR-10 FID score from 25.32 down to 10.87 (where lower is better) and improved CelebA 64x64 FID from 25.30 to 10.23. Furthermore, human perceptual evaluation scores on CelebA 64x64 nearly doubled, rising from 19.8% to 37.3%, matching established GAN benchmarks. Applying exponential moving averages proved essential, eliminating prominent color-shift artifacts and stabilizing checkpoint quality throughout training.

These results demonstrate that generative modeling can achieve state-of-the-art visual quality without the high training instability or complex hyperparameter tuning typical of adversarial frameworks. By replacing ad-hoc tuning with closed-form mathematical choices for noise schedules and step sizes, the framework significantly reduces engineering trial-and-error. For technical decision-makers, this lowers the risk and operational cost of training robust generative pipelines across high-dimensional modalities, while opening opportunities for seamless data synthesis and representation learning.

Organizations should adopt these five design principles—especially EMA tracking and analytical noise scheduling—when deploying score-based architectures. When compute budgets are constrained, teams should set sampling steps according to available hardware and calculate the step size analytically using the article’s formula rather than running expensive grid searches. As a next step, practitioners should explore fine-tuning hyperparameters for specific use cases and evaluate extending these noise principles to non-image modalities, such as audio, sensor streams, or behavioral data.

While the analytical derivations rely on simplified Gaussian assumptions and standard Euclidean distances, the empirical outcomes robustly transfer to complex real-world image datasets. Users should exercise caution regarding computational inference latency, as iterative sampling remains slower than single-pass generators, and should account for inherited dataset biases when deploying synthetic models in sensitive operational settings.

Stanford University diffusion probabilistic models generative model high resolution image score-based generative models score matching Langevin dynamics sampling

arXiv: 2006.09011

Paper: Generative Modeling by Estimating Gradients of the Data Distribution, Yang Song et al. (2019). This foundational paper introduced Noise-Conditional Score Networks (NCSN) and annealed Langevin dynamics, establishing the direct baseline framework and core failure modes that the source paper analyzes and improves upon.
Paper: Estimation of Non-Normalized Statistical Models by Score Matching, Aapo Hyvärinen (2005). This paper establishes the mathematical foundation of score matching, the fundamental objective utilized by score-based generative models to estimate the gradient of data distributions without tractable partition functions.

Paper: Score-Based Generative Modeling through Stochastic Differential Equations, Yang Song et al. (2021). This work generalizes the discrete multi-scale noise levels analyzed in the source into a unified continuous stochastic differential equation (SDE) framework, incorporating the architectural and training improvements introduced here.
Paper: Elucidating the Design Space of Diffusion-Based Generative Models, Tero Karras et al. (2022). This paper systematically dissects and refines the design space of diffusion and score-based models, directly building on the noise schedules, network preconditioning, and sampling stability techniques advanced by the source.
Paper: Denoising Diffusion Probabilistic Models, Jonathan Ho et al. (2020). This seminal work demonstrates that diffusion probabilistic models trained with score-matching objectives can scale effectively to high-resolution synthesis, echoing and expanding upon the high-dimensional sampling findings of the source.
Paper: Improved Denoising Diffusion Probabilistic Models, Alex Nichol et al. (2021). This work builds on practical high-resolution training and sampling insights from score-based diffusion methods to optimize noise schedules and reduce required sampling steps.
Paper: Cascaded Diffusion Models for High Fidelity Image Generation, Jonathan Ho et al. (2021). This paper extends high-resolution score/diffusion generation to higher scales via cascaded pipelines, utilizing conditioning augmentations to mitigate errors in multi-stage generation.
Paper: Scaling Rectified Flow Transformers for High-Resolution Image Synthesis, Patrick Esser et al. (2024). This work advances continuous generative flows and transformers for large-scale, high-resolution synthesis, extending beyond standard score-based and diffusion formulations.

Cover for Improved Techniques for Training Score-Based Generative Models

Abstract

Score-based generative models can produce high quality image samples comparable to GANs, without requiring adversarial optimization. However, existing training procedures are limited to images of low resolution (typically below 32x32), and can be unstable under some settings. We provide a new theoretical analysis of learning and sampling from score models in high dimensional spaces, explaining existing failure modes and motivating new solutions that generalize across datasets. To enhance stability, we also propose to maintain an exponential moving average of model weights. With these improvements, we can effortlessly scale score-based generative models to images with unprecedented resolutions ranging from 64x64 to 256x256. Our score-based models can generate high-fidelity samples that rival best-in-class GANs on various image datasets, including CelebA, FFHQ, and multiple LSUN categories.

Topics

diffusion probabilistic models generative model high resolution image low resolution optimization ranging image resolution

Keywords

score-based generative models score matching Langevin dynamics sampling conditional score network energy-based models FID scores Inception scores image generation vanilla NCSN

1 Introduction
2 Background
2.1 Langevin dynamics
2.2 Score-based generative modeling
3 Choosing noise scales
3.1 Initial noise scale
3.2 Other noise scales
3.3 Incorporating the noise information
4 Configuring annealed Langevin dynamics
5 Improving stability with moving average
6 Combining all techniques together
7 Conclusion
References
A Proofs
B Experimental details
B.1 Network architectures and hyperparameters
B.2 Additional settings
C Additional experimental results
C.1 Additional results without the denoising step
C.2 Training and sampling speed
C.3 Color shifts
C.4 Additional results on ablation studies
C.5 Generalization
C.5.1 Loss curves
C.5.2 Nearest neighbors
C.5.3 Additional interpolation results
C.6 Additional uncurated samples

Knowls

Knowl 1 — Initial Noise Scale Selection Based on Maximum Pairwise Distance
theoretical result
For an empirical dataset distribution $\hat{p}_\text{data}(\mathbf{x}) = \frac{1}{N}\sum_{i=1}^N \delta(\mathbf{x} = \mathbf{x}^{(i)})$ of $N$ data points in $\mathbb{R}^D$ perturbed by Gaussian noise with variance $\sigma_1^2$ , the perturbed density is given by $\hat{p}_{\sigma_1}(\mathbf{x}) = \frac{1}{N}\sum_{i=1}^N p^{(i)}(\mathbf{x})$ , where $p^{(i)}(\mathbf{x}) = \mathcal{N}(\mathbf{x} \mid \mathbf{x}^{(i)}, \sigma_1^2 \mathbf{I})$ . Defining the component weight function

$r^{(i)}(\mathbf{x}) = \frac{p^{(i)}(\mathbf{x})}{\sum_{k=1}^N p^{(k)}(\mathbf{x})}$

the score function decomposes as $\nabla_\mathbf{x} \log \hat{p}_{\sigma_1}(\mathbf{x}) = \sum_{i=1}^N r^{(i)}(\mathbf{x}) \nabla_\mathbf{x} \log p^{(i)}(\mathbf{x})$ . The expected transition weight from component $i$ to component $j$ is upper-bounded by:

$\mathbb{E}_{p^{(i)}(\mathbf{x})}[r^{(j)}(\mathbf{x})] \le \frac{1}{2} \exp\left( -\frac{\|\mathbf{x}^{(i)} - \mathbf{x}^{(j)}\|_2^2}{8\sigma_1^2} \right)$

If the initial noise standard deviation $\sigma_1$ is small relative to the Euclidean distance $\|\mathbf{x}^{(i)} - \mathbf{x}^{(j)}\|_2$ , the expectation decays exponentially, causing Langevin dynamics initialized near $\mathbf{x}^{(i)}$ to ignore other modes $\mathbf{x}^{(j)}$ . To ensure mode exploration and diverse sample generation, $\sigma_1$ must be chosen to be as large as the maximum Euclidean distance between all pairs of training data points (or estimated on a random subset of data points when $N$ is very large).
Knowl 2 — Geometric Noise Scale Schedule via High-Dimensional Radial Overlap
theoretical result
Let $\mathbf{x} \in \mathbb{R}^D \sim \mathcal{N}(0, \sigma^2 \mathbf{I})$ and $r = \|\mathbf{x}\|_2$ . The radial density of $\mathbf{x}$ in hyperspherical coordinates is

$p(r) = \frac{1}{2^{D/2-1}\Gamma(D/2)} \frac{r^{D-1}}{\sigma^D} \exp\left( -\frac{r^2}{2\sigma^2} \right)$

As dimensionality $D \to \infty$ , the radial variable converges in distribution according to $r - \sqrt{D}\sigma \xrightarrow{d} \mathcal{N}(0, \sigma^2/2)$ . For high-dimensional data, the radial density $p_{\sigma_i}(r)$ for noise scale $\sigma_i$ is concentrated within the three-sigma high-density interval $I_{i-1} = [m_{i-1} - 3s_{i-1}, m_{i-1} + 3s_{i-1}]$ , where $m_{i-1} = \sqrt{D}\sigma_{i-1}$ and $s_{i-1} = \sigma_{i-1}/\sqrt{2}$ .

To ensure samples from noise level $\sigma_i$ adequately cover the high-density regions of the preceding scale $\sigma_{i-1}$ , the probability mass of $p_{\sigma_i}(r)$ within $I_{i-1}$ is set to a constant overlap probability $C \in (0, 1)$ :

$p_{\sigma_i}(r \in I_{i-1}) = \Phi\left(\sqrt{2D}(\gamma_i - 1) + 3\gamma_i\right) - \Phi\left(\sqrt{2D}(\gamma_i - 1) - 3\gamma_i\right) = C$

where $\gamma_i = \sigma_{i-1}/\sigma_i$ and $\Phi(\cdot)$ is the cumulative distribution function of the standard normal distribution $\mathcal{N}(0, 1)$ . Enforcing a constant overlap $C \approx 0.5$ forces the ratio $\gamma_i$ to be constant across all scale pairs ( $\gamma_2 = \gamma_3 = \dots = \gamma_L = \gamma$ ), proving that the sequence of noise scales $\{\sigma_i\}_{i=1}^L$ should follow a geometric progression.
Knowl 3 — Variance Evolution and Step Size Selection in Annealed Langevin Dynamics
theoretical result
Consider applying Langevin dynamics targeting $p_{\sigma_i}(\mathbf{x}) = \mathcal{N}(\mathbf{x} \mid \mathbf{0}, \sigma_i^2 \mathbf{I})$ initialized with samples from the prior scale $\mathbf{x}_0 \sim p_{\sigma_{i-1}}(\mathbf{x}) = \mathcal{N}(\mathbf{0}, \sigma_{i-1}^2 \mathbf{I})$ . Using the step size $\alpha_i = \epsilon \cdot \frac{\sigma_i^2}{\sigma_L^2}$ , the Langevin iteration updates as $\mathbf{x}_{t+1} \leftarrow \mathbf{x}_t + \alpha_i \nabla_\mathbf{x}\log p_{\sigma_i}(\mathbf{x}_t) + \sqrt{2\alpha_i}\mathbf{z}_t$ with $\mathbf{z}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ . After $T$ iterations, the state distribution is $\mathbf{x}_T \sim \mathcal{N}(\mathbf{0}, s_T^2 \mathbf{I})$ , where the variance ratio is given in closed form by:

$\frac{s_T^2}{\sigma_i^2} = \left( 1 - \frac{\epsilon}{\sigma_L^2} \right)^{2T} \left( \gamma^2 - \frac{2\epsilon}{\sigma_L^2 - \sigma_L^2\left(1 - \frac{\epsilon}{\sigma_L^2}\right)^2} \right) + \frac{2\epsilon}{\sigma_L^2 - \sigma_L^2\left(1 - \frac{\epsilon}{\sigma_L^2}\right)^2}$

where $\gamma = \sigma_{i-1}/\sigma_i$ is the common ratio of the geometric noise progression. The ratio $s_T^2/\sigma_i^2$ is independent of the scale index $i$ and data dimension $D$ .

To optimize mixing across noise scales, one fixes $T$ according to a target computation budget (e.g., $T \in [3, 5]$ when the total number of noise levels $L$ is large) and selects the step size hyperparameter $\epsilon$ via grid search such that $s_T^2/\sigma_i^2$ is maximally close to $1$ .
Knowl 4 — Noise-Conditional Score Parameterization via Output Rescaling
model/method
For an isotropic Gaussian perturbation $p_\sigma(\mathbf{x}) = \mathcal{N}(\mathbf{x} \mid \mathbf{0}, \sigma^2 \mathbf{I})$ in $\mathbb{R}^D$ , the expected gradient norm satisfies $\mathbb{E}[\|\nabla_\mathbf{x} \log p_\sigma(\mathbf{x})\|_2] \approx \frac{\sqrt{D}}{\sigma}$ , indicating that the norm of the score function scales inversely with $\sigma$ .

Rather than modulating intermediate network activations via scale-and-bias parameters in conditional normalization layers (whose memory footprint scales linearly with the number of noise scales $L$ ), the noise conditional score network $\mathbf{s}_\theta(\mathbf{x}, \sigma)$ is parameterized by directly dividing the output of an unconditional score network $\mathbf{s}_\theta(\mathbf{x})$ by $\sigma$ :

$\mathbf{s}_\theta(\mathbf{x}, \sigma) = \frac{\mathbf{s}_\theta(\mathbf{x})}{\sigma}$

The network is trained using the multi-scale denoising score matching objective:

$\mathcal{L}(\theta) = \frac{1}{2L} \sum_{i=1}^L \mathbb{E}_{p_\text{data}(\mathbf{x})}\mathbb{E}_{p_{\sigma_i}(\tilde{\mathbf{x}}\mid\mathbf{x})}\left[ \left\| \sigma_i \mathbf{s}_\theta(\tilde{\mathbf{x}}, \sigma_i) + \frac{\tilde{\mathbf{x}} - \mathbf{x}}{\sigma_i} \right\|_2^2 \right]$

This parameterization enables score estimation across thousands of discrete or continuous noise scales without increasing model memory or parameters.
Knowl 5 — Exponential Moving Average of Parameters for Score-Based Models
model/method
During training of noise-conditional score networks, stochastic gradient updates produce high variance in the score estimation, resulting in severe sample quality volatility and global color shift artifacts across checkpoints.

To stabilize the learned score field, an exponential moving average (EMA) of network parameters is maintained. Letting $\theta_i$ denote the model parameters at optimization step $i$ , shadow parameters $\theta'$ are tracked via:

$\theta' \leftarrow m\theta' + (1 - m)\theta_i$

where $m \in [0, 1)$ is the momentum decay factor (typically $m = 0.999$ ). At inference time, score evaluation during annealed Langevin dynamics uses the averaged parameters $\mathbf{s}_{\theta'}(\mathbf{x}, \sigma)$ instead of $\mathbf{s}_{\theta_i}(\mathbf{x}, \sigma)$ . This eliminates visual color shifts and consistently lowers sample Fréchet Inception Distance (FID).
Knowl 6 — Annealed Langevin Dynamics with Tweedie Denoising
algorithm
The sampling procedure generates images by progressively sampling from noise-perturbed distributions $\{p_{\sigma_i}\}_{i=1}^L$ with descending noise standard deviations $\sigma_1 > \sigma_2 > \dots > \sigma_L$ . At each scale $\sigma_i$ , $T$ steps of Langevin dynamics are performed using step size $\alpha_i = \epsilon \cdot \sigma_i^2 / \sigma_L^2$ . After the final Langevin step at scale $\sigma_L$ , Tweedie's formula is applied as an analytical denoising step to subtract the residual Gaussian noise $\mathcal{N}(0, \sigma_L^2 \mathbf{I})$ .
Input: Noise scales { $\sigma_i$ } $_{i=1}^L$ , step size scaling factor $\epsilon$ , iterations per scale $T$ , score network $\mathbf{s}_\theta(\mathbf{x}, \sigma)$ , initial sample $\mathbf{x}_0$
Output: Denoised generated sample $\mathbf{x}_\text{final}$
for $i \leftarrow 1$ to $L$ do
     $\alpha_i \leftarrow \epsilon \cdot \sigma_i^2 / \sigma_L^2$
    for $t \leftarrow 1$ to $T$ do
        Draw $\mathbf{z}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$
         $\mathbf{x}_t \leftarrow \mathbf{x}_{t-1} + \alpha_i \mathbf{s}_\theta(\mathbf{x}_{t-1}, \sigma_i) + \sqrt{2\alpha_i} \mathbf{z}_t$
     $\mathbf{x}_0 \leftarrow \mathbf{x}_T$
$\mathbf{x}_\text{final} \leftarrow \mathbf{x}_T + \sigma_L^2 \mathbf{s}_\theta(\mathbf{x}_T, \sigma_L)$
return $\mathbf{x}_\text{final}$
Knowl 7 — Latent Spherical Interpolation via Langevin Noise Trajectories
model/method
Continuous semantic interpolations between two generated samples $\mathbf{x}^{(1)}$ and $\mathbf{x}^{(2)}$ are constructed by interpolating the injected Gaussian noise sequences used during annealed Langevin dynamics.

Let $\{\mathbf{z}_{ij}\}_{1 \le i \le L, 1 \le j \le T}$ denote the full tensor of standard Gaussian noise vectors injected at noise level $i$ and step $j$ during sampling. For two trajectories sharing the same initialization $\mathbf{x}_0$ but driven by noise realizations $\{\mathbf{z}_{ij}^{(1)}\}$ and $\{\mathbf{z}_{ij}^{(2)}\}$ , $N$ intermediate interpolated samples are produced. The $k$ -th intermediate sample ( $1 \le k \le N$ ) is generated by running annealed Langevin dynamics initialized at $\mathbf{x}_0$ with the spherically interpolated noise trajectory:

$\mathbf{z}_{ij}^{(k)} = \cos\left(\frac{k\pi}{2(N+1)}\right) \mathbf{z}_{ij}^{(1)} + \sin\left(\frac{k\pi}{2(N+1)}\right) \mathbf{z}_{ij}^{(2)}$

This procedure produces smooth, high-fidelity semantic transitions across complex image domains without requiring an explicit latent encoder.

Knowl 8 — Quantitative Generation Performance on CIFAR-10 and CelebA

data/table

NCSNv2 incorporates theoretical noise scale selection ( $\{\sigma_i\}$ ), network output rescaling ( $1/\sigma$ ), step size calibration ( $\{\alpha_i\}$ ), parameter EMA, and Tweedie denoising. It achieves lower FID scores on unconditional CIFAR-10 ( $32 \times 32$ ) and CelebA ( $64 \times 64$ ) compared to the baseline NCSN and competitive likelihood-based and GAN models.

Model	Inception $\uparrow$	FID $\downarrow$
CIFAR-10 Unconditional
PixelCNN	4.60	65.93
IGEBM	6.02	40.58
WGAN-GP	7.86 $\pm$ .07	36.4
SNGAN	8.22 $\pm$ .05	21.7
NCSN	8.87 $\pm$ .12	25.32
NCSN (w/ denoising)	7.32 $\pm$ .12	29.8
NCSNv2 (w/o denoising)	8.73 $\pm$ .13	31.75
NCSNv2 (w/ denoising)	8.40 $\pm$ .07	10.87
CelebA $64 \times 64$
NCSN (w/o denoising)	-	26.89
NCSN (w/ denoising)	-	25.30
NCSNv2 (w/o denoising)	-	28.86
NCSNv2 (w/ denoising)	-	10.23

The combined techniques along with Tweedie denoising improve the CIFAR-10 FID of score-based models from $25.32$ (NCSN) down to $10.87$ (NCSNv2), and the CelebA $64 \times 64$ FID from $25.30$ down to $10.23$ .

Knowl 9 — Human Perceptual Evaluation (HYPE-infinity) on CelebA

data/table

Human perceptual fidelity measured by the $\text{HYPE}_\infty$ benchmark (where higher percentages indicate higher realism, up to $50\%$ ) shows that NCSNv2 achieves sample quality competitive with progressive GAN architectures on CelebA $64 \times 64$ , whereas the original NCSN scores poorly due to color shifts.

Model	$\text{HYPE}_\infty$ (%) $\uparrow$	Fakes Error (%)	Reals Error (%)	Std.
StyleGAN*	50.7	62.2	39.3	1.3
ProgressiveGAN	40.3	46.2	34.4	0.9
BEGAN	10.0	6.2	13.8	1.6
WGAN-GP	3.8	1.7	5.9	0.6
NCSN	19.8	22.3	17.3	0.4
NCSNv2	37.3	49.8	24.8	0.5

*Evaluated with truncation tricks. While vanilla NCSN scored $19.8\%$ , NCSNv2 achieves $37.3\%$ , showing that the proposed techniques bring score-based generative sample fidelity to a level comparable to ProgressiveGAN ( $40.3\%$ ).

Knowl 10 — Hyperparameter and Scale Configurations for High-Resolution Datasets

data/table

NCSNv2 parameters $\sigma_1$ , $L$ , $T$ , and $\epsilon$ are configured analytically across varying image resolutions using Techniques 1–4. The initial noise scale $\sigma_1$ is set to the maximum pairwise dataset Euclidean distance, the number of noise levels $L$ is set to satisfy radial overlap $C \approx 0.5$ , and step size parameter $\epsilon$ is matched to $T$ using the closed-form variance relation.

Model	Dataset	$\sigma_1$	$L$	$T$	$\epsilon$	Batch Size	Iterations
NCSN	CIFAR-10 $32^2$	1	10	100	2e-5	128	300k
NCSN	CelebA $64^2$	1	10	100	2e-5	128	210k
NCSN	LSUN church $96^2$	1	10	100	2e-5	128	200k
NCSN	LSUN bedroom $128^2$	1	10	100	2e-5	64	150k
NCSNv2	CIFAR-10 $32^2$	50	232	5	6.2e-6	128	300k
NCSNv2	CelebA $64^2$	90	500	5	3.3e-6	128	210k
NCSNv2	LSUN church $96^2$	140	788	4	4.9e-6	128	200k
NCSNv2	LSUN bedroom/tower $128^2$	190	1086	3	1.8e-6	128	150k
NCSNv2	FFHQ $256^2$	348	2311	3	0.9e-7	32	80k

For higher image resolutions, $\sigma_1$ scales from $50$ to $348$ , which requires $L$ to increase from $232$ to $2311$ geometric noise levels with $T \in [3, 5]$ Langevin steps per level, scaling score-based generative modeling up to $256 \times 256$ images.

Coverage note — None was omitted; all primary theoretical propositions, derivation-free formulas (initial noise bound, radial Gaussian limit, variance ratio formula), algorithmic enhancements (Techniques 1-5), and key quantitative results across resolutions are fully represented.

References

1.Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pages 11895–11907, 2019.
2.Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
3.Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(Apr):695–709, 2005.
4.Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661–1674, 2011.
5.Martin Raphan and Eero P Simoncelli. Least squares estimation without priors or supervision. Neural computation, 23(2):374–420, 2011.
6.Gareth O Roberts, Richard L Tweedie, et al. Exponential convergence of langevin distributions and their discrete approximations. Bernoulli, 2(4):341–363, 1996.
7.Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 681–688, 2011.
8.Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, page 204, 2019.
9.Alexia Jolicoeur-Martineau, Rémi Piché-Taillefer, Ioannis Mitliagkas, and Rémi Tachet des Combes. Adversarial score matching and improved sampling for image generation. arXiv preprint arXiv:2009.05475, 2020.
10.Saeed Saremi and Aapo Hyvarinen. Neural empirical bayes. Journal of Machine Learning Research, 20:1–23, 2019.
11.Zengyi Li, Yubei Chen, and Friedrich T Sommer. Learning energy-based models in high-dimensional spaces with multi-scale denoising score matching. arXiv, pages arXiv–1910, 2019.
12.Zahra Kadkhodaie and Eero P Simoncelli. Solving linear inverse problems using the prior implicit in a denoiser. arXiv preprint arXiv:2007.13640, 2020.
13.Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems, pages 6626–6637, 2017.
14.Bradley Efron. Tweedie’s formula and selection bias. Journal of the American Statistical Association, 106(496):1602–1614, 2011.
15.Erik W Grafarend. Linear and nonlinear models: fixed effects, random effects, and mixed models. de Gruyter, 2006.
16.Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
17.Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelcnn decoders. In Advances in neural information processing systems, pages 4790–4798, 2016.
18.Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv:1903.08689, 2019.
19.Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in neural information processing systems, pages 5767–5777, 2017.
20.Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
21.Shane Barratt and Rishi Sharma. A note on the inception score. arXiv preprint arXiv:1801.01973, 2018.
22.Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems, pages 5228–5237, 2018.
23.Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems, pages 14837–14847, 2019.
24.Guosheng Lin, Anton Milan, Chunhua Shen, and Ian Reid. Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1925–1934, 2017.
25.Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289, 2015.
26.Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
27.Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.
28.Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4401–4410, 2019.
29.Sharon Zhou, Mitchell Gordon, Ranjay Krishna, Austin Narcomey, Li F Fei-Fei, and Michael Bernstein. Hype: A benchmark for human eye perceptual evaluation of generative models. In Advances in Neural Information Processing Systems, pages 3444–3456, 2019.
30.Jiaming Song and Stefano Ermon. Bridging the gap between f-gans and wasserstein gans. arXiv preprint arXiv:1910.09779, 2019.
31.Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237, 2019.
32.Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
33.David Berthelot, Thomas Schumm, and Luke Metz. Began: Boundary equilibrium generative adversarial networks. arXiv preprint arXiv:1703.10717, 2017.

Citation

MLA

Song, Y., and S. Ermon. “Improved Techniques for Training Score-Based Generative Models”. Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 12438–48, https://proceedings.neurips.cc/paper_files/paper/2020/file/92c3b916311a5517d9290576e3ea37ad-Paper.pdf.

APA

Song, Y., & Ermon, S. (2020). Improved Techniques for Training Score-Based Generative Models. Advances in Neural Information Processing Systems, 33, 12438–12448. https://proceedings.neurips.cc/paper_files/paper/2020/file/92c3b916311a5517d9290576e3ea37ad-Paper.pdf

Chicago

Song, Y., and S. Ermon. 2020. “Improved Techniques for Training Score-Based Generative Models”. Advances in Neural Information Processing Systems 33: 12438–48. https://proceedings.neurips.cc/paper_files/paper/2020/file/92c3b916311a5517d9290576e3ea37ad-Paper.pdf.

Harvard

Song, Y. and Ermon, S. (2020) “Improved Techniques for Training Score-Based Generative Models”, Advances in Neural Information Processing Systems. Curran Associates, Inc., pp. 12438–12448. Available at: https://proceedings.neurips.cc/paper_files/paper/2020/file/92c3b916311a5517d9290576e3ea37ad-Paper.pdf.

Vancouver

1. Song Y, Ermon S (2020) Improved Techniques for Training Score-Based Generative Models. In: Advances in Neural Information Processing Systems. Curran Associates, Inc., pp 12438–12448

BibTeX

@inproceedings{song2020improved,
  title = {Improved Techniques for Training Score-Based Generative Models},
  author = {Song, Yang and Ermon, Stefano},
  year = {2020},
  booktitle = {Advances in Neural Information Processing Systems},
  publisher = {Curran Associates, Inc.},
  volume = {33},
  pages = {12438-12448},
  url = {https://proceedings.neurips.cc/paper_files/paper/2020/file/92c3b916311a5517d9290576e3ea37ad-Paper.pdf}
}

Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

Improved Techniques for Training Score-Based Generative Models

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Initial Noise Scale Selection Based on Maximum Pairwise Distance

Knowl 2 — Geometric Noise Scale Schedule via High-Dimensional Radial Overlap

Knowl 3 — Variance Evolution and Step Size Selection in Annealed Langevin Dynamics

Knowl 4 — Noise-Conditional Score Parameterization via Output Rescaling

Knowl 5 — Exponential Moving Average of Parameters for Score-Based Models

Knowl 6 — Annealed Langevin Dynamics with Tweedie Denoising

Knowl 7 — Latent Spherical Interpolation via Langevin Noise Trajectories

Knowl 8 — Quantitative Generation Performance on CIFAR-10 and CelebA

Knowl 9 — Human Perceptual Evaluation (HYPE-infinity) on CelebA

Knowl 10 — Hyperparameter and Scale Configurations for High-Resolution Datasets

References

Citation

BibTeX

Access the Paper