Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting

Shikuang DengYuhang LiShanghang ZhangShi Gu

article2022ICLR350 citations

Proposes a temporal efficient training method that re-weights surrogate gradients in spiking neural networks to converge to flatter minima, substantially boosting generalization performance across standard vision and neuromorphic datasets.

Listen

Brain-inspired spiking neural networks (SNNs) offer major advantages for energy-efficient computing and fast processing on specialized neuromorphic hardware. However, training deep spiking networks directly from scratch has remained a critical bottleneck. Because spiking neurons rely on non-differentiable binary activations, standard gradient-based optimization cannot be directly applied. Conventional direct training methods use surrogate gradients to approximate derivatives, but this creates a fundamental mismatch between the loss landscape and gradient estimates. As a result, standard training easily gets trapped in sharp local minima, yielding poor generalizability on test data and high computational training costs.

The main objective of the article is to demonstrate that optimizing network outputs at every individual time step, rather than only evaluating the time-averaged final output, fundamentally improves generalization and training efficiency in deep spiking neural networks. To address the issue, the authors introduce the Temporal Efficient Training (TET) framework and a complementary Time Inheritance Training (TIT) acceleration scheme, evaluating their mathematical foundations and empirical performance across diverse image and neuromorphic datasets.

The authors designed a re-weighted loss function that constrains the pre-synaptic output distribution at each discrete time step alongside a regularizing mean squared error term. They evaluated the method using standard convolutional and residual network architectures—including ResNet-19, ResNet-34, and custom architectures—tested on standard static image benchmarks (CIFAR-10, CIFAR-100, and ImageNet) as well as the challenging neuromorphic dataset DVS-CIFAR10. They also carried out loss landscape visualizations, convergence proofs, and ablation studies comparing standard training against the new framework.

The analysis produced several vital findings. First, the proposed training method consistently outperformed existing state-of-the-art spiking models across all evaluated benchmarks. On the challenging DVS-CIFAR10 neuromorphic dataset, the method achieved an 83.17% top-1 accuracy, representing an improvement of over 10% compared to prior published results. Second, on CIFAR-100, the method improved accuracy by more than 3% across various simulation lengths, coming within 0.63% of equivalent artificial neural network baselines. Third, loss landscape analysis confirmed that the approach reliably guides optimization toward flatter, highly generalizable minima while standard training stalls in sharp valleys. Finally, the time inheritance scheme cut total training time roughly in half by first training on short simulation lengths before fine-tuning on longer time horizons.

These findings indicate that directly optimizing temporal dynamics resolves the long-standing generalization deficit of spiking networks without modifying their energy-efficient inference mechanisms. By converging to flatter loss regions, spiking models achieve performance parity with conventional neural networks while maintaining low computational and energy footprints. The halved training timeline substantially mitigates the development expense and carbon footprint associated with training temporal models.

Organizations developing low-power artificial intelligence, edge devices, or neuromorphic applications should adopt per-step temporal training and inheritance strategies to accelerate development cycles and boost model accuracy. Teams deploying these pipelines should tune the regularization strength carefully based on task type, using moderate penalties for static datasets and smaller penalties for sparse neuromorphic streams. Further work is recommended to validate the approach across larger-scale neuromorphic video streams and explore automatic hyperparameter scheduling during temporal scaling.

The reported evidence provides high confidence in the method's effectiveness across standard image classification benchmarks. Nevertheless, practitioners should exercise caution regarding specific limitations: neuromorphic datasets exhibit high noise and sparsity, which can disrupt early time-step training if regularization hyperparameters are miscalibrated. Additionally, while the training efficiency gains are substantial, the training phase still incurs recurrent memory overhead that requires careful hardware resource management.

Peking University Peng Cheng Laboratory University of Electronic Science and Technology of China Yale University backpropagation algorithm energy-efficient gradient algorithm Temporal Efficient Training spiking neural networks Gradient Re-weighting

arXiv: 2202.11946 Gus-Lab/temporal_efficient_training

Paper: Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-based optimization to spiking neural networks, Emre O. Neftci et al. (2019). This paper establishes the foundational framework of surrogate gradient learning in spiking neural networks to overcome the non-differentiability barrier, providing essential background for the optimization issues addressed by TET.
Paper: Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks, Yujie Wu et al. (2017). It introduces spatio-temporal backpropagation through iterative leaky integrate-and-fire dynamics with continuous derivative approximations, forming the direct training baseline that TET seeks to improve.
Paper: Deep Learning in Spiking Neural Networks, Amirhossein Tavanaei et al. (2018). This comprehensive survey outlines the key paradigms and challenges in direct training and gradient approximation for deep spiking neural networks.
Paper: Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation, Yoshua Bengio et al. (2013). It provides fundamental principles for estimating and propagating gradients through hard non-differentiable thresholds and stochastic binary units, underpinning modern surrogate gradient techniques.
Paper: Averaging Weights Leads to Wider Optima and Better Generalization, Pavel Izmailov et al. (2018). This work establishes the connection between converging to wider loss-landscape optima and achieving superior generalization, which motivates TET's gradient re-weighting objective.

No sufficiently relevant recommendations were found.

Cover for Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting

Abstract

Recently, brain-inspired spiking neuron networks (SNNs) have attracted widespread research interest because of their event-driven and energy-efficient characteristics. Still, it is difficult to efficiently train deep SNNs due to the non-differentiability of its activation function, which disables the typically used gradient descent approaches for traditional artificial neural networks (ANNs). Although the adoption of surrogate gradient (SG) formally allows for the back-propagation of losses, the discrete spiking mechanism actually differentiates the loss landscape of SNNs from that of ANNs, failing the surrogate gradient methods to achieve comparable accuracy as for ANNs. In this paper, we first analyze why the current direct training approach with surrogate gradient results in SNNs with poor generalizability. Then we introduce the temporal efficient training (TET) approach to compensate for the loss of momentum in the gradient descent with SG so that the training process can converge into flatter minima with better generalizability. Meanwhile, we demonstrate that TET improves the temporal scalability of SNN and induces a temporal inheritable training for acceleration. Our method consistently outperforms the SOTA on all reported mainstream datasets, including CIFAR-10/100 and ImageNet. Remarkably on DVS-CIFAR10, we obtained 83 $\%$ top-1 accuracy, over 10 $\%$ improvement compared to existing state of the art. Codes are available at \url{this https URL}.

Topics

backpropagation algorithm energy-efficient gradient algorithm spiking neural networks temporal scalability feedforward neural networks network architecture

Keywords

Temporal Efficient Training spiking neural networks Gradient Re-weighting neuromorphic computing Spatio-temporal backpropagation temporal spike sequence neuromorphic hardware neuromorphic data augmentation inter-spike intervals spike frequency

1 Introduction
2 Related Work
3 Preliminary
3.1 Iterative LIF Model
3.2 Surrogate gradient
3.3 Batch Normalization for SNN
4 Methodology
4.1 Formula of Training SNN with surrogate gradients
4.2 Convergence of Gradient Descent for SDT v.s. TET
4.3 Time Inheritance Training
5 Experiments
5.1 Model Validation and Ablation Study
5.2 Comparison to exiting works
6 Conclusion
7 Acknowledgment
References
References
A Appendix
A.1 Dataset and Training detail
A.2 ℒSDT\mathcal{L}_{\text{SDT}} loss landscape of ResNet-19
A.3 Effect of ℒMSE\mathcal{L}_{\text{MSE}}
A.4 statistical results
A.5 Time scalability robustness of SDT and TET.

Knowls

Knowl 1 — Temporal Efficient Training Objective Function
equation
In direct training of Spiking Neural Networks (SNNs), rather than computing the cross-entropy loss over the time-averaged output pre-synaptic potential, Temporal Efficient Training (TET) computes the cross-entropy loss at every individual time step. To regularize the potential distribution and prevent extreme outlier potentials at single timesteps, a mean squared error (MSE) regularization term is added.

For a total simulation duration $T$ , target one-hot label vector $\hat{y} \in \{0, 1\}^C$ for class label $y \in \{1, \dots, C\}$ , output layer pre-synaptic input vector $O(t) \in \mathbb{R}^C$ at timestep $t$ , cross-entropy loss function $\mathcal{L}_{\text{CE}}$ , and regularization weight hyperparameter $\lambda \in [0, 1]$ , the total training loss is: $\mathcal{L}_{\text{TOTAL}} = (1 - \lambda)\mathcal{L}_{\text{TET}} + \lambda \mathcal{L}_{\text{MSE}}$ where: $\mathcal{L}_{\text{TET}} = \frac{1}{T} \sum_{t=1}^T \mathcal{L}_{\text{CE}}(O(t), y)$ $\mathcal{L}_{\text{MSE}} = \frac{1}{T} \sum_{t=1}^T \text{MSE}(O(t), \phi)$ Here $\phi$ is a constant anchor set to the neuron firing threshold $V_{\text{th}}$ , and $\text{MSE}(O(t), \phi) = \frac{1}{C}\sum_{c=1}^C (O_c(t) - \phi)^2$ .

The gradient of $\mathcal{L}_{\text{TET}}$ with respect to network synaptic weights $W$ under a softmax inference function $S(\cdot)$ is: $\frac{\partial \mathcal{L}_{\text{TET}}}{\partial W} = \frac{1}{T} \sum_{t=1}^T [S(O(t)) - \hat{y}] \cdot \frac{\partial O(t)}{\partial W}$
Knowl 2 — Upper Bound of Standard Direct Training Loss by Temporal Efficient Training Loss
theoretical result
Let $T \in \mathbb{Z}^+$ be the total simulation duration, $y \in \{1, \dots, C\}$ be the true class label with one-hot encoding $\hat{y} \in \{0, 1\}^C$ , and $O(t) = (O_1(t), \dots, O_C(t))^T \in \mathbb{R}^C$ be the output layer pre-synaptic input vector at time step $t$ .

Let the Standard Direct Training (SDT) loss be defined as the cross-entropy of the temporal average output: $\mathcal{L}_{\text{SDT}} = \mathcal{L}_{\text{CE}}\left(\frac{1}{T} \sum_{t=1}^T O(t), y\right) = -\sum_{i=1}^C \hat{y}_i \log \left( S_i\left(\frac{1}{T}\sum_{t=1}^T O(t)\right) \right)$ where $S_i(z) = \frac{\exp(z_i)}{\sum_{j=1}^C \exp(z_j)}$ is the softmax probability for class $i$ .

Let the Temporal Efficient Training (TET) loss be defined as the average cross-entropy across all time steps: $\mathcal{L}_{\text{TET}} = \frac{1}{T}\sum_{t=1}^T \mathcal{L}_{\text{CE}}(O(t), y) = -\frac{1}{T}\sum_{t=1}^T \sum_{i=1}^C \hat{y}_i \log S_i(O(t))$

Then, $\mathcal{L}_{\text{SDT}}$ is strictly upper-bounded by $\mathcal{L}_{\text{TET}}$ : $\mathcal{L}_{\text{SDT}} \le \mathcal{L}_{\text{TET}}$ Consequently, driving $\mathcal{L}_{\text{TET}} \to 0$ during optimization guarantees that the standard inference loss $\mathcal{L}_{\text{SDT}} \to 0$ .
Knowl 3 — Gradient Re-Weighting and Flat-Minima Convergence in TET
model/method
In Standard Direct Training (SDT), the weight gradient is given by: $\frac{\partial \mathcal{L}_{\text{SDT}}}{\partial W} = \frac{1}{T} \sum_{t=1}^T [S(O_{\text{mean}}) - \hat{y}] \frac{\partial O(t)}{\partial W}$ where $O_{\text{mean}} = \frac{1}{T}\sum_{t=1}^T O(t)$ , $S(\cdot)$ is softmax, $\hat{y}$ is the one-hot target, and $W$ denotes the synaptic weights. When the network prediction $S(O_{\text{mean}})$ approximates $\hat{y}$ near a local minimum, the error term $S(O_{\text{mean}}) - \hat{y}$ approaches zero for all timesteps $t$ , ignoring the magnitudes of the partial derivatives $\frac{\partial O(t)}{\partial W}$ . Because surrogate gradient (SG) approximations introduce a mathematical mismatch between continuous surrogate derivatives and discrete step activations, accumulated momentum dissipates rapidly near local minima, trapping SDT in sharp, poorly generalizing local minima.

In Temporal Efficient Training (TET), the weight gradient is: $\frac{\partial \mathcal{L}_{\text{TET}}}{\partial W} = \frac{1}{T} \sum_{t=1}^T [S(O(t)) - \hat{y}] \frac{\partial O(t)}{\partial W}$ Because individual per-step outputs $O(t)$ at early timesteps rarely achieve zero classification error during training (even when the temporal average $O_{\text{mean}}$ is correct), the error term $S(O(t)) - \hat{y}$ remains non-zero. For $\mathcal{L}_{\text{TET}}$ to converge, the optimization is forced to drive the sensitivity term $\frac{\partial O(t)}{\partial W}$ toward zero. This gradient re-weighting increases the gradient norm around sharp minima and guides training toward flat local minima where parameter perturbations do not induce substantial changes in $O(t)$ , resulting in improved test generalization.
Knowl 4 — Time Inheritance Training
model/method
Time Inheritance Training (TIT) is an acceleration method for directly trained spiking neural networks (SNNs) that exploits the temporal scalability induced by Temporal Efficient Training (TET).

Because TET optimizes the output distribution independently at every time point rather than solely optimizing an integrated temporal aggregate, the learned representations transfer effectively across simulation durations. TIT executes in two stages:
1. Short-Latency Pre-training: Train the SNN from scratch for a standard schedule (e.g., 300 epochs) using a small simulation length $T_{\text{initial}}$ (e.g., $T=2$ ).
2. Target-Latency Fine-tuning: Increase the simulation length to the desired target duration $T_{\text{target}}$ (e.g., $T=4$ or $T=6$ ), initialize weights from the first stage, and fine-tune for a short schedule (e.g., 30 to 50 epochs) with a reduced learning rate (e.g., $10^{-4}$ ).
Assuming training cost per epoch scales linearly with simulation length ( $T \cdot t_s$ , where $t_s$ is single-step epoch time), training an SNN from scratch with $T=6$ for 300 epochs requires $1800 t_s$ . In contrast, TIT with $T_{\text{initial}}=2$ for 300 epochs followed by $T_{\text{target}}=6$ for 50 epochs requires $(300 \times 2 + 50 \times 6) t_s = 900 t_s$ , reducing the total training compute cost by $50\%$ while achieving equal or superior accuracy.
Knowl 5 — Algorithm for Temporal Efficient Training
algorithm
Temporal Efficient Training (TET) trains an SNN on mini-batches by evaluating per-step cross-entropy and MSE regularization during training, while classifying inputs based on the temporal average pre-synaptic potential across all time steps during inference.
Input: SNN model parameters $W$ ; Simulation duration $T$ ; Firing threshold $V_{\text{th}}$ ; Regularization weight $\lambda$ ; Target potential anchor $\phi = V_{\text{th}}$ ; Training dataset with $I_{\text{train}}$ iterations per epoch; Validation dataset with $I_{\text{val}}$ iterations per epoch.
Output: Trained SNN model parameters $W$ .
for $i = 1, 2, \dots, I_{\text{train}}$ do
    Obtain mini-batch training samples and one-hot labels $Y^{(i)}$
    for $t = 1, 2, \dots, T$ do
        Compute output layer pre-synaptic potential vector $O^{(i)}(t)$
    end for
    Compute per-step TET loss: $\mathcal{L}_{\text{TET}} = \frac{1}{T} \sum_{t=1}^T \mathcal{L}_{\text{CE}}(O^{(i)}(t), Y^{(i)})$
    Compute MSE regularization: $\mathcal{L}_{\text{MSE}} = \frac{1}{T} \sum_{t=1}^T \text{MSE}(O^{(i)}(t), \phi)$
    Compute total loss: $\mathcal{L}_{\text{TOTAL}} = (1 - \lambda)\mathcal{L}_{\text{TET}} + \lambda \mathcal{L}_{\text{MSE}}$
    Backpropagate gradients $\frac{\partial \mathcal{L}_{\text{TOTAL}}}{\partial W}$ using surrogate gradients for spiking activations
    Update parameters $W$ using optimizer
end for
for $i = 1, 2, \dots, I_{\text{val}}$ do
    Obtain mini-batch validation samples and labels $Y^{(i)}$
    for $t = 1, 2, \dots, T$ do
        Compute output layer pre-synaptic potential vector $O^{(i)}(t)$
    end for
    Compute temporal average output: $O_{\text{mean}}^{(i)} = \frac{1}{T} \sum_{t=1}^T O^{(i)}(t)$
    Predict class $\hat{c} = \arg\max_c (O_{\text{mean}}^{(i)})_c$ and compare with $Y^{(i)}$ for accuracy
end for
Hyperparameter choices:
- Neuron model: Leaky Integrate-and-Fire (LIF) with decay factor $\tau = 0.5$ , firing threshold $V_{\text{th}} = 1.0$ , reset potential $u(0) = 0$ , and non-leaky integration on the final layer.
- Surrogate gradient: triangular derivative $\frac{\partial a(t)}{\partial u(t)} = \frac{1}{\gamma^2}\max(0, \gamma - |u(t) - V_{\text{th}}|)$ with constraint factor $\gamma$ .
- Regularization parameter $\lambda$ : $\lambda = 0.05$ for CIFAR-10 and CIFAR-100; $\lambda = 0.001$ for ImageNet and DVS-CIFAR10.

Knowl 6 — Benchmark Accuracy Comparison Across Static and Neuromorphic Datasets

data/table

The table below compares Temporal Efficient Training (TET) and Time Inheritance Training (TIT) against prior state-of-the-art direct training methods (STBP, TSSL-BP, STBP-tdBN, SEW ResNet, LIAF) and conversion/hybrid methods on CIFAR-10, CIFAR-100, ImageNet, and DVS-CIFAR10. Across all benchmarks, TET outperforms previous SNN methods and approaches ANN-level accuracy. On DVS-CIFAR10, TET achieves an $11.47\%$ top-1 accuracy improvement over prior published results.

Dataset	Model	Methods	Architecture	Simulation Length ( $T$ )	Accuracy (%)
CIFAR-10	Rathi et al. (2019)	Hybrid training	ResNet-20	250	92.22
	Rathi Roy (2020)	Diet-SNN	ResNet-20	10	92.54
	Wu et al. (2018)	STBP	CIFARNet	12	89.83
	Wu et al. (2019)	STBP NeuNorm	CIFARNet	12	90.53
	Zhang Li (2020)	TSSL-BP	CIFARNet	5	91.41
	Zheng et al. (2021)	STBP-tdBN	ResNet-19	6	93.16
				4	92.92
				2	92.34
	TET (Ours)	TET	ResNet-19	6	94.50 $\pm$ 0.07
				4	94.44 $\pm$ 0.08
				2	94.16 $\pm$ 0.03
	ANN baseline	ANN	ResNet-19	1	94.97
CIFAR-100	Rathi et al. (2019)	Hybrid training	VGG-11	125	67.87
	Rathi Roy (2020)	Diet-SNN	ResNet-20	5	64.07
	Zheng et al. (2021)	STBP-tdBN	ResNet-19	6	71.12 $\pm$ 0.57
				4	70.86 $\pm$ 0.22
				2	69.41 $\pm$ 0.08
	TET (Ours)	TET	ResNet-19	6	74.72 $\pm$ 0.28
				4	74.47 $\pm$ 0.15
				2	72.87 $\pm$ 0.10
	ANN baseline	ANN	ResNet-19	1	75.35
ImageNet	Rathi et al. (2019)	Hybrid training	ResNet-34	250	61.48
	Sengupta et al. (2018)	SPIKE-NORM	ResNet-34	2500	69.96
	Zheng et al. (2021)	STBP-tdBN	Spiking-ResNet-34	6	63.72
	Fang et al. (2021)	SEW ResNet	SEW-ResNet-34	4	67.04
	TET (Ours)	TET	Spiking-ResNet-34	6	64.79
	TET (Ours)	TET	SEW-ResNet-34	4	68.00
DVS-CIFAR10	Zheng et al. (2021)	STBP-tdBN	ResNet-19	10	67.80
	Kugele et al. (2020)	Streaming Rollout	DenseNet	10	66.80
	Wu et al. (2021)	Conv3D	LIAF-Net	10	71.70
	Wu et al. (2021)	LIAF	LIAF-Net	10	70.40
	TET (Ours)	TET	VGGSNN	10	77.33 $\pm$ 0.21
	TET (Ours) with DA	TET	VGGSNN	10	83.17 $\pm$ 0.15

DA denotes neuromorphic data augmentation (random horizontal flip and random roll within 5 pixels). On CIFAR-100, TET yields a $>3\%$ accuracy gain across all simulation lengths ( $T=2, 4, 6$ ). On DVS-CIFAR10, TET without DA achieves $77.33\%$ (versus $73.3\%$ for SDT) and $83.17\%$ with DA, exceeding previous SOTA ( $71.70\%$ by Conv3D LIAF-Net) by $11.47\%$ .

Knowl 7 — Dynamic Escape from Suboptimal Local Minima by Switching from SDT to TET
empirical result
When a VGGSNN model is trained on DVS-CIFAR10 for 300 epochs using the Standard Direct Training loss ( $\mathcal{L}_{\text{SDT}}$ ), optimization becomes trapped in a sharp local minimum within the first 200 epochs where test accuracy plateaus and the Temporal Efficient Training loss ( $\mathcal{L}_{\text{TET}}$ ) remains elevated.

Switching the training loss from $\mathcal{L}_{\text{SDT}}$ to $\mathcal{L}_{\text{TET}}$ at epoch 200 causes an immediate and rapid decrease in both $\mathcal{L}_{\text{SDT}}$ and $\mathcal{L}_{\text{TET}}$ on the test set, accompanied by a steep improvement in test accuracy. This confirms experimentally that TET gradients provide the necessary driving force to escape suboptimal sharp local minima and converge into flatter, better-generalizing regions of the parameter space.

Knowl 8 — Per-Step Potential Accuracy and Temporal Scalability Under TET vs SDT

empirical result

Evaluating the classification accuracy of output membrane potential increments $O(t)$ at each distinct timestep $t$ demonstrates the temporal robustness of models trained with Temporal Efficient Training (TET, $\mathcal{L}_{\text{TET}}$ ) compared to Standard Direct Training (SDT, $\mathcal{L}_{\text{SDT}}$ ).

When ResNet-19 models trained on CIFAR-100 for 300 epochs at initial simulation lengths $T=3$ or $T=4$ have their simulation length extended to $T=8$ without fine-tuning, the single-step potential test accuracy across timesteps $t=1, \dots, 8$ is:

Method	$t=1$	$t=2$	$t=3$	$t=4$	$t=5$	$t=6$	$t=7$	$t=8$
SDT ( $T=3$ )	55.61%	57.95%	56.87%	55.09%	57.56%	53.54%	57.72%	54.04%
SDT ( $T=4$ )	37.96%	61.78%	55.03%	56.64%	57.47%	54.24%	58.74%	55.48%
TET ( $T=3$ )	65.97%	72.22%	71.78%	70.55%	71.90%	69.57%	72.15%	69.78%
TET ( $T=4$ )	62.17%	71.57%	71.05%	72.08%	71.77%	71.23%	71.81%	71.36%

TET achieves consistently higher per-step classification accuracy ( $>71\%$ for all $t \ge 2$ ) and substantially lower accuracy variance across timesteps ( $1.41$ and $0.08$ for TET vs $3.81$ and $4.04$ for SDT). When the simulation duration is further extended to $T=64$ without fine-tuning, TET models retain stable accuracy with an overall variance of $1.80$ , compared to an accuracy variance of $11.13$ for SDT models.

Knowl 9 — Energy Consumption and Inference Efficiency of TET-Trained SNNs
empirical result
In 45nm CMOS neuromorphic hardware, a synaptic addition operation consumes $0.9\,\text{pJ}$ while a synaptic floating-point multiplication consumes $4.6\,\text{pJ}$ . In SNN architectures where only the first layer performs floating-point multiplications on static inputs and subsequent layers perform event-driven binary additions, models trained via Temporal Efficient Training (TET) demonstrate superior trade-offs between energy consumption and classification accuracy.

On CIFAR-100 with ResNet-19:
- At $T=2$ , TET achieves $72.87\%$ accuracy with an energy consumption of $\approx 55\,\text{mJ}$ . This exceeds the accuracy of SDT at $T=2$ ( $69.41\%$ at $\approx 55\,\text{mJ}$ ) as well as SDT at $T=6$ ( $71.12\%$ , which requires $\approx 170\,\text{mJ}$ ), reducing energy consumption by over $67\%$ while achieving higher accuracy.
- Across all simulation lengths $T \in \{2, 3, 4, 5, 6\}$ , TET strictly dominates SDT on the accuracy-energy frontier.
Knowl 10 — Surrogate Gradient Mismatch as the Root Cause of SDT Deficit
empirical result
To isolate whether the generalization gap between Standard Direct Training (SDT) and Temporal Efficient Training (TET) stems specifically from the mismatch between non-differentiable step activations and surrogate gradients (SG), the Heaviside step firing function $a(t+1) = \Theta(u(t+1) - V_{\text{th}})$ was substituted with a smooth, differentiable Sigmoid activation $\sigma(k \cdot u)$ .

For a 5-layer convolutional network (CNN-5) trained on CIFAR-10 at simulation length $T=4$ across scaling factors $k \in \{1, 10, 20\}$ :
- $k=1$ : SDT achieves $88.00 \pm 0.15\%$ , TET achieves $87.63 \pm 0.38\%$ .
- $k=10$ : SDT achieves $88.83 \pm 0.32\%$ , TET achieves $89.31 \pm 0.15\%$ .
- $k=20$ : SDT achieves $88.50 \pm 0.32\%$ , TET achieves $88.64 \pm 0.28\%$ .
When the activation is genuinely differentiable and the backpropagated gradient matches the true loss gradient, SDT and TET attain statistically indistinguishable performance. In contrast, under discrete step activations with surrogate gradients (ResNet-19 on CIFAR-100), TET outperforms SDT by $3.25\%$ at $T=4$ ( $74.11\%$ vs $70.86\%$ ) and by $3.53\%$ at $T=6$ ( $74.65\%$ vs $71.12\%$ ), verifying that TET specifically remedies the surrogate gradient-loss mismatch.

Coverage note — Ablation experiments on specific $\lambda$ values across datasets and 2D loss landscape contour figures were omitted as their conclusions are subsumed within the gradient dynamics, algorithm specifications, and empirical benchmark results.

References

1.Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. Ieee Micro, 38(1):82–99, 2018.
2.Michael V DeBole, Brian Taba, Arnon Amir, Filipp Akopyan, Alexander Andreopoulos, William P Risk, Jeff Kusnitz, Carlos Ortega Otero, Tapan K Nayak, Rathinakumar Appuswamy, et al. Truenorth: Accelerating from zero to 64 million neurons in 10 years. Computer, 52(5):20–29, 2019.
3.Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee, 2009.
4.Lei Deng, Yujie Wu, Xing Hu, Ling Liang, Yufei Ding, Guoqi Li, Guangshe Zhao, Peng Li, and Yuan Xie. Rethinking the performance comparison between snns and anns. Neural Networks, 121:294 – 307, 2020.
5.Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. In International Conference on Learning Representations, 2020.
6.Peter U. Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, and Shih Chii Liu. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In Neural Networks (IJCNN), 2015 International Joint Conference on, 2015.
7.Peter U Diehl, Guido Zarrella, Andrew Cassidy, Bruno U Pedroni, and Emre Neftci. Conversion of artificial recurrent neural networks to spiking neural networks for low-power neuromorphic hardware. In 2016 IEEE International Conference on Rebooting Computing (ICRC), pp. 1–8. IEEE, 2016.
8.Steven K Esser, Paul A Merolla, John V Arthur, Andrew S Cassidy, Rathinakumar Appuswamy, Alexander Andreopoulos, David J Berg, Jeffrey L McKinstry, Timothy Melano, Davis R Barch, et al. Convolutional networks for fast, energy-efficient neuromorphic computing. Proceedings of the national academy of sciences, 113(41):11441–11446, 2016.
9.Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothee Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. arXiv preprint arXiv:2102.04159, 2021.
10.Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.
11.Bing Han and Kaushik Roy. Deep spiking neural network: Energy efficiency through time based coding. In European Conference on Computer Vision, 2020.
12.Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13558–13567, 2020.
13.Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448–456. PMLR, 2015.
14.Rahul Kidambi, Praneeth Netrapalli, Prateek Jain, and Sham Kakade. On the insufficiency of existing momentum schemes for stochastic optimization. In 2018 Information Theory and Applications Workshop (ITA), pp. 1–9. IEEE, 2018.
15.Seijoon Kim, Seongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: Spiking neural network for energy-efficient object detection. arXiv preprint arXiv:1903.06530, 2019.
16.Youngeun Kim and Priyadarshini Panda. Visual explanations from spiking neural networks using inter-spike intervals. Scientific reports, 11(1):1–14, 2021.
17.Youngeun Kim, Yeshwanth Venkatesha, and Priyadarshini Panda. Privatesnn: Fully privacy-preserving spiking neural networks. arXiv preprint arXiv:2104.03414, 2021.
18.Youngeun Kim, Yuhang Li, Hyoungseob Park, Yeshwanth Venkatesha, and Priyadarshini Panda. Neural architecture search for spiking neural networks. arXiv preprint arXiv:2201.10355, 2022.
19.Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
20.Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
21.Alex Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25(2), 2012.
22.Alexander Kugele, Thomas Pfeil, Michael Pfeiffer, and Elisabetta Chicca. Efficient processing of spatio-temporal data streams with spiking neural networks. Frontiers in Neuroscience, 14:439, 2020.
23.Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. Frontiers in neuroscience, 10:508, 2016.
24.Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 6391–6401, 2018.
25.Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar10-dvs: an event-stream dataset for object classification. Frontiers in neuroscience, 11:309, 2017.
26.Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, and Shi Gu. A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. arXiv preprint arXiv:2106.06984, 2021a.
27.Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differentiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in Neural Information Processing Systems, 34, 2021b.
28.Yuhang Li, Youngeun Kim, Hyoungseob Park, Tamar Geller, and Priyadarshini Panda. Neuromorphic data augmentation for training spiking neural networks. arXiv preprint arXiv:2203.06145, 2022.
29.Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51–63, 2019.
30.Nitin Rathi and Kaushik Roy. Diet-snn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks. arXiv preprint arXiv:2008.03658, 2020.
31.Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. In International Conference on Learning Representations, 2019.
32.Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, and Michael Pfeiffer. Theory and tools for the conversion of analog to spiking convolutional neural networks. arXiv: Statistics/Machine Learning, (1612.04052):0–0, 2016.
33.Ali Samadzadeh, Fatemeh Sadat Tabatabaei Far, Ali Javadi, Ahmad Nickabadi, and Morteza Haghir Chehreghani. Convolutional spiking neural networks for spatio-temporal feature extraction. arXiv preprint arXiv:2003.12346, 2020.
34.Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? In Proceedings of the 32nd international conference on neural information processing systems, pp. 2488–2498, 2018.
35.Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in Neuroence, 13, 2018.
36.Sumit Bam Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. In Advances in Neural Information Processing Systems, pp. 1412–1421, 2018.
37.Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
38.Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1311–1318, 2019.
39.Zhenzhi Wu, Hehui Zhang, Yihan Lin, Guoqi Li, Meng Wang, and Ye Tang. Liaf-net: Leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing. IEEE Transactions on Neural Networks and Learning Systems, 2021.
40.Yukun Yang, Wenrui Zhang, and Peng Li. Backpropagated neighborhood aggregation for accurate training of spiking neural networks. In International Conference on Machine Learning, pp. 11852–11862. PMLR, 2021.
41.Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. arXiv preprint arXiv:2002.10085, 2020.
42.Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11062–11070, 2021.
43.Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. Ieee Micro, 38(1):82–99, 2018.
44.Michael V DeBole, Brian Taba, Arnon Amir, Filipp Akopyan, Alexander Andreopoulos, William P Risk, Jeff Kusnitz, Carlos Ortega Otero, Tapan K Nayak, Rathinakumar Appuswamy, et al. Truenorth: Accelerating from zero to 64 million neurons in 10 years. Computer, 52(5):20–29, 2019.
45.Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee, 2009.
46.Lei Deng, Yujie Wu, Xing Hu, Ling Liang, Yufei Ding, Guoqi Li, Guangshe Zhao, Peng Li, and Yuan Xie. Rethinking the performance comparison between snns and anns. Neural Networks, 121:294 – 307, 2020.
47.Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. In International Conference on Learning Representations, 2020.
48.Peter U. Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, and Shih Chii Liu. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In Neural Networks (IJCNN), 2015 International Joint Conference on, 2015.
49.Peter U Diehl, Guido Zarrella, Andrew Cassidy, Bruno U Pedroni, and Emre Neftci. Conversion of artificial recurrent neural networks to spiking neural networks for low-power neuromorphic hardware. In 2016 IEEE International Conference on Rebooting Computing (ICRC), pp. 1–8. IEEE, 2016.
50.Steven K Esser, Paul A Merolla, John V Arthur, Andrew S Cassidy, Rathinakumar Appuswamy, Alexander Andreopoulos, David J Berg, Jeffrey L McKinstry, Timothy Melano, Davis R Barch, et al. Convolutional networks for fast, energy-efficient neuromorphic computing. Proceedings of the national academy of sciences, 113(41):11441–11446, 2016.
51.Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothee Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. arXiv preprint arXiv:2102.04159, 2021.
52.Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.
53.Bing Han and Kaushik Roy. Deep spiking neural network: Energy efficiency through time based coding. In European Conference on Computer Vision, 2020.
54.Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13558–13567, 2020.
55.Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448–456. PMLR, 2015.
56.Rahul Kidambi, Praneeth Netrapalli, Prateek Jain, and Sham Kakade. On the insufficiency of existing momentum schemes for stochastic optimization. In 2018 Information Theory and Applications Workshop (ITA), pp. 1–9. IEEE, 2018.
57.Seijoon Kim, Seongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: Spiking neural network for energy-efficient object detection. arXiv preprint arXiv:1903.06530, 2019.
58.Youngeun Kim and Priyadarshini Panda. Visual explanations from spiking neural networks using inter-spike intervals. Scientific reports, 11(1):1–14, 2021.
59.Youngeun Kim, Yeshwanth Venkatesha, and Priyadarshini Panda. Privatesnn: Fully privacy-preserving spiking neural networks. arXiv preprint arXiv:2104.03414, 2021.
60.Youngeun Kim, Yuhang Li, Hyoungseob Park, Yeshwanth Venkatesha, and Priyadarshini Panda. Neural architecture search for spiking neural networks. arXiv preprint arXiv:2201.10355, 2022.
61.Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
62.Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
63.Alex Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25(2), 2012.
64.Alexander Kugele, Thomas Pfeil, Michael Pfeiffer, and Elisabetta Chicca. Efficient processing of spatio-temporal data streams with spiking neural networks. Frontiers in Neuroscience, 14:439, 2020.
65.Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. Frontiers in neuroscience, 10:508, 2016.
66.Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 6391–6401, 2018.
67.Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar10-dvs: an event-stream dataset for object classification. Frontiers in neuroscience, 11:309, 2017.
68.Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, and Shi Gu. A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. arXiv preprint arXiv:2106.06984, 2021a.
69.Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differentiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in Neural Information Processing Systems, 34, 2021b.
70.Yuhang Li, Youngeun Kim, Hyoungseob Park, Tamar Geller, and Priyadarshini Panda. Neuromorphic data augmentation for training spiking neural networks. arXiv preprint arXiv:2203.06145, 2022.
71.Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51–63, 2019.
72.Nitin Rathi and Kaushik Roy. Diet-snn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks. arXiv preprint arXiv:2008.03658, 2020.
73.Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. In International Conference on Learning Representations, 2019.
74.Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, and Michael Pfeiffer. Theory and tools for the conversion of analog to spiking convolutional neural networks. arXiv: Statistics/Machine Learning, (1612.04052):0–0, 2016.
75.Ali Samadzadeh, Fatemeh Sadat Tabatabaei Far, Ali Javadi, Ahmad Nickabadi, and Morteza Haghir Chehreghani. Convolutional spiking neural networks for spatio-temporal feature extraction. arXiv preprint arXiv:2003.12346, 2020.
76.Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? In Proceedings of the 32nd international conference on neural information processing systems, pp. 2488–2498, 2018.
77.Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in Neuroence, 13, 2018.
78.Sumit Bam Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. In Advances in Neural Information Processing Systems, pp. 1412–1421, 2018.
79.Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
80.Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1311–1318, 2019.
81.Zhenzhi Wu, Hehui Zhang, Yihan Lin, Guoqi Li, Meng Wang, and Ye Tang. Liaf-net: Leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing. IEEE Transactions on Neural Networks and Learning Systems, 2021.
82.Yukun Yang, Wenrui Zhang, and Peng Li. Backpropagated neighborhood aggregation for accurate training of spiking neural networks. In International Conference on Machine Learning, pp. 11852–11862. PMLR, 2021.
83.Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. arXiv preprint arXiv:2002.10085, 2020.
84.Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11062–11070, 2021.

Citation

MLA

Deng, S., et al. “Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting”. arXiv, 2022, http://arxiv.org/abs/2202.11946v3.

APA

Deng, S., Li, Y., Zhang, S., & Gu, S. (2022). Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting. arXiv. http://arxiv.org/abs/2202.11946v3

Chicago

Deng, S., Y. Li, S. Zhang, and S. Gu. 2022. “Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting”. arXiv. http://arxiv.org/abs/2202.11946v3.

Harvard

Deng, S. et al. (2022) “Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2202.11946v3.

Vancouver

1. Deng S, Li Y, Zhang S, Gu S (2022) Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting. arXiv

BibTeX

@article{deng2022temporal,
  title = {Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting},
  author = {Deng, Shikuang and Li, Yuhang and Zhang, Shanghang and Gu, Shi},
  year = {2022},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2202.11946v3},
  eprint = {2202.11946}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Temporal Efficient Training Objective Function

Knowl 2 — Upper Bound of Standard Direct Training Loss by Temporal Efficient Training Loss

Knowl 3 — Gradient Re-Weighting and Flat-Minima Convergence in TET

Knowl 4 — Time Inheritance Training

Knowl 5 — Algorithm for Temporal Efficient Training