keyword
plasticity loss
Plasticity loss is a phenomenon in machine learning where an artificial neural network gradually loses its capacity to adapt and learn new information over the course of continuous training. Although a newly initialized model can readily update its internal representations to fit novel patterns, prolonged optimization—particularly in non-stationary environments such as continual learning and deep reinforcement learning—progressively diminishes this adaptability. This loss of flexibility can arise from factors such as dormant or saturated units, weight degradation, and adverse shifts in the geometry of the optimization landscape, ultimately preventing the network from effectively acquiring new behaviors or updating its predictions in response to fresh data.
2 items
Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning
Michal Nauman, Michal Bortkiewicz, Piotr Milos, Tomasz Trzcinski, Mateusz Ostaszewski, Marek Cygan
Why you should read this
Reveals through an extensive evaluation of over 60 actor-critic variants that general neural network regularizers outperform domain-specific reinforcement learning modifications, enabling basic Soft Actor-Critic agents to achieve state-of-the-art sample efficiency across diverse continuous control tasks.
Recent advancements in off-policy Reinforcement Learning (RL) have significantly improved sample efficiency, primarily due to the incorporation of various forms of regularization that enable more gradient update steps than traditional agents. However, many of these techniques have been tested in limited settings, often on tasks from single simulation benchmarks and against well-known algorithms rather than a range of regularization approaches. This limits our understanding of the specific mechanisms driving RL improvements. To address this, we implemented over 60 different off-policy agents, each integrating established regularization techniques from recent state-of-the-art algorithms. We tested these agents across 14 diverse tasks from 2 simulation benchmarks, measuring training metrics related to overestimation, overfitting, and plasticity loss — issues that motivate the examined regularization techniques. Our findings reveal that while the effectiveness of a specific regularization setup varies with the task, certain combinations consistently demonstrate robust and superior performance. Notably, a simple Soft Actor-Critic agent, appropriately regularized, reliably finds a better-performing policy within the training regime, which previously was achieved mainly through model-based approaches.
Added
2026-10-04
Understanding Plasticity in Neural Networks
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Ávila Pires, Razvan Pascanu, Will Dabney
Why you should read this
Reveals that neural network plasticity loss stems primarily from unfavorable changes in loss curvature rather than unit saturation, providing practical architectural and optimization techniques like layer normalization to maintain continual learning capacity in deep reinforcement learning.
Plasticity, the ability of a neural network to quickly change its predictions in response to new information, is essential for the adaptability and robustness of deep reinforcement learning systems. Deep neural networks are known to lose plasticity over the course of training even in relatively simple learning problems, but the mechanisms driving this phenomenon are still poorly understood. This paper conducts a systematic empirical analysis into plasticity loss, with the goal of understanding the phenomenon mechanistically in order to guide the future development of targeted solutions. We find that loss of plasticity is deeply connected to changes in the curvature of the loss landscape, but that it often occurs in the absence of saturated units. Based on this insight, we identify a number of parameterization and optimization design choices which enable networks to better preserve plasticity over the course of training. We validate the utility of these findings on larger-scale RL benchmarks in the Arcade Learning Environment.
Added
2026-09-26