Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

experience replay dataset

An experience replay dataset is a collection of transition records, typically composed of states, actions, rewards, and next states, gathered from past interactions between an agent or logging policy and an environment. In reinforcement learning, this dataset serves as a fixed or expanding repository from which learning algorithms sample batches of historical experiences to train and update policies or value functions. By decoupling the process of data collection from policy optimization, an experience replay dataset breaks the temporal correlations inherent in sequential decision-making, improves sample efficiency, and enables off-policy and offline reinforcement learning agents to learn effectively from previously recorded interactions without requiring real-time environmental exploration.

1 item

Off-Policy Deep Reinforcement Learning without Exploration

Off-Policy Deep Reinforcement Learning without Exploration

Scott Fujimoto, David Meger, Doina Precup

OrganizationsMcGill UniversityMila – Québec Artificial Intelligence Institute

Why you should read this

Introduces batch-constrained reinforcement learning to resolve extrapolation error, providing the first continuous control method capable of training effective deep reinforcement learning policies purely from fixed, offline datasets.

Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection. In this paper, we demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are incapable of learning with data uncorrelated to the distribution under the current policy, making them ineffective for this fixed batch setting. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space in order to force the agent towards behaving close to on-policy with respect to a subset of the given data. We present the first continuous control deep reinforcement learning algorithm which can learn effectively from arbitrary, fixed batch data, and empirically demonstrate the quality of its behavior in several tasks.

Added

2026-09-16