Off-Policy Deep Reinforcement Learning without Exploration

Scott FujimotoDavid MegerDoina Precup

article2018ICML2,164 citations

Introduces batch-constrained reinforcement learning to resolve extrapolation error, providing the first continuous control method capable of training effective deep reinforcement learning policies purely from fixed, offline datasets.

Listen

Deploying reinforcement learning in real-world settings often requires training agents entirely on static, pre-collected datasets because active real-time exploration can be dangerous, costly, or operationally impractical. The article demonstrates why conventional off-policy reinforcement learning methods fail when restricted to fixed historical data and evaluates a new method designed to learn reliably under these constraints.

The authors analyze the core mathematical and empirical reasons standard algorithms break down when unable to interact with their environments. To resolve these failures, the article introduces Batch-Constrained deep Q-learning, a framework that constrains the agent's actions to stay close to the historical dataset while seeking the highest expected reward. The method uses a generative model to propose familiar actions, applies small optimized adjustments, and penalizes unfamiliar future states through a dual-network evaluation scheme. This approach was tested against established continuous control algorithms across standard simulated robotic benchmarks and diverse data collection regimes, including expert, concurrent, and noisy demonstrations.

The article establishes several key findings:

Standard off-policy algorithms experience catastrophic failure and value estimation divergence in fixed-data settings because of extrapolation error, which causes agents to overestimate the performance of unfamiliar actions they cannot correct through real-time feedback.
In high-dimensional benchmarks, the proposed batch-constrained method was the only tested algorithm that consistently matched or exceeded the performance of the data-generating policies across all evaluated tasks.
The algorithm exhibited stable, non-divergent value estimates across all tests, remaining resilient even when trained on datasets containing substantial noise and suboptimal actions where traditional imitation learning failed.
Mathematical analysis confirms that restricting policy actions to the historical data distribution is strictly necessary to prevent value estimation bias in fixed-data environments.

These findings have major practical implications for organizations seeking to train autonomous decision-making systems from historical logs without online trial-and-error. Conventional methods introduce severe operational risk by pursuing unrealistic outcomes based on flawed extrapolation. The batch-constrained paradigm provides a safer, more predictable approach by ensuring algorithms optimize strictly within operational regions where historical evidence exists, bridging the gap between conservative imitation and unconstrained optimization.

Organizations considering offline reinforcement learning should avoid deploying standard off-policy algorithms on fixed datasets and instead implement batch-constrained mechanisms that model the data collection distribution. Future work should evaluate this approach on larger-scale, real-world operational datasets and explore improved generative architectures, as sensitivity to hyperparameters in complex environments can still introduce value instability if the underlying generative model fails to capture the true data distribution.

McGill University Mila – Québec Artificial Intelligence Institute deep reinforcement learning reinforcement learning agent action spaces Learning Reinforcement Learning Time Steps

arXiv: 1812.02900 sfujim/BCQ

Paper: Conservative Q-Learning for Offline Reinforcement Learning, Aviral Kumar et al. (2020). Conservative Q-Learning builds directly upon the batch-constraint insights of this work by introducing a rigorous lower-bound regularization technique for offline reinforcement learning.
Paper: Addressing Function Approximation Error in Actor-Critic Methods, Scott Fujimoto et al. (2018). TD3 analyzes the actor-critic value overestimation errors that motivate the action-restriction strategies needed for stable batch reinforcement learning.

Paper: Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems, Sergey Levine et al. (2020). This tutorial review generalizes the foundational batch-constrained principles established in the source paper into a broader taxonomy of offline reinforcement learning algorithms.
Paper: Decision Transformer: Reinforcement Learning via Sequence Modeling, Lili Chen et al. (2021). The Decision Transformer extends offline reinforcement learning beyond traditional temporal-difference and batch-constraint methods by framing policy learning as conditional sequence modeling.

Cover for Off-Policy Deep Reinforcement Learning without Exploration

Abstract

Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection. In this paper, we demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are incapable of learning with data uncorrelated to the distribution under the current policy, making them ineffective for this fixed batch setting. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space in order to force the agent towards behaving close to on-policy with respect to a subset of the given data. We present the first continuous control deep reinforcement learning algorithm which can learn effectively from arbitrary, fixed batch data, and empirically demonstrate the quality of its behavior in several tasks.

Topics

deep reinforcement learning reinforcement learning agent action spaces reinforcement learning method multi-agent reinforcement learning

Keywords

Learning Reinforcement Learning Time Steps Deep Reinforcement Learning Value Policy Batch Off-Policy Off-Policy Deep Reinforcement Reinforcement Learning Without

1 Introduction
2 Background
3 Extrapolation Error
3.1 Extrapolation Error in Deep Reinforcement Learning
4 Batch-Constrained Reinforcement Learning
4.1 Addressing Extrapolation Error in Finite MDPs
4.2 Batch-Constrained Deep Reinforcement Learning
5 Experiments
6 Related Work
7 Conclusion
References
A Missing Proofs
A.1 Proofs and Details from Section 4.1
A.2 Sketch of the Proof of Convergence of Q-Learning
B Missing Graphs
B.1 Extrapolation Error in Deep Reinforcement Learning
B.2 Complete Experimental Results
C Extrapolation Error in Kernel-Based Reinforcement Learning
D Additional Experiments
D.1 Ablation Study of Perturbation Model
D.2 Uncertainty Estimation for Batch-Constrained Reinforcement Learning
D.3 Random Behavioral Policy Study
E Missing Background
E.1 Variational Auto-Encoder
F Experimental Details
G Implementation Details

Knowls

Knowl 1 — Batch-Constrained Deep Q-Learning Policy Formulation via Generative and Perturbation Networks
model/method
Batch-Constrained deep Q-learning (BCQ) enforces a batch constraint in high-dimensional continuous action spaces by combining a state-conditioned generative model with an action perturbation model and deep Q-networks.

To restrict action selection to state-action pairs $(s, a)$ with high density under the fixed batch $\mathcal{B}$ , BCQ models the conditional action distribution $P_B(a|s)$ using a conditional Variational Auto-Encoder (VAE) $G_\omega(s) = \{E_{\omega_1}(s, a), D_{\omega_2}(s, z)\}$ . The encoder $E_{\omega_1}$ maps $(s, a)$ to Gaussian latent parameters $\mu, \sigma$ , and the decoder $D_{\omega_2}$ reconstructs the action from $(s, z)$ , where $z = \mu + \sigma \odot \epsilon$ with $\epsilon \sim \mathcal{N}(0, I)$ . The VAE is trained by minimizing the loss:

$\mathcal{L}_{\text{VAE}} = \sum_{(s,a) \in \mathcal{B}} \|D_{\omega_2}(s, z) - a\|^2 + \frac{1}{2J} D_{\text{KL}}(\mathcal{N}(\mu, \sigma) \parallel \mathcal{N}(0, I))$

where $J = 2 \dim(\mathcal{A})$ is the latent vector dimensionality, and the Gaussian KL divergence expands as:

$D_{\text{KL}}(\mathcal{N}(\mu, \sigma) \parallel \mathcal{N}(0, I)) = -\frac{1}{2} \sum_{j=1}^J \left( 1 + \log(\sigma_j^2) - \mu_j^2 - \sigma_j^2 \right)$

During inference, latent vectors $z$ are clipped to $[-0.5, 0.5]$ to avoid sampling out-of-distribution actions.

To increase action diversity while remaining close to the batch distribution, a perturbation model $\xi_\phi(s, a, \Phi)$ outputs an additive adjustment bounded in $[-\Phi, \Phi]$ via a $\tanh$ activation scaled by $\Phi$ . The perturbation network is trained with the deterministic policy gradient on actions sampled from $G_\omega(s)$ using critic $Q_{\theta_1}$ :

$\phi \leftarrow \arg\max_\phi \sum_{(s,a) \in \mathcal{B}} Q_{\theta_1}(s, a + \xi_\phi(s, a, \Phi)), \quad a \sim G_\omega(s)$

The resulting policy $\pi(s)$ samples $n$ candidate actions from $G_\omega(s)$ , perturbs each, and greedily selects the action that maximizes $Q_{\theta_1}$ :

$\pi(s) = \arg\max_{a_i + \xi_\phi(s, a_i, \Phi)} Q_{\theta_1}(s, a_i + \xi_\phi(s, a_i, \Phi)), \quad \{a_i \sim G_\omega(s)\}_{i=1}^n$
Knowl 2 — Batch-Constrained Deep Q-Learning (BCQ) Full Algorithm
algorithm
Batch-Constrained deep Q-learning (BCQ) trains an off-policy actor-critic agent on a fixed batch dataset $\mathcal{B}$ without online exploration by coupling a generative conditional VAE, a bounded action perturbation network, and twin Q-networks.
Algorithm: Batch-Constrained deep Q-learning (BCQ)
Input: Batch dataset $\mathcal{B}$ , horizon $T$ , target update rate $\tau = 0.005$ , mini-batch size $N = 100$ , maximum perturbation $\Phi = 0.05$ , candidate sample count $n = 10$ , minimum weighting $\lambda = 0.75$
Output: Policy networks $G_\omega, \xi_\phi$ and Q-networks $Q_{\theta_1}, Q_{\theta_2}$
Initialize Q-networks $Q_{\theta_1}, Q_{\theta_2}$ , perturbation network $\xi_\phi$ , and VAE $G_\omega = \{E_{\omega_1}, D_{\omega_2}\}$ with random parameters $\theta_1, \theta_2, \phi, \omega$
Initialize target networks: $\theta'_1 \leftarrow \theta_1$ , $\theta'_2 \leftarrow \theta_2$ , $\phi' \leftarrow \phi$
for $t = 1$ to $T$ do
    Sample mini-batch of $N$ transitions $(s, a, r, s')$ from $\mathcal{B}$
     $\mu, \sigma = E_{\omega_1}(s, a)$
    Sample $z \sim \mathcal{N}(\mu, \sigma)$
     $\tilde{a} = D_{\omega_2}(s, z)$
     $\omega \leftarrow \arg\min_\omega \sum (a - \tilde{a})^2 + \frac{1}{2J} D_{\text{KL}}(\mathcal{N}(\mu, \sigma) \parallel \mathcal{N}(0, I))$
    Sample $n$ actions per next state: $\{a_i \sim G_\omega(s')\}_{i=1}^n$
    Perturb each next-state candidate: $\{a_i \leftarrow a_i + \xi_{\phi'}(s', a_i, \Phi)\}_{i=1}^n$
    Compute target value: $y = r + \gamma \max_{a_i} [ \lambda \min_{j=1,2} Q_{\theta'_j}(s', a_i) + (1-\lambda) \max_{j=1,2} Q_{\theta'_j}(s', a_i) ]$
    Update critics: $\theta_j \leftarrow \arg\min_{\theta_j} \sum (y - Q_{\theta_j}(s, a))^2$ for $j \in \{1, 2\}$
    Sample current-state actions: $a \sim G_\omega(s)$
    Update perturbation network: $\phi \leftarrow \arg\max_\phi \sum Q_{\theta_1}(s, a + \xi_\phi(s, a, \Phi))$
    Update target networks: $\theta'_j \leftarrow \tau \theta_j + (1 - \tau)\theta'_j$ for $j \in \{1, 2\}$ , and $\phi' \leftarrow \tau \phi + (1 - \tau)\phi'$
end for
All networks are trained with the Adam optimizer at a learning rate of $10^{-3}$ and discount factor $\gamma = 0.99$ . Critic and perturbation networks have 2 hidden layers with 400 and 300 ReLU units; the VAE encoder and decoder use 2 hidden layers with 750 ReLU units.
Knowl 3 — Uncertainty-Penalized Learning Target in BCQ
equation
To penalize unfamiliar states and prevent value overestimation during off-policy updates on a fixed dataset $\mathcal{B}$ , BCQ trains twin Q-networks $Q_{\theta_1}$ and $Q_{\theta_2}$ using a convex combination of the minimum and maximum target critic evaluations:

$y = r + \gamma \max_{a_i} \left[ \lambda \min_{j=1,2} Q_{\theta'_j}(s', a_i) + (1 - \lambda) \max_{j=1,2} Q_{\theta'_j}(s', a_i) \right]$

where $\lambda \in [0.5, 1.0]$ is a fixed weighting hyperparameter (set to $\lambda = 0.75$ ), and candidate actions are given by:

$a_i = \tilde{a}_i + \xi_{\phi'}(s', \tilde{a}_i, \Phi), \quad \tilde{a}_i \sim G_\omega(s'), \quad i = 1, \dots, n$

Both Q-networks minimize the squared Bellman error against the shared target $y$ :

$\mathcal{L}_{\text{value}, j} = \frac{1}{|\mathcal{B}_{\text{mini}}|} \sum_{(s,a,r,s') \in \mathcal{B}_{\text{mini}}} \left( y - Q_{\theta_j}(s, a) \right)^2, \quad j \in \{1, 2\}$

Setting $\lambda > 0.5$ penalizes high variance across critic estimates in regions of state uncertainty, directing the policy toward actions that lead to familiar transitions present in the batch.
Knowl 4 — Extrapolation Error and Tabular Error Decomposition in Batch Reinforcement Learning
theoretical result
In a Markov decision process $M = (\mathcal{S}, \mathcal{A}, p_M, r, \gamma)$ , learning a Q-function exclusively from a fixed batch of transitions $\mathcal{B}$ introduces extrapolation error, defined as the discrepancy between the true value function $Q^\pi(s, a)$ in $M$ and the value function $Q_B^\pi(s, a)$ learned in the empirical batch MDP $M_B$ .

The batch MDP $M_B$ shares the state and action spaces of $M$ with an additional terminal state $s_{\text{init}}$ , having empirical transition probabilities:

$p_B(s'|s, a) = \frac{N(s, a, s')}{\sum_{\tilde{s}} N(s, a, \tilde{s})}$

if $\sum_{\tilde{s}} N(s, a, \tilde{s}) > 0$ , and $p_B(s_{\text{init}}|s, a) = 1$ otherwise with $r(s, a, s_{\text{init}}) = Q_0(s, a)$ . Performing standard tabular Q-learning by sampling uniformly from $\mathcal{B}$ converges to the optimal value function under $M_B$ .

The tabular extrapolation error $\epsilon_{\text{MDP}}(s, a) = Q^\pi(s, a) - Q_B^\pi(s, a)$ satisfies the Bellman-like recurrence:

$\epsilon_{\text{MDP}}(s, a) = \sum_{s'} (p_M(s'|s, a) - p_B(s'|s, a)) \left( r(s, a, s') + \gamma \sum_{a'} \pi(a'|s') Q_B^\pi(s', a') \right) + \sum_{s'} p_M(s'|s, a) \gamma \sum_{a'} \pi(a'|s') \epsilon_{\text{MDP}}(s', a')$

The total weighted extrapolation error under policy state-visitation $\mu^\pi(s)$ is given by:

$\epsilon_{\text{MDP}}^\pi = \sum_s \mu^\pi(s) \sum_a \pi(a|s) |\epsilon_{\text{MDP}}(s, a)|$

For any reward function, $\epsilon_{\text{MDP}}^\pi = 0$ if and only if $p_B(s'|s, a) = p_M(s'|s, a)$ for all $s' \in \mathcal{S}$ and all $(s, a)$ satisfying $\mu^\pi(s) > 0$ and $\pi(a|s) > 0$ .
Knowl 5 — Zero Extrapolation Error for Batch-Constrained Policies in Deterministic MDPs
theoretical result
Let a batch $\mathcal{B}$ be coherent, defined as satisfying $s' \in \mathcal{B}$ for all $(s, a, s') \in \mathcal{B}$ unless $s'$ is a terminal state. Let a policy $\pi$ be batch-constrained (denoted $\pi \in \Pi_B$ ) if $(s, a) \in \mathcal{B}$ for every $(s, a)$ with $\mu^\pi(s) > 0$ and $\pi(a|s) > 0$ .

In a deterministic MDP, for all reward functions, the total extrapolation error $\epsilon_{\text{MDP}}^\pi = 0$ if and only if the policy $\pi$ is batch-constrained. Furthermore, if $\mathcal{B}$ is coherent, at least one batch-constrained policy exists provided the initial state $s_0 \in \mathcal{B}$ .
Knowl 6 — Batch-Constrained Q-Learning (BCQL) and Convergence Guarantees
algorithm
Batch-Constrained Q-learning (BCQL) is a tabular reinforcement learning algorithm that restricts action updates to state-action pairs present in a batch $\mathcal{B}$ .

Given a sampled transition $(s, a, r, s') \in \mathcal{B}$ , BCQL applies the temporal difference update:

$Q(s, a) \leftarrow (1 - \alpha) Q(s, a) + \alpha \left( r + \gamma \max_{a' \text{ s.t. } (s', a') \in \mathcal{B}} Q(s', a') \right)$

BCQL possesses the following convergence guarantees:
1. Under Robbins-Monro conditions on learning rate $\alpha$ ( $\sum_t \alpha_t = \infty, \sum_t \alpha_t^2 < \infty$ ) and infinite state-action visitation across the true MDP, BCQL converges to the optimal value function $Q^*$ .
2. In a deterministic MDP with a coherent batch $\mathcal{B}$ , under Robbins-Monro conditions and uniform sampling from $\mathcal{B}$ , BCQL converges to $Q_B^{\pi^*}(s, a)$ , where $\pi^*(s) = \arg\max_{a \text{ s.t. } (s, a) \in \mathcal{B}} Q_B^{\pi^*}(s, a)$ is the optimal batch-constrained policy.
3. The converged policy $\pi^*$ satisfies $Q^{\pi^*}(s, a) \ge Q^\pi(s, a)$ for all batch-constrained policies $\pi \in \Pi_B$ and all $(s, a) \in \mathcal{B}$ , guaranteeing that BCQL matches or outperforms the data-generating behavioral policy from any starting state in $\mathcal{B}$ .
Knowl 7 — Extrapolation Error and Value Divergence in Standard Off-Policy Deep Actor-Critic Algorithms
empirical result
When standard deep actor-critic algorithms (such as DDPG and discretized DQN) are trained purely off-policy on fixed batch datasets without environment interaction, extrapolation error causes severe value function divergence and poor policy performance:
1. Final Buffer (1 million transitions collected by a DDPG agent trained with $\mathcal{N}(0, 0.5)$ exploration noise on Hopper-v1): Off-policy DDPG fails to match the average return of the batch trajectories, producing highly erratic critic estimates (fluctuating between $-4 \times 10^4$ and $+6 \times 10^4$ ).
2. Concurrent Learning (1 million transitions shared between an active behavioral DDPG agent with $\mathcal{N}(0, 0.1)$ noise and an offline DDPG agent): The offline agent achieves significantly lower returns (rarely exceeding 1000) than the behavioral agent (which reaches returns $\sim 2500$ ), with value estimates diverging over $1500$ , demonstrating that differences in initial policy state distributions are sufficient to trigger detrimental extrapolation error.
3. Imitation (1 million expert transitions collected by trained DDPG on Hopper-v1): The offline DDPG agent quickly selects out-of-distribution non-expert actions due to overestimation bias, driving Q-value estimates to diverge beyond $7 \times 10^7$ and collapsing policy return to near zero.
Knowl 8 — Comparative Performance and Value Stability of BCQ Across Continuous Control Tasks
empirical result
BCQ was evaluated across OpenAI Gym continuous control tasks (HalfCheetah-v1, Hopper-v1, Walker2d-v1) under four fixed batch regimes against DDPG, independently discretized DQN, feedforward Behavioral Cloning (BC), and a VAE-based Behavioral Cloning baseline (VAE-BC):
1. Final Buffer (1M exploration transitions): BCQ matches or exceeds the behavioral policy return (e.g., reaching $\sim 8000$ in HalfCheetah-v1, $\sim 1500$ in Hopper-v1, and $\sim 2000$ in Walker2d-v1), whereas DDPG and DQN fail.
2. Concurrent Learning (1M transitions): BCQ outperforms or matches the active behavioral agent across all environments and maintains stable Q-estimates matching Monte Carlo true returns.
3. Expert Imitation (1M expert transitions): BCQ maintains high return (matching BC and VAE-BC at $\sim 10000$ on HalfCheetah-v1, $\sim 3500$ on Hopper-v1, and $\sim 3500$ on Walker2d-v1) with completely stable value estimates, whereas DDPG and DQN diverge.
4. Imperfect Demonstrations (100k transitions with 30% uniform random action selection and $\mathcal{N}(0, 0.3)$ Gaussian noise): Standard BC and VAE-BC achieve poor returns (e.g., $<2000$ in HalfCheetah-v1), while BCQ rapidly attains high expert-level returns ( $>6000$ in HalfCheetah-v1, $>2500$ in Hopper-v1, $>2000$ in Walker2d-v1) in under $10^5$ iterations, successfully filtering noisy transitions.
Knowl 9 — Generative Action Constraints Versus Ensemble Uncertainty Minimization in Batch RL
empirical result
To evaluate whether uncertainty estimation alone can prevent extrapolation error in batch reinforcement learning, an actor-critic baseline was tested on the Hopper-v1 expert imitation task where the policy $\pi_\phi$ is trained to minimize the standard deviation across ensembles of Q-networks of size $N \in \{4, 10\}$ :

$\phi \leftarrow \arg\min_\phi \sum_{(s,a) \in \mathcal{B}} \sigma\left( \left\{ Q_{\theta_i}(s, a) \right\}_{i=1}^N \right)$

While ensemble variance minimization stabilizes the critic estimates (keeping values under 400), it fails to restrict the actor's actions to the expert distribution, yielding inferior policy returns (plateauing around 1000 to 1500 return) compared to BCQ (which achieves $\sim 3000$ return). Constraining the candidate action space via a conditional generative model provides an effective batch constraint without requiring computationally expensive large ensembles or delicate weighting between value maximization and uncertainty penalties.
Knowl 10 — Extrapolation Error in Kernel-Based Batch Reinforcement Learning
theoretical result
Extrapolation error affects non-parametric batch reinforcement learning methods such as Kernel-Based Reinforcement Learning (KBRL). Given a batch $\mathcal{B}$ , density kernel $k_\tau(s, s_B^a) = \phi(\|s - s_B^a\| / \tau)$ , and normalized weights $\kappa_\tau^a(s, s_B^a)$ , KBRL computes state-action values as:

$Q(s, a) = \sum_{(s_B^a, a, r, s_B'^a) \in \mathcal{B}} \kappa_\tau^a(s, s_B^a) \left[ r + \gamma V(s_B'^a) \right], \quad V(s') = \max_{a \text{ s.t. } (s', a) \in \mathcal{B}} Q(s', a)$

Consider a deterministic two-state MDP with states $\{s_0, s_1\}$ , actions $\{a_0, a_1\}$ , and transitions/rewards: $p_M(s_1|s_0, a_1) = 1$ with $r=1$ ; $p_M(s_0|s_1, a_0) = 1$ with $r=0$ ; and self-loops on $a_0$ at $s_0$ and $a_1$ at $s_1$ with $r=0$ . When provided with optimal batch $\mathcal{B} = \{(s_0, a_1, 1, s_1), (s_1, a_0, 0, s_0)\}$ , KBRL estimates:

$Q(\cdot, a_1) = \frac{1}{1 - \gamma^2}, \quad Q(\cdot, a_0) = \frac{\gamma}{1 - \gamma^2}$

Because the kernel assigns the same value across all states for each observed action, KBRL estimates $Q(s_1, a_1) = \frac{1}{1 - \gamma^2} > Q(s_1, a_0) = \frac{\gamma}{1 - \gamma^2}$ , leading $\arg\max_a Q(s, a)$ to select $a_1$ at $s_1$ (an unseen transition yielding reward 0 and self-loop), resulting in a degenerate policy that fails to learn the optimal behavior.

Coverage note — None was omitted; all major theoretical results (Theorems 1-4, Lemma 1, KBRL analysis), algorithmic components (BCQL, BCQ VAE and perturbation architecture, uncertainty target), and empirical evaluations (Hopper, HalfCheetah, Walker2d benchmarks, ensemble comparisons, toy random policy experiments) are fully covered.

References

1.Achiam, J., Held, D., Tamar, A., and Abbeel, P. Constrained policy optimization. In International Conference on Machine Learning, pp. 22–31, 2017.
2.Argall, B. D., Chernova, S., Veloso, M., and Browning, B. A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5):469–483, 2009.
3.Azizzadenesheli, K., Brunskill, E., and Anandkumar, A. Efficient exploration through bayesian deep q-networks. arXiv preprint arXiv:1802.04412, 2018.
4.Bertsekas, D. P. and Tsitsiklis, J. N. Neuro-Dynamic Programming. Athena scientific Belmont, MA, 1996.
5.Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym, 2016.
6.Buckman, J., Hafner, D., Tucker, G., Brevdo, E., and Lee, H. Sample-efficient reinforcement learning with stochastic ensemble value expansion. In Advances in Neural Information Processing Systems, pp. 8234–8244, 2018.
7.Chemali, J. and Lazaric, A. Direct policy iteration with demonstrations. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
8.Cheng, C.-A., Yan, X., Wagener, N., and Boots, B. Fast policy learning through imitation and reinforcement. arXiv preprint arXiv:1805.10413, 2018.
9.Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems 31, pp. 4759–4770, 2018.
10.Dayan, P. and Watkins, C. J. C. H. Q-learning. Machine learning, 8(3):279–292, 1992.
11.de Bruin, T., Kober, J., Tuyls, K., and Babuška, R. The importance of experience replay database composition in deep reinforcement learning. In Deep Reinforcement Learning Workshop, NIPS, 2015.
12.de Bruin, T., Kober, J., Tuyls, K., and Babuška, R. Improved deep reinforcement learning for robotics through distribution-based experience retention. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3947–3952. IEEE, 2016.
13.Dearden, R., Friedman, N., and Russell, S. Bayesian q-learning. In AAAI/IAAI, pp. 761–768, 1998.
14.Deisenroth, M. and Rasmussen, C. E. Pilco: A model-based and data-efficient approach to policy search. In International Conference on Machine Learning, pp. 465–472, 2011.
15.Duan, Y., Chen, X., Houthooft, R., Schulman, J., and Abbeel, P. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pp. 1329–1338, 2016.
16.Ernst, D., Geurts, P., and Wehenkel, L. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6(Apr):503–556, 2005.
17.Evans, O. Learning the preferences of ignorant, inconsistent agents. In AAAI, pp. 323–329, 2016.
18.Fujimoto, S., van Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, volume 80, pp. 1587–1596. PMLR, 2018.
19.Gal, Y., McAllister, R., and Rasmussen, C. E. Improving pilco with bayesian neural network dynamics models. In Data-Efficient Machine Learning workshop, International Conference on Machine Learning, 2016.
20.Gao, Y., Lin, J., Yu, F., Levine, S., and Darrell, T. Reinforcement learning from imperfect demonstrations. arXiv preprint arXiv:1802.05313, 2018.
21.Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., and Bengio, Y. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013.
22.Gordon, G. J. Stable function approximation in dynamic programming. In Machine Learning Proceedings 1995, pp. 261–268. Elsevier, 1995.
23.Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. Deep Reinforcement Learning that Matters. arXiv preprint arXiv:1709.06560, 2017.
24.Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Dulac-Arnold, G., et al. Deep q-learning from demonstrations. arXiv preprint arXiv:1704.03732, 2017.
25.Higuera, J. C. G., Meger, D., and Dudek, G. Synthesizing neural network controllers with probabilistic model based reinforcement learning. arXiv preprint arXiv:1803.02291, 2018.
26.Ho, J., Gupta, J., and Ermon, S. Model-free imitation learning with policy optimization. In International Conference on Machine Learning, pp. 2760–2769, 2016.
27.Hussein, A., Gaber, M. M., Elyan, E., and Jayne, C. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):21, 2017.
28.Isele, D. and Cosgun, A. Selective experience replay for lifelong learning. arXiv preprint arXiv:1802.10269, 2018.
29.Jaksch, T., Ortner, R., and Auer, P. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 11(Apr):1563–1600, 2010.
30.Jiang, N. and Li, L. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning, pp. 652–661, 2016.
31.Johannink, T., Bahl, S., Nair, A., Luo, J., Kumar, A., Loskyll, M., Ojea, J. A., Solowjow, E., and Levine, S. Residual reinforcement learning for robot control. arXiv preprint arXiv:1812.03201, 2018.
32.Kakade, S. and Langford, J. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning, volume 2, pp. 267–274, 2002.
33.Kim, B., Farahmand, A.-m., Pineau, J., and Precup, D. Learning from limited demonstrations. In Advances in Neural Information Processing Systems, pp. 2859–2867, 2013.
34.Kingma, D. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
35.Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
36.Konda, V. R. and Tsitsiklis, J. N. On actor-critic algorithms. SIAM journal on Control and Optimization, 42(4):1143–1166, 2003.
37.Lai, T. L. and Robbins, H. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 6(1):4–22, 1985.
38.Lange, S., Gabel, T., and Riedmiller, M. Batch reinforcement learning. In Reinforcement learning, pp. 45–73. Springer, 2012.
39.Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
40.Lin, L.-J. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3-4):293–321, 1992.
41.Liu, Y., Gottesman, O., Raghu, A., Komorowski, M., Faisal, A. A., Doshi-Velez, F., and Brunskill, E. Representation balancing mdps for off-policy policy evaluation. In Advances in Neural Information Processing Systems, pp. 2644–2653, 2018.
42.Lu, T., Schuurmans, D., and Boutilier, C. Non-delusional q-learning and value-iteration. In Advances in Neural Information Processing Systems, pp. 9971–9981, 2018.
43.McCloskey, M. and Cohen, N. J. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, volume 24, pp. 109–165. Elsevier, 1989.
44.Melo, F. S. Convergence of q-learning: A simple proof. Institute Of Systems and Robotics, Tech. Rep, pp. 1–4, 2001.
45.Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
46.Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems, pp. 1054–1062, 2016.
47.Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., and Abbeel, P. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 6292–6299. IEEE, 2018.
48.O’Donoghue, B., Osband, I., Munos, R., and Mnih, V. The uncertainty Bellman equation and exploration. In International Conference on Machine Learning, volume 80, pp. 3839–3848. PMLR, 2018.
49.Ormoneit, D. and Sen, S. Kernel-based reinforcement learning. Machine learning, 49(2-3):161–178, 2002.
50.Osband, I., Blundell, C., Pritzel, A., and Van Roy, B. Deep exploration via bootstrapped dqn. In Advances in Neural Information Processing Systems, pp. 4026–4034, 2016.
51.Osband, I., Aslanides, J., and Cassirer, A. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems 31, pp. 8626–8638, 2018.
52.Peshkin, L. and Shelton, C. R. Learning from scarce experience. In International Conference on Machine Learning, pp. 498–505, 2002.
53.Peters, J. and Mülling, K. Relative entropy policy search. In AAAI, pp. 1607–1612, 2010.
54.Pham, T.-H., De Magistris, G., Agravante, D. J., Chaudhury, S., Munawar, A., and Tachibana, R. Constrained exploration and recovery from experience shaping. arXiv preprint arXiv:1809.08925, 2018.
55.Piot, B., Geist, M., and Pietquin, O. Boosted bellman residual minimization handling expert demonstrations. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 549–564. Springer, 2014.
56.Precup, D., Sutton, R. S., and Dasgupta, S. Off-policy temporal-difference learning with function approximation. In International Conference on Machine Learning, pp. 417–424, 2001.
57.Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082, 2014.
58.Riedmiller, M. Neural fitted q iteration–first experiences with a data efficient neural reinforcement learning method. In European Conference on Machine Learning, pp. 317–328. Springer, 2005.
59.Schaal, S. Is imitation learning the route to humanoid robots? Trends in Cognitive Sciences, 3(6):233–242, 1999.
60.Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889–1897, 2015.
61.Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. Deterministic policy gradient algorithms. In International Conference on Machine Learning, pp. 387–395, 2014.
62.Silver, T., Allen, K., Tenenbaum, J., and Kaelbling, L. Residual policy learning. arXiv preprint arXiv:1812.06298, 2018.
63.Singh, S., Jaakkola, T., Littman, M. L., and Szepesvári, C. Convergence results for single-step on-policy reinforcement-learning algorithms. Machine learning, 38(3):287–308, 2000.
64.Sohn, K., Lee, H., and Yan, X. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems, pp. 3483–3491, 2015.
65.Strehl, A. L. and Littman, M. L. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8):1309–1331, 2008.
66.Sun, W., Venkatraman, A., Gordon, G. J., Boots, B., and Bagnell, J. A. Deeply aggrevated: Differentiable imitation learning for sequential prediction. In International Conference on Machine Learning, pp. 3309–3318, 2017.
67.Sun, W., Bagnell, J. A., and Boots, B. Truncated horizon policy search: Combining reinforcement learning & imitation learning. arXiv preprint arXiv:1805.11240, 2018.
68.Sutton, R. S. Learning to predict by the methods of temporal differences. Machine learning, 3(1):9–44, 1988.
69.Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
70.Thomas, P., Theocharous, G., and Ghavamzadeh, M. High confidence policy improvement. In International Conference on Machine Learning, pp. 2380–2388, 2015.
71.Thrun, S. and Schwartz, A. Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 Connectionist Models Summer School Hillsdale, NJ. Lawrence Erlbaum, 1993.
72.Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5026–5033. IEEE, 2012.
73.Touati, A., Satija, H., Romoff, J., Pineau, J., and Vincent, P. Randomized value functions via multiplicative normalizing flows. arXiv preprint arXiv:1806.02315, 2018.
74.Van Hasselt, H. Double q-learning. In Advances in Neural Information Processing Systems, pp. 2613–2621, 2010.
75.Van Hasselt, H., Guez, A., and Silver, D. Deep reinforcement learning with double q-learning. In AAAI, pp. 2094–2100, 2016.
76.Van Hoof, H., Neumann, G., and Peters, J. Non-parametric policy search with limited information loss. The Journal of Machine Learning Research, 18(1):2472–2517, 2017.
77.Večerík, M., Hester, T., Scholz, J., Wang, F., Pietquin, O., Piot, B., Heess, N., Rothörl, T., Lampe, T., and Riedmiller, M. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817, 2017.
78.Watkins, C. J. C. H. Learning from delayed rewards. PhD thesis, King’s College, Cambridge, 1989.
79.Xu, H., Li, Y., Tian, Y., Darrell, T., and Ma, T. Algorithmic framework for model-based reinforcement learning with theoretical guarantees. arXiv preprint arXiv:1807.03858, 2018.
80.Zhang, S. and Sutton, R. S. A deeper look at experience replay. arXiv preprint arXiv:1712.01275, 2017.

Citation

MLA

Fujimoto, S., et al. “Off-Policy Deep Reinforcement Learning Without Exploration”. arXiv, 2018, http://arxiv.org/abs/1812.02900v3.

APA

Fujimoto, S., Meger, D., & Precup, D. (2018). Off-Policy Deep Reinforcement Learning without Exploration. arXiv. http://arxiv.org/abs/1812.02900v3

Chicago

Fujimoto, S., D. Meger, and D. Precup. 2018. “Off-Policy Deep Reinforcement Learning Without Exploration”. arXiv. http://arxiv.org/abs/1812.02900v3.

Harvard

Fujimoto, S., Meger, D. and Precup, D. (2018) “Off-Policy Deep Reinforcement Learning without Exploration”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1812.02900v3.

Vancouver

1. Fujimoto S, Meger D, Precup D (2018) Off-Policy Deep Reinforcement Learning without Exploration. arXiv

BibTeX

@article{fujimoto2018off,
  title = {Off-Policy Deep Reinforcement Learning without Exploration},
  author = {Fujimoto, Scott and Meger, David and Precup, Doina},
  year = {2018},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1812.02900v3},
  eprint = {1812.02900}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Off-Policy Deep Reinforcement Learning without Exploration

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Batch-Constrained Deep Q-Learning Policy Formulation via Generative and Perturbation Networks

Knowl 2 — Batch-Constrained Deep Q-Learning (BCQ) Full Algorithm

Knowl 3 — Uncertainty-Penalized Learning Target in BCQ

Knowl 4 — Extrapolation Error and Tabular Error Decomposition in Batch Reinforcement Learning

Knowl 5 — Zero Extrapolation Error for Batch-Constrained Policies in Deterministic MDPs

Knowl 6 — Batch-Constrained Q-Learning (BCQL) and Convergence Guarantees

Knowl 7 — Extrapolation Error and Value Divergence in Standard Off-Policy Deep Actor-Critic Algorithms

Knowl 8 — Comparative Performance and Value Stability of BCQ Across Continuous Control Tasks

Knowl 9 — Generative Action Constraints Versus Ensemble Uncertainty Minimization in Batch RL

Knowl 10 — Extrapolation Error in Kernel-Based Batch Reinforcement Learning

References

Citation

BibTeX

Source Code

Access the Paper