Rainbow: Combining Improvements in Deep Reinforcement Learning

Matteo HesselJoseph ModayilHado Van HasseltTom SchaulGeorg OstrovskiWill DabneyDan HorganBilal PiotMohammad AzarDavid Silver

article2017AAAI2,706 citations

Integrates six key extensions to Deep Q-Networks into a unified agent that achieves state-of-the-art performance across 57 Atari games, accompanied by thorough ablation studies identifying which components drive its sample efficiency and final score improvements.

Listen

The article addresses the challenge of improving the DQN algorithm for deep reinforcement learning on complex sequential decision tasks such as Atari games, where multiple independent extensions had been proposed but their compatibility and combined value remained unclear. This matters because effective combinations could accelerate progress toward more capable agents without requiring entirely new architectures.

The article set out to evaluate whether six specific extensions to DQN—double Q-learning, prioritized replay, dueling networks, multi-step returns, distributional reinforcement learning, and noisy networks—could be integrated into a single agent and whether they deliver complementary gains.

The authors tested the integrated agent, named Rainbow, on all 57 games in the Atari 2600 benchmark using standardized training and evaluation protocols over 200 million frames, with direct comparisons to published baselines and controlled ablation experiments that removed one component at a time.

Rainbow reached a median human-normalized score of 231 percent in the no-ops regime and 153 percent in the human-starts regime, surpassing the best prior single-extension agents; it matched DQN’s final performance after only 7 million frames and exceeded all baselines within 44 million frames. Prioritized replay and multi-step learning produced the largest performance drops when removed, while distributional learning and noisy networks contributed clear but smaller aggregate gains; dueling networks and double Q-learning showed mixed per-game effects.

These results indicate that the extensions address distinct limitations and can be combined for substantially higher data efficiency and final performance, reducing the number of environment interactions needed to reach strong play and improving reliability across a wide range of games.

Further work should examine additional algorithmic components such as episodic memory or count-based exploration, test the same integration approach in policy-gradient and actor-critic methods, and explore more principled ways to handle raw rewards and action repetition.

The study used a single set of hyperparameters across all games after limited manual tuning, relied on reward clipping, and focused exclusively on value-based methods; results may therefore not generalize immediately to other domains or training regimes that differ markedly in compute or environment access.

Google deep reinforcement learning reinforcement learning Rainbow Deep Q-Network Double Q‑learning

arXiv: 1710.02298

Paper: Human-level control through deep reinforcement learning, Volodymyr Mnih et al. (2015). This seminal paper introduces Deep Q-Networks (DQN) for Atari games, providing the fundamental baseline architecture and training regime that Rainbow directly unifies and extends.
Paper: Deep Reinforcement Learning with Double Q-learning, Hado van Hasselt et al. (2016). This work introduces Double DQN to decouple action selection from value estimation, establishing one of the six essential algorithmic extensions integrated into the Rainbow agent.
Paper: Prioritized Experience Replay, Tom Schaul et al. (2016). This chapter introduces Prioritized Experience Replay to sample transitions based on temporal-difference error, another core component whose ablation is shown to cause severe performance drops in Rainbow.
Paper: Dueling Network Architectures for Deep Reinforcement Learning, Ziyu Wang et al. (2016). This chapter establishes the dueling network architecture that decouples state-value and action-advantage representations, forming one of the key structural extensions evaluated in Rainbow.
Paper: A Distributional Perspective on Reinforcement Learning, Marc G. Bellemare et al. (2017). This work formulates categorical distributional reinforcement learning (C51), which replaces scalar value estimation with probability distributions over returns and is directly integrated into Rainbow.
Paper: Asynchronous Methods for Deep Reinforcement Learning, Volodymyr Mnih et al. (2016). This paper presents multi-step asynchronous reinforcement learning updates, providing the foundation for the multi-step return mechanism integrated into the Rainbow architecture.
Paper: The Arcade Learning Environment: An Evaluation Platform for General Agents, Marc G. Bellemare et al. (2013). This paper establishes the Arcade Learning Environment (ALE) Atari 2600 benchmark protocol used as the primary empirical evaluation suite for Rainbow.

Paper: Distributional Reinforcement Learning with Quantile Regression, Will Dabney et al. (2018). This paper advances the categorical distributional RL component used in Rainbow by replacing fixed heuristic bins with quantile regression (QR-DQN) to directly minimize the Wasserstein distance.
Paper: Recurrent Experience Replay in Distributed Reinforcement Learning, Steven Kapturowski et al. (2019). This work builds on prioritized replay and multi-step DQN architectures to handle partially observable environments and recurrent networks via distributed experience replay (R2D2).
Paper: Mastering Atari with Discrete World Models, Danijar Hafner et al. (2021). This work introduces DreamerV2 and benchmarks its discrete world-model approach directly against Rainbow and other leading model-free agents across the Atari suite.
Paper: Deep Reinforcement Learning at the Edge of the Statistical Precipice, Rishabh Agarwal et al. (2021). This study critically assesses statistical evaluation methodologies and variability across deep RL algorithms on Atari benchmarks, analyzing results from benchmarks established by agents like Rainbow.
Paper: Deep Reinforcement Learning that Matters, Peter Henderson et al. (2018). This empirical investigation examines reproducibility, hyperparameter sensitivity, and evaluation protocols in deep reinforcement learning following the proliferation of complex integrated agents.

Cover for Rainbow: Combining Improvements in Deep Reinforcement Learning

Abstract

The deep reinforcement learning community has made several independent improvements to the DQN algorithm. However, it is unclear which of these extensions are complementary and can be fruitfully combined. This paper examines six extensions to the DQN algorithm and empirically studies their combination. Our experiments show that the combination provides state-of-the-art performance on the Atari 2600 benchmark, both in terms of data efficiency and final performance. We also provide results from a detailed ablation study that shows the contribution of each component to overall performance.

Topics

deep reinforcement learning reinforcement learning

Keywords

Rainbow Deep Q-Network Double Q‑learning Prioritized Experience Replay Dueling Network Multi‑step Learning Distributional Q‑learning Noisy Nets Atari 2600 Human‑normalized Performance

Introduction
Background
Extensions to DQN
The Integrated Agent
Experimental Methods
Analysis
Discussion
References

Knowls

Knowl 1 — Rainbow Deep Reinforcement Learning Agent
model/method
The Rainbow agent integrates six independent extensions to the Deep Q-Network (DQN) algorithm into a single value-based reinforcement learning architecture:
1. Double Q-learning: Decouples action selection from action evaluation at the bootstrap step to mitigate positive maximization bias.
2. Multi-step learning: Replaces single-step temporal-difference (TD) targets with truncated $n$ -step discounted returns to propagate rewards faster and adjust the bias-variance trade-off.
3. Distributional reinforcement learning: Models the full probability distribution of discounted returns using a categorical distribution over a fixed discrete support, rather than estimating scalar expected values.
4. Dueling network architecture: Decomposes value representations into a state-value stream and an action-advantage stream that share a convolutional encoder.
5. Prioritized experience replay: Samples transitions from the replay buffer proportionally to their learning potential, measured by the Kullback-Leibler (KL) divergence loss of the return distribution rather than scalar TD error.
6. Noisy Nets: Replaces conventional deterministic linear layers with stochastic linear layers possessing learnable perturbation parameters to achieve state-conditional, self-annealing exploration without $\epsilon$ -greedy random action selection.
All components share a unified convolutional feature extractor and parameter update rule.
Knowl 2 — Multi-Step Distributional Double Q-Learning Loss in Rainbow
equation
Rainbow defines its learning objective by combining multi-step returns, categorical distributional reinforcement learning, and Double Q-learning.

Given an $n$ -step transition $(S_t, A_t, R_{t+1:t+n}, \gamma_{t+1:t+n}, S_{t+n})$ , the truncated $n$ -step discounted return is defined as:

$R_t^{(n)} = \sum_{k=0}^{n-1} \gamma_t^{(k)} R_{t+k+1}, \quad \text{with } \gamma_t^{(k)} = \prod_{j=1}^k \gamma_{t+j}$

where $\gamma_{t+j} \in [0, 1]$ is the discount factor at step $t+j$ .

The bootstrap action $a^*_{t+n}$ at the next state $S_{t+n}$ is chosen greedily according to the expected values computed by the online parameter set $\theta$ (Double Q-learning selection):

$a^*_{t+n} = \arg\max_a q_\theta(S_{t+n}, a) = \arg\max_a \sum_{i=1}^{N_{\text{atoms}}} z_i p_\theta^i(S_{t+n}, a)$

where $z$ is a vector of $N_{\text{atoms}}$ discrete support points spanning $[v_{\min}, v_{\max}]$ .

The target distribution $d_t^{(n)}$ is formed by evaluating action $a^*_{t+n}$ using the target network parameter set $\bar{\theta}$ , shifting the support by $R_t^{(n)}$ , and contracting it by $\gamma_t^{(n)} = \prod_{j=1}^n \gamma_{t+j}$ :

$d_t^{(n)} = \left(R_t^{(n)} + \gamma_t^{(n)} z, \; p_{\bar{\theta}}(S_{t+n}, a^*_{t+n})\right)$

The network parameters $\theta$ are updated by minimizing the Kullback-Leibler divergence between the projected target distribution and the online distribution $d_t = (z, p_\theta(S_t, A_t))$ :

$\mathcal{L}_{\text{Rainbow}}(\theta) = D_{\text{KL}}\left(\Phi_z d_t^{(n)} \,\big\|\, d_t\right)$

where $\Phi_z$ is the $L_2$ -projection of the target support onto the discrete support $z$ .
Knowl 3 — Distributional Dueling Network Architecture in Rainbow
model/method
Rainbow adapts the dueling network architecture to output categorical return distributions over a fixed support vector $z = (z_1, \dots, z_{N_{\text{atoms}}})^\top \in \mathbb{R}^{N_{\text{atoms}}}$ where:

$z_i = v_{\min} + (i-1) \frac{v_{\max} - v_{\min}}{N_{\text{atoms}} - 1}, \quad i \in \{1, \dots, N_{\text{atoms}}\}$

A shared convolutional feature encoder $f_\xi(s)$ produces latent representation $\phi = f_\xi(s)$ . This representation feeds into:
1. A value stream $v_\eta(\phi) \in \mathbb{R}^{N_{\text{atoms}}}$ predicting value distributions independent of actions.
2. An advantage stream $a_\psi(\phi, a) \in \mathbb{R}^{N_{\text{atoms}} \times N_{\text{actions}}}$ predicting action-specific advantage distributions.
For each atom index $i \in \{1, \dots, N_{\text{atoms}}\}$ and action $a$ , the value and advantage outputs are merged and normalized via a softmax layer along the atom dimension:

$p_\theta^i(s, a) = \frac{\exp\left(v_\eta^i(\phi) + a_\psi^i(\phi, a) - \bar{a}_\psi^i(s)\right)}{\sum_{j=1}^{N_{\text{atoms}}} \exp\left(v_\eta^j(\phi) + a_\psi^j(\phi, a) - \bar{a}_\psi^j(s)\right)}$

where $\bar{a}_\psi^i(s) = \frac{1}{N_{\text{actions}}} \sum_{a'=1}^{N_{\text{actions}}} a_\psi^i(\phi, a')$ , and $\theta = \{\xi, \eta, \psi\}$ .

The expected action value $q_\theta(s, a)$ is recovered by computing the inner product $q_\theta(s, a) = z^\top p_\theta(s, a) = \sum_{i=1}^{N_{\text{atoms}}} z_i p_\theta^i(s, a)$ .
Knowl 4 — KL-Divergence Transition Prioritization for Prioritized Experience Replay
model/method
In Rainbow, transitions in the prioritized replay buffer are sampled according to their distributional loss rather than absolute scalar temporal-difference (TD) error. The sampling priority $p_t$ for transition $t$ is defined as:

$p_t \propto \left( D_{\text{KL}}\left(\Phi_z d_t^{(n)} \,\big\|\, d_t\right) \right)^\omega$

where:
- $d_t = (z, p_\theta(S_t, A_t))$ is the online predicted return distribution.
- $\Phi_z d_t^{(n)}$ is the projected multi-step Double Q target return distribution on support $z$ .
- $\omega \in [0, 1]$ is the prioritization exponent controlling the strength of prioritization.
New transitions are inserted into the replay buffer with maximum priority to bias sampling toward recent transitions. Non-uniform sampling bias during gradient descent is corrected using importance sampling weights $w_t = (N \cdot P(t))^{-\beta}$ , where $N$ is the current buffer size and $\beta$ is annealed linearly from an initial value to $1.0$ over training.

Prioritizing by KL loss provides greater robustness in stochastic environments because the KL divergence can continue to decrease even when environment transitions and returns are non-deterministic.
Knowl 5 — Noisy Linear Layers with Factorised Gaussian Noise for Exploration
model/method
Rainbow eliminates standard $\epsilon$ -greedy exploration entirely (setting $\epsilon = 0$ ) by replacing all fully connected linear layers in the value and advantage streams with noisy linear layers. A noisy linear layer transforms an input vector $x$ to output $y$ according to:

$y = (b + Wx) + \left(b_{\text{noisy}} \odot \epsilon^b + (W_{\text{noisy}} \odot \epsilon^w) x\right)$

where:
- $W$ and $b$ are standard learnable deterministic weight matrix and bias vector parameters.
- $W_{\text{noisy}}$ and $b_{ ext{noisy}}$ are learnable parameters scaling the stochastic perturbation.
- $\odot$ denotes the element-wise Hadamard product.
- $\epsilon^b$ and $\epsilon^w$ are zero-mean random noise vectors drawn from a factorised Gaussian noise distribution to reduce the number of independent random variables sampled per forward pass.
- Initial noise parameter $\sigma_0$ is set to $0.5$ .
Because the noise parameters are optimized by gradient descent alongside the deterministic parameters, the network learns to attenuate noise at different rates in different regions of the state space, enabling state-conditional, self-annealing exploration.

Knowl 6 — Atari 2600 Benchmark Performance Comparison of Rainbow Against Baselines

data/table

Rainbow was evaluated across the full suite of 57 Atari 2600 games from the Arcade Learning Environment. Median human-normalized performance scores across all 57 games were measured under two evaluation protocols: no-ops starts (up to 30 random no-op actions at episode initialization) and human starts (initialization from random points along human expert trajectories).

Agent	No-ops Starts Median	Human Starts Median
DQN	79%	68%
Double DQN	117%	110%
Prioritized DDQN	140%	128%
Dueling DDQN	151%	117%
A3C	–	116%
Noisy DQN	118%	102%
Distributional DQN (C51)	185%	125%
Rainbow	231%	153%

Rainbow outperforms all individual baseline algorithms by a substantial margin in both testing regimes. In terms of sample efficiency, Rainbow matches the final performance of DQN after 7 million frames and surpasses the final performance of all published baselines within 44 million frames.

Knowl 7 — Hyperparameter Configuration for the Rainbow Agent

experimental setup

Rainbow uses a single fixed hyperparameter configuration across all 57 Atari 2600 games, trained for 200 million frames (50 million environment steps with frame skip 4) using a single GPU per run.

Hyperparameter	Value
Replay buffer capacity	1,000,000 transitions
Minimum history to start learning	80,000 frames
Optimizer	Adam
Learning rate $\alpha$	$0.0000625$ ( $= 0.00025 / 4$ )
Adam $\epsilon$	$1.5 \times 10^{-4}$
Exploration $\epsilon$	$0.0$ (fully greedy)
Noisy Nets $\sigma_0$	$0.5$ (factorised Gaussian noise)
Target network update period	32,000 frames
Prioritization type	Proportional
Prioritization exponent $\omega$	$0.5$
Prioritization importance sampling $\beta$	$0.4 \to 1.0$ (linearly annealed)
Multi-step returns $n$	3
Distributional atoms $N_{\text{atoms}}$	51
Distributional support range $[v_{\min}, v_{\max}]$	$[-10, 10]$
Evaluation interval	Every 1,000,000 environment steps
Evaluation duration	500,000 frames
Maximum episode length	108,000 frames (30 min simulated play)

With prioritized replay, training updates can start after 80,000 frames rather than the 200,000 frames required by uniform replay DQN. Training for 200 million frames corresponds to approximately 10 days of wall-clock time on a single GPU (matching DQN's final score requires less than 10 hours).

Knowl 8 — Ablation Analysis of Individual Rainbow Components
empirical result
Evaluating six single-component ablations across all 57 Atari 2600 games identifies the relative contribution of each integrated extension to Rainbow's overall performance:
- Prioritized Experience Replay and Multi-Step Learning: These are the two most critical components. Removing either results in the largest drop in median normalized score and severely slows early learning. Removing multi-step returns also substantially harms final performance. Rainbow outperformed the ablated variant in 53 of 57 games for both prioritized replay and multi-step learning.
- Distributional Q-learning: Ranks third in overall impact. In the first 40 million frames, removing distributional RL yields no discernible difference in median performance, but the ablated agent lags behind in later training, particularly on games where scores approach or exceed human performance.
- Noisy Nets: Removing Noisy Nets and substituting annealed $\epsilon$ -greedy exploration degrades aggregate median performance and causes large drops on exploration-heavy games.
- Dueling Networks and Double Q-learning: Removing dueling networks or double Q-learning has minimal impact on aggregate median score across all 57 games, though their effects vary on individual games (dueling networks improved performance on games exceeding 200% of human level while slightly degrading games below 20%).
Knowl 9 — Interaction Between Distributional Support Truncation and Double Q-Learning
theoretical result
In Rainbow, ablating Double Q-learning produces only a small change in aggregate median score. This occurs because of an interaction between reward clipping and the fixed support of the categorical return distribution.

Because rewards are clipped to $[-1, 1]$ , true discounted returns in many Atari games exceed $+10$ , falling outside the predefined distributional support range $[v_{\min}, v_{\max}] = [-10, 10]$ . During the $L_2$ -projection $\Phi_z$ , probabilities assigned to returns exceeding $+10$ are projected onto the maximum boundary atom $z_{51} = 10$ . This projection causes systematic underestimation of high returns, which directly counteracts the maximization-induced overestimation bias of Q-learning. When the support is bounded in this manner, Double Q-learning provides little additional correction, though its contribution is expected to increase if the support range is expanded.

Coverage note — Prospective discussion in the final section concerning untested future additions (such as episodic memory, hierarchical reinforcement learning, auxiliary tasks, parallel training frameworks, and Pop-Art reward scaling) was omitted because these were not implemented or experimentally evaluated in the paper.

References

1.Bellemare, M. G.; Naddaf, Y.; Veness, J.; and Bowling, M. 2013. The arcade learning environment: An evaluation platform for general agents. J. Artif. Intell. Res. (JAIR) 47:253–279.
2.Bellemare, M. G.; Srinivasan, S.; Ostrovski, G.; Schaul, T.; Saxton, D.; and Munos, R. 2016. Unifying count-based exploration and intrinsic motivation. In NIPS.
3.Bellemare, M. G.; Dabney, W.; and Munos, R. 2017. A distributional perspective on reinforcement learning. In ICML.
4.Blundell, C.; Uria, B.; Pritzel, A.; Li, Y.; Ruderman, A.; Leibo, J. Z.; Rae, J.; Wierstra, D.; and Hassabis, D. 2016. Model-Free Episodic Control. ArXiv e-prints.
5.Dosovitskiy, A., and Koltun, V. 2016. Learning to act by predicting the future. CoRR abs/1611.01779.
6.Fortunato, M.; Azar, M. G.; Piot, B.; Menick, J.; Osband, I.; Graves, A.; Mnih, V.; Munos, R.; Hassabis, D.; Pietquin, O.; Blundell, C.; and Legg, S. 2017. Noisy networks for exploration. CoRR abs/1706.10295.
7.Hausknecht, M., and Stone, P. 2015. Deep recurrent Q-learning for partially observable MDPs. arXiv preprint arXiv:1507.06527.
8.He, F. S.; Liu, Y.; Schwing, A. G.; and Peng, J. 2016. Learning to play in a day: Faster deep reinforcement learning by optimality tightening. CoRR abs/1611.01606.
9.Jaderberg, M.; Mnih, V.; Czarnecki, W. M.; Schaul, T.; Leibo, J. Z.; Silver, D.; and Kavukcuoglu, K. 2016. Reinforcement learning with unsupervised auxiliary tasks. CoRR abs/1611.05397.
10.Kingma, D. P., and Ba, J. 2014. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR).
11.Kulkarni, T. D.; Narasimhan, K.; Saeedi, A.; and Tenenbaum, J. B. 2016a. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. CoRR abs/1604.06057.
12.Kulkarni, T. D.; Saeedi, A.; Gautam, S.; and Gershman, S. J. 2016b. Deep successor reinforcement learning. arXiv preprint arXiv:1606.02396.
13.Lin, L.-J. 1992. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning 8(3):293–321.
14.Mnih, V.; Kavukcuoglu, K.; Silver, D.; Graves, A.; Antonoglou, I.; Wierstra, D.; and Riedmiller, M. A. 2013. Playing atari with deep reinforcement learning. CoRR abs/1312.5602.
15.Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; Petersen, S.; Beattie, C.; Sadik, A.; Antonoglou, I.; King, H.; Kumaran, D.; Wierstra, D.; Legg, S.; and Hassabis, D. 2015. Human-level control through deep reinforcement learning. Nature 518(7540):529–533.
16.Mnih, V.; Badia, A. P.; Mirza, M.; Graves, A.; Lillicrap, T.; Harley, T.; Silver, D.; and Kavukcuoglu, K. 2016. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning.
17.Nair, A.; Srinivasan, P.; Blackwell, S.; Alcicek, C.; Fearon, R.; De Maria, A.; Panneershelvam, V.; Suleyman, M.; Beattie, C.; Petersen, S.; Legg, S.; Mnih, V.; Kavukcuoglu, K.; and Silver, D. 2015. Massively parallel methods for deep reinforcement learning. arXiv preprint arXiv:1507.04296.
18.O’Donoghue, B.; Munos, R.; Kavukcuoglu, K.; and Mnih, V. 2016. Pgq: Combining policy gradient and q-learning. CoRR abs/1611.01626.
19.Osband, I.; Blundell, C.; Pritzel, A.; and Roy, B. V. 2016. Deep exploration via bootstrapped dqn. In NIPS.
20.Salimans, T.; Ho, J.; Chen, X.; and Sutskever, I. 2017. Evolution strategies as a scalable alternative to reinforcement learning. CoRR abs/1703.03864.
21.Schaul, T.; Quan, J.; Antonoglou, I.; and Silver, D. 2015. Prioritized experience replay. In Proc. of ICLR.
22.Schulman, J.; Levine, S.; Moritz, P.; Jordan, M.; and Abbeel, P. 2015. Trust region policy optimization. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, 1889–1897. JMLR.org.
23.Sharma, S.; Lakshminarayanan, A. S.; and Ravindran, B. 2017. Learning to repeat: Fine grained action repetition for deep reinforcement learning. arXiv preprint arXiv:1702.06054.
24.Stadie, B. C.; Levine, S.; and Abbeel, P. 2015. Incentivizing exploration in reinforcement learning with deep predictive models. CoRR abs/1507.00814.
25.Sutton, R. S., and Barto, A. G. 1998. Reinforcement Learning: An Introduction. The MIT press, Cambridge MA.
26.Sutton, R. S. 1988. Learning to predict by the methods of temporal differences. Machine learning 3(1):9–44.
27.Tieleman, T., and Hinton, G. 2012. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning 4(2):26–31.
28.van Hasselt, H.; Guez, A.; Guez, A.; Hessel, M.; Mnih, V.; and Silver, D. 2016. Learning values across many orders of magnitude. In Advances in Neural Information Processing Systems 29, 4287–4295.
29.van Hasselt, H.; Guez, A.; and Silver, D. 2016. Deep reinforcement learning with double Q-learning. In Proc. of AAAI, 2094–2100.
30.van Hasselt, H. 2010. Double Q-learning. In Advances in Neural Information Processing Systems 23, 2613–2621.
31.Vezhnevets, A. S.; Osindero, S.; Schaul, T.; Heess, N.; Jaderberg, M.; Silver, D.; and Kavukcuoglu, K. 2017. Feudal networks for hierarchical reinforcement learning. CoRR abs/1703.01161.
32.Wang, Z.; Schaul, T.; Hessel, M.; van Hasselt, H.; Lanctot, M.; and de Freitas, N. 2016. Dueling network architectures for deep reinforcement learning. In Proceedings of The 33rd International Conference on Machine Learning, 1995–2003.

Citation

MLA

Hessel, M., et al. “Rainbow: Combining Improvements in Deep Reinforcement Learning”. arXiv, 2017, http://arxiv.org/abs/1710.02298v1.

APA

Hessel, M., Modayil, J., Hasselt, H. van ., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., & Silver, D. (2017). Rainbow: Combining Improvements in Deep Reinforcement Learning. arXiv. http://arxiv.org/abs/1710.02298v1

Chicago

Hessel, M., J. Modayil, H. van . Hasselt, et al. 2017. “Rainbow: Combining Improvements in Deep Reinforcement Learning”. arXiv. http://arxiv.org/abs/1710.02298v1.

Harvard

Hessel, M. et al. (2017) “Rainbow: Combining Improvements in Deep Reinforcement Learning”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1710.02298v1.

Vancouver

1. Hessel M, Modayil J, Hasselt H van, Schaul T, Ostrovski G, Dabney W, Horgan D, Piot B, Azar M, Silver D (2017) Rainbow: Combining Improvements in Deep Reinforcement Learning. arXiv

BibTeX

@article{hessel2017rainbow,
  title = {Rainbow: Combining Improvements in Deep Reinforcement Learning},
  author = {Hessel, Matteo and Modayil, Joseph and Hasselt, Hado van and Schaul, Tom and Ostrovski, Georg and Dabney, Will and Horgan, Dan and Piot, Bilal and Azar, Mohammad and Silver, David},
  year = {2017},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1710.02298v1},
  eprint = {1710.02298}
}

Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Rainbow: Combining Improvements in Deep Reinforcement Learning

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Rainbow Deep Reinforcement Learning Agent

Knowl 2 — Multi-Step Distributional Double Q-Learning Loss in Rainbow

Knowl 3 — Distributional Dueling Network Architecture in Rainbow

Knowl 4 — KL-Divergence Transition Prioritization for Prioritized Experience Replay

Knowl 5 — Noisy Linear Layers with Factorised Gaussian Noise for Exploration