Evolution Strategies as a Scalable Alternative to Reinforcement Learning

Tim SalimansJonathan HoXi ChenIlya Sutskever

article2017arXiv1,829 citations

Demonstrates that evolution strategies can match standard reinforcement learning methods on Atari and MuJoCo benchmarks while scaling efficiently across thousands of parallel CPU workers to achieve dramatic training speedups.

Listen

Modern artificial intelligence heavily relies on reinforcement learning to teach autonomous agents how to solve complex control and decision-making tasks. However, standard reinforcement learning algorithms rely on backpropagating gradients and estimating future reward values, which creates substantial communication overhead and makes it difficult to scale training across modern distributed computer clusters. The article addresses this operational bottleneck by evaluating whether Evolution Strategies, an older class of black-box optimization algorithms, can serve as a highly parallel, scalable alternative to standard reinforcement learning.

The article set out to demonstrate that Evolution Strategies can reliably train deep neural network policies to competitive performance levels on standard continuous robotic control and pixel-based video game benchmarks while achieving dramatic speedups through distributed computing. To evaluate this approach, the authors tested Evolution Strategies on continuous control environments in the MuJoCo physics simulator—including complex tasks like 3D humanoid walking—and 51 standard Atari 2600 games. The implementation eliminated heavy network communication by synchronizing random seeds across workers, allowing over a thousand parallel central processing units on commercial cloud hardware to communicate solely via single scalar reward values per episode.

The core finding is that Evolution Strategies scales near-linearly across large computing clusters, reducing training turnaround time from nearly a full day to mere minutes. Using 1,440 processor cores, the system solved the challenging 3D humanoid walking task in under 10 minutes, compared to roughly 11 hours on a single machine. In terms of sample efficiency, Evolution Strategies required approximately 3 to 10 times more environment interaction data than baseline algorithms on complex tasks, but this gap was offset by requiring roughly 3 times less computation per step because it avoids gradient backpropagation and value function approximation entirely. On the Atari benchmark, one hour of parallel Evolution Strategies training matched the computational cost of standard 24-hour reinforcement learning baselines, outperforming the established asynchronous actor-critic baseline on 23 games while falling short on 28. Furthermore, the approach exhibited robust exploration behavior and was completely invariant to action frequency and delayed rewards.

These findings indicate that computing turnaround time and engineering simplicity can often outweigh pure data efficiency. Because Evolution Strategies only evaluates complete episodes and perturbs policy parameters directly, it eliminates common reinforcement learning failure modes such as exploding gradients and high sensitivity to time-discounting parameters. Organizations can leverage standard cloud compute instances or low-precision inference hardware without requiring specialized high-bandwidth networking or complex gradient synchronization. Decision-makers facing long training cycles or delayed-reward problems should consider Evolution Strategies as a practical option whenever high-volume simulation data can be generated in parallel.

Next steps include applying Evolution Strategies to meta-learning and tasks with long time horizons or non-differentiable network components. Although Evolution Strategies proves highly effective when massive parallel simulation is available, its reliance on higher overall sample counts means it remains less suitable for physical systems where data collection is slow, expensive, or limited to real time.

OpenAI communication function approximation mdp evolution strategies reinforcement learning black-box optimization

arXiv: 1703.03864 openai/evolution-strategies-starter

Paper: Asynchronous Methods for Deep Reinforcement Learning, Volodymyr Mnih et al. (2016). Its asynchronous actor-critic results establish the distributed reinforcement-learning baseline that this paper directly compares against.
Paper: Benchmarking Deep Reinforcement Learning for Continuous Control, Yan Duan et al. (2016). Its standardized MuJoCo control suite supplies the benchmark context for assessing the source’s evolutionary approach against gradient-based methods.
Paper: Human-level control through deep reinforcement learning, Volodymyr Mnih et al. (2015). Its Atari DQN work establishes the pixel-based benchmark and reinforcement-learning baseline used to interpret the source’s game results.
Paper: Continuous control with deep reinforcement learning, T. Lillicrap et al. (2015). Its DDPG method provides a key continuous-control comparison, clarifying the sample-efficiency tradeoff discussed in the source.
Paper: High-Dimensional Continuous Control Using Generalized Advantage Estimation, John Schulman et al. (2016). Its policy-gradient and value-function framework makes clear which gradient-based training machinery the source’s evolution-strategy alternative avoids.
Paper: A Brief Survey of Deep Reinforcement Learning, Kai Arulkumaran et al. (2017). Its survey organizes the deep-RL algorithms and Atari and MuJoCo benchmarks that frame the source’s comparisons.

Paper: Recurrent World Models Facilitate Policy Evolution, David Ha et al. (2018). It carries evolution strategies from direct policy optimization into a learned world model, showing how policy evolution can train agents in an imagined environment.
Paper: Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning, Xin Qiu et al. (2025). It extends the source’s gradient-free, parallel evolution-strategy approach to fine-tuning billion-parameter language models on reasoning tasks.

Cover for Evolution Strategies as a Scalable Alternative to Reinforcement Learning

Abstract

We explore the use of Evolution Strategies (ES), a class of black box optimization algorithms, as an alternative to popular MDP-based RL techniques such as Q-learning and Policy Gradients. Experiments on MuJoCo and Atari show that ES is a viable solution strategy that scales extremely well with the number of CPUs available: By using a novel communication strategy based on common random numbers, our ES implementation only needs to communicate scalars, making it possible to scale to over a thousand parallel workers. This allows us to solve 3D humanoid walking in 10 minutes and obtain competitive results on most Atari games after one hour of training. In addition, we highlight several advantages of ES as a black box optimization technique: it is invariant to action frequency and delayed rewards, tolerant of extremely long horizons, and does not need temporal discounting or value function approximation.

Topics

communication function approximation mdp q-learning random numbers value function approximation black-box optimization optimization problems

Keywords

evolution strategies reinforcement learning black-box optimization policy gradients neural network policies Atari games continuous control

1 Introduction
2 Evolution Strategies
2.1 Scaling and parallelizing ES
2.2 The impact of network parameterization
3 Smoothing in parameter space versus smoothing in action space
3.1 When is ES better than policy gradients?
3.2 Problem dimensionality
3.3 Advantages of not calculating gradients
4 Experiments
4.1 MuJoCo
4.2 Atari
4.3 Parallelization
4.4 Invariance to temporal resolution
5 Related work
6 Conclusion
References

Knowls

Knowl 1 — Natural Evolution Strategies Objective and Parameter Gradient Estimator
model/method
Evolution Strategies (ES) for reinforcement learning optimizes policy parameters $\theta \in \mathbb{R}^d$ by maximizing the expected return under an isotropic Gaussian parameter distribution $p_\psi(\theta) = \mathcal{N}(\psi, \sigma^2 I)$ , where $\psi = \theta$ is the parameter mean and $\sigma > 0$ is a fixed scalar standard deviation. For an environment return function $F(\theta)$ , the stochastic objective represents a Gaussian-blurred version of $F$ :

$F_{\text{ES}}(\theta) = \mathbb{E}_{\epsilon \sim \mathcal{N}(0, I)} [F(\theta + \sigma \epsilon)]$

Using the score function identity $\nabla_\psi \mathbb{E}_{\theta \sim p_\psi} [F(\theta)] = \mathbb{E}_{\theta \sim p_\psi} [F(\theta) \nabla_\psi \log p_\psi(\theta)]$ , the gradient with respect to $\theta$ is:

$\nabla_\theta F_{\text{ES}}(\theta) = \frac{1}{\sigma} \mathbb{E}_{\epsilon \sim \mathcal{N}(0, I)} \{ F(\theta + \sigma \epsilon) \epsilon \}$

Because $\mathbb{E}_{\epsilon \sim \mathcal{N}(0, I)} [F(\theta) \epsilon / \sigma] = 0$ , this estimator is mathematically equivalent to high-dimensional randomized finite differences:

$\nabla_\theta F_{\text{ES}}(\theta) = \mathbb{E}_{\epsilon \sim \mathcal{N}(0, I)} \left[ \frac{F(\theta + \sigma \epsilon) - F(\theta)}{\sigma} \epsilon \right]$

This smoothing removes non-smoothness introduced by discontinuous environment transitions or discrete actions, allowing gradient-based stochastic optimization without calculating analytical derivatives via backpropagation.
Knowl 2 — Parallelized Evolution Strategies via Synchronized Random Seeds
algorithm
Parallelized Evolution Strategies coordinates $n$ parallel workers by synchronizing pseudo-random seeds before optimization. Because every worker can reconstruct the perturbations $\epsilon_j \in \mathbb{R}^d$ generated by all other workers $j \in \{1, \dots, n\}$ , workers only communicate scalar episode returns $F_j \in \mathbb{R}$ , requiring $O(1)$ communication bandwidth per rollout rather than communicating parameter-sized gradient vectors.
Input: Learning rate $\alpha$ , noise standard deviation $\sigma$ , initial policy parameters $\theta_0$ , population size $n$
Initialize: $n$ parallel workers with synchronized pseudo-random number seeds and shared parameters $\theta_0$
for $t = 0, 1, 2, \dots$ do
    for each worker $i = 1, \dots, n$ in parallel do
        Sample perturbation $\epsilon_i \sim \mathcal{N}(0, I)$
        Evaluate policy $\theta_t + \sigma \epsilon_i$ on environment rollout to compute return $F_i = F(\theta_t + \sigma \epsilon_i)$
    end for
    Broadcast scalar return $F_i$ from each worker $i$ to all other workers
    for each worker $i = 1, \dots, n$ in parallel do
        Reconstruct perturbations $\epsilon_j$ for all $j = 1, \dots, n$ using known random seeds
        Compute fitness-shaped rank-transformed values or raw returns $F_j$
        Update parameters $\theta_{t+1} \leftarrow \theta_t + \frac{\alpha}{n \sigma} \sum_{j=1}^n F_j \epsilon_j$
    end for
end for
Practical variance reduction and stabilization enhancements include:
- Antithetic sampling (mirrored sampling): Perturbations are evaluated in complementary pairs $(\epsilon_i, -\epsilon_i)$ .
- Fitness shaping: Returns $F_j$ are replaced by a rank transformation before computing the gradient step to eliminate outlier distortion and prevent premature convergence to local optima.
- Weight decay: An $L_2$ regularization penalty is applied to $\theta$ to prevent parameter magnitudes from dominating perturbation scale $\sigma$ .
- Dynamic episode length capping: Episode length is truncated at $m = 2 \times \text{mean}(\text{episode steps})$ to guarantee at least $50\%$ CPU utilization across parallel workers.
Knowl 3 — Estimator Variance Scaling in Parameter-Space Smoothing vs Action-Space Smoothing
theoretical result
For an agent executing a sequence of actions $a = (a_1, \dots, a_T)$ over an episode of length $T$ and receiving total return $R(a)$ :
- Policy Gradients (action-space smoothing) introduce stochasticity into the action distribution $p(a; \theta) = \prod_{t=1}^T p(a_t; \theta)$ with gradient: $\nabla_\theta F_{\text{PG}}(\theta) = \mathbb{E}_\epsilon \{ R(a(\epsilon, \theta)) \nabla_\theta \log p(a(\epsilon, \theta); \theta) \}$ Assuming simple Monte Carlo estimation (REINFORCE) with a baseline, the variance scales as: $\operatorname{Var}[\nabla_\theta F_{\text{PG}}(\theta)] \approx \operatorname{Var}[R(a)] \sum_{t=1}^T \operatorname{Var}[\nabla_\theta \log p(a_t; \theta)]$ Because $\nabla_\theta \log p(a; \theta)$ is a sum of $T$ uncorrelated terms, the policy gradient estimator variance grows linearly with the episode horizon $T$ .
- Evolution Strategies (parameter-space smoothing) introduces stochasticity into parameters $\tilde{\theta} = \theta + \xi$ where $\xi \sim \mathcal{N}(0, \sigma^2 I)$ with gradient: $\nabla_\theta F_{\text{ES}}(\theta) = \mathbb{E}_\xi \{ R(a(\xi, \theta)) \nabla_\theta \log p(\tilde{\theta}(\xi, \theta); \theta) \}$ Its variance estimator satisfies: $\operatorname{Var}[\nabla_\theta F_{\text{ES}}(\theta)] \approx \operatorname{Var}[R(a)] \operatorname{Var}[\nabla_\theta \log p(\tilde{\theta}; \theta)]$
The ES variance term $\operatorname{Var}[\nabla_\theta \log p(\tilde{\theta}; \theta)]$ is independent of the episode horizon $T$ . While policy gradient methods rely on reward discounting or learned value functions to reduce effective horizon (which biases gradients if actions have long-lasting effects), Evolution Strategies remains unbiased and exhibits lower variance on long-horizon, delayed-reward problems without value function approximations.
Knowl 4 — Exploration Enhancement via Virtual Batch Normalization and Action Discretization
model/method
Because Evolution Strategies relies solely on parameter perturbations $\theta + \sigma \epsilon$ for exploration rather than stochastic action selection, standard neural architectures can suffer from insufficient exploration. Two structural parameterizations mitigate this issue:
1. Virtual Batch Normalization (VBN) for visual policies (e.g., Atari convolutional neural networks): Standard Gaussian perturbations to convolutional weights often produce policies that output identical actions regardless of the input frame. VBN normalizes network activations using normalization statistics computed from a fixed reference minibatch sampled once at initialization. This causes early randomly perturbed networks to become sensitive to minor pixel variations across frames, inducing the diversity of action selection necessary to uncover sparse initial rewards.
2. Action Discretization for continuous control policies (e.g., MuJoCo robotics tasks): Standard multilayer perceptrons producing continuous control actions can be overly smooth with respect to parameter perturbations. Discretizing continuous action dimensions into uniform discrete bins (e.g., 10 bins per action dimension) forces discontinuous policy outputs with respect to inputs and parameters, generating a wide variety of exploratory behaviors during rollouts.
Knowl 5 — Distributed Scaling and Linear Speedup on MuJoCo 3D Humanoid
empirical result
Evolution Strategies was evaluated on the 3D Humanoid walking continuous control task in OpenAI Gym (MuJoCo physics simulator) using parallel worker clusters deployed on Amazon EC2:
- On a single 18-core CPU instance, ES required 657 minutes (~11 hours) of wall-clock time to achieve a target score of 6,000 (comparable to state-of-the-art reinforcement learning baselines).
- When scaled to 1,440 CPU cores across 80 instances, ES solved the task to the same 6,000 score threshold in 10 minutes (median of 7 independent runs).
- The wall-clock scaling from 18 cores to 1,440 cores exhibits linear speedup with core count, enabled by minimal $O(1)$ scalar-only inter-worker communication.
- Unlike policy gradient algorithms such as Trust Region Policy Optimization (TRPO), ES discovered multiple distinct locomotion gaits on 3D Humanoid, including walking sideways and walking backwards.

Knowl 6 — Sample Complexity Comparison Between Evolution Strategies and TRPO on MuJoCo Benchmarks

data/table

The sample complexity of Evolution Strategies was evaluated against Trust Region Policy Optimization (TRPO) across six continuous robotic control tasks in OpenAI Gym / MuJoCo. Both algorithms used identical neural network policies (multilayer perceptrons with two 64-unit hidden layers and tanh nonlinearities). The table reports the environment interaction timesteps required by TRPO and ES to achieve 25%, 50%, 75%, and 100% of TRPO's final score at 5 million timesteps, averaged over 6 random seeds.

Environment	% TRPO final score	TRPO score	TRPO timesteps	ES timesteps	ES / TRPO timesteps
HalfCheetah	25%	-1.35	$9.05 \times 10^5$	$1.36 \times 10^5$	0.15
	50%	793.55	$1.70 \times 10^6$	$8.28 \times 10^5$	0.49
	75%	1589.83	$3.34 \times 10^6$	$1.42 \times 10^6$	0.42
	100%	2385.79	$5.00 \times 10^6$	$2.88 \times 10^6$	0.58
Hopper	25%	877.45	$7.29 \times 10^5$	$3.83 \times 10^5$	0.53
	50%	1718.16	$1.03 \times 10^6$	$3.73 \times 10^6$	3.64
	75%	2561.11	$1.59 \times 10^6$	$9.63 \times 10^6$	6.05
	100%	3403.46	$4.56 \times 10^6$	$3.16 \times 10^7$	6.94
InvertedDoublePendulum	25%	2358.98	$8.73 \times 10^5$	$3.98 \times 10^5$	0.46
	50%	4609.68	$9.65 \times 10^5$	$4.66 \times 10^5$	0.48
	75%	6874.03	$1.07 \times 10^6$	$5.30 \times 10^5$	0.49
	100%	9104.07	$4.39 \times 10^6$	$5.39 \times 10^6$	1.23
InvertedPendulum	25%	276.59	$2.21 \times 10^5$	$6.25 \times 10^4$	0.28
	50%	519.15	$2.73 \times 10^5$	$1.43 \times 10^5$	0.52
	75%	753.17	$3.25 \times 10^5$	$2.55 \times 10^5$	0.78
	100%	1000.00	$5.17 \times 10^5$	$4.55 \times 10^5$	0.88
Swimmer	25%	41.97	$1.04 \times 10^6$	$5.88 \times 10^5$	0.56
	50%	70.73	$1.82 \times 10^6$	$8.52 \times 10^5$	0.47
	75%	99.68	$2.33 \times 10^6$	$1.23 \times 10^6$	0.53
	100%	128.25	$4.59 \times 10^6$	$1.39 \times 10^6$	0.30
Walker2d	25%	957.68	$1.55 \times 10^6$	$6.43 \times 10^5$	0.41
	50%	1916.48	$2.27 \times 10^6$	$1.29 \times 10^7$	5.69
	75%	2872.81	$2.89 \times 10^6$	$2.31 \times 10^7$	8.02
	100%	3830.03	$4.81 \times 10^6$	$3.79 \times 10^7$	7.88

On simpler control environments (HalfCheetah, Swimmer, InvertedPendulum), ES attains equal or superior sample efficiency compared to TRPO ( $0.30\times$ to $0.88\times$ timesteps at 100% TRPO performance). On harder locomotion tasks (Hopper, Walker2d), ES solves the tasks within a $7\times$ to $8\times$ sample complexity penalty relative to TRPO.

Knowl 7 — Atari 2600 Benchmark Performance of Evolution Strategies vs A3C and DQN

data/table

Evolution Strategies was trained on 51 Atari 2600 games using raw pixel inputs and feedforward convolutional neural networks across 720 parallel CPUs on Amazon EC2 for 1 billion frames (1 hour wall-clock time). Because ES omits backpropagation and value function estimation, 1 billion frames under ES requires equivalent total neural network compute to 320 million frames of Asynchronous Advantage Actor-Critic (A3C, 1 day wall-clock time). Deterministic evaluation scores are averaged over 10 test episodes with up to 30 random initial no-ops.

Game	DQN	A3C FF (1 day)	HyperNEAT	ES FF (1 hour)	A2C FF
Amidar	133.4	283.9	184.4	112.0	548.2
Assault	3332.3	3746.1	912.6	1673.9	2026.6
Asterix	124.5	6723.0	2340.0	1440.0	3779.7
Asteroids	697.1	3009.4	1694.0	1562.0	1733.4
Atlantis	76108.0	772392.0	61260.0	1267410.0	2872644.8
Bank Heist	176.3	946.0	214.0	225.0	724.1
Battle Zone	17560.0	11340.0	36200.0	16600.0	8406.2
Beam Rider	8672.4	13235.9	1412.8	744.0	4438.9
Berzerk	–	1433.4	1394.0	686.0	720.6
Bowling	41.2	36.2	135.8	30.0	28.9
Boxing	25.8	33.7	16.4	49.8	95.8
Breakout	303.9	551.6	2.8	9.5	368.5
Centipede	3773.1	3306.5	25275.2	7783.9	2773.3
Chopper Command	3046.0	4669.0	3960.0	3710.0	1700.0
Crazy Climber	50992.0	101624.0	0.0	26430.0	100034.4
Demon Attack	12835.2	84997.5	14620.0	1166.5	23657.7
Double Dunk	21.6	0.1	2.0	0.2	3.2
Enduro	475.6	82.2	93.6	95.0	0.0
Fishing Derby	2.3	13.6	49.8	49.0	33.9
Freeway	25.8	0.1	29.0	31.0	0.0
Frostbite	157.4	180.1	2260.0	370.0	266.6
Gopher	2731.8	8442.8	364.0	582.0	6266.2
Gravitar	216.5	269.5	370.0	805.0	256.2
Ice Hockey	3.8	4.7	10.6	4.1	4.9
Kangaroo	2696.0	106.0	800.0	11200.0	1357.6
Krull	3864.0	8066.6	12601.4	8647.2	6411.5
Montezuma's Revenge	50.0	53.0	0.0	0.0	0.0
Name This Game	5439.9	5614.0	6742.0	4503.0	5532.8
Phoenix	–	28181.8	1762.0	4041.0	14104.7
Pit Fall	–	123.0	0.0	0.0	8.2
Pong	16.2	11.4	17.4	21.0	20.8
Private Eye	298.2	194.4	10747.4	100.0	100.0
Q*Bert	4589.8	13752.3	695.0	147.5	15758.6
River Raid	4065.3	10001.2	2616.0	5009.0	9856.9
Road Runner	9264.0	31769.0	3220.0	16590.0	33846.9
Robotank	58.5	2.3	43.8	11.9	2.2
Seaquest	2793.9	2300.2	716.0	1390.0	1763.7
Skiing	–	13700.0	7983.6	15442.5	15245.8
Solaris	–	1884.8	160.0	2090.0	2265.0
Space Invaders	1449.7	2214.7	1251.0	678.5	951.9
Star Gunner	34081.0	64393.0	2720.0	1470.0	40065.6
Tennis	2.3	10.2	0.0	4.5	11.2
Time Pilot	5640.0	5825.0	7340.0	4970.0	4637.5
Tutankham	32.4	26.1	23.6	130.3	194.3
Up and Down	3311.3	54525.4	43734.0	67974.0	75785.9
Venture	54.0	19.0	0.0	760.0	0.0
Video Pinball	20228.1	185852.6	0.0	22834.8	46470.1
Wizard of Wor	246.0	5278.0	3360.0	3480.0	1587.5
Yars Revenge	–	7270.8	24096.4	16401.7	8963.5
Zaxxon	831.0	2659.0	3000.0	6380.0	5.6

Across the 51 evaluated Atari games, ES outperforms A3C on 23 games and performs worse on 28 games, while demonstrating substantial advantages on games requiring sustained exploratory trajectories or delayed rewards (e.g., Kangaroo: 11,200.0 vs 106.0; Venture: 760.0 vs 19.0; Freeway: 31.0 vs 0.1; Frostbite: 370.0 vs 180.1).

Knowl 8 — Invariance of Evolution Strategies to Temporal Resolution and Frame-Skip
empirical result
In MDP-based reinforcement learning algorithms (e.g., Q-learning and policy gradients), the action frequency or simulator frame-skip parameter $\kappa$ critically dictates learning success. Lower frame-skip values dramatically expand the effective trajectory step horizon $T$ , degrading gradient estimation variance and credit assignment.

Because Evolution Strategies operates directly on total episode returns without temporal discounting, its gradient estimator is invariant to action frequency. When tested on the Atari game Pong across frame-skip values $\kappa \in \{1, 2, 3, 4\}$ , the ES learning curves remain virtually identical, with each configuration converging to optimal performance in approximately 100 parameter updates.
Knowl 9 — Computational and Architectural Advantages of Gradient-Free Policy Optimization
model/method
Optimizing neural policies via Evolution Strategies offers distinct computational and architectural benefits over analytical gradient algorithms:
1. Compute and memory reductions: By eliminating the backpropagation backward pass and critic/value function training, ES reduces per-episode computation by approximately two-thirds ( $66\%$ ) and drastically cuts memory footprint by avoiding activation storage.
2. Hardware acceleration on inference devices: ES rollouts require only forward passes, enabling training on low-precision inference-only accelerators (such as TPUs or binary neural networks) where backpropagation memory requirements or gradient precision limits prevent standard gradient descent.
3. Immunity to exploding gradients in RNNs: Smoothing the objective function in parameter space bounds the effective curvature of the optimization landscape, preventing exploding gradient failures in recurrent neural network policies without explicit gradient clipping.
4. Support for non-differentiable components: Architectures containing non-differentiable operations, such as hard visual attention mechanisms, can be optimized directly without surrogate relaxation.
Knowl 10 — Independence of ES Optimization from Nominal Parameter Dimension
theoretical result
Although zero-order gradient estimation on general non-smooth objectives theoretically exhibits sample complexity scaling linearly with the parameter dimension $d$ , the empirical performance of Evolution Strategies on neural network policies is governed by the intrinsic problem difficulty rather than the nominal number of parameters.

For instance, duplicating all features in a linear model $\hat{y} = x \cdot w$ by setting $x' = (x, x)$ doubles the nominal parameter dimension, but the optimization trajectory under ES remains identical provided the perturbation scale $\sigma$ and learning rate $\alpha$ are halved.

Empirically, using larger convolutional neural network architectures with ES on Atari 2600 games improves average performance over smaller architectures, mirroring the optimization landscape advantages of overparameterized deep networks (which contain fewer poor local minima) observed in supervised deep learning.

Coverage note — None was omitted; all contributed algorithms, mathematical derivations, parameterization techniques, distributed systems designs, and empirical control/Atari benchmark results were converted into knowls.

References

1.Alex Braylan, Mark Hollenbeck, Elliot Meyerson, and Risto Miikkulainen. Frame skip is a powerful parameter for learning to play atari. Space, 1600:1800, 2005.
2.Dimo Brockhoff, Anne Auger, Nikolaus Hansen, Dirk V Arnold, and Tim Hohm. Mirrored sampling and sequential selection for evolution strategies. In International Conference on Parallel Problem Solving from Nature, pages 11–21. Springer, 2010.
3.Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. arXiv preprint arXiv:1606.01540, 2016.
4.Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In Proceedings of the 33rd International Conference on Machine Learning (ICML), 2016a.
5.Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RL2 : Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779, 2016b.
6.John C Duchi, Michael I Jordan, Martin J Wainwright, and Andre Wibisono. Optimal rates for zero-order convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory, 61(5):2788–2806, 2015.
7.John Geweke. Antithetic acceleration of monte carlo integration in bayesian inference. Journal of Econometrics, 38(1-2):73–89, 1988.
8.Tobias Glasmachers, Tom Schaul, and Jürgen Schmidhuber. A natural evolution strategy for multi-objective optimization. In International Conference on Parallel Problem Solving from Nature, pages 627–636. Springer, 2010a.
9.Tobias Glasmachers, Tom Schaul, Sun Yi, Daan Wierstra, and Jürgen Schmidhuber. Exponential natural evolution strategies. In Proceedings of the 12th annual conference on Genetic and evolutionary computation, pages 393–400. ACM, 2010b.
10.Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, 9(2):159–195, 2001.
11.Matthew Hausknecht, Joel Lehman, Risto Miikkulainen, and Peter Stone. A neuroevolution approach to general atari game playing. IEEE Transactions on Computational Intelligence and AI in Games, 6(4):355–366, 2014.
12.Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
13.Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. arXiv preprint arXiv:1704.04760, 2017.
14.Kenji Kawaguchi. Deep learning without poor local minima. In Advances In Neural Information Processing Systems, pages 586–594, 2016.
15.Jan Koutník, Faustino Gomez, and Jürgen Schmidhuber. Evolving neural networks in compressed weight space. In Proceedings of the 12th annual conference on Genetic and evolutionary computation, pages 619–626. ACM, 2010.
16.Jan Koutník, Giuseppe Cuccu, Jürgen Schmidhuber, and Faustino Gomez. Evolving large-scale neural networks for vision-based reinforcement learning. In Proceedings of the 15th annual conference on Genetic and evolutionary computation, pages 1061–1068. ACM, 2013.
17.Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
18.Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, 2016.
19.Yurii Nesterov. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization, 22(2):341–362, 2012.
20.Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, pages 1–40, 2011.
21.Andrew Ng, Adam Coates, Mark Diel, Varun Ganapathi, Jamie Schulte, Ben Tse, Eric Berger, and Eric Liang. Autonomous inverted helicopter flight via reinforcement learning. Experimental Robotics IX, pages 363–372, 2006.
22.Ronald Parr and Stuart Russell. Reinforcement learning with hierarchies of machines. Advances in neural information processing systems, pages 1043–1049, 1998.
23.I. Rechenberg and M. Eigen. Evolutionsstrategie: Optimierung Technischer Systeme nach Prinzipien der Biologischen Evolution. Frommann-Holzboog Stuttgart, 1973.
24.Sebastian Risi and Julian Togelius. Neuroevolution in games: State of the art and open challenges. IEEE Transactions on Computational Intelligence and AI in Games, 2015.
25.Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems, pages 2226–2234, 2016.
26.Tom Schaul, Tobias Glasmachers, and Jürgen Schmidhuber. High dimensions and heavy tails for natural evolution strategies. In Proceedings of the 13th annual conference on Genetic and evolutionary computation, pages 845–852. ACM, 2011.
27.Juergen Schmidhuber and Jieyu Zhao. Direct policy search and uncertain policy evaluation. In Aaai spring symposium on search under uncertain and incomplete information, stanford univ, pages 119–124, 1998.
28.Jürgen Schmidhuber, Daan Wierstra, Matteo Gagliolo, and Faustino Gomez. Training recurrent networks by evolino. Neural computation, 19(3):757–779, 2007.
29.John Schulman, Sergey Levine, Pieter Abbeel, Michael I Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, pages 1889–1897, 2015.
30.H.-P. Schwefel. Numerische optimierung von computer-modellen mittels der evolutionsstrategie. 1977.
31.Frank Sehnke, Christian Osendorfer, Thomas Rückstieß, Alex Graves, Jan Peters, and Jürgen Schmidhuber. Parameter-exploring policy gradients. Neural Networks, 23(4):551–559, 2010.
32.David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016.
33.James C Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE transactions on automatic control, 37(3):332–341, 1992.
34.Rupesh Kumar Srivastava, Jürgen Schmidhuber, and Faustino Gomez. Generalized compressed network search. In International Conference on Parallel Problem Solving from Nature, pages 337–346. Springer, 2012.
35.Kenneth O Stanley, David B D’Ambrosio, and Jason Gauci. A hypercube-based encoding for evolving large-scale neural networks. Artificial life, 15(2):185–212, 2009.
36.Freek Stulp and Olivier Sigaud. Policy improvement methods: Between black-box optimization and episodic reinforcement learning. 2012.
37.Yi Sun, Daan Wierstra, Tom Schaul, and Juergen Schmidhuber. Efficient natural evolution strategies. In Proceedings of the 11th Annual conference on Genetic and evolutionary computation, pages 539–546. ACM, 2009.
38.Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pages 5026–5033. IEEE, 2012.
39.Nicolas Usunier, Gabriel Synnaeve, Zeming Lin, and Soumith Chintala. Episodic exploration for deep deterministic policies: An application to starcraft micromanagement tasks. arXiv preprint arXiv:1609.02993, 2016.
40.Sjoerd van Steenkiste, Jan Koutník, Kurt Driessens, and Jürgen Schmidhuber. A wavelet-based encoding for neuroevolution. In Proceedings of the 2016 on Genetic and Evolutionary Computation Conference, pages 517–524. ACM, 2016.
41.Daan Wierstra, Tom Schaul, Jan Peters, and Juergen Schmidhuber. Natural evolution strategies. In Evolutionary Computation, 2008. CEC 2008.(IEEE World Congress on Computational Intelligence). IEEE Congress on, pages 3381–3387. IEEE, 2008.
42.Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. Journal of Machine Learning Research, 15(1):949–980, 2014.
43.Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992.
44.Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, volume 14, pages 77–81, 2015.
45.Sun Yi, Daan Wierstra, Tom Schaul, and Jürgen Schmidhuber. Stochastic search using the natural gradient. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 1161–1168. ACM, 2009.

Citation

MLA

Salimans, T., et al. “Evolution Strategies as a Scalable Alternative to Reinforcement Learning”. arXiv, 2017, http://arxiv.org/abs/1703.03864v2.

APA

Salimans, T., Ho, J., Chen, X., Sidor, S., & Sutskever, I. (2017). Evolution Strategies as a Scalable Alternative to Reinforcement Learning. arXiv. http://arxiv.org/abs/1703.03864v2

Chicago

Salimans, T., J. Ho, X. Chen, S. Sidor, and I. Sutskever. 2017. “Evolution Strategies as a Scalable Alternative to Reinforcement Learning”. arXiv. http://arxiv.org/abs/1703.03864v2.

Harvard

Salimans, T. et al. (2017) “Evolution Strategies as a Scalable Alternative to Reinforcement Learning”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1703.03864v2.

Vancouver

1. Salimans T, Ho J, Chen X, Sidor S, Sutskever I (2017) Evolution Strategies as a Scalable Alternative to Reinforcement Learning. arXiv

BibTeX

@article{salimans2017evolution,
  title = {Evolution Strategies as a Scalable Alternative to Reinforcement Learning},
  author = {Salimans, Tim and Ho, Jonathan and Chen, Xi and Sidor, Szymon and Sutskever, Ilya},
  year = {2017},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1703.03864v2},
  eprint = {1703.03864}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Published with permission

Evolution Strategies as a Scalable Alternative to Reinforcement Learning

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Natural Evolution Strategies Objective and Parameter Gradient Estimator

Knowl 2 — Parallelized Evolution Strategies via Synchronized Random Seeds

Knowl 3 — Estimator Variance Scaling in Parameter-Space Smoothing vs Action-Space Smoothing

Knowl 4 — Exploration Enhancement via Virtual Batch Normalization and Action Discretization

Knowl 5 — Distributed Scaling and Linear Speedup on MuJoCo 3D Humanoid