OpenAI Gym

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba
OpenAI

Abstract

OpenAI Gym$^{1}$ is a toolkit for reinforcement learning research. It includes a growing collection of benchmark problems that expose a common interface, and a website where people can share their results and compare the performance of algorithms. This whitepaper discusses the components of OpenAI Gym and the design decisions that went into the software.

$^{1}$ gym.openai.com

Executive Summary: Reinforcement learning has made rapid progress by combining deep neural networks with established methods such as policy gradients and Q-learning, yet meaningful comparisons across algorithms remain difficult because existing benchmark collections differ in interfaces, versioning practices, and ease of use. Without shared, reproducible tasks, researchers cannot reliably distinguish genuine advances in sample efficiency or final performance from differences in experimental setup.

OpenAI Gym was developed to supply a single, extensible toolkit that presents a growing set of reinforcement-learning environments through a uniform interface, together with a public website for posting and comparing results. The project deliberately limits its scope to the environment side of the agent–environment interaction, leaving researchers free to implement any learning procedure they choose.

The toolkit was assembled by integrating and extending prior benchmark collections, notably the Arcade Learning Environment for Atari games and the RLLab suite for continuous control. At launch it offered hundreds of tasks spanning classic control problems, algorithmic sequences, Atari titles, board games, and simulated 2-D and 3-D robots driven by the MuJoCo physics engine. Every environment carries a strict version identifier so that performance numbers remain comparable across software updates; by default each run is instrumented to record learning curves and occasional video.

The resulting system shows that a common interface can be maintained without restricting algorithmic style, that sample complexity can be tracked alongside final scores, and that public code sharing can substitute for hidden test sets when assessing generalization. These features lower the barrier to entry for new researchers and make it easier for the community to judge whether reported gains reflect genuine methodological improvement rather than tuning to a narrow set of tasks.

Because the environments are currently limited to simulation, the principal next step is integration with physical robots to test whether algorithms developed inside Gym transfer to hardware. Additional planned extensions include multi-agent scenarios and sequences of tasks that support curriculum and transfer learning. The main limitations at present are the absence of real-world validation and the reliance on user-submitted write-ups rather than automated verification of results; both can be addressed by the extensions already outlined.

1. Introduction

Section Summary: Reinforcement learning is a type of machine learning focused on training systems to make sequences of decisions, and recent combinations with deep learning have produced strong results on hard problems without heavy customization. To support further progress, the field needs reliable, shared test problems for comparing algorithms, and OpenAI Gym supplies a convenient collection of such tasks with a common interface, built-in versioning for consistent results, and an accompanying website for posting and viewing performance scores. The package draws from earlier benchmarks while aiming to be more accessible and to expand over time.

Reinforcement learning (RL) is the branch of machine learning that is concerned with making sequences of decisions. RL has a rich mathematical theory and has found a variety of practical applications [1]. Recent advances that combine deep learning with reinforcement learning have led to a great deal of excitement in the field, as it has become evident that general algorithms such as policy gradients and Q-learning can achieve good performance on difficult problems, without problem-specific engineering [2, 3, 4].

To build on recent progress in reinforcement learning, the research community needs good benchmarks on which to compare algorithms. A variety of benchmarks have been released, such as the Arcade Learning Environment (ALE) [5], which exposed a collection of Atari 2600 games as reinforcement learning problems, and recently the RLLab benchmark for continuous control [6], to which we refer the reader for a survey on other RL benchmarks, including [7, 8, 9, 10, 11]. OpenAI Gym aims to combine the best elements of these previous benchmark collections, in a software package that is maximally convenient and accessible. It includes a diverse collection of tasks (called environments) with a common interface, and this collection will grow over time. The environments are versioned in a way that will ensure that results remain meaningful and reproducible as the software is updated.

Alongside the software library, OpenAI Gym has a website (gym.openai.com) where one can find scoreboards for all of the environments, showcasing results submitted by users. Users are encouraged to provide links to source code and detailed instructions on how to reproduce their results.

2. Background

Section Summary: Reinforcement learning involves an agent that interacts with an environment by taking actions and receiving observations along with rewards, with the aim of maximizing some measure of total reward over time. The environment is formally described as a partially observable Markov decision process, and OpenAI Gym focuses on the episodic version of this setup, where each episode begins from a random starting state and continues until a terminal condition is reached. The goal is to achieve strong performance across episodes as efficiently as possible, as shown in a basic code example of sequential agent-environment steps.

Reinforcement learning assumes that there is an agent that is situated in an environment. Each step, the agent takes an action, and it receives an observation and reward from the environment. An RL algorithm seeks to maximize some measure of the agent's total reward, as the agent interacts with the environment. In the RL literature, the environment is formalized as a partially observable Markov decision process (POMDP) [12].

OpenAI Gym focuses on the episodic setting of reinforcement learning, where the agent's experience is broken down into a series of episodes. In each episode, the agent's initial state is randomly sampled from a distribution, and the interaction proceeds until the environment reaches a terminal state. The goal in episodic reinforcement learning is to maximize the expectation of total reward per episode, and to achieve a high level of performance in as few episodes as possible.

The following code snippet shows a single episode with 100 timesteps. It assumes that there is an object called agent, which takes in the observation at each timestep, and an object called env, which is the environment. OpenAI Gym does not include an agent class or specify what interface the agent should use; we just include an agent here for demonstration purposes.

    ob0 = env.reset() # sample environment state, return first observation
    a0 = agent.act(ob0) # agent chooses first action
    ob1, rew0, done0, info0 = env.step(a0) # environment returns observation, 
    # reward, and boolean flag indicating if the episode is complete.
    a1 = agent.act(ob1)
    ob2, rew1, done1, info1 = env.step(a1)
    ...
    a99 = agent.act(o99)
    ob100, rew99, done99, info2 = env.step(a99)
    # done99 == True  =>  terminal

3. Design Decisions

Section Summary: The design of OpenAI Gym focuses on standardizing only the environments rather than the agents themselves, giving users flexibility to implement learning in different styles such as incremental updates or batch processing. It prioritizes measuring how quickly an algorithm improves, known as sample complexity, over final performance alone, since extra computation can artificially boost results without reflecting true algorithmic quality. The system also favors collaborative peer review through shared code and detailed writeups over competitive rankings, supported by strict environment versioning to keep results comparable and default monitoring to track progress and generate shareable data.

The design of OpenAI Gym is based on the authors' experience developing and comparing reinforcement learning algorithms, and our experience using previous benchmark collections. Below, we will summarize some of our design decisions.

Environments, not agents. Two core concepts are the agent and the environment. We have chosen to only provide an abstraction for the environment, not for the agent. This choice was to maximize convenience for users and allow them to implement different styles of agent interface. First, one could imagine an "online learning" style, where the agent takes (observation, reward, done) as an input at each timestep and performs learning updates incrementally. In an alternative "batch update" style, a agent is called with observation as input, and the reward information is collected separately by the RL algorithm, and later it is used to compute an update. By only specifying the agent interface, we allow users to write their agents with either of these styles.

Emphasize sample complexity, not just final performance. The performance of an RL algorithm on an environment can be measured along two axes: first, the final performance; second, the amount of time it takes to learn—the sample complexity. To be more specific, final performance refers to the average reward per episode, after learning is complete. Learning time can be measured in multiple ways, one simple scheme is to count the number of episodes before a threshold level of average performance is exceeded. This threshold is chosen per-environment in an ad-hoc way, for example, as 90% of the maximum performance achievable by a very heavily trained agent.

Both final performance and sample complexity are very interesting, however, arbitrary amounts of computation can be used to boost final performance, making it a comparison of computational resources rather than algorithm quality.

Encourage peer review, not competition. The OpenAI Gym website allows users to compare the performance of their algorithms. One of its inspiration is Kaggle, which hosts a set of machine learning contests with leaderboards. However, the aim of the OpenAI Gym scoreboards is not to create a competition, but rather to stimulate the sharing of code and ideas, and to be a meaningful benchmark for assessing different methods.

RL presents new challenges for benchmarking. In the supervised learning setting, performance is measured by prediction accuracy on a test set, where the correct outputs are hidden from contestants. In RL, it's less straightforward to measure generalization performance, except by running the users' code on a collection of unseen environments, which would be computationally expensive. Without a hidden test set, one must check that an algorithm did not "overfit" on the problems it was tested on (for example, through parameter tuning).

We would like to encourage a peer review process for interpreting results submitted by users. Thus, OpenAI Gym asks users to create a Writeup describing their algorithm, parameters used, and linking to code. Writeups should allow other users to reproduce the results. With the source code available, it is possible to make a nuanced judgement about whether the algorithm "overfit" to the task at hand.

{Strict versioning for environments. If an environment changes, results before and after the change would be incomparable. To avoid this problem, we guarantee than any changes to an environment will be accompanied by an increase in version number. For example, the initial version of the CartPole task is named Cartpole-v0, and if its functionality changes, the name will be updated to Cartpole-v1.

{Monitoring by default. By default, environments are instrumented with a Monitor, which keeps track of every time step (one step of simulation) and reset (sampling a new initial state) are called. The Monitor's behavior is configurable, and it can record a video periodically. It also is sufficient to produce learning curves. The videos and learning curve data can be easily posted to the OpenAI Gym website.

4. Environments

Section Summary: OpenAI Gym includes a growing collection of simulated environments, each structured as a partially observable decision process for testing reinforcement learning methods. At launch these covered small-scale control tasks, algorithmic problems like sequence manipulation, classic Atari games, board games such as Go, and simulated two- and three-dimensional robots that rely on physics engines. Later additions have expanded the set with environments built on Box2D and the Doom engine.

OpenAI Gym contains a collection of Environments (POMDPs), which will grow over time. See Figure 1 for examples. At the time of Gym's initial beta release, the following environments were included:

• Classic control and toy text: small-scale tasks from the RL literature.
• Algorithmic: perform computations such as adding multi-digit numbers and reversing sequences. Most of these tasks require memory, and their difficulty can be chosen by varying the sequence length.
• Atari: classic Atari games, with screen images or RAM as input, using the Arcade Learning Environment [5].
• Board games: currently, we have included the game of Go on 9x9 and 19x19 boards, where the Pachi engine [13] serves as an opponent.
• 2D and 3D robots: control a robot in simulation. These tasks use the MuJoCo physics engine, which was designed for fast and accurate robot simulation [14]. A few of the tasks are adapted from RLLab [6].

Since the initial release, more environments have been created, including ones based on the open source physics engine Box2D or the Doom game engine via VizDoom [15].

5. Future Directions

Section Summary: In the future, OpenAI Gym plans to support tasks where multiple agents must work together or compete against one another. It also aims to introduce sequences of progressively harder problems so that algorithms can learn from one task and apply that knowledge to the next, rather than starting from scratch each time. Finally, the platform hopes to connect directly with real robotic hardware to test these methods outside of simulation.

In the future, we hope to extend OpenAI Gym in several ways.

• Multi-agent setting. It will be interesting to eventually include tasks in which agents must collaborate or compete with other agents.
• Curriculum and transfer learning. Right now, the tasks are meant to be solved from scratch. Later, it will be more interesting to consider sequences of tasks, so that the algorithm is trained on one task after the other. Here, we will create sequences of increasingly difficult tasks, which are meant to be solved in order.
• Real-world operation. Eventually, we would like to integrate the Gym API with robotic hardware, validating reinforcement learning algorithms in the real world.

References

Section Summary: This references section compiles foundational books and research papers on reinforcement learning, a method where algorithms learn decision-making by trial and error in simulated environments. It highlights influential works on deep learning techniques for games and control tasks, along with tools like physics engines, game platforms, and software libraries used to run and test such experiments. The list also notes benchmarks and past competitions that have helped standardize progress in the field.

[1] Dimitri P Bertsekas, Dimitri P Bertsekas, Dimitri P Bertsekas, and Dimitri P Bertsekas. Dynamic programming and optimal control. Athena Scientific Belmont, MA, 1995.

[2] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, Sadik Beattie, C., Antonoglou A., H. I., King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.

[3] J. Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz. Trust region policy optimization. In ICML, pages 1889–1897, 2015.

[4] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. arXiv preprint arXiv:1602.01783, 2016.

[5] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The Arcade Learning Environment: An evaluation platform for general agents. J. Artif. Intell. Res., 47:253–279, 2013.

[6] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. arXiv preprint arXiv:1604.06778, 2016.

[7] A. Geramifard, C. Dann, R. H. Klein, W. Dabney, and J. P. How. RLPy: A value-function-based reinforcement learning framework for education and research. J. Mach. Learn. Res., 16:1573–1578, 2015.

[8] B. Tanner and A. White. RL-Glue: Language-independent software for reinforcement-learning experiments. J. Mach. Learn. Res., 10:2133–2136, 2009.

[9] T. Schaul, J. Bayer, D. Wierstra, Y. Sun, M. Felder, F. Sehnke, T. Rückstieß, and J. Schmidhuber. PyBrain. J. Mach. Learn. Res., 11:743–746, 2010.

[10] S. Abeyruwan. RLLib: Lightweight standard and on/off policy reinforcement learning library (C++). http://web.cs.miami.edu/home/saminda/rilib.html, 2013.

[11] Christos Dimitrakakis, Guangliang Li, and Nikoalos Tziortziotis. The reinforcement learning competition 2014. AI Magazine, 35(3):61–65, 2014.

[12] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.

[13] Petr Baudiš and Jean-loup Gailly. Pachi: State of the art open source go program. In Advances in Computer Games, pages 24–38. Springer, 2011.

[14] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pages 5026–5033. IEEE, 2012.

[15] Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski. Vizdoom: A doom-based ai research platform for visual reinforcement learning. arXiv preprint arXiv:1605.02097, 2016.