The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games

Chao YuAkash VeluEugene VinitskyYu WangA. BayenYi Wu

article2022NeurIPS2,777 citations

Demonstrates that Proximal Policy Optimization matches or outperforms specialized off-policy methods across major cooperative multi-agent benchmarks, establishing simple on-policy learning as a competitive baseline through actionable implementation and tuning guidelines.

Listen

Cooperative multi-agent reinforcement learning has seen strong results from off-policy algorithms such as QMIX and MADDPG, while on-policy methods like PPO have been underused because they are widely viewed as less sample-efficient and harder to tune in multi-agent environments. This perception stands in contrast to the on-policy algorithms that powered recent large-scale successes in games such as StarCraft II and Dota 2. The article therefore set out to test whether a properly configured PPO can deliver competitive performance across standard cooperative benchmarks without any domain-specific algorithmic changes.

The authors conducted a systematic empirical evaluation of two PPO variants—MAPPO, which uses a centralized value function, and IPPO, which remains fully decentralized—on four widely used testbeds: multi-agent particle environments, StarCraft Multi-Agent Challenge, Google Research Football, and Hanabi. They compared both variants against strong off-policy baselines, performed equivalent hyperparameter searches for all methods, and measured both final returns and sample efficiency under controlled compute budgets.

Across the majority of tasks, MAPPO and IPPO reached final performance that matched or exceeded the off-policy baselines while using a comparable number of environment steps. In StarCraft, PPO variants achieved win rates within one standard deviation of the best reported results on most maps; in Google Research Football they outperformed QMIX by large margins and beat a method that used large-scale pre-training on four of five scenarios; in Hanabi they produced the highest average scores for three- and four-player games. Five practical factors proved especially important: value normalization, the inclusion of both global and agent-specific features in the value-function input, limiting training to roughly ten epochs on hard tasks, keeping the PPO clipping parameter at or below 0.2, and using sufficiently large batch sizes.

These results indicate that simple, well-tuned PPO implementations can serve as strong, easy-to-deploy baselines for cooperative multi-agent problems and that the presumed sample-efficiency gap between on-policy and off-policy methods is smaller than commonly assumed when modern implementation practices are followed. Practitioners can therefore consider PPO as a first-line option rather than defaulting to more complex off-policy frameworks, provided they adopt the identified configuration guidelines.

Further validation on continuous-action domains, competitive settings, and heterogeneous agent populations would strengthen the conclusions, as would theoretical analysis of why the recommended practices improve stability. Until such work is completed, the empirical findings should be treated as reliable guidance within the discrete-action, cooperative regimes examined in the study.

Shanghai Qi Zhi Institute Tsinghua University University of California Berkeley agent systems multi-agent multi-agent reinforcement learning Proximal Policy Optimization Multi‑Agent Reinforcement Learning Sample Efficiency

arXiv: 2103.01955 marlbenchmark/on-policy

Paper: Proximal Policy Optimization Algorithms, John Schulman et al. (2017). Introduces the core Proximal Policy Optimization (PPO) algorithm and clipped surrogate objective that the source adapts and systematically evaluates in multi-agent cooperative environments.
Paper: Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments, Ryan Lowe et al. (2017). Establishes the centralized training with decentralized execution (CTDE) paradigm and MADDPG framework that serves as a primary multi-agent actor-critic baseline in the source's empirical study.
Paper: Counterfactual Multi-Agent Policy Gradients, Jakob N. Foerster et al. (2017). Presents the Counterfactual Multi-Agent (COMA) policy gradient framework using a centralized critic on multi-agent cooperative benchmarks like StarCraft micromanagement.
Paper: Trust Region Policy Optimization, John Schulman et al. (2015). Formulates the foundational trust region optimization theory and policy gradient bounds from which PPO is derived.
Paper: Deep Reinforcement Learning that Matters, Peter Henderson et al. (2018). Provides the methodological analysis on deep RL reproducibility, implementation details, and tuning practices that motivated the source paper's controlled benchmarking of PPO.
Paper: Markov Games as a Framework for Multi-Agent Reinforcement Learning, M. Littman (1994). Introduces Markov games as the foundational mathematical framework for multi-agent reinforcement learning problems.
Paper: Emergent Tool Use From Multi-Agent Autocurricula, Bowen Baker et al. (2020). Demonstrates the practical viability and emergent behaviors of scaling PPO with centralized critics in complex multi-agent simulated environments.

Paper: Reinforcement Learning: An Overview, Kevin P. Murphy (2024). Surveys the broader landscape of modern reinforcement learning, situating multi-agent actor-critic algorithms and policy optimization within contemporary AI paradigms.
Paper: Empirical Design in Reinforcement Learning, Andrew Patterson et al. (2024). Deepens the experimental and statistical evaluation methodologies required to rigorously test and compare reinforcement learning algorithms across complex benchmark suites.

Cover for The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games

Abstract

Proximal Policy Optimization (PPO) is a ubiquitous on-policy reinforcement learning algorithm but is significantly less utilized than off-policy learning algorithms in multi-agent settings. This is often due to the belief that PPO is significantly less sample efficient than off-policy methods in multi-agent systems. In this work, we carefully study the performance of PPO in cooperative multi-agent settings. We show that PPO-based multi-agent algorithms achieve surprisingly strong performance in four popular multi-agent testbeds: the particle-world environments, the StarCraft multi-agent challenge, Google Research Football, and the Hanabi challenge, with minimal hyperparameter tuning and without any domain-specific algorithmic modifications or architectures. Importantly, compared to competitive off-policy methods, PPO often achieves competitive or superior results in both final returns and sample efficiency. Finally, through ablation studies, we analyze implementation and hyperparameter factors that are critical to PPO’s empirical performance, and give concrete practical suggestions regarding these factors. Our results show that when using these practices, simple PPO-based methods can be a strong baseline in cooperative multi-agent reinforcement learning. Source code is released at https://github.com/marlbenchmark/on-policy.

1 Introduction
2 Related Works
3 PPO in Multi-Agent Settings
3.1 Preliminaries
3.2 MAPPO and IPPO
3.3 Implementation Details
4 Main Results
4.1 Testbeds, Baselines, and Common Experimental Setup
4.2 MPE Testbed
4.3 SMAC Testbed
4.4 Google Football Testbed
4.5 Hanabi Testbed
5 Factors Influential to PPO’s Performance
5.1 Value Normalization
5.2 Input Representation to Value Function
5.3 Training Data Usage
5.4 PPO Clipping
5.5 PPO Batch Size
6 Conclusion
A MAPPO Details
B Testing domains
C Training details
C.1 Implementation
C.2 Parameter Sharing
C.3 Death Masking
C.4 Hyperparameters
D Additional Results
D.1 Additional SMAC Results
D.2 Additional GRF Results
E Ablation Studies
References

Knowls

Knowl 1 — Multi-Agent Proximal Policy Optimization (MAPPO) and Independent PPO (IPPO)
model/method
Cooperative multi-agent reinforcement learning is formalized as a decentralized partially observable Markov decision process (DEC-POMDP) defined by the tuple $\langle \mathcal{S}, \mathcal{A}, \mathcal{O}, R, P, n, \gamma \rangle$ :
- $\mathcal{S}$ is the global state space.
- $\mathcal{A}$ is the action space shared by all agents $i \in \{1, \dots, n\}$ .
- $o_i = \mathcal{O}(s; i)$ is the local observation received by agent $i$ when the environment is in global state $s \in \mathcal{S}$ .
- $P(s' \mid s, A)$ is the transition probability from state $s$ to state $s'$ given the joint action $A = (a_1, \dots, a_n) \in \mathcal{A}^n$ .
- $R(s, A)$ is the shared team reward function.
- $\gamma \in [0, 1)$ is the discount factor.
Each agent chooses actions using an actor network parameterizing a decentralized policy $\pi_\theta(a_i \mid o_i)$ , maximizing the expected discounted return: $J(\theta) = \mathbb{E}_{A^t, s^t} \left[ \sum_{t=0}^\infty \gamma^t R(s^t, A^t) \right]$ where $A^t = (a_1^t, \dots, a_n^t)$ is the joint action at timestep $t$ .

Actor-critic PPO is applied to this setting with separate neural networks for the policy $\pi_\theta$ and value function $V_\phi$ :
1. Multi-Agent PPO (MAPPO): Follows the Centralized Training with Decentralized Execution (CTDE) framework. The critic $V_\phi$ receives centralized state information containing global features during training, while the actor $\pi_\theta(a_i \mid o_i)$ executes solely on local observations $o_i$ .
2. Independent PPO (IPPO): Both the actor $\pi_\theta(a_i \mid o_i)$ and the critic $V_\phi(o_i)$ take only the agent's local observation $o_i$ as input.
In environments with homogeneous agents (identical observation and action spaces), parameter sharing is used such that all agents share identical policy parameters $\theta$ and value function parameters $\phi$ . Both MAPPO and IPPO utilize Generalized Advantage Estimation (GAE) with advantage normalization and clipped objectives.
Knowl 2 — Running Value Target Standardization (Value Normalization)
model/method
In cooperative multi-agent reinforcement learning, target returns can change drastically throughout training as policies evolve across all agents, causing instability in critic updates. Value normalization stabilizes value learning by standardizing the targets of the value network using running empirical estimates of the mean $\mu_V$ and standard deviation $\sigma_V$ of the value targets.

During training, the value network $V_\phi$ regresses toward normalized targets: $\mathcal{L}(\phi) = \frac{1}{|B|} \sum_{k \in B} \left( V_\phi(s_k) - \frac{\hat{R}_k - \mu_V}{\sigma_V} \right)^2$ where $\hat{R}_k$ is the computed target return for transition $k$ (e.g., via Generalized Advantage Estimation) and $B$ is a training batch.

When calculating advantage estimates for policy optimization, the network predictions are denormalized back to the original scale: $\tilde{V}_\phi(s_t) = V_\phi(s_t) \cdot \sigma_V + \mu_V$ $\hat{A}_t = \sum_{l=0}^{\infty} (\gamma \lambda)^l \left( r_{t+l} + \gamma \tilde{V}_\phi(s_{t+l+1}) - \tilde{V}_\phi(s_{t+l}) \right)$ where $\gamma$ is the discount factor and $\lambda$ is the GAE parameter. This target standardization stabilizes optimization without restricting value predictions across environments with widely differing reward scales.
Knowl 3 — Centralized Value Function State Representations: CL, EP, AS, and FP
model/method
In multi-agent centralized training with decentralized execution (CTDE), the choice of input representation provided to the centralized value function $V_\phi$ strongly influences training quality. Four representations are distinguished:
1. Concatenation of Local Observations (CL): Formed by concatenating all $n$ agents' local observations, $s_{\text{CL}} = (o_1, o_2, \dots, o_n)$ . Its dimension scales linearly with team size $n$ , making learning difficult in large-agent settings, and it omits global environment features unobserved by any agent.
2. Environment-Provided Global State (EP): The environment-native global state $s_{\text{EP}}$ . While it captures overall environment context, it often omits crucial agent-specific local features, such as agent IDs and local available action masks.
3. Agent-Specific Global State (AS): Combines global and agent-specific local features for agent $i$ by concatenating the environment state with agent $i$ 's local observation: $s_{\text{AS}, i} = (s_{\text{EP}}, o_i)$ This provides full information but introduces feature redundancy because $s_{\text{EP}}$ and $o_i$ often duplicate identical state variables.
4. Feature-Pruned Agent-Specific Global State (FP): Formed by taking $s_{\text{AS}, i}$ and removing redundant overlapping features shared between $s_{\text{EP}}$ and $o_i$ . FP preserves both global environment context and agent-specific local information while minimizing input dimensionality.
Knowl 4 — Training Epoch and Mini-batch Configuration in Multi-Agent PPO
model/method
While single-agent continuous control commonly re-uses collected experience across tens of epochs and divides rollouts into 32 to 64 mini-batches, excessive sample reuse degrades performance in multi-agent environments due to non-stationarity induced by concurrent policy updates across agents.

Best practices for training data usage in multi-agent PPO:
1. Restrict the number of training epochs per update to at most 10 epochs on difficult tasks, and at most 15 epochs on easy tasks. Using 5 to 10 epochs prevents policy deterioration in complex cooperative coordination tasks.
2. Avoid splitting rollout data into multiple mini-batches. Using 1 mini-batch (full-batch gradient descent over the collected on-policy rollout) or at most 2 mini-batches yields superior stability and final performance compared to finer partitions (such as 4 mini-batches), as larger mini-batch sizes reduce gradient variance during on-policy updates.
Knowl 5 — Clipping Parameter Calibration in Multi-Agent PPO
model/method
PPO restricts update magnitudes via a clipping parameter $\epsilon$ , which bounds the policy probability ratio $r_t(\theta) = \frac{\pi_\theta(a_t \mid o_t)}{\pi_{\theta_{\text{old}}}(a_t \mid o_t)}$ within $[1 - \epsilon, 1 + \epsilon]$ and applies symmetric clipping to value loss updates.

In cooperative multi-agent reinforcement learning:
- The clipping threshold should satisfy $\epsilon \le 0.2$ .
- Large clipping parameters (such as $\epsilon \in \{0.2, 0.3, 0.5\}$ ) permit excessive policy shift per gradient step, destabilizing learning due to multi-agent non-stationarity.
- Small clipping values, such as $\epsilon = 0.05$ , provide stable learning dynamics and lower variance across seeds at the cost of slower initial convergence on difficult maps. Within $(0, 0.2]$ , $\epsilon$ acts as an explicit control parameter balancing training stability and learning speed.
Knowl 6 — Batch Size Scaling Dynamics in Multi-Agent PPO
model/method
In on-policy policy gradient methods such as MAPPO, trajectory batches collected in parallel are used to estimate gradients for policy and value function objectives. The batch size per update exhibits threshold dynamics in cooperative multi-agent domains:
- A critical minimal batch-size threshold is required for MAPPO to learn effective coordination. Below this threshold, MAPPO fails to solve complex coordination tasks or plateaus at suboptimal returns.
- Increasing batch size up to the critical threshold improves both final asymptotic performance and sample efficiency (the number of environment interactions needed to attain target success benchmarks such as 80%–90% win rate).
- Scaling batch size far beyond this critical point maintains peak win rates but reduces sample efficiency, requiring more total interactions to achieve equivalent performance milestones. The recommended practice is to identify the minimal batch size that reaches peak task performance.

Knowl 7 — Empirical Evaluation of MAPPO and IPPO on the StarCraft Multi-Agent Challenge (SMAC)

data/table

Evaluation of MAPPO (with Feature-Pruned FP and Agent-Specific AS critic inputs), Independent PPO (IPPO), QMix, and RODE across 23 maps in the StarCraft Multi-Agent Challenge (SMAC). Performance is measured as the median evaluation win rate percentage over 6 random seeds (with standard deviations in parentheses). MAPPO, IPPO, and QMix are trained up to 10M environment steps or convergence. Columns with an asterisk ( $*$ ) compare MAPPO to RODE using the exact timestep budget from RODE's published benchmark results.

Map	MAPPO(FP)	MAPPO(AS)	IPPO	QMix	RODE*	MAPPO*(FP)
2m_vs_1z	100.0 (0.0)	100.0 (0.0)	100.0 (0.0)	95.3 (5.2)	/	100.0 (0.0)
3m	100.0 (0.0)	100.0 (1.5)	100.0 (0.0)	96.9 (1.3)	/	100.0 (0.0)
2s_vs_1sc	100.0 (0.0)	100.0 (0.0)	100.0 (1.5)	96.9 (2.9)	100.0 (0.0)	100.0 (0.0)
2s3z	100.0 (0.7)	100.0 (1.5)	100.0 (0.0)	95.3 (2.5)	100.0 (0.0)	96.9 (1.5)
3s_vs_3z	100.0 (0.0)	100.0 (0.0)	100.0 (0.0)	96.9 (12.5)	/	100.0 (0.0)
3s_vs_4z	100.0 (1.3)	98.4 (1.6)	99.2 (1.5)	97.7 (1.7)	/	100.0 (2.1)
so_many_baneling	100.0 (0.0)	100.0 (0.7)	100.0 (1.5)	96.9 (2.3)	/	100.0 (1.5)
8m	100.0 (0.0)	100.0 (0.0)	100.0 (0.7)	97.7 (1.9)	/	100.0 (0.0)
MMM	96.9 (0.6)	93.8 (1.5)	96.9 (0.0)	95.3 (2.5)	/	93.8 (2.6)
1c3s5z	100.0 (0.0)	96.9 (2.6)	100.0 (0.0)	96.1 (1.7)	100.0 (0.0)	100.0 (0.0)
bane_vs_bane	100.0 (0.0)	100.0 (0.0)	100.0 (0.0)	100.0 (0.0)	100.0 (46.4)	100.0 (0.0)
3s_vs_5z	100.0 (0.6)	99.2 (1.4)	100.0 (0.0)	98.4 (2.4)	78.9 (4.2)	98.4 (5.5)
2c_vs_64zg	100.0 (0.0)	100.0 (0.0)	98.4 (1.3)	92.2 (4.0)	100.0 (0.0)	96.9 (3.1)
8m_vs_9m	96.9 (0.6)	96.9 (0.6)	96.9 (0.7)	92.2 (2.0)	/	84.4 (5.1)
25m	100.0 (1.5)	100.0 (4.0)	100.0 (0.0)	85.9 (7.1)	/	96.9 (3.1)
5m_vs_6m	89.1 (2.5)	88.3 (1.2)	87.5 (2.3)	75.8 (3.7)	71.1 (9.2)	65.6 (14.1)
3s5z	96.9 (0.7)	96.9 (1.9)	96.9 (1.5)	88.3 (2.9)	93.8 (2.0)	71.9 (11.8)
10m_vs_11m	96.9 (4.8)	96.9 (1.2)	93.0 (7.4)	95.3 (1.0)	95.3 (2.2)	81.2 (8.3)
MMM2	90.6 (2.8)	87.5 (5.1)	86.7 (7.3)	87.5 (2.6)	89.8 (6.7)	51.6 (21.9)
3s5z_vs_3s6z	84.4 (34.0)	63.3 (19.2)	82.8 (19.1)	82.8 (5.3)	96.8 (25.11)	75.0 (36.3)
27m_vs_30m	93.8 (2.4)	85.9 (3.8)	69.5 (11.8)	39.1 (9.8)	96.8 (1.5)	93.8 (3.8)
6h_vs_8z	88.3 (3.7)	85.9 (30.9)	84.4 (33.3)	9.4 (2.0)	78.1 (37.0)	78.1 (5.6)
corridor	100.0 (1.2)	98.4 (0.8)	98.4 (3.1)	84.4 (2.5)	65.6 (32.1)	93.8 (3.5)

Across SMAC maps, MAPPO achieves performance at least as high as QMix while using an identical sample budget, and achieves win rates comparable or superior to RODE in 10 out of 14 maps under matched timestep constraints.

Knowl 8 — Evaluation on Google Research Football (GRF) Academy Scenarios

data/table

Evaluation of MAPPO against off-policy baselines on Google Research Football (GRF) academy scenarios under a shared dense-reward setting. All algorithms except TiKick are trained from scratch for 25M environment steps (50M steps on CA(hard) and Corner). TiKick combines offline pre-training on human expert data with online RL fine-tuning. Results report average evaluation success rate percentages (and standard deviations) over 6 seeds across 100 test rollouts per evaluation.

Scenario	MAPPO	QMix	CDS	TiKick
3v.1	88.03 (1.06)	8.12 (2.83)	76.60 (3.27)	76.88 (3.15)
CA(easy)	87.76 (1.34)	15.98 (2.85)	63.28 (4.89)	/
CA(hard)	77.38 (4.81)	3.22 (1.60)	58.35 (5.56)	73.09 (2.08)
Corner	65.53 (2.19)	16.10 (3.00)	3.80 (0.54)	33.00 (3.01)
PS	94.92 (0.68)	8.05 (3.66)	94.15 (2.54)	/
RPS	76.83 (1.81)	8.08 (4.71)	62.38 (4.56)	79.12 (2.06)

MAPPO outperforms QMix across all scenarios, outperforms CDS without requiring intrinsic exploration rewards, and exceeds TiKick on 4 out of 5 scenarios despite learning from scratch without human demonstration pre-training.

Knowl 9 — Evaluation on Hanabi-Full with Varying Player Counts

data/table

Evaluation of MAPPO and IPPO compared to off-policy value-decomposition baselines (Simplified Action Decoder [SAD] and Value Decomposition Networks [VDN]) on the full-scale Hanabi benchmark across 2 to 5 players. In Hanabi, agents cannot observe their own cards; MAPPO supplies the agent's own card features to the centralized value function during training. Algorithms are evaluated over at least 3 random seeds across 10,000 evaluation games, trained for at most $10^{10}$ environment steps without auxiliary tasks.

# Players	Metric	MAPPO	IPPO	SAD	VDN
2	Avg.	23.89 (0.02)	24.00 (0.02)	23.87 (0.03)	23.83 (0.03)
	Best	24.23 (0.01)	24.19 (0.02)	24.01 (0.01)	23.96 (0.01)
3	Avg.	23.77 (0.20)	23.25 (0.33)	23.69 (0.05)	23.71 (0.06)
	Best	24.01 (0.01)	23.87 (0.03)	23.93 (0.01)	23.99 (0.01)
4	Avg.	23.57 (0.13)	22.52 (0.37)	23.27 (0.26)	23.03 (0.15)
	Best	23.71 (0.01)	23.06 (0.03)	23.81 (0.01)	23.79 (0.00)
5	Avg.	23.04 (0.10)	20.75 (0.56)	22.06 (0.23)	21.28 (0.12)
	Best	23.16 (0.01)	22.54 (0.02)	23.01 (0.01)	21.80 (0.01)

While IPPO is competitive with MAPPO in the 2-player setting, its performance declines as the number of agents increases. MAPPO outperforms IPPO, SAD, and VDN in 3, 4, and 5-player games, demonstrating the utility of a centralized critic in complex cooperative card reasoning games.

Knowl 10 — Empirical Performance of MAPPO and IPPO on Multi-Agent Particle Environments (MPE)
empirical result
Across the three cooperative Multi-Agent Particle Environments (MPE) evaluated over 10 random seeds using a fixed budget of $2 \times 10^6$ environment steps:
1. Physical Deception (Spread): Homogeneous agents must cooperate to cover target landmarks while avoiding inter-agent collisions. MAPPO matches the final return and sample efficiency of QMix and MADDPG.
2. Simple Reference (Reference): Homogeneous agents must navigate to landmarks whose target assignments are observed only by the partner agent. MAPPO matches QMix and converges to higher final returns than IPPO.
3. Cooperative Communication (Comm): Heterogeneous agents (speakers and listeners) cooperate without parameter sharing. MAPPO exceeds the final return of MADDPG and matches QMix while maintaining comparable sample efficiency.
Across all three MPE benchmarks, MAPPO and IPPO demonstrate that on-policy policy gradient methods can match the sample efficiency of off-policy algorithms (QMix and MADDPG) in continuous-state, discrete-action particle domains.
Knowl 11 — Scope and Limitations of the Empirical MARL PPO Benchmark
limitation
The empirical findings and guidelines presented for MAPPO and IPPO have several explicit boundaries:
1. Discrete Action Spaces: All evaluated benchmarks (MPE, SMAC, GRF, and Hanabi) use discrete action spaces; continuous multi-agent control settings are not evaluated.
2. Exclusively Cooperative Settings: The study focuses exclusively on fully cooperative multi-agent tasks with shared rewards, and does not assess competitive or general-sum games.
3. Agent Homogeneity: The majority of environments feature homogeneous agents where parameter sharing is standard; heterogeneous systems are limited to the Comm scenario in MPE.
4. Empirical Nature: The conclusions and recommendations are derived empirically without formal theoretical proofs or convergence rate guarantees under multi-agent non-stationarity.

Coverage note — None was omitted; all contributed models, guidelines, benchmark evaluations across four domains, and stated limitations are fully represented.

References

1.Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on-policy deep actor-critic methods? a large-scale study. In International Conference on Learning Representations, 2021.
2.Bowen Baker, Ingmar Kanitscheider, Todor M. Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
3.Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The Hanabi challenge: A new frontier for AI research. Artificial Intelligence, 280:103216, 2020.
4.Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. CoRR, abs/1912.06680, 2019.
5.Filippos Christianos, Georgios Papoudakis, Arrasy Rahman, and Stefano V. Albrecht. Scaling multi-agent reinforcement learning with selective parameter sharing, 2021.
6.Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. AAAI/IAAI, 1998(746-752):2, 1998.
7.Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip H. S. Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533, 2020.
8.Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pages 1329–1338, 2016.
9.Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study on ppo and trpo. In International Conference on Learning Representations, 2020.
10.Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pages 1407–1416, 2018.
11.Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
12.Jakob N Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. arXiv preprint arXiv:1709.04326, 2017.
13.Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018.
14.Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https://github.com/hill-a/stable-baselines, 2018.
15.Hengyuan Hu and Jakob N Foerster. Simplified action decoder for deep multi-agent reinforcement learning. In International Conference on Learning Representations, 2020.
16.Shiyu Huang, Wenze Chen, Longfei Zhang, Shizhen Xu, Ziyang Li, Fengming Zhu, Deheng Ye, Ting Chen, and Jun Zhu. Tikick: Towards playing multi-agent football full games from single-agent demonstrations, 2021.
17.Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. A closer look at deep policy gradients. In International Conference on Learning Representations, 2020.
18.Shariq Iqbal, Christian A. Schröder de Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, and Fei Sha. Ai-qmix: Attention and imagination for dynamic multi-agent reinforcement learning. CoRR, abs/2006.04222, 2020.
19.Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, and Sylvain Gelly. Google research football: A novel reinforcement learning environment. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 4501–4510. AAAI Press, 2020.
20.Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. Celebrating diversity in shared multi-agent reinforcement learning, 2021.
21.Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pages 157–163. Elsevier, 1994.
22.Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. Neural Information Processing Systems (NIPS), 2017.
23.Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. arXiv preprint arXiv:1703.04908, 2017.
24.Frans A Oliehoek, Christopher Amato, et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
25.Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
26.Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. In NeurIPS, 2020.
27.Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4295–4304. PMLR, 10–15 Jul 2018.
28.Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob N. Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. CoRR, abs/1902.04043, 2019.
29.John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
30.John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.
31.Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pages 5887–5896. PMLR, 2019.
32.Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 2085–2087, 2018.
33.J. K. Terry, Nathaniel Grammel, Ananth Hari, Luis Santos, and Benjamin Black. Revisiting parameter sharing in multi-agent deep reinforcement learning, 2021.
34.George Tucker, Surya Bhupatiraju, Shixiang Gu, Richard Turner, Zoubin Ghahramani, and Sergey Levine. The mirage of action-dependent baselines in reinforcement learning. In International conference on machine learning, pages 5015–5024. PMLR, 2018.
35.Oriol Vinyals, Igor Babuschkin, M Wojciech Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, H David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, P John Agapiou, Max Jaderberg, S Alexander Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, L Tom Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, pages 1–5, 2019.
36.Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. {QPLEX}: Duplex dueling multi-agent q-learning. In International Conference on Learning Representations, 2021.
37.Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. RODE: Learning roles to decompose multi-agent tasks. In International Conference on Learning Representations, 2021.

Citation

MLA

Yu, C., et al. “The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games”. arXiv, 2021, http://arxiv.org/abs/2103.01955v4.

APA

Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A., & Wu, Y. (2021). The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games. arXiv. http://arxiv.org/abs/2103.01955v4

Chicago

Yu, C., A. Velu, E. Vinitsky, et al. 2021. “The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games”. arXiv. http://arxiv.org/abs/2103.01955v4.

Harvard

Yu, C. et al. (2021) “The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2103.01955v4.

Vancouver

1. Yu C, Velu A, Vinitsky E, Gao J, Wang Y, Bayen A, Wu Y (2021) The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games. arXiv

BibTeX

@article{yu2021the,
  title = {The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games},
  author = {Yu, Chao and Velu, Akash and Vinitsky, Eugene and Gao, Jiaxuan and Wang, Yu and Bayen, Alexandre and Wu, Yi},
  year = {2021},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2103.01955v4},
  eprint = {2103.01955}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Multi-Agent Proximal Policy Optimization (MAPPO) and Independent PPO (IPPO)

Knowl 2 — Running Value Target Standardization (Value Normalization)

Knowl 3 — Centralized Value Function State Representations: CL, EP, AS, and FP

Knowl 4 — Training Epoch and Mini-batch Configuration in Multi-Agent PPO

Knowl 5 — Clipping Parameter Calibration in Multi-Agent PPO

Knowl 6 — Batch Size Scaling Dynamics in Multi-Agent PPO

Knowl 7 — Empirical Evaluation of MAPPO and IPPO on the StarCraft Multi-Agent Challenge (SMAC)

Knowl 8 — Evaluation on Google Research Football (GRF) Academy Scenarios

Knowl 9 — Evaluation on Hanabi-Full with Varying Player Counts

Knowl 10 — Empirical Performance of MAPPO and IPPO on Multi-Agent Particle Environments (MPE)

Knowl 11 — Scope and Limitations of the Empirical MARL PPO Benchmark

References

Citation

BibTeX

Source Code

Access the Paper