Back

Counterfactual Multi-Agent Policy Gradients

Jakob N. FoersterGregory FarquharTriantafyllos AfourasNantas NardelliShimon Whiteson

article2017AAAI2,749 citationsOutstanding Student Paper Award

Introduces a multi-agent policy gradient method that resolves the multi-agent credit assignment problem in cooperative reinforcement learning by pairing a centralized critic with an efficient counterfactual baseline to isolate each agent's individual contribution.

Listen

Cooperative multi-agent tasks such as autonomous vehicle coordination and network packet routing require decentralized policies because of partial observability, communication limits, and exponentially growing joint action spaces. Existing reinforcement learning methods struggle with credit assignment when agents receive only global rewards, and independent learning approaches often fail to coordinate effectively.

The article introduces counterfactual multi-agent (COMA) policy gradients to learn effective decentralized policies through centralized training. COMA employs a centralized critic that conditions on the full state and joint actions, paired with decentralized actors that condition only on local histories, and replaces standard advantage estimates with a counterfactual baseline that isolates each agent's contribution.

The method was evaluated on a decentralized StarCraft unit micromanagement benchmark featuring limited fields of view and discrete actions across four scenarios with three to five units. Thirty-five independent trials compared COMA against independent actor-critic variants and ablated central-critic versions, measuring win rates over training episodes.

COMA achieved the highest final win rates in every scenario, for example 81 percent on the five-marine map versus 58–71 percent for baselines, and reached strong policies faster and more stably. Its best agents matched or approached published results from fully centralized controllers that had access to the complete state and macro-actions, despite operating under stricter decentralization constraints.

These results indicate that the counterfactual baseline successfully mitigates multi-agent credit assignment noise while the centralized critic supplies accurate value estimates unavailable to independent methods. The approach therefore enables reliable coordination in partially observable settings without requiring extra simulations or hand-crafted local rewards.

The authors recommend extending COMA to larger agent populations and developing more sample-efficient variants suitable for real-world deployment such as self-driving vehicles. They note that the current evaluation is limited to small scenarios with discrete actions and that coordinated exploration remains challenging as the number of agents grows; readers should therefore treat performance claims as preliminary until tested on larger problems.

University of Oxfordactor criticagent systemsobservabilityCounterfactual Multi‑Agent (COMA) Policy GradientCentralised CriticDecentralised Actor
arXiv: 1705.08926oxwhirl/pymarl
Cover for Counterfactual Multi-Agent Policy Gradients

Abstract

Cooperative multi-agent systems can be naturally used to model many real world problems, such as network packet routing and the coordination of autonomous vehicles. There is a great need for new reinforcement learning methods that can efficiently learn decentralised policies for such systems. To this end, we propose a new multi-agent actor-critic method called counterfactual multi-agent (COMA) policy gradients. COMA uses a centralised critic to estimate the Q-function and decentralised actors to optimise the agents' policies. In addition, to address the challenges of multi-agent credit assignment, it uses a counterfactual baseline that marginalises out a single agent's action, while keeping the other agents' actions fixed. COMA also uses a critic representation that allows the counterfactual baseline to be computed efficiently in a single forward pass. We evaluate COMA in the testbed of StarCraft unit micromanagement, using a decentralised variant with significant partial observability. COMA significantly improves average performance over other multi-agent actor-critic methods in this setting, and the best performing agents are competitive with state-of-the-art centralised controllers that get access to the full state.

Topics

actor criticagent systemsobservabilityreinforcement learning methodself-driving autosingle-agentsoftware agentsagent basedautonomous agents

Keywords

Counterfactual Multi‑Agent (COMA) Policy GradientCentralised CriticDecentralised ActorMulti‑Agent Credit AssignmentStarCraft MicromanagementAdvantage FunctionCounterfactual BaselineIndependent Actor‑CriticTD(λ)Difference Reward

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 Background
  • 4 Methods
  • Independent Actor-Critic
  • Counterfactual Multi-Agent Policy Gradients
  • 5 Experimental Setup
  • 6 Results
  • 7 Conclusions & Future Work
  • References
  • A Proof of Lemma
  • B Training Details and Hyperparameters
  • C Algorithm

Knowls

  1. Knowl 1 — Counterfactual Advantage Function in COMA

    equation

    In Counterfactual Multi-Agent (COMA) policy gradients, credit assignment for each agent a∈{1,…,n}a \in \{1, \dots, n\} is performed by comparing the centralized action-value of the executed joint action u=(ua,u−a)∈Un\mathbf{u} = (u^a, \mathbf{u}^{-a}) \in \mathcal{U}^n to a counterfactual baseline that marginalizes out agent aa's action while holding the actions of all other agents u−a\mathbf{u}^{-a} fixed:

    Aa(s,u)=Q(s,u)−∑u′a∈Uπa(u′a∣τa)Q(s,(u−a,u′a))A^a(s, \mathbf{u}) = Q(s, \mathbf{u}) - \sum_{u'^a \in \mathcal{U}} \pi^a(u'^a \mid \tau^a) Q(s, (\mathbf{u}^{-a}, u'^a))

    where s∈Ss \in \mathcal{S} is the global environment state, τa∈T\tau^a \in \mathcal{T} is the local action-observation history of agent aa, πa(u′a∣τa)\pi^a(u'^a \mid \tau^a) is the stochastic policy of agent aa, and Q(s,u)Q(s, \mathbf{u}) is the state-action value function estimated by a centralized critic. This advantage evaluates the specific contribution of agent aa's action uau^a relative to its expected return under its current policy without requiring extra environment rollouts, default actions, or external reward models.

  2. Knowl 2 — Efficient Centralised Critic Architecture for Counterfactual Evaluation

    model/method

    To evaluate the counterfactual baseline ∑u′aπa(u′a∣τa)Q(s,(u−a,u′a))\sum_{u'^a} \pi^a(u'^a \mid \tau^a) Q(s, (\mathbf{u}^{-a}, u'^a)) without requiring ∣U∣|\mathcal{U}| separate evaluations of the joint action-value network for each agent, COMA uses a specialized critic architecture.

    For agent aa, the input to the centralized critic network consists of the global state ss (or concatenated observations), the local observation oao^a, and the joint action of all other agents u−a\mathbf{u}^{-a}. Rather than taking the joint action u\mathbf{u} and outputting a single scalar, the critic network outputs a vector of ∣U∣|\mathcal{U}| values, where each output element represents Q(s,(u−a,u′a))Q(s, (\mathbf{u}^{-a}, u'^a)) for a specific action u′a∈Uu'^a \in \mathcal{U} available to agent aa.

    This parameterization allows the counterfactual advantage for agent aa to be calculated in a single forward pass of the actor and critic networks. Using a single shared centralized critic, all QQ-values across all nn agents can be computed simultaneously in a single batched forward pass.

  3. Knowl 3 — Zero Expected Policy Gradient Contribution of the Counterfactual Baseline

    theoretical result

    Let b(s,u−a)=∑u′a∈Uπa(u′a∣τa)Q(s,(u−a,u′a))b(s, \mathbf{u}^{-a}) = \sum_{u'^a \in \mathcal{U}} \pi^a(u'^a \mid \tau^a) Q(s, (\mathbf{u}^{-a}, u'^a)) denote the counterfactual baseline for agent aa. Under the state distribution dπ(s)d^\pi(s) induced by the joint policy π(u∣s)=∏aπa(ua∣τa)\pi(\mathbf{u} \mid s) = \prod_a \pi^a(u^a \mid \tau^a), the expected gradient contribution of this baseline across all agents is identically zero:

    gb=−Eπ[∑a∇θlog⁡πa(ua∣τa)b(s,u−a)]=0g_b = -\mathbb{E}_\pi \left[ \sum_a \nabla_\theta \log \pi^a(u^a \mid \tau^a) b(s, \mathbf{u}^{-a}) \right] = 0

    Because the baseline b(s,u−a)b(s, \mathbf{u}^{-a}) does not depend on the specific action uau^a chosen by agent aa, summing πa(ua∣τa)∇θlog⁡πa(ua∣τa)=∇θπa(ua∣τa)\pi^a(u^a \mid \tau^a) \nabla_\theta \log \pi^a(u^a \mid \tau^a) = \nabla_\theta \pi^a(u^a \mid \tau^a) over all ua∈Uu^a \in \mathcal{U} yields ∇θ∑uaπa(ua∣τa)=∇θ(1)=0\nabla_\theta \sum_{u^a} \pi^a(u^a \mid \tau^a) = \nabla_\theta (1) = 0. Consequently, introducing the counterfactual baseline reduces gradient variance without altering the expected policy gradient or introducing recursive self-consistency issues between the policy and value functions.

  4. Knowl 4 — Local Convergence of COMA Policy Gradients

    theoretical result

    For an actor-critic algorithm parameterised by actor parameters θ={θ1,…,θn}\theta = \{\theta^1, \dots, \theta^n\} updating via the COMA policy gradient:

    gk=Eπ[∑a∇θklog⁡πa(ua∣τa)Aa(s,u)]g_k = \mathbb{E}_\pi \left[ \sum_a \nabla_{\theta_k} \log \pi^a(u^a \mid \tau^a) A^a(s, \mathbf{u}) \right]

    with a compatible critic trained using temporal difference learning TD(1)\text{TD}(1), the policy parameters converge to a local maximum of the expected discounted total reward J(θ)=Eπ[R0]J(\theta) = \mathbb{E}_\pi [R_0], satisfying:

    lim inf⁡k→∞∥∇J(θk)∥=0with probability 1\liminf_{k \to \infty} \|\nabla J(\theta_k)\| = 0 \quad \text{with probability 1}

    provided that the policy π\pi is differentiable, the update timescales for the critic QQ and actor π\pi are sufficiently slow (with π\pi updated sufficiently slower than QQ), and QQ uses a representation compatible with π\pi.

  5. Knowl 5 — Centralised Critic Loss with Bootstrapped TD(lambda)

    model/method

    The centralized critic fc(⋅,θc)f^c(\cdot, \theta^c) in COMA is trained on-policy to estimate action values using deep neural network adaptations of TD(λ)\text{TD}(\lambda). The parameters θc\theta^c are updated by minibatch gradient descent to minimize the squared error loss:

    Lt(θc)=(yt(λ)−fc(⋅t,θc))2\mathcal{L}_t(\theta^c) = \left( y^{(\lambda)}_t - f^c(\cdot_t, \theta^c) \right)^2

    where the λ\lambda-return target yt(λ)y^{(\lambda)}_t is defined by:

    yt(λ)=(1−λ)∑n=1∞λn−1Gt(n)y^{(\lambda)}_t = (1 - \lambda) \sum_{n=1}^\infty \lambda^{n-1} G^{(n)}_t

    and the nn-step returns are computed as:

    Gt(n)=∑l=1nγl−1rt+l+γnfc(⋅t+n,θc−)G^{(n)}_t = \sum_{l=1}^n \gamma^{l-1} r_{t+l} + \gamma^n f^c(\cdot_{t+n}, \theta^{c-})

    Here, θc−\theta^{c-} denotes the parameters of a target network copied periodically from θc\theta^c to stabilize training, γ∈[0,1)\gamma \in [0, 1) is the discount factor, and λ=0.8\lambda = 0.8 is the trace decay parameter.

  6. Knowl 6 — Decentralised StarCraft Micromanagement Benchmark Environment

    experimental setup

    The StarCraft unit micromanagement benchmark tests cooperative multi-agent reinforcement learning in combat scenarios against the built-in StarCraft AI. To impose meaningful decentralization and partial observability, agents have a restricted circular field of view equal to the firing range of ranged units.

    Key characteristics include:

    • Partial Observability and Invalid Actions: Units cannot observe entities outside their field of view. Attack-move macro-actions (built-in pathfinding toward targets) are disabled; units can only attack enemies within range. Issuing attack commands on dead or out-of-range units results in no action, enlarging the effective exploration space.
    • Action Space: Discrete actions comprising move[direction], attack[enemy_id], stop, and noop.
    • Combat Scenarios: Symmetric unit setups comprising 3 marines (3m), 5 marines (5m), 5 wraiths (5w), and 2 dragoons with 3 zealots (2d_3z).
    • Reward Signal: Shared global reward given at each step by the total damage inflicted on opponent units minus half the damage taken. Killing an enemy awards 10 points; winning awards the team's remaining health plus 200 points.

  7. Knowl 7 — Decentralised Actor Architecture and Action Selection

    model/method

    In COMA, all agents share parameters in a single decentralized recurrent actor network to accelerate learning while maintaining separate hidden states.

    • Network Architecture: The actor consists of a 128-bit Gated Recurrent Unit (GRU) with fully connected input and output layers. It receives the local action-observation history τa\tau^a, which includes an agent-specific ID and normalized relative coordinates/attributes of visible units within its local field of view.
    • Action Selection: Policy outputs zz are transformed into action selection probabilities via an ϵ\epsilon-bounded softmax distribution:

    P(u)=(1−ϵ) softmax(z)u+ϵ∣U∣P(u) = (1 - \epsilon) \, \text{softmax}(z)_u + \frac{\epsilon}{|\mathcal{U}|}

    where ϵ\epsilon is linearly annealed from 0.50.5 down to 0.020.02 across the first 750750 training episodes.

  8. Knowl 8 — Win Rate Comparison on Decentralised StarCraft Micromanagement

    data/table

    The performance of COMA and competing baseline methods evaluated on StarCraft micromanagement across 1000 evaluation episodes (mean win percentage with 95% confidence intervals in parentheses, along with maximum win percentages):

    Local Field of View (Decentralised) Full FoV, Central Control
    Map heur. IAC-V IAC-Q cnt-V cnt-QV COMA (mean / best) heur. DQN GMEZO
    3m 35 47 (3) 56 (6) 83 (3) 83 (5) 87 (3) / 98 74 - -
    5m 66 63 (2) 58 (3) 67 (5) 71 (9) 81 (5) / 95 98 99 100
    5w 70 18 (5) 57 (5) 65 (3) 76 (1) 82 (3) / 98 82 70 74
    2d_3z 63 27 (9) 19 (21) 36 (6) 39 (5) 47 (5) / 65 68 61 90

    COMA achieves higher mean win rates than all decentralized baselines (independent actor-critics IAC-V and IAC-Q) and centralized critic baselines (central-V and central-QV). Despite being restricted to local fields of view and decentralized execution, the best-performing COMA models achieve win rates competitive with state-of-the-art centralized architectures (DQN and GMEZO) that operate with full field of view and macro-actions.

  9. Knowl 9 — Ablation Analysis of Centralisation and Counterfactual Baselines

    empirical result

    Empirical ablations across StarCraft combat scenarios demonstrate two key design conclusions:

    1. Value of Critic Centralisation: Centralized critics (cnt-V, cnt-QV, and COMA) consistently outperform decentralized independent actor-critics (IAC-V and IAC-Q). Even though IAC shares early network layers between actor and critic, the improved accuracy of policy evaluation enabled by conditioning on global state information outweighs the parameter-sharing benefit.
    2. Value of Counterfactual Marginalisation: COMA strictly dominates cnt-QV (which estimates advantage as Q(s,u)−V(s)Q(s, \mathbf{u}) - V(s) using simultaneously trained QQ and VV networks) in both sample efficiency and final win rate. This confirms that deducting an agent-marginalized counterfactual baseline Q(s,u−a,⋅)Q(s, \mathbf{u}^{-a}, \cdot) provides superior multi-agent credit assignment compared to subtracting a state-value baseline V(s)V(s).

  10. Knowl 10 — Scalability and Exploration Bottlenecks in COMA

    limitation

    The primary bottleneck in scaling COMA to settings with large numbers of agents is multi-agent exploration rather than critic centralization. As the number of agents grows, uncoordinated exploration by teammate agents generates noisy learning signals for the centralized critic.

    Additionally, while COMA computes counterfactual baselines analytically for discrete action spaces in a single forward pass, continuous action spaces require estimating the expectation ∑u′aπa(u′a∣τa)Q(s,(u−a,u′a))\sum_{u'^a} \pi^a(u'^a \mid \tau^a) Q(s, (\mathbf{u}^{-a}, u'^a)) via Monte Carlo sampling or restricting the policy and critic to functional families (such as Gaussian distributions) that allow analytical integration.

Coverage note — None was omitted; all contributed theory, methods, architecture details, experimental benchmarks, tabular performance comparisons, ablations, and stated limitations are fully covered.

References

  1. 1.Busoniu, L.; Babuska, R.; and De Schutter, B. 2008. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems Man and Cybernetics Part C Applications and Reviews 38(2):156.
  2. 2.Cao, Y.; Yu, W.; Ren, W.; and Chen, G. 2013. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Transactions on Industrial informatics 9(1):427–438.
  3. 3.Chang, Y.-H.; Ho, T.; and Kaelbling, L. P. 2003. All learning is local: Multi-agent learning in global reward games. In NIPS, 807–814.
  4. 4.Cho, K.; van Merriënboer, B.; Bahdanau, D.; and Bengio, Y. 2014. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259.
  5. 5.Colby, M. K.; Curran, W.; and Tumer, K. 2015. Approximating difference evaluations with local information. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, 1659–1660. International Foundation for Autonomous Agents and Multiagent Systems.
  6. 6.Collobert, R.; Kavukcuoglu, K.; and Farabet, C. 2011. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop.
  7. 7.Das, A.; Kottur, S.; Moura, J. M.; Lee, S.; and Batra, D. 2017. Learning cooperative visual dialog agents with deep reinforcement learning. arXiv preprint arXiv:1703.06585.
  8. 8.Foerster, J.; Assael, Y. M.; de Freitas, N.; and Whiteson, S. 2016. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, 2137–2145.
  9. 9.Foerster, J.; Nardelli, N.; Farquhar, G.; Torr, P.; Kohli, P.; Whiteson, S.; et al. 2017. Stabilising experience replay for deep multi-agent reinforcement learning. In Proceedings of The 34th International Conference on Machine Learning.
  10. 10.Gupta, J. K.; Egorov, M.; and Kochenderfer, M. 2017. Cooperative multi-agent control using deep reinforcement learning.
  11. 11.Hausknecht, M., and Stone, P. 2015. Deep recurrent q-learning for partially observable mdps. arXiv preprint arXiv:1507.06527.
  12. 12.Hochreiter, S., and Schmidhuber, J. 1997. Long short-term memory. Neural computation 9(8):1735–1780.
  13. 13.Jorge, E.; Kågebäck, M.; and Gustavsson, E. 2016. Learning to play guess who? and inventing a grounded language as a consequence. arXiv preprint arXiv:1611.03218.
  14. 14.Konda, V. R., and Tsitsiklis, J. N. 2000. Actor-critic algorithms. In Advances in neural information processing systems, 1008–1014.
  15. 15.Kraemer, L., and Banerjee, B. 2016. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing 190:82–94.
  16. 16.Lazaridou, A.; Peysakhovich, A.; and Baroni, M. 2016. Multi-agent cooperation and the emergence of (natural) language. arXiv preprint arXiv:1612.07182.
  17. 17.Leibo, J. Z.; Zambaldi, V.; Lanctot, M.; Marecki, J.; and Graepel, T. 2017. Multi-agent reinforcement learning in sequential social dilemmas. arXiv preprint arXiv:1702.03037.
  18. 18.Lowe, R.; Wu, Y.; Tamar, A.; Harb, J.; Abbeel, P.; and Mordatch, I. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. arXiv preprint arXiv:1706.02275.
  19. 19.Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; et al. 2015. Human-level control through deep reinforcement learning. Nature 518(7540):529–533.
  20. 20.Mordatch, I., and Abbeel, P. 2017. Emergence of grounded compositional language in multi-agent populations. arXiv preprint arXiv:1703.04908.
  21. 21.Oliehoek, F. A.; Spaan, M. T. J.; and Vlassis, N. 2008. Optimal and approximate Q-value functions for decentralized POMDPs. 32:289–353.
  22. 22.Omidshafiei, S.; Pazis, J.; Amato, C.; How, J. P.; and Vian, J. 2017. Deep decentralized multi-task multi-agent rl under partial observability. arXiv preprint arXiv:1703.06182.
  23. 23.Peng, P.; Yuan, Q.; Wen, Y.; Yang, Y.; Tang, Z.; Long, H.; and Wang, J. 2017. Multiagent bidirectionally-coordinated nets for learning to play starcraft combat games. arXiv preprint arXiv:1703.10069.
  24. 24.Proper, S., and Tumer, K. 2012. Modeling difference rewards for multiagent learning. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems-Volume 3, 1397–1398. International Foundation for Autonomous Agents and Multiagent Systems.
  25. 25.Schulman, J.; Moritz, P.; Levine, S.; Jordan, M. I.; and Abbeel, P. 2015. High-dimensional continuous control using generalized advantage estimation. CoRR abs/1506.02438.
  26. 26.Shoham, Y., and Leyton-Brown, K. 2009. Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations. New York: Cambridge University Press.
  27. 27.Sukhbaatar, S.; Fergus, R.; et al. 2016. Learning multiagent communication with backpropagation. In Advances in Neural Information Processing Systems, 2244–2252.
  28. 28.Sutton, R. S.; McAllester, D. A.; Singh, S. P.; Mansour, Y.; et al. 1999. Policy gradient methods for reinforcement learning with function approximation. In NIPS, volume 99, 1057–1063.
  29. 29.Sutton, R. S. 1988. Learning to predict by the methods of temporal differences. Machine learning 3(1):9–44.
  30. 30.Synnaeve, G.; Nardelli, N.; Auvolat, A.; Chintala, S.; Lacroix, T.; Lin, Z.; Richoux, F.; and Usunier, N. 2016. Torchcraft: a library for machine learning research on real-time strategy games. arXiv preprint arXiv:1611.00625.
  31. 31.Tampuu, A.; Matiisen, T.; Kodelja, D.; Kuzovkin, I.; Korjus, K.; Aru, J.; Aru, J.; and Vicente, R. 2015. Multiagent cooperation and competition with deep reinforcement learning. arXiv preprint arXiv:1511.08779.
  32. 32.Tan, M. 1993. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, 330–337.
  33. 33.Tumer, K., and Agogino, A. 2007. Distributed agent-based air traffic flow management. In Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems, 255. ACM.
  34. 34.Usunier, N.; Synnaeve, G.; Lin, Z.; and Chintala, S. 2016. Episodic exploration for deep deterministic policies: An application to starcraft micromanagement tasks. arXiv preprint arXiv:1609.02993.
  35. 35.Weaver, L., and Tao, N. 2001. The optimal reward baseline for gradient-based reinforcement learning. In Proceedings of the Seventeenth conference on Uncertainty in artificial intelligence, 538–545. Morgan Kaufmann Publishers Inc.
  36. 36.Weyns, D.; Helleboogh, A.; and Holvoet, T. 2005. The packet-world: A test bed for investigating situated multiagent systems. In Software Agent-Based Applications, Platforms and Development Kits. Springer. 383–408.
  37. 37.Williams, R. J. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8(3-4):229–256.
  38. 38.Wolpert, D. H., and Tumer, K. 2002. Optimal payoff functions for members of collectives. In Modeling complexity in economic and social systems. World Scientific. 355–369.
  39. 39.Yang, E., and Gu, D. 2004. Multiagent reinforcement learning for multi-robot systems: A survey. Technical report, tech. rep.
  40. 40.Ye, D.; Zhang, M.; and Yang, Y. 2015. A multi-agent framework for packet routing in wireless sensor networks. sensors 15(5):10026–10047.
  41. 41.Ying, W., and Dayong, S. 2005. Multi-agent framework for third party logistics in e-commerce. Expert Systems with Applications 29(2):431–436.

Citation

MLA
Foerster, J., et al. “Counterfactual Multi-Agent Policy Gradients”. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, 2018, https://doi.org/10.1609/aaai.v32i1.11794.
APA
Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., & Whiteson, S. (2018). Counterfactual Multi-Agent Policy Gradients. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). https://doi.org/10.1609/aaai.v32i1.11794
Chicago
Foerster, J., G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. 2018. “Counterfactual Multi-Agent Policy Gradients”. Proceedings of the AAAI Conference on Artificial Intelligence 32 (1). https://doi.org/10.1609/aaai.v32i1.11794.
Harvard
Foerster, J. et al. (2018) “Counterfactual Multi-Agent Policy Gradients”, Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). Available at: https://doi.org/10.1609/aaai.v32i1.11794.
Vancouver
1. Foerster J, Farquhar G, Afouras T, Nardelli N, Whiteson S (2018) Counterfactual Multi-Agent Policy Gradients. Proceedings of the AAAI Conference on Artificial Intelligence. https://doi.org/10.1609/aaai.v32i1.11794

BibTeX

@article{Foerster_2018, title={Counterfactual Multi-Agent Policy Gradients}, volume={32}, ISSN={2159-5399}, url={http://dx.doi.org/10.1609/aaai.v32i1.11794}, DOI={10.1609/aaai.v32i1.11794}, number={1}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, publisher={Association for the Advancement of Artificial Intelligence (AAAI)}, author={Foerster, Jakob and Farquhar, Gregory and Afouras, Triantafyllos and Nardelli, Nantas and Whiteson, Shimon}, year={2018}, month=Apr }
Metadata:Crossref

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF