QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

Tabish RashidMikayel SamvelyanChristian Schroeder de WittGregory FarquharJakob FoersterShimon Whiteson

article2018ICML2,149 citations

Introduces QMIX, a multi-agent reinforcement learning algorithm that enforces monotonic value function factorisation to guarantee policy consistency between centralised training and decentralised execution on complex cooperative tasks like StarCraft II.

Listen

Coordinating teams of autonomous agents—such as autonomous vehicles or robotic swarms—requires systems that can make fast, independent decisions while pursuing a shared goal. While agents frequently have access to comprehensive global data in simulated training environments, real-world deployment is restricted by limited local observations and constrained communication. The core technical challenge is effectively leveraging full global data during centralized training while guaranteeing that individual agents can act independently and coherently during live execution.

This article evaluates QMIX, a novel reinforcement learning framework designed to train decentralized policies using a centralized, non-linear combination of individual agent values. The objective is to demonstrate that structurally enforcing a monotonic relationship between per-agent utility functions and the overall joint value function produces superior coordination and performance over existing multi-agent methods.

To test this approach, the researchers evaluated QMIX across a cooperative matrix game and a suite of complex multi-agent combat scenarios within the StarCraft II Learning Environment. The benchmark setups ranged from small homogeneous teams (e.g., 3 to 8 identical units) to larger, heterogeneous teams with diverse capabilities. The architecture was rigorously tested against standard baselines—independent Q-learning (IQL) and Value-Decomposition Networks (VDN)—alongside architectural ablations that systematically isolated the impact of non-linear mixing and centralized state conditioning.

The evaluation yielded several key operational findings. First, QMIX consistently achieved the highest win rates across all benchmark scenarios, notably outperforming existing value-based methods in learning speed and final performance. Second, the advantage of QMIX was most pronounced in complex heterogeneous team scenarios, such as the 3 Stalkers and 5 Zealots task, where prior factored methods failed to match even simple baseline heuristics. Third, qualitative policy analysis revealed that QMIX successfully learned sophisticated collaborative behaviors, such as defensive shielding and coordinated focus firing, whereas baseline approaches degraded into suboptimal or unstable movement patterns. Finally, ablation experiments confirmed that both non-linear factorisation and integration of global state features via hypernetworks are critical to achieving high performance in complex environments.

These findings indicate that teams of autonomous systems can achieve complex tactical coordination without requiring inter-agent communication during deployment, significantly reducing operational bandwidth demands and hardware vulnerability risks. Organizations deploying cooperative automated systems can achieve higher task success rates and greater policy stability by utilizing centralized off-policy training while maintaining scalable, decentralized execution.

Decision-makers considering deployment should note that QMIX relies on a monotonicity constraint, meaning it cannot represent scenarios where an agent's optimal immediate action strictly depends on the simultaneous actions of another agent in a non-monotonic manner. Future development should explore advanced exploration strategies and validate the method on larger swarms with wider unit diversity. Overall, there is high confidence that QMIX represents a robust and scalable architecture for cooperative multi-agent coordination under partial observability.

Russian-Armenian University University of Oxford communication constraints multi-agent multi-agent reinforcement learning Learning QMIX Value

arXiv: 1803.11485 oxwhirl/pymarl

Paper: Counterfactual Multi-Agent Policy Gradients, Jakob N. Foerster et al. (2017). Introduces the centralized training with decentralized execution framework and StarCraft micromanagement benchmark that motivate and contextualize QMIX's value-based approach.
Paper: Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments, Ryan Lowe et al. (2017). Establishes the standard paradigm of centralized training for decentralized execution in multi-agent reinforcement learning, providing essential context for QMIX's factorized value learning.
Paper: Dueling Network Architectures for Deep Reinforcement Learning, Ziyu Wang et al. (2016). Presents value network architecture innovations that underpin deep value function estimation used in modern multi-agent deep Q-learning methods.
Paper: Deep Reinforcement Learning with Double Q-learning, Hado van Hasselt et al. (2016). Addresses maximization bias in deep Q-learning, providing fundamental stability techniques utilized when training multi-agent deep action-value networks.
Paper: Human-level control through deep reinforcement learning, Volodymyr Mnih et al. (2015). Provides the foundational Deep Q-Network algorithm and experience replay framework that QMIX extends to cooperative multi-agent settings.
Paper: Markov Games as a Framework for Multi-Agent Reinforcement Learning, M. Littman (1994). Introduces Markov games as the standard theoretical formulation for multi-agent reinforcement learning problems that QMIX solves.
Paper: Technical Note: Q-Learning, CHRISTOPHER J.C.H. WATKINS et al. (2004). Supplies the core mathematical principles and convergence foundations of Q-learning underlying value-based reinforcement learning algorithms.

Paper: The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games, Chao Yu et al. (2022). Re-evaluates cooperative multi-agent benchmarks by directly comparing on-policy policy gradient methods against standard off-policy factorized value baselines like QMIX.

Cover for QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

Abstract

In many real-world settings, a team of agents must coordinate their behaviour while acting in a decentralised way. At the same time, it is often possible to train the agents in a centralised fashion in a simulated or laboratory setting, where global state information is available and communication constraints are lifted. Learning joint action-values conditioned on extra state information is an attractive way to exploit centralised learning, but the best strategy for then extracting decentralised policies is unclear. Our solution is QMIX, a novel value-based method that can train decentralised policies in a centralised end-to-end fashion. QMIX employs a network that estimates joint action-values as a complex non-linear combination of per-agent values that condition only on local observations. We structurally enforce that the joint-action value is monotonic in the per-agent values, which allows tractable maximisation of the joint action-value in off-policy learning, and guarantees consistency between the centralised and decentralised policies. We evaluate QMIX on a challenging set of StarCraft II micromanagement tasks, and show that QMIX significantly outperforms existing value-based multi-agent reinforcement learning methods.

Topics

communication constraints multi-agent multi-agent reinforcement learning reinforcement learning method software agents value functions factorization agent based

Keywords

Learning QMIX Value Value Function Agent Reinforcement Learning Multi-Agent Reinforcement Learning Monotonic Value Function Deep Multi-Agent Reinforcement Value Function Factorisation

1 Introduction
2 Related Work
3 Background
3.1 Deep QQ-Learning
3.2 Deep Recurrent QQ-Learning
3.3 Independent QQ-Learning
3.4 Value Decomposition Networks
4 QMIX
4.1 Representational Complexity
5 Two-Step Game
6 Experimental Setup
6.1 Decentralised StarCraft II Micromanagement
6.2 Ablations
7 Results
7.1 Main Results
7.2 Ablation Results
7.3 Learned Policies
8 Conclusion
References
A QMIX
A.1 Representational Complexity
B Two Step Game
B.1 Architecture and Training
B.2 Learned Value Functions
B.3 Results
C StarCraft II Setup
C.1 Environment Features
C.2 Architecture and Training
D StarCraft II Results

Knowls

Knowl 1 — Monotonicity Condition for Factorised Joint Action-Values
theoretical result
In a cooperative multi-agent task modeled as a Decentralised Partially Observable Markov Decision Process (Dec-POMDP) with $n$ agents $\mathcal{A} = \{1, \dots, n\}$ , joint action $\mathbf{u} = (u^1, \dots, u^n) \in \mathcal{U}^n$ , and joint action-observation history $\boldsymbol{\tau} = (\tau^1, \dots, \tau^n)$ , a centralised action-value function $Q_{tot}(\boldsymbol{\tau}, \mathbf{u})$ can be decentralised into individual agent utilities $Q_a(\tau^a, u^a)$ such that centralised and decentralised greedy action choices coincide if:

$\operatorname{argmax}_{\mathbf{u}} Q_{tot}(\boldsymbol{\tau}, \mathbf{u}) = \begin{pmatrix} \operatorname{argmax}_{u^1} Q_1(\tau^1, u^1) \\ \vdots \\ \operatorname{argmax}_{u^n} Q_n(\tau^n, u^n) \end{pmatrix}$

A sufficient condition to guarantee this consistency is to enforce a global monotonicity constraint on the relationship between the joint action-value $Q_{tot}$ and each individual agent value function $Q_a$ :

$\frac{\partial Q_{tot}}{\partial Q_a} \ge 0, \quad \forall a \in \mathcal{A}$

Enforcing this constraint enables each agent $a$ to select actions greedily with respect to its own local utility $Q_a(\tau^a, u^a)$ during decentralised execution, while allowing the joint action maximisation $\operatorname{argmax}_{\mathbf{u}} Q_{tot}(\boldsymbol{\tau}, \mathbf{u})$ required during off-policy centralised training to be computed in $\mathcal{O}(n)$ time rather than scaling exponentially $\mathcal{O}(|\mathcal{U}|^n)$ in the number of agents.
Knowl 2 — QMIX Neural Architecture and Hypernetwork Design
model/method
The QMIX architecture consists of three core components designed to enforce monotonicity while conditioning joint value estimates on the global environment state $s$ :
1. Agent Networks: Each agent $a \in \{1, \dots, n\}$ has a Deep Recurrent Q-Network (DRQN) that receives the current local observation $o_t^a$ and previous action $u_{t-1}^a$ as input. The network comprises a fully connected layer, a 64-unit Gated Recurrent Unit (GRU) cell to maintain an internal representation of the action-observation history $\tau^a$ , and a fully connected linear layer outputting individual utilities $Q_a(\tau^a, u^a)$ for each discrete action $u^a \in \mathcal{U}$ . Agent network parameters are shared across all agents, with a one-hot encoding of the agent ID concatenated to the input.
2. Mixing Network: A feed-forward neural network with one hidden layer of 32 units and an Exponential Linear Unit (ELU) non-linearity. It takes the vector of chosen per-agent values $(Q_1(\tau^1, u^1), \dots, Q_n(\tau^n, u^n))$ as input and outputs the scalar joint action-value $Q_{tot}(\boldsymbol{\tau}, \mathbf{u}, s)$ . To guarantee the monotonicity condition $\frac{\partial Q_{tot}}{\partial Q_a} \ge 0$ , all mixing network layer weights are constrained to be non-negative.
3. State-Conditioned Hypernetworks: Neural networks that take the full global state $s$ as input and generate the weights and biases of the mixing network. For each layer of the mixing network, a hypernetwork consisting of a single linear layer followed by an absolute value activation function produces positive weight matrices. Mixing network biases are generated by separate linear hypernetworks without sign constraints, except for the final layer bias, which is produced by a 2-layer hypernetwork with 32 hidden units and a ReLU activation. This design allows the joint value $Q_{tot}$ to depend arbitrarily and non-monotonically on the global state $s$ while maintaining strictly monotonic mixing with respect to individual agent utilities $Q_a$ .
Knowl 3 — QMIX End-to-End Training Algorithm
algorithm
QMIX optimizes the centralised joint action-value function by minimizing the temporal-difference (TD) squared loss across batches of sampled episodic trajectories:
Initialize replay buffer D to capacity 5000 episodes
Initialize agent networks Q_a and mixing network parameters theta
Initialize target network parameters theta^- = theta
for each training episode do
    Initialize environment state s_0, observations o_0^a for all agents a in {1, ..., n}
    for step t = 0, 1, 2, ... until episode termination do
        for each agent a in {1, ..., n} do
            With probability epsilon select random action u_t^a in U
            Otherwise select u_t^a = argmax_{u^a} Q_a(tau_t^a, u^a; theta)
        end for
        Execute joint action u_t = (u_t^1, ..., u_t^n), observe reward r_t, next state s_{t+1}, and observations o_{t+1}
        Store transition in current episode trajectory
    end for
    Store complete episode trajectory in replay buffer D
    Anneal epsilon linearly from 1.0 to 0.05 over 50,000 steps
    if |D| >= batch_size (32 episodes) then
        Sample a batch of b = 32 unrolled episodes uniformly from D
        Compute target values for each transition in the batch:
            y^{tot} = r_t + gamma * Q_{tot}(tau_{t+1}, u*, s_{t+1}; theta^-)
            where u* = (argmax_{u^1} Q_1(tau_{t+1}^1, u^1; theta^-), ..., argmax_{u^n} Q_n(tau_{t+1}^n, u^n; theta^-))
        Compute loss:
            L(theta) = (1 / b) * sum_{i=1}^b (y_i^{tot} - Q_{tot}(tau_{t,i}, u_{t,i}, s_{t,i}; theta))^2
        Update theta using RMSprop optimizer with learning rate 5e-4, alpha = 0.99, no momentum
    end if
    if episode_count mod 200 == 0 then
        Update target parameters theta^- = theta
    end if
end for
Knowl 4 — Representational Limits and Universal Approximation of Monotonic Mixing
limitation
In fully observable environments, QMIX can represent any joint action-value function $Q_{tot}$ that factorises into a non-linear monotonic combination of individual agent value functions, because neural networks with non-negative weights and arbitrary state-dependent hypernetworks act as universal approximators for the class of continuous monotonic functions.

However, QMIX has strict representational limitations:
1. Non-Monotonic Value Functions: QMIX cannot represent joint value functions where an agent's optimal action depends non-monotonically on the simultaneous actions of other agents. For example, in a two-agent matrix game with payoffs $\begin{pmatrix} 2 & 1 \\ 1 & 8 \end{pmatrix}$ , Agent 1's best action is $A$ if Agent 2 picks $A$ (payoff 2 vs 1), but $B$ if Agent 2 picks $B$ (payoff 8 vs 1). Because the action ordering changes conditionally on the other agent's choice, such games violate the monotonicity constraint $\frac{\partial Q_{tot}}{\partial Q_a} \ge 0$ and cannot be correctly factorised by QMIX.
2. Decentralised Partial Observability: In a Dec-POMDP, an agent's history $\tau^a$ is a partial observation of the true global state $s$ . If partial observability causes an agent's local utility ranking $Q_a(\tau^a, u) > Q_a(\tau^a, u')$ to contradict the true global ordering $Q_{tot}(s, (\mathbf{u}^{-a}, u)) < Q_{tot}(s, (\mathbf{u}^{-a}, u'))$ , the monotonic mixing network cannot recover the true joint action-value function.

Knowl 5 — Two-Step Cooperative Matrix Game Comparison

empirical result

To evaluate representational capacity under full exploration ( $\epsilon = 1.0$ , discount factor $\gamma = 0.99$ ), methods are tested on a two-step matrix game for two agents. At step 1, Agent 1 chooses Action $A$ (transitioning to State 2A) or Action $B$ (transitioning to State 2B), while Agent 2's action has no effect. At step 2, both agents choose actions from $\{A, B\}$ and receive payoff $R$ :

State 2A Payoff Matrix: $\begin{pmatrix} 7 & 7 \\ 7 & 7 \end{pmatrix}$
State 2B Payoff Matrix: $\begin{pmatrix} 0 & 1 \\ 1 & 8 \end{pmatrix}$

State 2B represents a monotonic payoff requiring non-linear representation, where the optimal joint return is 8 via joint action $(B, B)$ , whereas State 2A offers a uniform payoff of 7.

VDN, VDN-S, and QMIX-Lin cannot represent the non-linear monotonic matrix of State 2B (learning joint values of $\approx 6.51$ for State 2B $(B,B)$ vs $\approx 7.02$ for State 2A), causing them to pick State 1 Action $A$ and obtain a suboptimal reward of 7. QMIX and QMIX-NS correctly learn the value of 8.00 for State 2B $(B,B)$ and $\approx 7.92$ for State 1 Action $B$ , successfully recovering the optimal policy.

Knowl 6 — StarCraft II Multi-Agent Micromanagement Experimental Setup
experimental setup
Decentralised combat micromanagement scenarios are constructed in the StarCraft II Learning Environment (SC2LE) with symmetrical allied and enemy armies. Allied units are controlled by individual decentralized RL agents; enemy units are controlled by the built-in medium heuristic AI.
- Action Space: Discrete set per agent containing move[north], move[south], move[east], move[west], attack[enemy_id] (valid only if enemy is within shooting range of 6), stop, and noop. Built-in automated retaliatory fire and auto-targeting when idle are explicitly disabled.
- Local Observations: Restricted to a circular sight range of radius 9. Contains feature vectors for all visible units (allied and enemy): distance, relative $x$ , relative $y$ , and unit type.
- Global State: Map-relative unit coordinates, health points (HP), shield points, weapon cooldown, and previous actions of all allied agents.
- Unit Statistics: Marines (45 HP), Stalkers (80 HP, 50 Shields), Zealots (100 HP, 80 Shields), Colossi (200 HP, 150 Shields).
- Reward Function: Scaled sum of total damage dealt to enemy units, $+10$ points for killing an enemy unit, and $+200$ points for eliminating the enemy army, normalized so that the maximum cumulative episode reward is 20.
- Evaluation Scenarios & Time Limits: Homogeneous maps: 3 Marines (3m, 60 steps), 5 Marines (5m, 60 steps), 8 Marines (8m, 120 steps). Heterogeneous maps: 2 Stalkers and 3 Zealots (2s_3z, 120 steps), 3 Stalkers and 5 Zealots (3s_5z, 150 steps), 1 Colossus, 3 Stalkers, and 5 Zealots (1c_3s_5z, 200 steps).
Knowl 7 — Performance of QMIX on StarCraft II Micromanagement Scenarios
empirical result
Across six combat maps in StarCraft II (evaluated over 20 independent training runs with greedy evaluation win rates measured every 100 episodes over 20 test episodes), QMIX consistently outperforms Independent Q-Learning (IQL) and Value-Decomposition Networks (VDN):
- Homogeneous Scenarios (3m, 5m, 8m):
  - On 3m, QMIX reaches a test win rate of $\approx 90\%$ , outperforming VDN ( $\approx 70\%$ ) and a heuristic closest-target baseline ( $76\%$ ).
  - On 5m, QMIX achieves $>95\%$ win rate, exceeding VDN ( $\approx 75\%$ ) and the heuristic baseline ( $60\%$ ).
  - On 8m, both QMIX and VDN converge to $\approx 95\%$ win rate, matching the heuristic baseline ( $95\%$ ).
  - IQL fails across all maps, exhibiting unstable training and win rates below $50\%$ on 3m and 8m due to environment non-stationarity.
- Heterogeneous Scenarios (2s_3z, 3s_5z, 1c_3s_5z):
  - On 2s_3z, QMIX reaches $\approx 95\%$ win rate, while VDN plateaus at $\approx 75\%$ and the heuristic baseline achieves $82\%$ .
  - On 3s_5z, QMIX reaches $\approx 75\%$ win rate, whereas VDN fails to exceed $\approx 20\%$ (heuristic baseline: $45\%$ ).
  - On 1c_3s_5z, QMIX reaches $\approx 70\%$ win rate, whereas VDN remains at $\approx 25\%$ (heuristic baseline: $70\%$ ).
Knowl 8 — Ablation Analysis of Non-Linear Mixing and State Conditioning
empirical result
To isolate the contributions of hypernetwork-driven state conditioning and non-linear mixing, three ablations of QMIX were evaluated on StarCraft II micromanagement:
1. QMIX-NS (Mixing network with absolute value weights learned directly without hypernetwork state conditioning): On homogeneous maps (3m), QMIX-NS performs comparably to QMIX. However, on heterogeneous maps (3s_5z and 1c_3s_5z), QMIX-NS collapses to a win rate of $\approx 30\%$ , performing on par with VDN.
2. QMIX-Lin (Mixing network without a hidden layer, performing a state-dependent weighted linear sum of $Q_a$ generated via hypernetworks): QMIX-Lin achieves $\approx 50\%$ win rate on 3s_5z and 1c_3s_5z, outperforming linear VDN but falling substantially short of full non-linear QMIX ( $\approx 70\text{--}75\%$ ).
3. VDN-S (Standard additive VDN augmented with a state-value baseline network $V(s)$ implemented with a 32-unit ReLU hidden layer): VDN-S shows negligible improvement over standard VDN on heterogeneous maps (2s_3z, 3s_5z, 1c_3s_5z), confirming that an additive state baseline cannot compensate for linear factorisation limits.
These comparisons demonstrate that while linear factorisation suffices for simple homogeneous combat, high performance in complex heterogeneous multi-agent settings requires both non-linear mixing of agent utilities and state-conditioned mixing weights.
Knowl 9 — Tactical Coordination Strategies Discovered by QMIX
empirical result
Qualitative inspection of learned policies in StarCraft II demonstrates distinct combat tactics between factorisation methods:
- Flanking and Focus Firing in Homogeneous Combat (8m): Both QMIX and VDN learn to spread Marines into a semicircle concave formation before firing, allowing agents to attack incoming enemies simultaneously from multiple angles rather than engaging head-on.
- Role-Differentiated Defense in Heterogeneous Combat (2s_3z and 3s_5z): In StarCraft II mechanics, melee Zealots counter ranged Stalkers. VDN and VDN-S fail to learn unit positioning, rushing into weapon range without protecting vulnerable units or matching unit types. QMIX discovers a specialized defensive formation: allied Zealots advance forward to block and engage enemy Zealots while allied Stalkers kite backwards to maintain maximum firing distance, only focusing down enemy Stalkers once enemy Zealots are eliminated.
- Exploration Dynamics: Early in training, both VDN and IQL exhibit a performance peak by executing immediate attacks (matching the heuristic baseline), followed by a performance drop as exploration causes agents to move rather than fire. VDN successfully recovers by learning coordinated move-and-fire maneuvers, whereas IQL fails to recover due to non-stationarity across simultaneously learning agents.

Coverage note — None was omitted. All primary contributions—the monotonicity condition, QMIX hypernetwork and mixing architecture, training objective, representational capacity analysis, two-step matrix game, StarCraft II benchmark results, ablations, and policy behavioral analysis—are fully covered.

References

1.Busoniu, L., Babuska, R., and De Schutter, B. A Comprehensive Survey of Multiagent Reinforcement Learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(2):156–172, 2008.
2.Cao, Y., Yu, W., Ren, W., and Chen, G. An Overview of Recent Progress in the Study of Distributed Multiagent Coordination. IEEE Transactions on Industrial Informatics, 9(1):427–438, 2012.
3.Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS 2014 Workshop on Deep Learning, 2014.
4.Dugas, C., Bengio, Y., Blisle, F., Nadeau, C., and Garcia, R. Incorporating functional knowledge in neural networks. Journal of Machine Learning Research, 10:1239–1262, 2009.
5.Foerster, J., Nardelli, N., Farquhar, G., Afouras, T., Torr, P. H. S., Kohli, P., and Whiteson, S. Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning. In Proceedings of The 34th International Conference on Machine Learning, pp. 1146–1155, 2017.
6.Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. Counterfactual multi-agent policy gradients. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
7.Guestrin, C., Koller, D., and Parr, R. Multiagent Planning with Factored MDPs. In Advances in Neural Information Processing Systems, pp. 1523–1530. MIT Press, 2002.
8.Gupta, J. K., Egorov, M., and Kochenderfer, M. Cooperative Multi-agent Control Using Deep Reinforcement Learning. In Autonomous Agents and Multiagent Systems, pp. 66–83. Springer, 2017.
9.Ha, D., Dai, A., and Le, Q. V. HyperNetworks. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
10.Hausknecht, M. and Stone, P. Deep Recurrent Q-Learning for Partially Observable MDPs. In AAAI Fall Symposium on Sequential Decision Making for Intelligent Agents, 2015.
11.Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
12.Hüttenrauch, M., Šošić, A., and Neumann, G. Guided Deep Reinforcement Learning for Swarm Systems. In AAMAS 2017 Autonomous Robots and Multirobot Systems (ARMS) Workshop, 2017.
13.Jorge, E., Kågebäck, M., and Gustavsson, E. Learning to play guess who? and inventing a grounded language as a consequence. In NIPS 2016 Workshop on Deep Reinforcement Learning, 2016.
14.Kok, J. R. and Vlassis, N. Collaborative Multiagent Reinforcement Learning by Payoff Propagation. Journal of Machine Learning Research, 7:1789–1828, 2006.
15.Kraemer, L. and Banerjee, B. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190:82–94, 2016.
16.Leibo, J. Z., Zambaldi, V., Lanctot, M., Marecki, J., and Graepel, T. Multi-agent reinforcement learning in sequential social dilemmas. In Proceedings of the 16th Conference on Autonomous Agents and Multiagent Systems, pp. 464–473, 2017.
17.Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, O. P., and Mordatch, I. Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems, pp. 6382–6393, 2017.
18.Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., and others. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
19.Oliehoek, F. A. and Amato, C. A Concise Introduction to Decentralized POMDPs. SpringerBriefs in Intelligent Systems. Springer, 2016.
20.Oliehoek, F. A., Spaan, M. T. J., and Vlassis, N. Optimal and Approximate Q-value Functions for Decentralized POMDPs. Journal of Artificial Intelligence Research, 32(1):289–353, 2008.
21.Omidshafiei, S., Pazis, J., Amato, C., How, J. P., and Vian, J. Deep Decentralized Multi-task Multi-Agent RL under Partial Observability. In Proceedings of the 34th International Conference on Machine Learning, pp. 2681–2690, 2017.
22.Peng, P., Wen, Y., Yang, Y., Yuan, Q., Tang, Z., Long, H., and Wang, J. Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games. arXiv preprint arXiv:1703.10069, 2017.
23.Pinkus, A. Approximation theory of the mlp model in neural networks. Acta numerica, 8:143–195, 1999.
24.Sukhbaatar, S., Fergus, R., and others. Learning multiagent communication with backpropagation. In Advances in Neural Information Processing Systems, pp. 2244–2252, 2016.
25.Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., and Graepel, T. Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward. In Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems, 2017.
26.Synnaeve, G., Nardelli, N., Auvolat, A., Chintala, S., Lacroix, T., Lin, Z., Richoux, F., and Usunier, N. TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games. arXiv preprint arXiv:1611.00625, 2016.
27.Tampuu, A., Matiisen, T., Kodelja, D., Kuzovkin, I., Korjus, K., Aru, J., Aru, J., and Vicente, R. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 2017.
28.Tan, M. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the Tenth International Conference on Machine Learning, pp. 330–337, 1993.
29.Usunier, N., Synnaeve, G., Lin, Z., and Chintala, S. Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
30.Vinyals, O., Ewalds, T., Bartunov, S., Georgiev, P., Vezhnevets, A. S., Yeo, M., Makhzani, A., Küttler, H., Agapiou, J., Schrittwieser, J., Quan, J., Gaffney, S., Petersen, S., Simonyan, K., Schaul, T., van Hasselt, H., Silver, D., Lillicrap, T., Calderone, K., Keet, P., Brunasso, A., Lawrence, D., Ekermo, A., Repp, J., and Tsing, R. StarCraft II: A New Challenge for Reinforcement Learning. arXiv preprint arXiv:1708.04782, 2017.
31.Watkins, C. Learning from delayed rewards. PhD thesis, University of Cambridge England, 1989.
32.Yang, E. and Gu, D. Multiagent reinforcement learning for multi-robot systems: A survey. Technical report, 2004.

Citation

MLA

Rashid, T., et al. “QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning”. arXiv, 2018, http://arxiv.org/abs/1803.11485v2.

APA

Rashid, T., Samvelyan, M., Witt, C. S. de ., Farquhar, G., Foerster, J., & Whiteson, S. (2018). QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. arXiv. http://arxiv.org/abs/1803.11485v2

Chicago

Rashid, T., M. Samvelyan, C. S. de . Witt, G. Farquhar, J. Foerster, and S. Whiteson. 2018. “QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning”. arXiv. http://arxiv.org/abs/1803.11485v2.

Harvard

Rashid, T. et al. (2018) “QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1803.11485v2.

Vancouver

1. Rashid T, Samvelyan M, Witt CS de, Farquhar G, Foerster J, Whiteson S (2018) QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. arXiv

BibTeX

@article{rashid2018qmix,
  title = {QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning},
  author = {Rashid, Tabish and Samvelyan, Mikayel and Witt, Christian Schroeder de and Farquhar, Gregory and Foerster, Jakob and Whiteson, Shimon},
  year = {2018},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1803.11485v2},
  eprint = {1803.11485}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Method	IQL	VDN	VDN-S	QMIX-Lin	QMIX-NS	QMIX
Final Test Reward	7	7	7	7	8	8

QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Monotonicity Condition for Factorised Joint Action-Values

Knowl 2 — QMIX Neural Architecture and Hypernetwork Design

Knowl 3 — QMIX End-to-End Training Algorithm

Knowl 4 — Representational Limits and Universal Approximation of Monotonic Mixing

Knowl 5 — Two-Step Cooperative Matrix Game Comparison

Knowl 6 — StarCraft II Multi-Agent Micromanagement Experimental Setup

Knowl 7 — Performance of QMIX on StarCraft II Micromanagement Scenarios

Knowl 8 — Ablation Analysis of Non-Linear Mixing and State Conditioning

Knowl 9 — Tactical Coordination Strategies Discovered by QMIX

References

Citation

BibTeX

Source Code

Access the Paper