Learning to Communicate with Deep Multi-Agent Reinforcement Learning

Jakob N. FoersterYannis AssaelNando de FreitasShimon Whiteson

article2016NeurIPS1,975 citations

Proposes Reinforced Inter-Agent Learning (RIAL) and Differentiable Inter-Agent Learning (DIAL), enabling partially observable multi-agent systems to learn effective communication protocols end-to-end by backpropagating gradients across agent channels during centralized training.

Listen

Coordinating autonomous systems—such as fleets of delivery robots, self-driving vehicles, or distributed sensors—requires effective communication. In real-world deployments, these systems often face partial observability, meaning no single agent possesses complete information about the environment, as well as strict bandwidth limitations that restrict data transmission. Handcrafting communication protocols for every multi-agent deployment is costly, inflexible, and increasingly intractable as team sizes grow.

To address this challenge, the article sets out to demonstrate how autonomous agents can automatically learn and discover discrete communication protocols from scratch using deep multi-agent reinforcement learning. Specifically, the researchers evaluated whether allowing inter-agent gradient flow during centralized training enables agents to discover effective communication strategies for complex, sequential tasks.

The authors designed and evaluated two deep learning frameworks within a centralized training and decentralized execution paradigm: Reinforced Inter-Agent Learning (RIAL) and Differentiable Inter-Agent Learning (DIAL). RIAL combines recurrent neural networks with standard reinforcement learning, treating other agents as part of the external environment. In contrast, DIAL allows continuous real-valued messages to pass directly between agents during simulated training, enabling end-to-end backpropagation of error feedback across the communication channel before discretizing the messages into binary signals for real-world execution. Both methods were benchmarked on a cooperative coordination riddle ("The Switch Riddle") and complex multi-agent visual recognition challenges using the MNIST dataset.

The experimental findings demonstrate three primary outcomes: First, DIAL substantially outperformed RIAL across all tasks, discovering optimal communication protocols faster and scaling effectively to multi-step challenges where RIAL completely failed to learn. Second, parameter sharing among agents proved critical; sharing network parameters significantly accelerated learning speed and prevented failure modes when scaling to larger team sizes. Third, injecting continuous noise into the communication bottleneck during centralized training was essential to force learned representations into discrete modes, preventing information over-encoding and ensuring seamless transitions to discrete execution.

These results demonstrate that end-to-end differentiable communication offers a practical, scalable mechanism for automated protocol discovery, reducing the trial-and-error exploration typically required in multi-agent reinforcement learning. For organizations developing robotic fleets or distributed software agents, this approach lowers engineering overhead and improves system coordination in bandwidth-constrained environments. Practitioners should adopt centralized training with parameter sharing and differentiable message channels when training cooperative multi-agent systems, ensuring appropriate channel noise calibration based on deployment bandwidth limits.

While these findings offer strong confidence in the viability of automated protocol learning, the evaluation remains constrained to controlled synthetic tasks and small agent groups. Further research and pilot testing on larger real-world deployments are necessary to assess scalability, conversational complexity, and robustness against real-world packet loss and transmission latency.

CIFAR Google University of Oxford communication channels communication protocols computer vision Learning Agents Agent

arXiv: 1605.06676

Paper: Human-level control through deep reinforcement learning, Volodymyr Mnih et al. (2015). This seminal paper introduces Deep Q-Networks (DQN) with experience replay and target networks, establishing the deep reinforcement learning foundations that RIAL and DIAL adapt to multi-agent settings.
Paper: Deep Recurrent Q-Learning for Partially Observable MDPs, Matthew Hausknecht et al. (2015). This work introduces Deep Recurrent Q-Networks to handle partially observable environments via recurrent memory, providing the essential architectural framework used by agents to communicate over sequential time steps.
Paper: Markov Games as a Framework for Multi-Agent Reinforcement Learning, M. Littman (1994). This foundational paper formalizes multi-agent reinforcement learning within Markov games, laying the conceptual groundwork for multi-agent coordination and decentralized decision-making.

Paper: Counterfactual Multi-Agent Policy Gradients, Jakob N. Foerster et al. (2017). This work advances the centralized training with decentralized execution paradigm established in multi-agent deep RL by introducing counterfactual baselines for multi-agent credit assignment.
Paper: Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments, Ryan Lowe et al. (2017). This paper extends centralized training and decentralized execution to actor-critic frameworks operating in continuous action spaces and mixed cooperative-competitive environments.
Paper: QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning, Tabish Rashid et al. (2018). This paper develops monotonic value function factorization to scale centralized training with decentralized execution in complex cooperative multi-agent tasks.
Paper: The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games, Chao Yu et al. (2022). This study evaluates on-policy policy optimization variants within centralized-critic cooperative multi-agent environments, extending the benchmark results and methodologies seen across MARL literature.

Cover for Learning to Communicate with Deep Multi-Agent Reinforcement Learning

Abstract

We consider the problem of multiple agents sensing and acting in environments with the goal of maximising their shared utility. In these environments, agents must learn communication protocols in order to share information that is needed to solve the tasks. By embracing deep neural networks, we are able to demonstrate end-to-end learning of protocols in complex environments inspired by communication riddles and multi-agent computer vision problems with partial observability. We propose two approaches for learning in these domains: Reinforced Inter-Agent Learning (RIAL) and Differentiable Inter-Agent Learning (DIAL). The former uses deep Q-learning, while the latter exploits the fact that, during learning, agents can backpropagate error derivatives through (noisy) communication channels. Hence, this approach uses centralised learning but decentralised execution. Our experiments introduce new environments for studying the learning of communication protocols and present a set of engineering innovations that are essential for success in these domains.

1 Introduction
2 Related Work
3 Background
4 Setting
5 Methods
5.1 Reinforced Inter-Agent Learning
5.2 Differentiable Inter-Agent Learning
6 Experiments
6.1 Model Architecture
6.2 Switch Riddle
6.3 MNIST Games
6.4 Effect of Channel Noise
7 Conclusions
References
A DIAL Details
B MNIST Games: Further Analysis
C Effect of Noise: Further Analysis

Knowls

Knowl 1 — Differentiable Inter-Agent Learning (DIAL)
model/method
Differentiable Inter-Agent Learning (DIAL) is a framework for learning communication protocols in cooperative multi-agent reinforcement learning problems with partial observability under the paradigm of centralized learning with decentralized execution (CTDE).

During centralized learning, communication channels between agents are modeled as continuous, real-valued connections rather than discrete action spaces. This allows message activations $m_t^a$ output by sending agent $a$ to be passed directly into receiving agents as continuous features. Consequently, network gradients calculated from the reinforcement learning loss (e.g., Deep Q-Network loss on task-level environment actions $u$ ) at the receiving agent backpropagate directly through the communication channel into the parameters of the sending agent. This end-to-end gradient passing provides rich inter-agent feedback, bypassing the high variance and sparse reward challenges of trial-and-error exploration over discrete communication actions.

During decentralized execution, the continuous messages are discretized into allowed communication actions using a Discretise/Regularise Unit, allowing execution to remain completely decentralized across limited-bandwidth discrete channels.
Knowl 2 — Discretise/Regularise Unit (DRU)
model/method
The Discretise/Regularise Unit (DRU) is a transformation applied to outgoing continuous message activations $m_t^a \in \mathbb{R}$ produced by an agent $a$ at time-step $t$ , enabling differentiable training across agents during centralized learning while enforcing discrete messaging during decentralized execution.

The DRU operation is defined as:

$\text{DRU}(m_t^a) = \begin{cases} \text{Logistic}(\mathcal{N}(m_t^a, \sigma)), & \text{during centralized training} \\ \mathbf{1}_{\{m_t^a > 0\}}, & \text{during decentralized execution} \end{cases}$

where $\mathcal{N}(m_t^a, \sigma)$ adds zero-mean Gaussian noise with standard deviation $\sigma$ to the message, $\text{Logistic}(z) = \frac{1}{1 + e^{-z}}$ , and $\mathbf{1}_{\{m_t^a > 0\}}$ is the indicator step function that maps positive values to $1$ and non-positive values to $0$ .

During training, adding Gaussian noise perturbs values in the sensitive, high-gradient regime of the logistic sigmoid, penalizing intermediate values and encouraging the network to push activations into the saturated tails where noise perturbation is minimized. This regularizes channel capacity and minimizes discretization error when switching to the hard threshold during decentralized execution.
Knowl 3 — DIAL Training Algorithm
algorithm
The DIAL algorithm trains a centralized neural network (C-Net) with parameters $\theta$ and target parameters $\theta^{-}$ across $n$ agents over episodes of length up to $T$ steps.
Input: Step size $\alpha$ , target reset interval $C$ , channel noise standard deviation $\sigma$ , exploration rate $\epsilon$
Initialize online parameters $\theta_1$ and target parameters $\theta^{-}_1 = \theta_1$
for each episode $e = 1, 2, \dots$ do
    Initialize environment state $s_1$ , $t = 0$ , and hidden state $h_0^a = 0$ for each agent $a \in \{1, \dots, n\}$
    while $s_t$ is not terminal and $t < T$ do
         $t = t + 1$
        for each agent $a \in \{1, \dots, n\}$ do
            Obtain incoming messages $\hat{m}_{t-1}^{a'}$ from other agents $a' \neq a$
            Compute $Q(o_t^a, \hat{m}_{t-1}^{a'}, h_{t-1}^a, u_{t-1}^a, a, \cdot; \theta_i)$ and continuous message $m_t^a$ using C-Net
            With probability $\epsilon$ , select random action $u_t^a \in U$ ; otherwise $u_t^a = \arg\max_u Q(o_t^a, \hat{m}_{t-1}^{a'}, h_{t-1}^a, u_{t-1}^a, a, u; \theta_i)$
            Compute regularized message $\hat{m}_t^a = \text{Logistic}(\mathcal{N}(m_t^a, \sigma))$
        end for
        Execute environment actions $\{u_t^a\}$ , observe shared reward $r_t$ and next state $s_{t+1}$
    end while
    Reset gradient accumulator $\nabla\theta = 0$
    for $t = T$ down to $1$ do
        for each agent $a \in \{1, \dots, n\}$ do
            if $s_t$ is terminal then
                 $y_t^a = r_t$
            else
                 $y_t^a = r_t + \gamma \max_u Q(o_{t+1}^a, \hat{m}_t^{a'}, h_t^a, u_t^a, a, u; \theta^{-}_i)$
            end if
             $\Delta Q_t^a = y_t^a - Q(o_t^a, \hat{m}_{t-1}^{a'}, h_{t-1}^a, u_{t-1}^a, a, u_t^a; \theta_i)$
             $\nabla\theta = \nabla\theta + \frac{\partial}{\partial\theta} (\Delta Q_t^a)^2$
             $\mu_t^a = \mathbf{1}_{\{t < T-1\}} \sum_{a' \neq a} \left( \frac{\partial}{\partial \hat{m}_t^a} (\Delta Q_{t+1}^{a'})^2 + \mu_{t+1}^{a'} \frac{\partial \hat{m}_{t+1}^{a'}}{\partial \hat{m}_t^a} \right)$
             $\nabla\theta = \nabla\theta + \mu_t^a \frac{\partial \text{DRU}(m_t^a)}{\partial m_t^a} \frac{\partial m_t^a}{\partial\theta}$
        end for
    end for
    Update parameters $\theta_{i+1} = \theta_i + \alpha \nabla\theta$
    Every $C$ steps, update target network $\theta^{-}_i = \theta_i$
end for
Knowl 4 — Reinforced Inter-Agent Learning (RIAL)
model/method
Reinforced Inter-Agent Learning (RIAL) combines Deep Recurrent Q-Networks (DRQN) with independent Q-learning for multi-agent communication and action selection. Each agent $a$ estimates Q-values conditioned on its private observation $o_t^a$ , incoming discrete messages $m_{t-1}^{a'}$ , internal recurrent hidden state $h_{t-1}^a$ , previous environment action $u_{t-1}^a$ , and previous message $m_{t-1}^a$ .

To prevent action space explosion from joint action-message pairs $|U| \times |M|$ , RIAL factors the output into two separate heads:
1. $Q_u(o_t^a, m_{t-1}^{a'}, h_{t-1}^a, u_{t-1}^a, m_{t-1}^a, a, u^a)$ for environment actions $u^a \in U$ .
2. $Q_m(o_t^a, m_{t-1}^{a'}, h_{t-1}^a, u_{t-1}^a, m_{t-1}^a, a, m^a)$ for communication actions $m^a \in M$ .
Action selection performs separate $\epsilon$ -greedy selections over $|U|$ and $|M|$ , reducing computational and parameter complexity from $|U||M|$ to $|U| + |M|$ .

Two critical modifications make RIAL functional in multi-agent domains:
1. Experience replay is disabled because simultaneous multi-agent learning causes non-stationarity that invalidates stored replay transitions.
2. Parameter sharing across all agents is employed to reduce parameter space and accelerate learning, with agent index $a$ passed as an input feature to allow individual agent specialization.
Knowl 5 — Agent Neural Network Architecture with Recurrent Embedding
model/method
In both RIAL and DIAL, each agent $a$ at time-step $t$ uses a shared modular neural architecture:
1. Embedding Stage: Input features comprising observation $o_t^a$ , incoming message $\hat{m}_{t-1}^{a'}$ , previous action $u_{t-1}^a$ , and agent index $a$ are mapped to 128-dimensional vectors. The combined embedding $z_t^a$ is formed via element-wise summation:
$z_t^a = \text{TaskMLP}(o_t^a) + \text{MLP}_{(|M|, 128)}(\text{BatchNorm}(\hat{m}_{t-1}^{a'})) + \text{Lookup}(u_{t-1}^a) + \text{Lookup}(a)$

Applying batch normalization to the incoming message $\hat{m}_{t-1}^{a'}$ significantly improves training stability and speed.
1. Recurrent State: The embedding $z_t^a$ is passed through a 2-layer Gated Recurrent Unit (GRU) with 128 hidden units per layer to approximate the action-observation history:
$h_{1,t}^a = \text{GRU}_{[128, 128]}(z_t^a, h_{1,t-1}^a)$ $h_{2,t}^a = \text{GRU}_{[128, 128]}(h_{1,t}^a, h_{2,t-1}^a)$
1. Output Heads: The top GRU hidden state $h_{2,t}^a$ passes through a 2-layer MLP with 128 hidden units:
- In RIAL: outputs $|U|$ environment Q-values and $|M|$ communication Q-values.
- In DIAL: outputs $|U|$ environment Q-values and $|m|$ continuous communication activations $m_t^a$ .
Knowl 6 — Inter-Agent Gradient vs Temporal-Difference Learning in Antisymmetric Settings
theoretical result
In cooperative games with antisymmetric reward structures and private state observations, reinforcement learning via standard temporal-difference (TD) error updates fails to learn communication protocols, whereas inter-agent backpropagation (DIAL) succeeds.

Consider a two-agent setting with random private digits $s^1, s^2 \in \{0, 1\}$ , binary action $u^2 \in \{1, 2\}$ , and antisymmetric reward $r(s^1, s^2, u^2) = (-1)^{(s^1 + s^2 + u^2)}$ . Agent 1 sends a 1-bit message $m^1$ to Agent 2. Before a coordinated protocol is established, Agent 2 acts arbitrarily. Averaged over $s^1$ , the expected reward is zero, causing the expected TD error for Agent 1 (the sender) to evaluate to zero for any message $m^1$ :

$\mathbb{E}\left[ \Delta Q(s^1, m^1) \right] = Q(s^1, m^1) - \mathbb{E}_{s^2, u^2}\left[ r(s^2, u^2, s^1) \right] = 0 - 0 = 0$

Consequently, Q-learning algorithms like RIAL receive no directional gradient to update communication actions.

In contrast, DIAL computes parameter gradients through the receiver's Q-function with respect to the outgoing message:

$\mathbb{E}[\nabla_\theta] = \mathbb{E}_{s^2, u^2}\left[ \left( Q(s^2, m^1, u^2) - r(s^2, u^2, s^1) \right) \frac{\partial}{\partial m^1} Q(s^2, m^1, u^2) \frac{\partial}{\partial \theta} m^1(s^1) \right]$

Because the gradient incorporates the receiver's action and private observation directly, terms across $+1$ and $-1$ reward outcomes do not cancel out, providing a direct learning signal to shape informative messages.
Knowl 7 — Probability Distribution and Capacity of Noisy Logistic Communication Channel
equation
When a continuous message activation $m \in \mathbb{R}$ is perturbed by additive Gaussian noise $\mathcal{N}(0, \sigma^2)$ and passed through a standard logistic sigmoid function, the probability density function $P(\hat{m} \mid m)$ of the regularized output $\hat{m} \in (0, 1)$ is given by:

$P(\hat{m} \mid m) = \frac{1}{\sqrt{2\pi} \sigma \hat{m}(1 - \hat{m})} \exp\left( -\frac{\left(m - \log\left(\frac{1}{\hat{m}} - 1\right)\right)^2}{2\sigma^2} \right)$

For two activations $m_1, m_2$ to transmit distinguishable symbols with error rate below $\epsilon$ , their output distributions must satisfy:

$\left(\max_{\hat{m}} \text{ s.t. } P(\hat{m} \mid m_1) > \epsilon\right) \le \left(\min_{\hat{m}} \text{ s.t. } P(\hat{m} \mid m_2) > \epsilon\right)$

Assuming the decodable linear input range of the logistic function is approximately $[-5, 5]$ (corresponding to outputs between $0.01$ and $0.99$ ), setting $\sigma \ge 2$ requires distinct message modes to be at least $6\sigma \approx 12$ units apart, effectively restricting the decodable information capacity of the continuous channel to exactly 1 bit.
Knowl 8 — Switch Riddle Multi-Agent Benchmark Formulation
experimental setup
The Switch Riddle benchmark evaluates multi-agent protocol discovery under extreme partial observability and sparse rewards.
- Environment Setup: $n$ agents (prisoners) are isolated. At each time step $t$ , the environment selects one agent $a$ uniformly at random with replacement to visit a room containing a single light switch (a 1-bit memory state).
- Observations: Agent $a$ receives private observation $o_t^a = 1$ if inside the room and $o_t^a = 0$ otherwise. If in the room, the agent also observes the previous switch position $m_{t-1}^{a'} \in \{0, 1\}$ .
- Action Space: If in the room, agent $a$ selects an environment action $u_t^a \in \{\text{"None"}, \text{"Tell"}\}$ and a communication action $m_t^a \in \{\text{"Off"} (0), \text{"On"} (1)\}$ . If outside, its action is fixed to $\text{"None"}$ .
- Reward and Termination: The episode ends when any agent chooses $\text{"Tell"}$ or at maximum step $T = 4n - 6$ . Reward $r_t = 0$ at all steps except when an agent chooses $\text{"Tell"}$ , yielding $r_t = +1$ if every prisoner has visited the room at least once, and $r_t = -1$ otherwise.
- Policy Space Complexity: At step $t$ , each agent has $3^t$ possible observation histories and 4 action-message choices. The multi-agent policy search space is $4^{n(3^{T+1}-3)/2} = 4^{n 3^{O(n)}}$ . For $n = 4$ , the state-policy search space is $4^{88572}$ .
Knowl 9 — Empirical Performance and Protocol Discovery on the Switch Riddle
empirical result
Evaluations on the Switch Riddle for $n = 3$ and $n = 4$ agents demonstrate clear differences between communication algorithms:
- For $n = 3$ agents: DIAL with parameter sharing reaches optimal normalized reward ( $1.0$ ) in under 1,000 epochs. RIAL with parameter sharing also reaches optimal performance but requires approximately 5,000 epochs. Baselines without communication (NoComm) plateau around 0.6–0.7.
- For $n = 4$ agents: DIAL with parameter sharing outperforms all methods, reaching optimal reward in approximately 20,000 epochs. RIAL with parameter sharing converges substantially slower. RIAL without parameter sharing (RIAL-NS) fails entirely to outperform the NoComm baseline, showing that parameter sharing is necessary for coordinating independent RL agents.
- Discovered Protocol ( $n = 3$ ): Sampled episodes reveal that DIAL discovers an optimal counting strategy: on Day 1, the agent turns the switch "On". On subsequent days (Day 2 and 3+), if an agent enters for the first time, it inspects whether the switch is "On" or "Off" to deduce whether one or two agents have previously visited, turning the switch "Off" or declaring "Tell" when the count is complete.
Knowl 10 — Empirical Protocol Learning on Vision-Based MNIST Multi-Agent Games
empirical result
In cooperative vision-based digit coordination games using raw $28 \times 28$ MNIST images:
1. Colour-Digit MNIST: Two agents receive colored digit images (size $2 \times 28 \times 28$ ) with hidden color labels $c^a \in \{0, 1\}$ and digit values $d^a \in \{0, \dots, 9\}$ . In step 1, each sends a 1-bit message; in step 2, each takes a binary action $u_2^a$ . The shared payoff $r = r(1) + r(2)$ with $r(a) = 2(-1)^{u_2^a + c^a + d^{a'}} + (-1)^{u_2^a + d^a + c^{a'}}$ rewards coordinating on digit parity (odd/even) over color. DIAL learns the optimal parity-encoding protocol, reaching normalized reward near $1.0$ . In contrast, RIAL fluctuates around suboptimal local minima (reward $\approx 0.5$ ) because its TD-error updates cancel out under stochastic rewards.
2. Multi-Step MNIST: Two agents communicate over 5 consecutive steps with 1 bit transmitted per step (4 received messages total) to guess the other agent's digit $d^{a'} \in \{0, \dots, 9\}$ . DIAL achieves optimal reward by learning a 4-bit binary encoding/decoding scheme across time steps. RIAL completely fails to learn on Multi-Step MNIST (normalized reward remains at 0).
Knowl 11 — Effect of Channel Noise on Discretization and Continuous Over-Encoding
empirical result
The standard deviation $\sigma$ of the Gaussian noise in the DRU determines whether emergent communication protocols are discrete and robust during decentralized execution:
- Without Noise ( $\sigma = 0$ ): Network activations cluster closely around zero (the continuous linear region of the logistic sigmoid). The network exploits continuous message bandwidth during training to over-encode information. When continuous messages are thresholded to $\{0, 1\}$ during execution, performance collapses due to massive discretization error.
- With Noise ( $\sigma > 0$ , e.g., $\sigma = 2.0$ ): The noise penalizes activations near zero, forcing activations into two distinct saturated modes at the positive and negative extremes (large positive and negative logits). This produces near-zero discretization error when evaluating with hard step binarization $\mathbf{1}_{\{m > 0\}}$ .
- Incentive for Over-Encoding: In Multi-Step MNIST with fewer available time steps than $\lceil\log_2(10)\rceil = 4$ bits (e.g., 2 steps allowing only 1 bit), low noise ( $\sigma = 0.5$ ) allows partial continuous over-encoding during training, yielding an execution/training reward ratio of $\approx 0.7$ . When sufficient bits are available (5 steps, 4 bits), normalized test performance exceeds 1.0, demonstrating successful discrete regularized protocol emergence.

Coverage note — None was omitted; all key contributions including methods (DIAL, RIAL, DRU), network architecture, theoretical/gradient analysis, benchmarks, and experimental results are covered.

References

1.L. Kraemer and B. Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190:82–94, 2016.
2.V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
3.J. N. Foerster, Y. M. Assael, N. de Freitas, and S. Whiteson. Learning to communicate to solve riddles with deep distributed recurrent q-networks. arXiv preprint arXiv:1602.02672, 2016.
4.M. Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In ICML, 1993.
5.F. S. Melo, M. Spaan, and S. J. Witwicki. QueryPOMDP: POMDP-based communication in multiagent systems. In Multi-Agent Systems, pages 189–204. 2011.
6.L. Panait and S. Luke. Cooperative multi-agent learning: The state of the art. Autonomous Agents and Multi-Agent Systems, 11(3):387–434, 2005.
7.C. Zhang and V. Lesser. Coordinating multi-agent reinforcement learning with limited communication. In AAMAS, volume 2, pages 1101–1108, 2013.
8.T. Kasai, H. Tenmoto, and A. Kamiya. Learning of communication codes in multi-agent reinforcement learning problem. In IEEE Soft Computing in Industrial Applications, pages 1–6, 2008.
9.C. L. Giles and K. C. Jim. Learning communication for multi-agent systems. In Innovative Concepts for Agent-Based Systems, pages 377–390. Springer, 2002.
10.K. Gregor, I. Danihelka, A. Graves, and D. Wierstra. Draw: A recurrent neural network for image generation. arXiv preprint arXiv:1502.04623, 2015.
11.M. Courbariaux and Y. Bengio. BinaryNet: Training deep neural networks with weights and activations constrained to +1 or -1. arXiv preprint arXiv:1602.02830, 2016.
12.G. Hinton and R. Salakhutdinov. Discovering binary codes for documents by learning deep generative models. Topics in Cognitive Science, 3(1):74–91, 2011.
13.R. S. Sutton and A. G. Barto. Introduction to reinforcement learning. MIT Press, 1998.
14.A. Tampuu, T. Matiisen, D. Kodelja, I. Kuzovkin, K. Korjus, J. Aru, J. Aru, and R. Vicente. Multiagent cooperation and competition with deep reinforcement learning. arXiv preprint arXiv:1511.08779, 2015.
15.Y. Shoham and K. Leyton-Brown. Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations. Cambridge University Press, New York, 2009.
16.E. Zawadzki, A. Lipson, and K. Leyton-Brown. Empirically evaluating multiagent learning algorithms. arXiv preprint 1401.8074, 2014.
17.M. Hausknecht and P. Stone. Deep recurrent Q-learning for partially observable MDPs. arXiv preprint arXiv:1507.06527, 2015.
18.F. A. Oliehoek, M. T. J. Spaan, and N. Vlassis. Optimal and approximate Q-value functions for decentralized POMDPs. JAIR, 32:289–353, 2008.
19.K. Narasimhan, T. Kulkarni, and R. Barzilay. Language understanding for text-based games using deep reinforcement learning. arXiv preprint arXiv:1506.08941, 2015.
20.K. Cho, B. van Merriënboer, D. Bahdanau, and Y. Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014.
21.S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
22.J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
23.R. Jozefowicz, W. Zaremba, and I. Sutskever. An empirical exploration of recurrent network architectures. In ICML, pages 2342–2350, 2015.
24.S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, pages 448–456, 2015.
25.W. Wu. 100 prisoners and a lightbulb. Technical report, OCF, UC Berkeley, 2002.
26.Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
27.M. Studdert-Kennedy. How did language go discrete? In M. Tallerman, editor, Language Origins: Perspectives on Evolution, chapter 3. Oxford University Press, 2005.

Citation

MLA

Foerster, J. N., et al. “Learning to Communicate with Deep Multi-Agent Reinforcement Learning”. arXiv, 2016, http://arxiv.org/abs/1605.06676v2.

APA

Foerster, J. N., Assael, Y. M., Freitas, N. de ., & Whiteson, S. (2016). Learning to Communicate with Deep Multi-Agent Reinforcement Learning. arXiv. http://arxiv.org/abs/1605.06676v2

Chicago

Foerster, J. N., Y. M. Assael, N. de . Freitas, and S. Whiteson. 2016. “Learning to Communicate with Deep Multi-Agent Reinforcement Learning”. arXiv. http://arxiv.org/abs/1605.06676v2.

Harvard

Foerster, J.N. et al. (2016) “Learning to Communicate with Deep Multi-Agent Reinforcement Learning”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1605.06676v2.

Vancouver

1. Foerster JN, Assael YM, Freitas N de, Whiteson S (2016) Learning to Communicate with Deep Multi-Agent Reinforcement Learning. arXiv

BibTeX

@article{foerster2016learning,
  title = {Learning to Communicate with Deep Multi-Agent Reinforcement Learning},
  author = {Foerster, Jakob N. and Assael, Yannis M. and Freitas, Nando de and Whiteson, Shimon},
  year = {2016},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1605.06676v2},
  eprint = {1605.06676}
}

Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Published with permission

Learning to Communicate with Deep Multi-Agent Reinforcement Learning

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Differentiable Inter-Agent Learning (DIAL)

Knowl 2 — Discretise/Regularise Unit (DRU)

Knowl 3 — DIAL Training Algorithm

Knowl 4 — Reinforced Inter-Agent Learning (RIAL)

Knowl 5 — Agent Neural Network Architecture with Recurrent Embedding

Knowl 6 — Inter-Agent Gradient vs Temporal-Difference Learning in Antisymmetric Settings

Knowl 7 — Probability Distribution and Capacity of Noisy Logistic Communication Channel

Knowl 8 — Switch Riddle Multi-Agent Benchmark Formulation

Knowl 9 — Empirical Performance and Protocol Discovery on the Switch Riddle

Knowl 10 — Empirical Protocol Learning on Vision-Based MNIST Multi-Agent Games

Knowl 11 — Effect of Channel Noise on Discretization and Continuous Over-Encoding

References

Citation

BibTeX

Access the Paper