Generative Adversarial Imitation Learning

article2016NeurIPS3,788 citations

Proposes a model-free imitation learning framework that bypasses the computational bottleneck of inverse reinforcement learning by directly matching expert state-action distributions using generative adversarial training.

Listen

The paper addresses the challenge of teaching an agent to perform complex tasks by observing expert demonstrations alone, without further queries to the expert or any reward signals. Standard approaches either copy actions directly from the data, which fails when errors compound in large state spaces, or recover an implicit reward function first through inverse reinforcement learning and then optimize a policy, which incurs repeated expensive reinforcement-learning steps inside the training loop. Both limitations become acute in high-dimensional continuous control problems such as humanoid locomotion.

The work therefore sets out to derive and test a method that directly produces a policy whose state-action distribution closely matches the expert’s, while remaining practical for large environments and arbitrary neural-network policies.

The authors first characterize the policy recovered by regularized inverse reinforcement learning as the solution to an occupancy-measure matching problem. They then instantiate this characterization with a new regularizer whose dual corresponds to the Jensen-Shannon divergence between the learner’s and expert’s state-action distributions. The resulting algorithm alternates between training a discriminator network to distinguish expert trajectories from those generated by the current policy and updating the policy with a trust-region policy-optimization step that minimizes the discriminator’s output. Experiments compare the method against behavioral cloning and two linear apprenticeship-learning baselines on nine physics-based tasks, using expert datasets that range from one to several hundred trajectories.

Across the majority of tasks the new algorithm reaches 70–100 percent of expert performance with far fewer demonstrations than the baselines require; on the 376-dimensional humanoid task it matches expert returns exactly for every dataset size tested, while behavioral cloning stays below 60 percent. The performance gap is largest on the highest-dimensional problems, where the competing methods either collapse to random behavior or need substantially more data.

These results indicate that imitation learning can be made both sample-efficient in expert data and computationally tractable without hand-crafted features or inner-loop reinforcement learning. Consequently, the approach lowers the barrier to deploying imitation in settings where expert interaction is costly or unsafe and where reward engineering is difficult.

Further gains in environment-sample efficiency are likely if the policy is first pre-trained by behavioral cloning before adversarial fine-tuning; combining the method with learned dynamics models and limited expert queries during training would address the remaining interaction cost. The main limitations are that the algorithm remains model-free and therefore interaction-heavy, and that all reported results come from simulated MuJoCo environments whose dynamics are fully known during training.

Stanford University generative adversarial networks imitation learning model free Generative Adversarial Imitation Learning Inverse Reinforcement Learning Occupancy Measure

arXiv: 1606.03476 openai/imitation

Paper: A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, Stephane Ross et al. (2010). Understanding DAgger provides crucial background on iterative imitation learning and the problem of compounding errors in sequential prediction addressed by generative adversarial approaches.
Paper: f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization, Sebastian Nowozin et al. (2016). F-GAN generalizes generative adversarial training across variational divergences, offering the foundational mathematical machinery for adversarial divergence minimization utilized in policy-space imitation.
Paper: Algorithms for Inverse Reinforcement Learning, Andrew Y. Ng et al. (2000). Ng and Russell's inverse reinforcement learning framework establishes the traditional two-step paradigm of cost recovery and policy extraction that generative adversarial imitation learning aims to bypass.
Paper: A Natural Policy Gradient, Sham M. Kakade (2001). Natural policy gradient optimization provides key theoretical foundations for updating parameterized policies along covariant directions in complex Markov decision processes.
Paper: Policy Gradient Methods for Reinforcement Learning with Function Approximation, Richard S. Sutton et al. (1999). The policy gradient theorem forms the core mathematical basis for updating policies directly from experience without relying entirely on intermediate value functions.
Paper: Actor-Critic Algorithms, Vijay R. Konda et al. (1999). Actor-critic algorithms introduce two-time-scale stochastic approximation methods that link policy updates and value function estimation in continuous reinforcement learning.

Paper: Proximal Policy Optimization Algorithms, John Schulman et al. (2017). Proximal policy optimization builds directly upon policy gradient foundations to stabilize updates through clipped surrogate objectives.
Paper: High-Dimensional Continuous Control Using Generalized Advantage Estimation, John Schulman et al. (2016). Generalized advantage estimation builds on policy optimization frameworks to resolve high-dimensional credit assignment challenges with tunable bias-variance trade-offs.
Paper: Hindsight Experience Replay, Marcin Andrychowicz et al. (2017). Hindsight experience replay extends standard reinforcement learning to handle sparse binary reward environments by repurposing failed trajectories as successful alternative goals.
Paper: Conservative Q-Learning for Offline Reinforcement Learning, Aviral Kumar et al. (2020). Conservative Q-learning extends offline reinforcement learning by penalizing unseen actions to prevent overestimation errors in static datasets.
Paper: Diffusion policy: Visuomotor policy learning via action diffusion, Cheng Chi et al. (2023). Diffusion policy adapts continuous generation techniques to visuomotor control, addressing multimodal action distributions in robotic manipulation.
Paper: Deep reinforcement learning from human preferences, Paul F. Christiano et al. (2017). Deep reinforcement learning from human preferences extends reward-based alignment techniques to settings where explicit reward functions are absent.

Cover for Generative Adversarial Imitation Learning

Abstract

Consider learning a policy from example expert behavior, without interaction with the expert or access to reinforcement signal. One approach is to recover the expert's cost function with inverse reinforcement learning, then extract a policy from that cost function with reinforcement learning. This approach is indirect and can be slow. We propose a new general framework for directly extracting a policy from data, as if it were obtained by reinforcement learning following inverse reinforcement learning. We show that a certain instantiation of our framework draws an analogy between imitation learning and generative adversarial networks, from which we derive a model-free imitation learning algorithm that obtains significant performance gains over existing model-free methods in imitating complex behaviors in large, high-dimensional environments.

Topics

generative adversarial networks imitation learning model free learning from demonstration

Keywords

Generative Adversarial Imitation Learning Inverse Reinforcement Learning Occupancy Measure Jensen‑Shannon Divergence Discriminator Network Trust Region Policy Optimization Policy Gradient Expert Trajectory Model‑Free Imitation Algorithm Causal Entropy Regularization

1 Introduction
2 Background
3 Characterizing the induced optimal policy
4 Practical occupancy measure matching
5 Generative adversarial imitation learning
6 Experiments
7 Discussion and outlook
Acknowledgments
References
A Proofs
A.1 Proofs for Section 3
A.2 Proofs for Section 5
B Environments and detailed results

Knowls

Knowl 1 — Generative Adversarial Imitation Learning Algorithm
algorithm
Generative Adversarial Imitation Learning (GAIL) directly learns a parameterized policy $\pi_\theta$ from expert demonstration trajectories by finding a saddle point of a discriminator-guided objective, bypassing intermediate inverse reinforcement learning cost reconstruction.

The algorithm alternates between training a discriminator network $D_w: \mathcal{S} \times \mathcal{A} \to (0, 1)$ parameterized by $w$ using Adam to distinguish generated state-action pairs from expert state-action pairs, and updating policy parameters $\theta$ using Trust Region Policy Optimization (TRPO) with surrogate cost function $c(s, a) = \log(D_w(s, a))$ .
Input: Expert trajectories $\tau_E \sim \pi_E$ , initial policy parameters $\theta_0$ , initial discriminator parameters $w_0$ , entropy regularizer weight $\lambda \ge 0$
for $i = 0, 1, 2, \dots$ do
    Sample trajectories $\tau_i \sim \pi_{\theta_i}$
    Compute discriminator gradient $g_w$ :
         $g_w = \hat{\mathbb{E}}_{\tau_i}[\nabla_w \log(D_w(s, a))] + \hat{\mathbb{E}}_{\tau_E}[\nabla_w \log(1 - D_w(s, a))]$
    Update discriminator parameters $w_{i+1}$ from $w_i$ using an Adam step on $g_w$
    Compute policy gradient $g_\theta$ using surrogate cost $c(s,a) = \log(D_{w_{i+1}}(s, a))$ :
         $g_\theta = \hat{\mathbb{E}}_{\tau_i}[\nabla_\theta \log \pi_\theta(a|s) Q(s, a)] - \lambda \nabla_\theta H(\pi_\theta)$
        where $Q(\bar{s}, \bar{a}) = \hat{\mathbb{E}}_{\tau_i}[\sum_{t=0}^\infty \gamma^t \log(D_{w_{i+1}}(s_t, a_t)) \mid s_0 = \bar{s}, a_0 = \bar{a}]$
    Update policy parameters $\theta_{i+1}$ from $\theta_i$ using a TRPO step (KL-constrained natural gradient step) with $g_\theta$
end for
Implementation specifications:
- Policy $\pi_\theta$ and discriminator $D_w$ are parameterised as multi-layer perceptrons with two hidden layers of 100 units each and $\tanh$ activations.
- Value functions share the same neural network architecture as the policy and use Generalized Advantage Estimation (GAE) with $\gamma = 0.995$ and $\lambda_{\text{GAE}} = 0.97$ .
- In most continuous control tasks, causal entropy weight is set to $\lambda = 0$ , with $\lambda = 10^{-3}$ or $10^{-2}$ used when entropy regularization is needed.
Knowl 2 — Induced Policy as Convex Conjugate of Cost Regularizer in Maximum Causal Entropy IRL
theoretical result
In a Markov Decision Process with state space $\mathcal{S}$ , action space $\mathcal{A}$ , discount factor $\gamma \in (0, 1)$ , and transition distribution $P(s' \mid s, a)$ , let $\Pi$ be the set of stationary stochastic policies, and let $\mathcal{D} = \{\rho_\pi : \pi \in \Pi\}$ be the set of valid occupancy measures, where $\rho_\pi(s, a) = \pi(a \mid s) \sum_{t=0}^\infty \gamma^t P(s_t = s \mid \pi)$ .

Let $H(\pi) = \mathbb{E}_\pi[-\log \pi(a \mid s)]$ denote the $\gamma$ -discounted causal entropy of policy $\pi$ . For a closed, proper, convex cost regularizer $\psi: \mathbb{R}^{\mathcal{S} \times \mathcal{A}} \to \bar{\mathbb{R}}$ , regularized maximum causal entropy Inverse Reinforcement Learning (IRL) finds a cost function $\tilde{c} \in \text{IRL}_\psi(\pi_E)$ via: $\text{IRL}_\psi(\pi_E) = \arg\max_{c \in \mathbb{R}^{\mathcal{S} \times \mathcal{A}}} -\psi(c) + \min_{\pi \in \Pi} \left( -H(\pi) + \mathbb{E}_\pi[c(s, a)] \right) - \mathbb{E}_{\pi_E}[c(s, a)]$

Running reinforcement learning on the output cost, defined as $\text{RL}(c) = \arg\min_{\pi \in \Pi} -H(\pi) + \mathbb{E}_\pi[c(s, a)]$ , yields the induced optimal policy: $\text{RL} \circ \text{IRL}_\psi(\pi_E) = \arg\min_{\pi \in \Pi} -H(\pi) + \psi^*(\rho_\pi - \rho_{\pi_E})$ where $\psi^*: \mathbb{R}^{\mathcal{S} \times \mathcal{A}} \to \bar{\mathbb{R}}$ is the convex conjugate of $\psi$ , given by $\psi^*(x) = \sup_{y \in \mathbb{R}^{\mathcal{S} \times \mathcal{A}}} x^T y - \psi(y)$ .

Thus, running RL on costs learned by regularized maximum causal entropy IRL directly minimizes the discrepancy between the learner's occupancy measure and the expert's occupancy measure as measured by $\psi^*$ , regularized by causal entropy.
Knowl 3 — Duality Between Unregularized IRL and Exact Occupancy Measure Matching
theoretical result
When no cost regularization is applied (the regularizer $\psi(c)$ is a constant function over all cost functions $c \in \mathbb{R}^{\mathcal{S} \times \mathcal{A}}$ ), the maximum causal entropy Inverse Reinforcement Learning (IRL) problem is the Lagrangian dual of an exact occupancy measure matching optimization problem:
$ho) \quad \text{subject to} \quad \rho(s, a) = \rho_{\pi_E}(s, a) \quad \forall s \in \mathcal{S}, a \in \mathcal{A}$$ where $\bar{H}(\rho) = -\sum_{s, a} \rho(s, a) \log \left( \frac{\rho(s, a)}{\sum_{a'} \rho(s, a')} \right)$ is the strictly concave causal entropy written as a function of the occupancy measure $\rho$, and $\mathcal{D}$ is the convex set of valid occupancy measures. Key duality properties: 1. The cost function values $c(s, a)$ serve as Lagrange multipliers (dual variables) for the equality constraints $\rho(s, a) = \rho_{\pi_E}(s, a)$. 2. Recovering the cost function $\tilde{c} \in \text{IRL}_\psi(\pi_E)$ corresponds to solving the dual problem. 3. Running reinforcement learning on $\tilde{c}$, given by $\tilde{\pi} \in \text{RL}(\tilde{c})$, recovers the unique primal optimum $\tilde{\rho} = \arg\min_{\rho \in \mathcal{D}} \bar{L}(\rho, \tilde{c}) = \rho_{\pi_E}$. 4. By strong duality, the induced optimal policy matches the expert's occupancy measure exactly: $\rho_{\tilde{\pi}} = \rho_{\pi_E}$.$
Knowl 4 — GAIL Cost Regularizer and Occupancy Measure Jensen-Shannon Divergence
model/method
To combine exact occupancy measure matching capabilities with tractability in large environments, Generative Adversarial Imitation Learning uses the cost regularizer $\psi_{\text{GA}}: \mathbb{R}^{\mathcal{S} \times \mathcal{A}} \to \bar{\mathbb{R}}$ : $\psi_{\text{GA}}(c) = \begin{cases} \mathbb{E}_{\pi_E}[g(c(s, a))] & \text{if } c < 0 \\ +\infty & \text{otherwise} \end{cases}$ where $g(x) = -x - \log(1 - e^x)$ for $x < 0$ , and $+\infty$ otherwise.

The convex conjugate $\psi_{\text{GA}}^*$ evaluated at the occupancy measure difference $\rho_\pi - \rho_{\pi_E}$ equals the optimal binary classification negative log-loss: $\psi_{\text{GA}}^*(\rho_\pi - \rho_{\pi_E}) = \max_{D \in (0, 1)^{\mathcal{S} \times \mathcal{A}}} \mathbb{E}_\pi[\log(D(s, a))] + \mathbb{E}_{\pi_E}[\log(1 - D(s, a))]$ where $D: \mathcal{S} \times \mathcal{A} \to (0, 1)$ is a discriminative classifier distinguishing state-action pairs of $\pi$ from those of $\pi_E$ .

This optimal loss is, up to a constant shift of $-\log 4$ , the Jensen-Shannon divergence $D_{\text{JS}}(\rho_\pi, \rho_{\pi_E}) = D_{\text{KL}}(\rho_\pi \parallel (\rho_\pi + \rho_{\pi_E})/2) + D_{\text{KL}}(\rho_{\pi_E} \parallel (\rho_\pi + \rho_{\pi_E})/2)$ . Consequently, the optimization problem $\min_\pi \psi_{\text{GA}}^*(\rho_\pi - \rho_{\pi_E}) - \lambda H(\pi) = \min_\pi D_{\text{JS}}(\rho_\pi, \rho_{\pi_E}) - \lambda H(\pi)$ minimizes a true squared metric between occupancy measures, enabling exact expert imitation.
Knowl 5 — Equivalence of Apprenticeship Learning to Indicator Cost-Regularized IRL
theoretical result
Apprenticeship learning finds a policy $\pi$ that outperforms the expert $\pi_E$ across a convex cost class $\mathcal{C} \subset \mathbb{R}^{\mathcal{S} \times \mathcal{A}}$ by optimizing $\min_\pi \max_{c \in \mathcal{C}} \mathbb{E}_\pi[c(s, a)] - \mathbb{E}_{\pi_E}[c(s, a)]$ .

With causal entropy regularization $-H(\pi)$ and the indicator regularizer $\delta_{\mathcal{C}}(c)$ ( $0$ if $c \in \mathcal{C}$ , $+\infty$ otherwise), the apprenticeship learning objective becomes: $\min_\pi -H(\pi) + \max_{c \in \mathbb{R}^{\mathcal{S} \times \mathcal{A}}} \left( -\delta_{\mathcal{C}}(c) + \sum_{s, a} (\rho_\pi(s, a) - \rho_{\pi_E}(s, a))c(s, a) \right) = \min_\pi -H(\pi) + \delta_{\mathcal{C}}^*(\rho_\pi - \rho_{\pi_E})$ where $\delta_{\mathcal{C}}^*$ is the convex conjugate of $\delta_{\mathcal{C}}$ .

This establishes that entropy-regularized apprenticeship learning is equivalent to running RL following maximum causal entropy IRL with cost regularizer $\psi = \delta_{\mathcal{C}}$ . Consequently:
1. $\mathcal{C}_{\text{linear}} = \{\sum_i w_i f_i : \|w\|_2 \le 1\}$ corresponds to feature expectation matching, which minimizes $\ell_2$ distance $\|\mathbb{E}_\pi[f(s, a)] - \mathbb{E}_{\pi_E}[f(s, a)]\|_2$ .
2. $\mathcal{C}_{\text{convex}} = \{\sum_i w_i f_i : \sum_i w_i = 1, w_i \ge 0\}$ corresponds to Multiplicative Weights Apprenticeship Learning (MWAL) and Linear Programming Apprenticeship Learning (LPAL), minimizing worst-case basis cost $\max_i \mathbb{E}_\pi[f_i(s, a)] - \mathbb{E}_{\pi_E}[f_i(s, a)]$ .
3. Restricting costs to linear combinations of basis functions forces $\pi_E$ to be encoded in a small subspace; if $\mathcal{C}$ does not contain a cost function explaining expert behavior, apprenticeship learning fails to imitate the expert.
Knowl 6 — Cost Regularizer Construction from Decreasing Convex Surrogate Loss Functions
theoretical result
Let $\phi: \mathbb{R} \to \mathbb{R}$ be a strictly decreasing convex surrogate loss function for binary classification, and let $T$ be the range of $-\phi$ . Define $g_\phi: \mathbb{R} \to \bar{\mathbb{R}}$ and $\psi_\phi: \mathbb{R}^{\mathcal{S} \times \mathcal{A}} \to \bar{\mathbb{R}}$ by: $g_\phi(x) = \begin{cases} -x + \phi(-\phi^{-1}(-x)) & \text{if } x \in T \\ +\infty & \text{otherwise} \end{cases}$ $\psi_\phi(c) = \begin{cases} \sum_{s, a} \rho_{\pi_E}(s, a) g_\phi(c(s, a)) & \text{if } c(s, a) \in T \quad \forall s, a \\ +\infty & \text{otherwise} \end{cases}$

Then:
1. $\psi_\phi$ is closed, proper, and convex.
2. Its convex conjugate satisfies $\psi_\phi^*(\rho_\pi - \rho_{\pi_E}) = -R_\phi(\rho_\pi, \rho_{\pi_E})$ , where $R_\phi$ is the minimum expected binary classification risk: $R_\phi(\rho_\pi, \rho_{\pi_E}) = \sum_{s, a} \min_{\gamma \in \mathbb{R}} \left( \rho_\pi(s, a)\phi(\gamma) + \rho_{\pi_E}(s, a)\phi(-\gamma) \right)$
3. Reinforcement learning on the output of $\psi_\phi$ -regularized IRL directly minimizes the entropy-regularized surrogate risk: $\text{RL} \circ \text{IRL}_{\psi_\phi}(\pi_E) = \arg\min_\pi -H(\pi) - R_\phi(\rho_\pi, \rho_{\pi_E})$
Because minimum expected risks for convex surrogate loss functions correspond to $f$ -divergences, this result provides a general procedure to construct imitation learning algorithms minimizing arbitrary $f$ -divergences between occupancy measures.
Knowl 7 — Policy Gradient Formula for Causal Entropy
theoretical result
For a parameterized stochastic policy $\pi_\theta(a \mid s)$ operating in a discounted Markov Decision Process with discount factor $\gamma$ , the gradient of the causal entropy $H(\pi_\theta) = \mathbb{E}_{\pi_\theta}[-\log \pi_\theta(a \mid s)]$ with respect to policy parameters $\theta$ is: $\nabla_\theta \mathbb{E}_{\pi_\theta}[-\log \pi_\theta(a \mid s)] = \mathbb{E}_{\pi_\theta}\left[ \nabla_\theta \log \pi_\theta(a \mid s) Q_{\log}(s, a) \right]$ where $Q_{\log}(\bar{s}, \bar{a})$ is the expected cumulative negative log-policy cost: $Q_{\log}(\bar{s}, \bar{a}) = \mathbb{E}_{\pi_\theta}\left[ \sum_{t=0}^\infty \gamma^t (-\log \pi_\theta(a_t \mid s_t)) \,\middle|\, s_0 = \bar{s}, a_0 = \bar{a} \right]$

This demonstrates that the causal entropy policy gradient matches the standard reinforcement learning policy gradient evaluated with the fixed cost function $c_{\log}(s, a) = -\log \pi_\theta(a \mid s)$ .

Knowl 8 — Empirical Benchmark Performance Across Continuous Control Tasks

data/table

GAIL was evaluated against Behavioral Cloning (BC), Feature Expectation Matching (FEM), and Game-Theoretic Apprenticeship Learning (GTAL) across 9 physics-based control tasks in OpenAI Gym and MuJoCo over varying numbers of expert trajectories (each consisting of $\approx 50$ state-action pairs).

The table reports performance (mean $\pm$ standard deviation over 50 trajectories; Cartpole, Mountain Car, Acrobot, and Reacher further averaged over 7 random initializations):

Task	Trajectories	Behavioral Cloning	FEM	GTAL	GAIL (Ours)
Cartpole-v0	1	$72.02 \pm 35.82$	$200.00 \pm 0.00$	$200.00 \pm 0.00$	$200.00 \pm 0.00$
	10	$177.19 \pm 52.83$	$199.75 \pm 3.50$	$200.00 \pm 0.00$	$200.00 \pm 0.00$
Acrobot-v0	1	$-130.60 \pm 55.08$	$-133.14 \pm 60.80$	$-81.35 \pm 22.40$	$-77.26 \pm 18.03$
	10	$-95.09 \pm 33.33$	$-77.22 \pm 18.51$	$-94.32 \pm 46.51$	$-78.91 \pm 15.76$
Mountain Car-v0	1	$-136.76 \pm 34.44$	$-100.97 \pm 12.54$	$-115.48 \pm 36.35$	$-101.55 \pm 10.32$
	10	$-123.14 \pm 28.26$	$-100.48 \pm 8.14$	$-120.05 \pm 36.66$	$-100.83 \pm 11.40$
HalfCheetah-v1	4	$-493.62 \pm 246.58$	$734.01 \pm 84.59$	$1008.14 \pm 280.42$	$4515.70 \pm 549.49$
	25	$3718.58 \pm 1856.22$	$502.29 \pm 375.78$	$869.55 \pm 447.90$	$4840.07 \pm 95.36$
Hopper-v1	4	$50.57 \pm 0.95$	$3571.98 \pm 6.35$	$3065.21 \pm 147.79$	$3614.22 \pm 7.17$
	25	$3383.96 \pm 657.61$	$3331.05 \pm 3.55$	$3458.82 \pm 5.40$	$3560.85 \pm 3.09$
Walker-v1	4	$32.18 \pm 1.25$	$3648.17 \pm 327.41$	$4945.90 \pm 65.97$	$4877.98 \pm 2848.37$
	25	$1599.36 \pm 1456.59$	$4368.15 \pm 267.17$	$4687.80 \pm 186.22$	$6832.01 \pm 254.64$
Ant-v1	4	$1611.75 \pm 359.54$	$-2052.51 \pm 49.41$	$-5743.81 \pm 723.48$	$3186.80 \pm 903.57$
	25	$3235.73 \pm 1186.38$	$-5122.12 \pm 703.19$	$-3271.37 \pm 226.66$	$4132.90 \pm 878.67$
Humanoid-v1	80	$1397.06 \pm 1057.84$	$5093.12 \pm 583.11$	$5096.43 \pm 24.96$	$10200.73 \pm 1324.47$
	240	$5660.53 \pm 3600.70$	$5192.34 \pm 24.59$	$5145.94 \pm 21.13$	$10361.94 \pm 61.28$

Reference expert returns are: Cartpole $200.00 \pm 0.00$ , Acrobot $-75.25 \pm 10.94$ , Mountain Car $-98.75 \pm 8.71$ , HalfCheetah $4463.46 \pm 105.83$ , Hopper $3571.38 \pm 184.20$ , Walker $6717.08 \pm 845.62$ , Ant $4228.37 \pm 424.16$ , and Humanoid $9575.40 \pm 1750.80$ .

GAIL achieves full expert-level performance on complex high-dimensional benchmarks (HalfCheetah, Hopper, Walker, Ant, Humanoid) across all tested dataset sizes, whereas FEM and GTAL collapse on Ant-v1 (performing worse than random policies) and BC fails to exceed 60% of expert performance on Humanoid-v1.

Knowl 9 — Sample Complexity Limitations of GAIL
limitation
While Generative Adversarial Imitation Learning is sample-efficient with respect to expert demonstration trajectories (matching expert performance with only 4 to 25 trajectories on most tasks), it exhibits high sample complexity with respect to environment interaction during training.

Key constraints:
1. As a model-free method, estimating the policy gradient and discriminator expectations requires extensive environment simulation—typically 5,000 to 50,000 state-action pairs per iteration over 200 to 1,500 iterations (amounting to up to $7.5 \times 10^7$ state-action samples for Humanoid-v1).
2. The total amount of environment interaction required is comparable to training an expert policy from scratch using TRPO with ground-truth reinforcement signals.
3. The learner does not interact with or query the expert during training (unlike interactive imitation learning methods such as DAgger), relying entirely on undirected exploration in the environment to align occupancy measures.

Coverage note — None was omitted; all key theoretical proofs (characterization, duality, surrogate loss construction, entropy gradient), algorithms, empirical benchmarks, and limitations are fully covered.

References

1.P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the 21st International Conference on Machine Learning, 2004.
2.A. G. Barto, R. S. Sutton, and C. W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. Systems, Man and Cybernetics, IEEE Transactions on, (5):834–846, 1983.
3.M. Bloem and N. Bambos. Infinite time horizon maximum causal entropy inverse reinforcement learning. In Decision and Control (CDC), 2014 IEEE 53rd Annual Conference on, pages 4911–4916. IEEE, 2014.
4.S. Boyd and L. Vandenberghe. Convex optimization. Cambridge university press, 2004.
5.G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI Gym. arXiv preprint arXiv:1606.01540, 2016.
6.T. M. Cover and J. A. Thomas. Elements of information theory. John Wiley & Sons, 2012.
7.C. Finn, S. Levine, and P. Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In Proceedings of the 33rd International Conference on Machine Learning, 2016.
8.A. Geramifard, C. Dann, R. H. Klein, W. Dabney, and J. P. How. Rlpy: A value-function-based reinforcement learning framework for education and research. JMLR, 2015.
9.I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NIPS, pages 2672–2680, 2014.
10.J.-B. Hiriart-Urruty and C. Lemaréchal. Convex Analysis and Minimization Algorithms, volume 305. Springer, 1996.
11.J. Ho, J. K. Gupta, and S. Ermon. Model-free imitation learning with policy optimization. In Proceedings of the 33rd International Conference on Machine Learning, 2016.
12.D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
13.S. Levine and P. Abbeel. Learning neural network policies with guided policy search under unknown dynamics. In Advances in Neural Information Processing Systems, pages 1071–1079, 2014.
14.S. Levine and V. Koltun. Continuous inverse optimal control with locally optimal examples. In Proceedings of the 29th International Conference on Machine Learning, pages 41–48, 2012.
15.S. Levine, Z. Popovic, and V. Koltun. Nonlinear inverse reinforcement learning with gaussian processes. In Advances in Neural Information Processing Systems, pages 19–27, 2011.
16.P. W. Millar. The minimax principle in asymptotic statistical theory. In Ecole d’Eté de Probabilités de Saint-Flour XI—1981, pages 75–265. Springer, 1983.
17.A. W. Moore and T. Hall. Efficient memory-based learning for robot control. 1990.
18.A. Y. Ng and S. Russell. Algorithms for inverse reinforcement learning. In ICML, 2000.
19.X. Nguyen, M. J. Wainwright, and M. I. Jordan. On surrogate loss functions and f-divergences. The Annals of Statistics, pages 876–904, 2009.
20.D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88–97, 1991.
21.M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
22.N. D. Ratliff, D. Silver, and J. A. Bagnell. Learning to search: Functional gradient techniques for imitation learning. Autonomous Robots, 27(1):25–53, 2009.
23.S. Ross and D. Bagnell. Efficient reductions for imitation learning. In AISTATS, pages 661–668, 2010.
24.S. Ross, G. J. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS, pages 627–635, 2011.
25.S. Russell. Learning agents for uncertain environments. In Proceedings of the Eleventh Annual Conference on Computational Learning Theory, pages 101–103. ACM, 1998.
26.J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In Proceedings of The 32nd International Conference on Machine Learning, pages 1889–1897, 2015.
27.J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.
28.U. Syed and R. E. Schapire. A game-theoretic approach to apprenticeship learning. In Advances in Neural Information Processing Systems, pages 1449–1456, 2007.
29.U. Syed, M. Bowling, and R. E. Schapire. Apprenticeship learning using linear programming. In Proceedings of the 25th International Conference on Machine Learning, pages 1032–1039, 2008.
30.E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pages 5026–5033. IEEE, 2012.
31.B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In AAAI, AAAI’08, 2008.
32.B. D. Ziebart, J. A. Bagnell, and A. K. Dey. Modeling interaction via the principle of maximum causal entropy. In ICML, pages 1255–1262, 2010.

Citation

MLA

Ho, J., and S. Ermon. “Generative Adversarial Imitation Learning”. arXiv, 2016, http://arxiv.org/abs/1606.03476v1.

APA

Ho, J., & Ermon, S. (2016). Generative Adversarial Imitation Learning. arXiv. http://arxiv.org/abs/1606.03476v1

Chicago

Ho, J., and S. Ermon. 2016. “Generative Adversarial Imitation Learning”. arXiv. http://arxiv.org/abs/1606.03476v1.

Harvard

Ho, J. and Ermon, S. (2016) “Generative Adversarial Imitation Learning”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1606.03476v1.

Vancouver

1. Ho J, Ermon S (2016) Generative Adversarial Imitation Learning. arXiv

BibTeX

@article{ho2016generative,
  title = {Generative Adversarial Imitation Learning},
  author = {Ho, Jonathan and Ermon, Stefano},
  year = {2016},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1606.03476v1},
  eprint = {1606.03476}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Published with permission

Generative Adversarial Imitation Learning

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Generative Adversarial Imitation Learning Algorithm

Knowl 2 — Induced Policy as Convex Conjugate of Cost Regularizer in Maximum Causal Entropy IRL

Knowl 3 — Duality Between Unregularized IRL and Exact Occupancy Measure Matching

Knowl 4 — GAIL Cost Regularizer and Occupancy Measure Jensen-Shannon Divergence

Knowl 5 — Equivalence of Apprenticeship Learning to Indicator Cost-Regularized IRL

Knowl 6 — Cost Regularizer Construction from Decreasing Convex Surrogate Loss Functions