Constrained Policy Optimization

Joshua AchiamDavid HeldAviv TamarPieter Abbeel

article2017ICML2,039 citations

Introduces Constrained Policy Optimization (CPO), the first general-purpose policy search algorithm for constrained reinforcement learning that provides theoretical guarantees of constraint satisfaction throughout the entire training process in high-dimensional control tasks.

Listen

Modern reinforcement learning enables complex autonomous behaviors in high-dimensional tasks, but standard approaches grant agents complete freedom to explore actions via trial and error. In high-stakes settings such as industrial robotics and physical human-robot interaction, unconstrained exploration can lead to equipment damage, plant destruction, or severe safety hazards to personnel. Incorporating formal safety constraints is therefore essential. The article addresses this challenge by designing an optimization method that enables neural network controllers to maximize mission rewards while consistently enforcing auxiliary safety constraints throughout the entire training lifecycle.

The objective of the article is to develop and evaluate Constrained Policy Optimization (CPO), the first general-purpose policy search algorithm for constrained reinforcement learning that provides theoretical guarantees for near-constraint satisfaction at every policy update during training. The researchers set out to demonstrate that this framework can effectively balance performance and safety across high-dimensional, continuous simulated robotic control domains.

The researchers established a novel theoretical bound connecting policy return differences to average distribution divergences, which formally justifies using surrogate objectives within a local trust region framework. They then translated this theory into a scalable computational approach by locally linearizing objectives and cost constraints while taking a second-order approximation of the step-size limit. The method was evaluated using simulated locomotion tasks featuring three agents of increasing structural complexity: a point-mass, a quadruped ant robot, and a high-dimensional humanoid. Tasks included navigating circular areas while respecting planar boundary constraints and collecting target items while avoiding hazardous bombs.

The experimental findings show that CPO successfully drives constraint returns directly to specified safety thresholds across all tested robotic systems without sacrificing reward optimization. In head-to-head comparisons, CPO consistently outperformed standard primal-dual optimization, which exhibited unstable spikes in constraint violations during training and proved fragile with respect to hyperparameter tuning and dual variable initialization. Additionally, ablation analyses revealed that augmenting safety constraints with cost shaping—specifically by penalizing the predicted short-term probability of entering an unsafe state—almost completely mitigated practical approximation errors. Finally, standard fixed-penalty methods proved unworkable, as slight shifts in penalty weights resulted in either complete disregard for constraints or overly conservative agents that failed to learn any useful task behavior.

These results demonstrate that formal constrained optimization provides a robust, principled foundation for safe machine learning in continuous control domains. By dynamically calculating constraint enforcement parameters at every update step rather than relying on brittle manual tuning or delayed feedback, the framework minimizes operational risks, prevents policy degradation, and adheres to safety boundaries throughout training. This marks a critical step toward deploying automated learning systems in real-world environments governed by strict physical and operational constraints.

Organizations evaluating automated physical systems should consider constrained trust-region frameworks over heuristic or fixed-penalty reward tuning when safety bounds are explicit. For deployment, engineering teams should incorporate auxiliary safety models to shape costs around boundary regions and smooth sparse failure signals. Further work is required before directly executing this method on physical hardware: research must bridge the simulation-to-reality gap, evaluate multi-constraint scenarios beyond single-constraint benchmarks, and mitigate the small residual constraint violations that stem from sample approximation and linear estimation errors.

OpenAI University of California Berkeley general purpose neural networks optimization constrained policy optimization policy gradient safety constraints

arXiv: 1705.10528 jachiam/cpo

Paper: Trust Region Policy Optimization, John Schulman et al. (2015). Trust Region Policy Optimization establishes the theoretical foundation and trust-region machinery that Constrained Policy Optimization directly extends to accommodate safety constraints.
Paper: A comprehensive survey on safe reinforcement learning, Javier García et al. (2015). This comprehensive survey outlines the taxonomy and fundamental challenges of safe reinforcement learning that motivate CPO's constrained policy search formulation.
Paper: High-Dimensional Continuous Control Using Generalized Advantage Estimation, John Schulman et al. (2016). Generalized Advantage Estimation provides the essential low-variance advantage estimation framework utilized by CPO during policy and constraint updates.
Paper: Concrete Problems in AI Safety, Dario Amodei et al. (2016). This seminal paper frames key AI safety problems and demonstrates why explicit constraint satisfaction is critical during reinforcement learning exploration.
Paper: Policy Gradient Methods for Reinforcement Learning with Function Approximation, Richard S. Sutton et al. (1999). This foundational work derives the policy gradient theorem that underlies all policy optimization and actor-critic methods used in CPO.
Paper: Actor-Critic Algorithms, Vijay R. Konda et al. (1999). This paper establishes the theoretical convergence and two-time-scale architecture of actor-critic algorithms upon which modern continuous policy search methods build.

Paper: Proximal Policy Optimization Algorithms, John Schulman et al. (2017). Proximal Policy Optimization presents a simpler, first-order clipped surrogate alternative to trust-region policy search methods like TRPO and CPO.
Paper: Gradient Surgery for Multi-Task Learning, Tianhe Yu et al. (2020). This work introduces gradient projection techniques to resolve conflicting objectives in multi-task optimization, offering an alternative mechanism for handling trade-offs in policy search.
Paper: Stable-Baselines3: Reliable Reinforcement Learning Implementations, A. Raffin et al. (2021). Stable-Baselines3 offers standardized, reliable implementations of modern deep reinforcement learning algorithms that build upon the trust-region and policy gradient paradigms.

Cover for Constrained Policy Optimization

Abstract

For many applications of reinforcement learning it can be more convenient to specify both a reward function and constraints, rather than trying to design behavior through the reward function. For example, systems that physically interact with or around humans should satisfy safety constraints. Recent advances in policy search algorithms (Mnih et al., 2016, Schulman et al., 2015, Lillicrap et al., 2016, Levine et al., 2016) have enabled new capabilities in high-dimensional control, but do not consider the constrained setting.

We propose Constrained Policy Optimization (CPO), the first general-purpose policy search algorithm for constrained reinforcement learning with guarantees for near-constraint satisfaction at each iteration. Our method allows us to train neural network policies for high-dimensional control while making guarantees about policy behavior all throughout training. Our guarantees are based on a new theoretical result, which is of independent interest: we prove a bound relating the expected returns of two policies to an average divergence between them. We demonstrate the effectiveness of our approach on simulated robot locomotion tasks where the agent must satisfy constraints motivated by safety.

Topics

general purpose neural networks optimization optimization problems reward function theoretical result expected return reinforcement learning method robotic arms

Keywords

constrained policy optimization policy gradient safety constraints primal-dual optimization policy divergence constraint costs deep reinforcement learning

1 Introduction
2 Related Work
3 Preliminaries
4 Constrained Markov Decision Processes
5 Constrained Policy Optimization
5.1 Policy Performance Bounds
5.2 Trust Region Methods
5.3 Trust Region Optimization for Constrained MDPs
6 Practical Implementation
6.1 Approximately Solving the CPO Update
6.2 Feasibility
6.3 Tightening Constraints via Cost Shaping
7 Connections to Prior Work
8 Experiments
8.1 Evaluating CPO and Comparison Analysis
8.2 Ablation on Cost Shaping
8.3 Constraint vs. Fixed Penalty
9 Discussion
References
10 Appendix
10.1 Proof of Policy Performance Bound
10.1.1 Preliminaries
10.1.2 Main Results
10.2 Proof of Analytical Solution to LQCLP
10.3 Experimental Parameters
10.3.1 Environments
10.3.2 Algorithm Parameters
10.3.3 Primal-Dual Optimization Implementation

Knowls

Knowl 1 — Policy Performance Difference Bounds via Average Action Divergence
theoretical result
Let $M = (\mathcal{S}, \mathcal{A}, R, P, \mu, \gamma)$ be a Markov Decision Process with state space $\mathcal{S}$ , action space $\mathcal{A}$ , reward function $R(s, a, s')$ , transition distribution $P(s' \mid s, a)$ , initial state distribution $\mu$ , and discount factor $\gamma \in [0, 1)$ . Let $J(\pi) = \mathbb{E}_{\tau \sim \pi}[\sum_{t=0}^\infty \gamma^t R(s_t, a_t, s_{t+1})]$ denote the expected discounted return of a stationary policy $\pi$ , and let $d^\pi(s) = (1 - \gamma) \sum_{t=0}^\infty \gamma^t P(s_t = s \mid \pi)$ denote the discounted future state visitation distribution.

For any reference function $f: \mathcal{S} \to \mathbb{R}$ and any two policies $\pi$ and $\pi'$ , define the generalized temporal-difference discrepancy $\delta_f(s, a, s') = R(s, a, s') + \gamma f(s') - f(s)$ , the maximum expected discrepancy $\epsilon^{\pi'}_f = \max_{s} |\mathbb{E}_{a \sim \pi'(\cdot \mid s), s' \sim P(\cdot \mid s, a)}[\delta_f(s, a, s')]|$ , and the surrogate discrepancy:

$L_{\pi, f}(\pi') = \mathbb{E}_{s \sim d^\pi, a \sim \pi, s' \sim P} \left[ \left( \frac{\pi'(a \mid s)}{\pi(a \mid s)} - 1 \right) \delta_f(s, a, s') \right]$

The policy performance difference is bounded above and below by:

$D^+_{\pi, f}(\pi') \ge J(\pi') - J(\pi) \ge D^-_{\pi, f}(\pi')$

where

$D^\pm_{\pi, f}(\pi') = \frac{L_{\pi, f}(\pi')}{1 - \gamma} \pm \frac{2\gamma \epsilon^{\pi'}_f}{(1 - \gamma)^2} \mathbb{E}_{s \sim d^\pi} [D_{TV}(\pi' \| \pi)[s]]$

and $D_{TV}(\pi' \| \pi)[s] = \frac{1}{2} \sum_{a \in \mathcal{A}} |\pi'(a \mid s) - \pi(a \mid s)|$ is the total variation divergence at state $s$ . When $\pi' = \pi$ , the upper and lower bounds both equal zero identically.

Setting $f = V^\pi(s)$ yields the performance lower bound:

$J(\pi') - J(\pi) \ge \frac{1}{1 - \gamma} \mathbb{E}_{s \sim d^\pi, a \sim \pi'} \left[ A^\pi(s, a) \right] - \frac{2\gamma \epsilon^{\pi'}}{(1 - \gamma)^2} \mathbb{E}_{s \sim d^\pi} [D_{TV}(\pi' \| \pi)[s]]$

where $A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s)$ is the advantage function and $\epsilon^{\pi'} = \max_s |\mathbb{E}_{a \sim \pi'}[A^\pi(s, a)]|$ .

Similarly, for an auxiliary cost function $C_i(s, a, s')$ with expected discounted return $J_{C_i}(\pi)$ , setting $f = V^\pi_{C_i}$ yields the cost upper bound:

$J_{C_i}(\pi') - J_{C_i}(\pi) \le \frac{1}{1 - \gamma} \mathbb{E}_{s \sim d^\pi, a \sim \pi'} \left[ A^\pi_{C_i}(s, a) \right] + \frac{2\gamma \epsilon^{\pi'}_{C_i}}{(1 - \gamma)^2} \mathbb{E}_{s \sim d^\pi} [D_{TV}(\pi' \| \pi)[s]]$

where $\epsilon^{\pi'}_{C_i} = \max_s |\mathbb{E}_{a \sim \pi'}[A^\pi_{C_i}(s, a)]|$ . Applying Pinsker's inequality and Jensen's inequality allows substituting $\mathbb{E}_{s \sim d^\pi}[D_{TV}(\pi' \| \pi)[s]] \le \sqrt{\frac{1}{2} \bar{D}_{KL}(\pi' \| \pi)}$ , where $\bar{D}_{KL}(\pi' \| \pi) = \mathbb{E}_{s \sim d^\pi}[D_{KL}(\pi'(\cdot \mid s) \| \pi(\cdot \mid s))]$ .
Knowl 2 — Constrained Policy Optimization Trust Region Formulation and Worst-Case Violation Bound
model/method
In a Constrained Markov Decision Process (CMDP) with auxiliary cost functions $C_1, \dots, C_m$ and limits $d_1, \dots, d_m$ , the goal is to find $\pi^* = \arg\max_{\pi \in \Pi_C} J(\pi)$ subject to $J_{C_i}(\pi) \le d_i$ for each $i \in \{1, \dots, m\}$ .

Constrained Policy Optimization (CPO) optimizes parametrized policies $\pi_\theta \in \Pi_\theta$ using a trust region update at step $k$ that replaces off-policy constraint evaluation with on-policy surrogate expectations under the current state visitation distribution $d^{\pi_k}$ :

$\pi_{k+1} = \arg\max_{\pi \in \Pi_\theta} \mathbb{E}_{s \sim d^{\pi_k}, a \sim \pi} \left[ A^{\pi_k}(s, a) \right]$

$\text{s.t. } J_{C_i}(\pi_k) + \frac{1}{1 - \gamma} \mathbb{E}_{s \sim d^{\pi_k}, a \sim \pi} \left[ A^{\pi_k}_{C_i}(s, a) \right] \le d_i, \quad \forall i = 1, \dots, m$

$\bar{D}_{KL}(\pi \| \pi_k) \le \delta$

where $\bar{D}_{KL}(\pi \| \pi_k) = \mathbb{E}_{s \sim d^{\pi_k}}[D_{KL}(\pi(\cdot \mid s) \| \pi_k(\cdot \mid s))]$ and $\delta > 0$ is the trust region radius.

For any set of policies $\Pi_\theta$ containing $\pi_k$ , every iterate $\pi_{k+1}$ generated by this update satisfies the worst-case constraint violation bound:

$J_{C_i}(\pi_{k+1}) \le d_i + \frac{\sqrt{2\delta}\gamma \epsilon^{\pi_{k+1}}_{C_i}}{(1 - \gamma)^2}$

where $\epsilon^{\pi_{k+1}}_{C_i} = \max_s |\mathbb{E}_{a \sim \pi_{k+1}}[A^{\pi_k}_{C_i}(s, a)]|$ .
Knowl 3 — Constrained Policy Optimization Algorithm
algorithm
Constrained Policy Optimization (CPO) solves the trust region CMDP update by locally linearizing the objective and cost constraints and approximating the average KL divergence with its second-order Taylor expansion around parameter vector $\theta_k$ .
Input: Initial policy parameters $\theta_0 \in \mathbb{R}^n$ , trust region step size $\delta > 0$ , constraint bounds $d_1, \dots, d_m$ , line search backtrack budget $J$ , backtrack decay $s \in (0, 1)$
for $k = 0, 1, 2, \dots$ do
    Sample trajectory batch $\mathcal{D} = \{\tau\}$ from policy $\pi_{\theta_k}$
    Estimate objective gradient $g = \nabla_\theta \mathbb{E}_{s \sim d^{\pi_k}, a \sim \pi_\theta}[A^{\pi_k}(s, a)] \vert_{\theta = \theta_k}$
    For each constraint $i \in \{1, \dots, m\}$ :
        Estimate constraint gradient $b_i = \nabla_\theta \frac{1}{1-\gamma} \mathbb{E}_{s \sim d^{\pi_k}, a \sim \pi_\theta}[A_{C_i}^{\pi_k}(s, a)] \vert_{\theta = \theta_k}$
        Compute current constraint margin $c_i = J_{C_i}(\pi_k) - d_i$
    Form matrix $B = [b_1, \dots, b_m]$ and vector $c = [c_1, \dots, c_m]^T$
    Estimate Fisher Information Matrix $H = \nabla_\theta^2 \bar{D}_{KL}(\pi_\theta \| \pi_{\theta_k}) \vert_{\theta = \theta_k}$
    Compute $H^{-1}g$ and $H^{-1}b_i$ using conjugate gradient
    if linearized primal problem $\max_\theta g^T(\theta - \theta_k)$ s.t. $c + B^T(\theta - \theta_k) \le 0, \frac{1}{2}(\theta - \theta_k)^T H (\theta - \theta_k) \le \delta$ is feasible then
        Solve dual program $\max_{\lambda \ge 0, \nu \ge 0} -\frac{1}{2\lambda}(g^T H^{-1} g - 2 r^T \nu + \nu^T S \nu) + \nu^T c - \frac{\lambda \delta}{2}$ for $\lambda^*, \nu^*$ , where $r = g^T H^{-1} B$ and $S = B^T H^{-1} B$
        Compute search direction proposal $\Delta \theta = \frac{1}{\lambda^*} H^{-1}(g - B \nu^*)$
    else
        Compute recovery search direction to purely decrease constraint value: $\Delta \theta = -\sqrt{\frac{2\delta}{B^T H^{-1} B}} H^{-1} B$
    end if
    Find smallest integer $j \in \{0, 1, \dots, J\}$ such that $\theta = \theta_k + s^j \Delta \theta$ satisfies sample surrogate constraints and $\bar{D}_{KL}(\pi_\theta \| \pi_{\theta_k}) \le \delta$
    Set $\theta_{k+1} = \theta_k + s^j \Delta \theta$
end for
The algorithm avoids explicit $n \times n$ matrix inversion of the Fisher Information Matrix $H$ by employing conjugate gradient steps on Fisher-vector products. For a single constraint ( $m = 1$ ), the dual parameters $\lambda^*$ and $\nu^*$ are obtained analytically without an inner-loop numerical solver.
Knowl 4 — Analytical Solution to Linear-Quadratic Constrained Linear Program
theoretical result
Consider the optimization problem with linear objective, one linear inequality constraint, and one convex quadratic constraint:

$p^* = \min_{x \in \mathbb{R}^n} g^T x \quad \text{s.t.} \quad b^T x + c \le 0, \quad x^T H x \le \delta$

where $g, b \in \mathbb{R}^n$ , $c, \delta \in \mathbb{R}$ , $\delta > 0$ , and $H \in \mathbb{R}^{n \times n}$ is symmetric positive definite ( $H \succ 0$ ). Let $q = g^T H^{-1} g$ , $r = g^T H^{-1} b$ , and $s = b^T H^{-1} b$ .
1. Cauchy-Schwarz Property: $qs \ge r^2$ , guaranteeing $q - r^2/s \ge 0$ .
2. Geometric Feasibility Check: The unconstrained minimum of $x^T H x$ $x^{T} H x$ on the hyperplane $b^T x + c = 0$ $b^{T} x + c = 0$ is $x^* = c H^{-1} b / s$ $x^{*} = c H^{- 1} b / s$ , with norm $x^{*T} H x^* = c^2/s$ $x^{* T} H x^{*} = c^{2} / s$ .
  - If $c^2/s > \delta$ and $c > 0$ , the constraint halfspace and the ellipsoid do not intersect, and the problem is infeasible.
  - If $c^2/s > \delta$ and $c < 0$ , the quadratic trust region lies strictly within the feasible halfspace $b^T x + c \le 0$ , so the linear constraint is inactive.
3. Optimal Primal Point: When strictly feasible, strong duality holds and the optimal point is:
$x^* = -\frac{1}{\lambda^*} H^{-1}(g + \nu^* b)$

where

$\nu^* = \left( \frac{\lambda^* c - r}{s} \right)_+ = \max\left(0, \frac{\lambda^* c - r}{s}\right)$

and $\lambda^*$ is found by maximizing the piecewise dual over intervals $\Lambda_a = \{\lambda \ge 0 : \lambda c - r > 0\}$ and $\Lambda_b = \{\lambda \ge 0 : \lambda c - r \le 0\}$ :

$\lambda_a^* = \text{Proj}\left( \sqrt{\frac{q - r^2/s}{\delta - c^2/s}}, \Lambda_a \right), \quad \lambda_b^* = \text{Proj}\left( \sqrt{\frac{q}{\delta}}, \Lambda_b \right)$

$\lambda^* = \begin{cases} \lambda_a^* & \text{if } f_a(\lambda_a^*) \ge f_b(\lambda_b^*) \\ \lambda_b^* & \text{otherwise} \end{cases}$

where $f_a(\lambda) = \frac{1}{2\lambda}(r^2/s - q) + \frac{\lambda}{2}(c^2/s - \delta) - \frac{rc}{s}$ , $f_b(\lambda) = -\frac{1}{2}(q/\lambda + \lambda \delta)$ , and $\text{Proj}(x, [a, b]) = \max(a, \min(b, x))$ .
Knowl 5 — State Visitation Distribution Divergence Bound under Policy Shift
theoretical result
Let $M = (\mathcal{S}, \mathcal{A}, R, P, \mu, \gamma)$ be an MDP with finite state and action spaces and discount factor $\gamma \in [0, 1)$ . For any two stationary policies $\pi$ and $\pi'$ , let $d^\pi(s) = (1 - \gamma) \sum_{t=0}^\infty \gamma^t P(s_t = s \mid \pi)$ denote the discounted future state visitation distribution.

The $L_1$ total variation divergence between the state visitation distributions $d^{\pi'}$ and $d^\pi$ is bounded by the expected total variation divergence between the policies averaged over the baseline state distribution $d^\pi$ :

$\|d^{\pi'} - d^\pi\|_1 \le \frac{2\gamma}{1 - \gamma} \mathbb{E}_{s \sim d^\pi} [D_{TV}(\pi' \| \pi)[s]]$

where $D_{TV}(\pi' \| \pi)[s] = \frac{1}{2} \sum_{a \in \mathcal{A}} |\pi'(a \mid s) - \pi(a \mid s)|$ .

This bound tightens prior results that depended on $\max_s D_{TV}(\pi' \| \pi)[s]$ , allowing policy performance differences to be bounded by expectations easily estimated from on-policy samples drawn from $d^\pi$ .
Knowl 6 — Worst-Case Performance Degradation Bound for Trust Region Policy Updates
theoretical result
Let $\pi_k, \pi_{k+1} \in \Pi_\theta$ be successive policy iterates related by the trust region update:

$\pi_{k+1} = \arg\max_{\pi \in \Pi_\theta} \mathbb{E}_{s \sim d^{\pi_k}, a \sim \pi} \left[ A^{\pi_k}(s, a) \right] \quad \text{s.t.} \quad \bar{D}_{KL}(\pi \| \pi_k) \le \delta$

where $\bar{D}_{KL}(\pi \| \pi_k) = \mathbb{E}_{s \sim d^{\pi_k}}[D_{KL}(\pi(\cdot \mid s) \| \pi_k(\cdot \mid s))]$ and $\delta > 0$ is the step size.

The expected return difference between $\pi_{k+1}$ and $\pi_k$ is lower bounded by:

$J(\pi_{k+1}) - J(\pi_k) \ge -\frac{\sqrt{2\delta}\gamma \epsilon^{\pi_{k+1}}}{(1 - \gamma)^2}$

where $\epsilon^{\pi_{k+1}} = \max_s |\mathbb{E}_{a \sim \pi_{k+1}}[A^{\pi_k}(s, a)]|$ and $\gamma \in [0, 1)$ is the discount factor. This result guarantees that the worst-case degradation of a trust region policy step is strictly bounded by a function proportional to $\sqrt{\delta}$ .
Knowl 7 — Constraint Tightening via Learned Cost Shaping
model/method
To counteract policy approximation errors, finite sample estimation errors, and theoretical surrogate slackness, constraints in Constrained Policy Optimization (CPO) are tightened using cost shaping:

$C_i^+(s, a, s') = C_i(s, a, s') + \alpha \Delta_i(s, a, s')$

where $C_i(s, a, s')$ is the original auxiliary cost, $\Delta_i: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to \mathbb{R}_+$ is an auxiliary penalty, and $\alpha > 0$ is a scaling coefficient.

In environments with binary safety indicators ( $C(s) = 1$ in unsafe states $U$ and $0$ otherwise), $\Delta(s, a, s')$ is defined as the probability that the agent will enter an unsafe state within a lookahead horizon of $T$ time steps: $P_\phi(s \to U)$ . This probability is modeled by a neural network with parameter vector $\phi$ (e.g., a single hidden layer of 32 units with sigmoid output) trained via Adam to minimize failure prediction error on collected rollouts. This cost shaping tightens the feasible set and smooths out sparse constraint signals.
Knowl 8 — Empirical Comparison of CPO Against Primal-Dual Optimization and TRPO
empirical result
Constrained Policy Optimization (CPO) was evaluated against Primal-Dual Optimization (PDO) and unconstrained Trust Region Policy Optimization (TRPO) across continuous robot locomotion tasks (Point-Circle, Ant-Circle, Humanoid-Circle, Point-Gather, Ant-Gather) using neural network policies with two hidden layers of sizes (64, 32).
1. Constraint Satisfaction During Training: CPO drives the constraint return directly to the constraint threshold $d$ and maintains feasibility throughout training across all environments. In contrast, PDO frequently oscillates, overshoots, and severely violates constraints during learning (e.g., suffering a large spike in constraint cost in Ant-Circle around iteration 350 when the agent learns to run).
2. Sensitivity to Hyperparameters: PDO dual variables updated via $\nu_{k+1} = (\nu_k + \alpha(J_C(\pi_k) - d))_+$ are highly sensitive to dual learning rate $\alpha$ and initial value $\nu_0$ . Setting $\nu_0 = 1000$ guarantees safety initially but substantially impairs reward optimization. CPO computes optimal dual variables per step from scratch, eliminating dual learning rate tuning.
3. Unconstrained Baselines: TRPO achieves high task returns but severely violates safety constraints, confirming that optimal unconstrained policies are infeasible for the CMDP tasks.
Knowl 9 — Sensitivity of Fixed Penalty Optimization Compared to CPO
empirical result
Fixed Penalty Optimization (FPO), which optimizes $R(s, a, s') - \lambda C^+(s, a, s')$ with unconstrained TRPO for fixed weights $\lambda \in \{1, 5, 50\}$ , was compared against CPO on the Ant-Circle environment where the safety constraint threshold is $d = 10$ .
- $\lambda = 1$ : The agent ignores the safety constraint, maximizing reward (reaching returns $> 2000$ ) while accumulating massive constraint costs ( $C^+$ return $\approx 30$ , far exceeding the limit of 10).
- $\lambda = 5$ (less than one order of magnitude increase): The penalty dominates, producing an overly conservative policy that minimizes cost but never learns locomotion, acquiring near-zero reward.
- $\lambda = 50$ : The agent completely fails to acquire task reward.
In contrast, CPO dynamically adjusts the effective Lagrange multiplier at each iteration to enforce $J_{C^+}(\pi) \le 10$ while successfully maximizing locomotion reward without manual penalty coefficient sweeps.

Knowl 10 — Hyperparameters and Experimental Configurations for CPO Locomotion Benchmarks

data/table

All policy networks are Gaussian with state-dependent mean outputs from a MLP with two hidden layers of sizes (64, 32) and $\tanh$ activations, and separate learnable log-standard deviation parameters. Value functions for reward and constraints share the same architecture. Advantage estimation uses GAE with discount factor $\gamma = 0.995$ , regular advantage discount $\lambda_{\text{GAE}} = 0.95$ , and KL step size $\delta_{\text{KL}} = 0.01$ .

Parameter	Point-Circle	Ant-Circle	Humanoid-Circle	Point-Gather	Ant-Gather
Batch size (steps)	50,000	100,000	50,000	50,000	100,000
Rollout length (steps)	50–65	500	1000	15	500
Constraint limit $d$	5	10	10	0.1	0.2
Failure horizon $T$	5	20	20	N/A	20
Failure predictor SGD steps	25	25	25	N/A	10
Predictor coefficient $\alpha$	1	1	1	N/A	0.01
$\lambda_{\text{GAE}}^C$ (constraint GAE)	1	0.5	0.5	1	0.5

For higher-dimensional robots (Ant and Humanoid), setting the constraint advantage parameter $\lambda_{\text{GAE}}^C < 1$ (e.g., 0.5) is critical; setting $\lambda_{\text{GAE}}^C = 1$ causes significant overestimation of constraint gradient magnitudes, resulting in unsafe policy updates.

Coverage note — All major theoretical contributions (Theorem 1, Corollaries 1-3, Propositions 1-2, Theorem 2, Lemma 3), algorithm formulations (CPO, conjugate gradient subproblem, recovery direction, cost shaping), and empirical validation benchmarks were converted into knowls. Intermediate proof steps (such as Lemma 1 and Lemma 2 derivations) were omitted in accordance with the rules against intermediate proof steps.

References

1.Altman, Eitan. Constrained Markov Decision Processes. pp. 260, 1999. ISSN 01676377. doi: 10.1016/0167-6377(96)00003-X.
2.Amodei, Dario, Olah, Chris, Steinhardt, Jacob, Christiano, Paul, Schulman, John, and Mané, Dan. Concrete Problems in AI Safety. arXiv, 2016. URL http://arxiv.org/abs/1606.06565.
3.Bou Ammar, Haitham, Tutunov, Rasul, and Eaton, Eric. Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regret. International Conference on Machine Learning, 37:19, 2015. URL http://arxiv.org/abs/1505.0579.
4.Boyd, Stephen, Xiao, Lin, and Mutapcic, Almir. Subgradient methods. Lecture Notes of Stanford EE392, 2003. URL http://xxpt.ynjgy.com/resource/data/20100601/U/stanford201001010/02-subgrad{_}method{_}notes.pdf.
5.Chow, Yinlam, Ghavamzadeh, Mohammad, Janson, Lucas, and Pavone, Marco. Risk-Constrained Reinforcement Learning with Percentile Risk Criteria. Journal of Machine Learning Research, 1(xxxx):1–49, 2015.
6.Csiszar, I and Körner, J. Information Theory: Coding Theorems for Discrete Memoryless Systems. Book, 244:452, 1981. ISSN 0895-4801. doi: 10.2307/2529636. URL http://www.getcited.org/pub/102082957.
7.Duan, Yan, Chen, Xi, Schulman, John, and Abbeel, Pieter. Benchmarking Deep Reinforcement Learning for Continuous Control. The 33rd International Conference on Machine Learning (ICML 2016) (2016), 48:14, 2016. URL http://arxiv.org/abs/1604.06778.
8.García, Javier and Fernández, Fernando. A Comprehensive Survey on Safe Reinforcement Learning. Journal of Machine Learning Research, 16:1437–1480, 2015. ISSN 15337928.
9.Gu, Shixiang, Lillicrap, Timothy, Ghahramani, Zoubin, Turner, Richard E., and Levine, Sergey. Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic. In International Conference on Learning Representations, 2017. URL http://arxiv.org/abs/1611.02247.
10.Held, David, Mccarthy, Zoe, Zhang, Michael, Shentu, Fred, and Abbeel, Pieter. Probabilistically Safe Policy Transfer. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2017.
11.Jiang, Nan and Li, Lihong. Doubly Robust Off-policy Value Evaluation for Reinforcement Learning. International Conference on Machine Learning, 2015. URL http://arxiv.org/abs/1511.03722.
12.Kakade, Sham and Langford, John. Approximately Optimal Approximate Reinforcement Learning. Proceedings of the 19th International Conference on Machine Learning, pp. 267–274, 2002. URL http://www.cs.cmu.edu/afs/cs/Web/People/jcl/papers/aoarl/Final.pdf.
13.Levine, Sergey, Finn, Chelsea, Darrell, Trevor, and Abbeel, Pieter. End-to-End Training of Deep Visuomotor Policies. Journal of Machine Learning Research, 17:1–40, 2016. ISSN 15337928. doi: 10.1007/s13398-014-0173-7.2.
14.Lillicrap, Timothy P., Hunt, Jonathan J., Pritzel, Alexander, Heess, Nicolas, Erez, Tom, Tassa, Yuval, Silver, David, and Wierstra, Daan. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016. ISBN 2200000006. doi: 10.1561/2200000006.
15.Lipton, Zachary C., Gao, Jianfeng, Li, Lihong, Chen, Jianshu, and Deng, Li. Combating Deep Reinforcement Learning’s Sisyphean Curse with Intrinsic Fear. In arXiv, 2017. ISBN 2004012439. URL http://arxiv.org/abs/1611.01211.
16.Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei a, Veness, Joel, Bellemare, Marc G, Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K, Ostrovski, Georg, Petersen, Stig, Beattie, Charles, Sadik, Amir, Antonoglou, Ioannis, King, Helen, Kumaran, Dharshan, Wierstra, Daan, Legg, Shane, and Hassabis, Demis. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015. ISSN 0028-0836. doi: 10.1038/nature14236. URL http://dx.doi.org/10.1038/nature14236.
17.Mnih, Volodymyr, Badia, Adrià Puigdomènech, Mirza, Mehdi, Graves, Alex, Lillicrap, Timothy P., Harley, Tim, Silver, David, and Kavukcuoglu, Koray. Asynchronous Methods for Deep Reinforcement Learning. pp. 1–28, 2016. URL http://arxiv.org/abs/1602.01783.
18.Moldovan, Teodor Mihai and Abbeel, Pieter. Safe Exploration in Markov Decision Processes. Proceedings of the 29th International Conference on Machine Learning, 2012. URL http://arxiv.org/abs/1205.4810.
19.Ng, Andrew Y., Harada, Daishi, and Russell, Stuart. Policy invariance under reward transformations : Theory and application to reward shaping. Sixteenth International Conference on Machine Learning, 3:278–287, 1999. doi: 10.1.1.48.345.
20.Peters, Jan and Schaal, Stefan. Reinforcement learning of motor skills with policy gradients. Neural Networks, 21 (4):682–697, 2008. ISSN 08936080. doi: 10.1016/j.neunet.2008.02.003.
21.Pirotta, Matteo, Restelli, Marcello, and Calandriello, Daniele. Safe Policy Iteration. Proceedings of the 30th International Conference on Machine Learning, 28, 2013.
22.Schulman, John, Moritz, Philipp, Jordan, Michael, and Abbeel, Pieter. Trust Region Policy Optimization. International Conference on Machine Learning, 2015.
23.Schulman, John, Moritz, Philipp, Levine, Sergey, Jordan, Michael, and Abbeel, Pieter. High-Dimensional Continuous Control Using Generalized Advantage Estimation. arXiv, 2016.
24.Shalev-Shwartz, Shai, Shammah, Shaked, and Shashua, Amnon. Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving. arXiv, 2016. URL http://arxiv.org/abs/1610.03295.
25.Silver, David, Huang, Aja, Maddison, Chris J., Guez, Arthur, Sifre, Laurent, van den Driessche, George, Schrittwieser, Julian, Antonoglou, Ioannis, Panneershelvam, Veda, Lanctot, Marc, Dieleman, Sander, Grewe, Dominik, Nham, John, Kalchbrenner, Nal, Sutskever, Ilya, Lillicrap, Timothy, Leach, Madeleine, Kavukcuoglu, Koray, Graepel, Thore, and Hassabis, Demis. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587): 484–489, 2016. ISSN 0028-0836. doi: 10.1038/nature16961. URL http://dx.doi.org/10.1038/nature16961.
26.Sutton, Richard S and Barto, Andrew G. Introduction to Reinforcement Learning. Learning, 4(1996):1–5, 1998. ISSN 10743529. doi: 10.1.1.32.7692. URL http://dl.acm.org/citation.cfm?id=551283.
27.Uchibe, Eiji and Doya, Kenji. Constrained reinforcement learning from intrinsic and extrinsic rewards. 2007 IEEE 6th International Conference on Development and Learning, ICDL, (February):163–168, 2007. doi: 10.1109/DEVLRN.2007.4354030.

Citation

MLA

Achiam, J., et al. “Constrained Policy Optimization”. arXiv, 2017, http://arxiv.org/abs/1705.10528v1.

APA

Achiam, J., Held, D., Tamar, A., & Abbeel, P. (2017). Constrained Policy Optimization. arXiv. http://arxiv.org/abs/1705.10528v1

Chicago

Achiam, J., D. Held, A. Tamar, and P. Abbeel. 2017. “Constrained Policy Optimization”. arXiv. http://arxiv.org/abs/1705.10528v1.

Harvard

Achiam, J. et al. (2017) “Constrained Policy Optimization”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1705.10528v1.

Vancouver

1. Achiam J, Held D, Tamar A, Abbeel P (2017) Constrained Policy Optimization. arXiv

BibTeX

@article{achiam2017constrained,
  title = {Constrained Policy Optimization},
  author = {Achiam, Joshua and Held, David and Tamar, Aviv and Abbeel, Pieter},
  year = {2017},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1705.10528v1},
  eprint = {1705.10528}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Constrained Policy Optimization

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Policy Performance Difference Bounds via Average Action Divergence

Knowl 2 — Constrained Policy Optimization Trust Region Formulation and Worst-Case Violation Bound

Knowl 3 — Constrained Policy Optimization Algorithm

Knowl 4 — Analytical Solution to Linear-Quadratic Constrained Linear Program

Knowl 5 — State Visitation Distribution Divergence Bound under Policy Shift

Knowl 6 — Worst-Case Performance Degradation Bound for Trust Region Policy Updates

Knowl 7 — Constraint Tightening via Learned Cost Shaping

Knowl 8 — Empirical Comparison of CPO Against Primal-Dual Optimization and TRPO

Knowl 9 — Sensitivity of Fixed Penalty Optimization Compared to CPO