Back

OptNet: Differentiable Optimization as a Layer in Neural Networks

Brandon AmosJ. Kolter

article2017ICML1,342 citationsOutstanding Paper Award at the ICML Theoretical Foundations Workshop

Introduces OptNet, an architecture that embeds quadratic optimization problems directly into neural networks as differentiable layers, enabling end-to-end learning that can explicitly model and satisfy hard constraints.

Listen

Modern deep learning relies heavily on standard layers, such as convolutional and fully connected layers, to extract representations from data. However, these traditional architectures struggle to capture strict mathematical constraints and complex structural dependencies between variables. In many real-world domains—including physical systems, logical reasoning, and control—inference is naturally framed as a constrained optimization problem. The article introduces OptNet, a novel neural network architecture that integrates exact, constrained quadratic optimization problems directly as individual layers within end-to-end trainable deep networks.

To achieve this, the authors develop mathematical and algorithmic techniques for training networks containing optimization layers. They derive exact gradient calculations through the layer by implicitly differentiating the underlying optimality conditions (the Karush-Kuhn-Tucker conditions), supporting both equality and inequality constraints without relying on approximations or unrolling iterative loops. To overcome the computational bottleneck of standard solvers, they introduce a specialized GPU-based primal-dual interior point method. This solver executes batch optimization operations in parallel and reuses matrix factorizations from the forward pass, computing backward gradients with virtually no additional computational overhead.

Empirical evaluations across synthetic benchmarks and structured tasks demonstrate four primary findings. First, the custom GPU batch solver achieves dramatic computational speedups, solving batches of quadratic programs more than twenty-five times faster than commercial solvers like Gurobi (executing a batch of 128 problems in 0.18 seconds compared to 4.7 seconds). Second, theoretical analysis proves that an OptNet layer can represent operations that require exponentially many units in standard two-layer rectified linear unit networks. Third, in signal denoising experiments, fine-tuning an optimization layer initialized with total variation differencing improves the test mean squared error by 12% over classical total variation methods, outperforming standard fully connected networks. Fourth, in a 4x4 mini-Sudoku task with 9,000 training examples, OptNet successfully learns the underlying hard logical constraints purely from input-output examples and generalizes well to unseen puzzles, whereas a ten-layer convolutional baseline overfits and fails to learn the required logic.

These findings indicate that embedding domain-relevant optimization structures directly into neural networks significantly improves data efficiency, interpretability, and generalizability for tasks governed by hard constraints. By reducing network depth and parameter count while preserving expressive power, OptNet narrows the gap between purely statistical deep learning and rigorous mathematical modeling. Practitioners can deploy this approach to learn physical or rule-based constraints directly from raw data without manual specification.

Teams considering this architecture should evaluate whether their target problems involve strict structural constraints or domain-specific optimization frameworks. Next steps should focus on applying OptNet to specialized domains like control systems or structured prediction, while utilizing sparse matrix techniques to scale the solver to higher dimensions. Users should note that exact quadratic programming incurs cubic computational complexity relative to the number of variables and constraints, making current implementations practical primarily for layer sizes under 1,000 dimensions, and model training may require additional hyperparameter tuning due to scale-invariant parameter manifolds.

Carnegie Mellon Universitybackpropagation algorithmgpunetwork architectureOptNetdifferentiable optimizationoptimization layers
arXiv: 1703.00443
Cover for OptNet: Differentiable Optimization as a Layer in Neural Networks

Abstract

This paper presents OptNet, a network architecture that integrates optimization problems (here, specifically in the form of quadratic programs) as individual layers in larger end-to-end trainable deep networks. These layers encode constraints and complex dependencies between the hidden states that traditional convolutional and fully-connected layers often cannot capture. We explore the foundations for such an architecture: we show how techniques from sensitivity analysis, bilevel optimization, and implicit differentiation can be used to exactly differentiate through these layers and with respect to layer parameters; we develop a highly efficient solver for these layers that exploits fast GPU-based batch solves within a primal-dual interior point method, and which provides backpropagation gradients with virtually no additional cost on top of the solve; and we highlight the application of these approaches in several problems. In one notable example, the method is learns to play mini-Sudoku (4x4) given just input and output games, with no a-priori information about the rules of the game; this highlights the ability of OptNet to learn hard constraints better than other neural architectures.

Topics

backpropagation algorithmgpunetwork architectureoptimizationoptimization problemslayeredarchitecture designs

Keywords

OptNetdifferentiable optimizationoptimization layersconvex optimizationbilevel optimizationargmin differentiationstructured prediction energy networksconstrained optimization

Table of Contents

  • 1 Introduction
  • 2 Background and related work
  • 3 OptNet: solving optimization within a neural network
  • 3.1 An efficient batched QP solver
  • 3.1.1 Efficiently computing gradients
  • 3.2 Properties and representational power
  • 3.3 Limitations of the method
  • 4 Experimental results
  • 4.1 Batch QP solver performance
  • 4.2 Total variation denoising
  • 4.2.1 Baseline: Total variation denoising
  • 4.2.2 Baseline: Learning with a fully-connected neural network
  • 4.2.3 Learning the differencing operator
  • 4.2.4 Fine-tuning and improving the total variation solution
  • 4.3 MNIST
  • 4.4 Sudoku
  • 5 Conclusion
  • References
  • A MNIST Experiment
  • B Denoising Experiment Details
  • C Representational power of the QP OptNet layer
  • C.1 Proof of Theorem
  • C.2 Proof of Theorem
  • C.3 Proof of Theorem

Knowls

  1. Knowl 1 — OptNet Quadratic Optimization Layer Formulation

    model/method

    An OptNet layer is a neural network layer whose output zi+1∈Rnz_{i+1} \in \mathbb{R}^n is defined as the solution to a constrained quadratic program (QP) parameterized by the activations of the preceding layer ziz_i:

    zi+1=argmin⁡z∈Rn12zTQ(zi)z+q(zi)Tzsubject toA(zi)z=b(zi),G(zi)z≤h(zi)z_{i+1} = \operatorname{argmin}_{z \in \mathbb{R}^n} \frac{1}{2} z^T Q(z_i) z + q(z_i)^T z \quad \text{subject to} \quad A(z_i) z = b(z_i), \quad G(z_i) z \le h(z_i)

    where zz is the optimization variable. The problem parameters are:

    • Q(zi)∈Rn×nQ(z_i) \in \mathbb{R}^{n \times n}, a positive semidefinite matrix (Q(zi)⪰0Q(z_i) \succeq 0),
    • q(zi)∈Rnq(z_i) \in \mathbb{R}^n, the linear objective vector,
    • A(zi)∈Rm×nA(z_i) \in \mathbb{R}^{m \times n} and b(zi)∈Rmb(z_i) \in \mathbb{R}^m, the linear equality constraint matrix and vector,
    • G(zi)∈Rp×nG(z_i) \in \mathbb{R}^{p \times n} and h(zi)∈Rph(z_i) \in \mathbb{R}^p, the linear inequality constraint matrix and vector.

    Any subset of the parameter tuple {Q,q,A,b,G,h}\{Q, q, A, b, G, h\} can be learnable network weights, fixed problem constants, or differentiable functions of the previous layer activations ziz_i.

  2. Knowl 2 — Exact Backpropagation through QP Layers via KKT Matrix Differentials

    equation

    For a quadratic program layer with optimal primal solution z∗∈Rnz^* \in \mathbb{R}^n, optimal equality dual variables ν∗∈Rm\nu^* \in \mathbb{R}^m, and optimal inequality dual variables λ∗∈R≥0p\lambda^* \in \mathbb{R}^p_{\ge 0}, the derivatives of a scalar loss ℓ\ell with respect to the input parameters are computed by differentiating the Karush-Kuhn-Tucker (KKT) optimality conditions.

    Given the upstream loss gradient ∂ℓ∂z∗∈R1×n\frac{\partial \ell}{\partial z^*} \in \mathbb{R}^{1 \times n}, the intermediate adjoint variables dz∈Rnd_z \in \mathbb{R}^n, dλ∈Rpd_\lambda \in \mathbb{R}^p, and dν∈Rmd_\nu \in \mathbb{R}^m are computed by solving the linear system:

    [dzdλdν]=−[QGTD(λ∗)ATGD(Gz∗−h)0A00]−1[(∂ℓ∂z∗)T00]\begin{bmatrix} d_z \\ d_\lambda \\ d_\nu \end{bmatrix} = - \begin{bmatrix} Q & G^T D(\lambda^*) & A^T \\ G & D(G z^* - h) & 0 \\ A & 0 & 0 \end{bmatrix}^{-1} \begin{bmatrix} \left(\frac{\partial \ell}{\partial z^*}\right)^T \\ 0 \\ 0 \end{bmatrix}

    where D(v)D(v) denotes a diagonal matrix formed from vector vv.

    The gradients of ℓ\ell with respect to each QP parameter are then given by:

    ∇Qℓ=12(dz(z∗)T+z∗dzT),∇qℓ=dz\nabla_Q \ell = \frac{1}{2} \left( d_z (z^*)^T + z^* d_z^T \right), \quad \nabla_q \ell = d_z

    ∇Aℓ=dν(z∗)T+ν∗dzT,∇bℓ=−dν\nabla_A \ell = d_\nu (z^*)^T + \nu^* d_z^T, \quad \nabla_b \ell = -d_\nu

    ∇Gℓ=D(λ∗)dλ(z∗)T+λ∗dzT,∇hℓ=−D(λ∗)dλ\nabla_G \ell = D(\lambda^*) d_\lambda (z^*)^T + \lambda^* d_z^T, \quad \nabla_h \ell = -D(\lambda^*) d_\lambda

    These parameter gradients are subsequently backpropagated to previous layers ziz_i using the standard multivariate chain rule.

  3. Knowl 3 — Primal-Dual Interior Point Solver with Zero-Overhead Backward Pass

    algorithm

    A custom GPU-based primal-dual interior point method (PDIPM) solves batches of quadratic programs in parallel while yielding the backward pass gradients without requiring an additional matrix factorization.

    During forward solve iterations over primal variable z∈Rnz \in \mathbb{R}^n, slack variable s∈Rps \in \mathbb{R}^p (s>0s > 0), and dual variables λ∈Rp\lambda \in \mathbb{R}^p and ν∈Rm\nu \in \mathbb{R}^m, the solver factorizes a symmetrized KKT matrix KsymK_{\text{sym}}:

    Ksym=[Q0GTAT0D(λ/s)I0GI00A000]K_{\text{sym}} = \begin{bmatrix} Q & 0 & G^T & A^T \\ 0 & D(\lambda / s) & I & 0 \\ G & I & 0 & 0 \\ A & 0 & 0 & 0 \end{bmatrix}

    At convergence to the optimal solution (z∗,s∗,λ∗,ν∗)(z^*, s^*, \lambda^*, \nu^*), the matrix factorization (such as the LU decomposition) of KsymK_{\text{sym}} is retained. The backward pass gradients are obtained by solving the linear system:

    Ksym[dzdsd~λdν]=[−(∂ℓ∂z∗)T000]K_{\text{sym}} \begin{bmatrix} d_z \\ d_s \\ \tilde{d}_\lambda \\ d_\nu \end{bmatrix} = \begin{bmatrix} -\left(\frac{\partial \ell}{\partial z^*}\right)^T \\ 0 \\ 0 \\ 0 \end{bmatrix}

    where d~λ=D(λ∗)dλ\tilde{d}_\lambda = D(\lambda^*) d_\lambda. Because solving this linear system via back-substitution requires only quadratic time O((n+p+m)2)O((n+p+m)^2) once the cubic O((n+p+m)3)O((n+p+m)^3) factorization of KsymK_{\text{sym}} is computed during the forward pass, the computational overhead of the backward pass is negligible compared to the forward solve.

  4. Knowl 4 — Subdifferentiability and Almost Everywhere Differentiability of OptNet Layers

    theoretical result

    Let z∗(θ)z^*(\theta) denote the optimal solution mapping of an OptNet layer with parameter set θ={Q,q,A,b,G,h}\theta = \{Q, q, A, b, G, h\}.

    If the quadratic objective matrix is strictly positive definite (Q≻0Q \succ 0) and the equality constraint matrix AA has full row rank:

    1. The solution mapping z∗(θ)z^*(\theta) is continuous and subdifferentiable everywhere. That is, the Clarke generalized subdifferential ∂z∗(θ)\partial z^*(\theta) is non-empty for all parameter configurations θ\theta.
    2. The mapping z∗(θ)z^*(\theta) is differentiable (the Clarke generalized subdifferential contains a single unique Jacobian matrix) for all θ\theta except on a set of Lebesgue measure zero.

    The measure-zero non-differentiable set consists of degenerate QP solutions where an inequality constraint is active ((Gz∗−h)i=0(G z^* - h)_i = 0) and its associated Lagrange multiplier is simultaneously zero (λi∗=0\lambda^*_i = 0).

  5. Knowl 5 — Exact Representation of Piecewise Linear Functions and ReLU Layers by OptNet

    theoretical result

    OptNet layers can represent arbitrary elementwise piecewise linear activations and standard linear-plus-ReLU layers:

    1. Any elementwise continuous piecewise linear function f:Rn→Rnf: \mathbb{R}^n \to \mathbb{R}^n with kk linear regions per dimension can be represented exactly as an OptNet layer using O(nk)O(nk) parameters.
    2. A standard feedforward layer with weight matrix W∈Rn×mW \in \mathbb{R}^{n \times m}, bias vector b∈Rnb \in \mathbb{R}^n, and rectified linear unit (ReLU) activation, zi+1=max⁡{Wzi+b,0}z_{i+1} = \max\{W z_i + b, 0\}, can be represented exactly by an OptNet layer with O(mn)O(mn) parameters via the non-negative quadratic program:

    zi+1=argmin⁡z∈Rn∥z−Wzi−b∥22subject toz≥0z_{i+1} = \operatorname{argmin}_{z \in \mathbb{R}^n} \|z - W z_i - b\|_2^2 \quad \text{subject to} \quad z \ge 0

  6. Knowl 6 — Exponential Separation Between OptNet and Two-Layer ReLU Networks

    theoretical result

    There exist scalar-valued functions f:Rn→Rf: \mathbb{R}^n \to \mathbb{R} representable compactly by an OptNet layer with pp parameters that cannot be represented exactly over all of Rn\mathbb{R}^n by any two-layer ReLU network of the form f′(x)=∑i=1mwimax⁡{aiTx+bi,0}f'(x) = \sum_{i=1}^m w_i \max\{a_i^T x + b_i, 0\}, and which require an exponential number of parameters O(cp)O(c^p) (for some constant c>1c > 1) for a two-layer ReLU network to approximate over a bounded domain.

    For example, the pointwise maximum of three linear functions, f(x)=max⁡{a1Tx,a2Tx,a3Tx}f(x) = \max\{a_1^T x, a_2^T x, a_3^T x\} with a1,a2,a3∈Rna_1, a_2, a_3 \in \mathbb{R}^n, is represented exactly by an OptNet layer with O(n)O(n) parameters:

    argmin⁡z∈Rz2subject toaiTx≤z(i=1,2,3)\operatorname{argmin}_{z \in \mathbb{R}} z^2 \quad \text{subject to} \quad a_i^T x \le z \quad (i = 1, 2, 3)

    In contrast, sums of single ReLUs produce creases that span the entire input space, preventing exact representation of pointwise maxima whose creases terminate at intersections. Similarly, Euclidean projection onto the probability simplex (argmin⁡z≥0,1Tz=1∥z−x∥22\operatorname{argmin}_{z \ge 0, \mathbf{1}^T z = 1} \|z - x\|_2^2) can be directly represented by a single OptNet layer but cannot be represented in closed form by a standard single feedforward layer.

  7. Knowl 7 — Batched GPU QP Solver Runtime Performance

    empirical result

    In computational benchmarks comparing the custom batched GPU interior point solver (qpth) executed on an NVIDIA Titan X GPU against Gurobi running on a quad-core Intel Core i7-5960X CPU @ 3.00 GHz on batches of random strictly convex QPs (n=100n = 100 variables, p=100p = 100 inequality constraints):

    • For a minibatch size of 128 problems, the batched GPU solver solves all 128 instances simultaneously in an average of 0.18 seconds.
    • Gurobi solving the 128 instances sequentially requires an average of 4.7 seconds (over 26 times slower).

    Minibatch parallelization on GPU hardware eliminates CPU-GPU memory transfer overheads and enables practical end-to-end backpropagation through optimization layers during neural network training.

  8. Knowl 8 — Learning Combinatorial Sudoku Rules End-to-End Without Prior Knowledge

    empirical result

    On a 4×44 \times 4 mini-Sudoku puzzle dataset (9,000 training puzzles and 1,000 held-out test puzzles), an OptNet layer learned the underlying logical constraint rules purely from input-output examples without being provided the rules of the game.

    The input was provided as a 4×4×44 \times 4 \times 4 binary tensor (one-hot vectors for known cell digits and all zeros for empty cells), and the target was a 4×4×44 \times 4 \times 4 one-hot solution tensor. The OptNet architecture employed a generic standard-form QP layer with learnable equality constraints Ax=bA x = b, non-negativity x≥0x \ge 0, linear term qq set to the input tensor, and Q=0.1IQ = 0.1 I.

    While a 10-layer convolutional network baseline (512 3×33 \times 3 filters per layer) achieved low training loss through memorization but failed to generalize on held-out test puzzles, OptNet converged to near-zero test error by learning the true constraint matrix AA and vector bb governing the puzzle's row, column, and block constraints.

  9. Knowl 9 — 1D Total Variation Denoising and Difference Operator Learning

    data/table

    The table below compares Mean Squared Error (MSE) on a 1D piecewise-constant signal denoising task corrupted by independent Gaussian noise across four methods: a fully connected neural network (FC Net), a pure OptNet layer with a randomly initialized difference matrix DD, standard convex Total Variation (TV) denoising with grid-searched regularizer λ≈13\lambda \approx 13, and an OptNet layer initialized with the first-order TV difference operator and fine-tuned end-to-end (OptNet Tuned TV):

    Method Train MSE Test MSE
    FC Net 18.5 29.8
    Pure OptNet 52.9 53.3
    Total Variation 16.3 16.5
    OptNet Tuned TV 13.8 14.4

    Standard TV denoising solves argmin⁡z12∥y−z∥22+λ∥Dz∥1\operatorname{argmin}_z \frac{1}{2}\|y - z\|_2^2 + \lambda \|Dz\|_1 with fixed Di=ei−ei+1D_i = e_i - e_{i+1}. Pure OptNet learns an interpretable sparse differencing operator with alternating signs, but suffers from over-regularization when trained from scratch. Fine-tuning the operator parameters within OptNet reduces test MSE by 12% relative to analytical Total Variation (from 16.5 to 14.4).

  10. Knowl 10 — Computational and Optimization Limitations of OptNet Layers

    limitation

    OptNet layers present three core practical limitations:

    1. Cubic Computational Complexity: Solving the KKT system exactly via interior point methods scales cubically with the number of variables and constraints (O((n+m+p)3)O((n + m + p)^3)), compared to quadratic complexity (O(n2)O(n^2)) for standard feedforward layers. This restricts practical layer sizes to hidden dimensions of roughly n≤1000n \le 1000.
    2. Dense GPU Matrix Operations: Standard batched implementations use dense linear algebra operations and do not exploit matrix sparsity patterns (e.g., via sparse fill-reducing orderings), limiting scaling on structured large-scale problems.
    3. Parameter Invariance Manifolds: The optimization problem is invariant under certain parameter transformations (for instance, multiplying any row of AA and bb by a positive scalar does not alter the feasible space or optimal primal solution z∗z^*). These invariant manifolds can complicate gradient-based training, requiring parameterizations like Q=LLT+ϵIQ = L L^T + \epsilon I to ensure strict positive definiteness and specialized hyperparameter tuning.

Coverage note — None was omitted; all primary methodological formulations, theoretical results, core algorithms, empirical benchmarks (Sudoku, Denoising, Solver runtimes), and stated limitations are covered. The minor MNIST sanity check from the supplement was omitted as the authors noted it served only to confirm baseline gradient flow with no significant performance distinction.

References

  1. 1.Amos, Brandon, Xu, Lei, and Kolter, J Zico. Input convex neural networks. In Proceedings of the International Conference on Machine Learning, 2017.
  2. 2.Belanger, David and McCallum, Andrew. Structured prediction energy networks. In Proceedings of the International Conference on Machine Learning, 2016.
  3. 3.Belanger, David, Yang, Bishan, and McCallum, Andrew. End-to-end learning for structured prediction energy networks. In Proceedings of the International Conference on Machine Learning, 2017.
  4. 4.Bertsekas, Dimitri P. Nonlinear programming. Athena scientific Belmont, 1999.
  5. 5.Bonnans, J Frederic and Shapiro, Alexander. Perturbation analysis of optimization problems. Springer Science & Business Media, 2013.
  6. 6.Boyd, Stephen and Vandenberghe, Lieven. Convex optimization. Cambridge university press, 2004.
  7. 7.Brakel, Philemon, Stroobandt, Dirk, and Schrauwen, Ben- jamin. Training energy-based models for time-series imputation. Journal of Machine Learning Research, 14(1): 2771–2797, 2013.
  8. 8.Chen, Liang-Chieh, Schwing, Alexander G, Yuille, Alan L, and Urtasun, Raquel. Learning deep structured models. In Proceedings of the International Conference on Machine Learning, 2015.
  9. 9.Clarke, Frank H. Generalized gradients and applications. Transactions of the American Mathematical Society, 205:247–262, 1975.
  10. 10.Domke, Justin. Generic methods for optimization-based modeling. In AISTATS, volume 22, pp. 318–326, 2012.
  11. 11.Dontchev, Asen L and Rockafellar, R Tyrrell. Implicit functions and solution mappings. Springer Monogr. Math., 2009.
  12. 12.Duchi, John, Shalev-Shwartz, Shai, Singer, Yoram, and Chandra, Tushar. Efficient projections onto the l 1-ball for learning in high dimensions. In Proceedings of the 25th international conference on Machine learning, pp. 272–279, 2008.
  13. 13.Fiacco, Anthony V and Ishizuka, Yo. Sensitivity and stability analysis for nonlinear programming. Annals of Operations Research, 27(1):215–235, 1990.
  14. 14.Goodfellow, Ian, Mirza, Mehdi, Courville, Aaron, and Bengio, Yoshua. Multi-prediction deep boltzmann machines. In Advances in Neural Information Processing Systems, pp. 548–556, 2013.
  15. 15.Goodfellow, Ian, Pouget-Abadie, Jean, Mirza, Mehdi, Xu, Bing, Warde-Farley, David, Ozair, Sherjil, Courville, Aaron, and Bengio, Yoshua. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672–2680, 2014.
  16. 16.Gould, Stephen, Fernando, Basura, Cherian, Anoop, Anderson, Peter, Santa Cruz, Rodrigo, and Guo, Edison. On differentiating parameterized argmin and argmax problems with application to bi-level optimization. arXiv preprint arXiv:1607.05447, 2016.
  17. 17.Griewank, Andreas and Walther, Andrea. Evaluating derivatives: principles and techniques of algorithmic differentiation. SIAM, 2008.
  18. 18.Johnson, Matthew, Duvenaud, David K, Wiltschko, Alex, Adams, Ryan P, and Datta, Sandeep R. Composing graphical models with neural networks for structured representations and fast inference. In Advances in Neural Information Processing Systems, pp. 2946–2954, 2016.
  19. 19.Kennedy, Michael Peter and Chua, Leon O. Neural networks for nonlinear programming. IEEE Transactions on Circuits and Systems, 35(5):554–562, 1988.
  20. 20.Kingma, Diederik and Ba, Jimmy. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  21. 21.Kunisch, Karl and Pock, Thomas. A bilevel optimization approach for parameter learning in variational models. SIAM Journal on Imaging Sciences, 6(2):938–983, 2013.
  22. 22.LeCun, Yann, Chopra, Sumit, Hadsell, Raia, Ranzato, M, and Huang, F. A tutorial on energy-based learning. Predicting structured data, 1:0, 2006.
  23. 23.Lillo, Walter E, Loh, Mei Heng, Hui, Stefen, and Zak, Stanislaw H. On solving constrained optimization problems with neural networks: A penalty method approach. IEEE Transactions on neural networks, 4(6):931–940, 1993.
  24. 24.Lotstedt, Per. Numerical simulation of time-dependent contact and friction problems in rigid body mechanics. SIAM journal on scientific and statistical computing, 5 (2):370–393, 1984.
  25. 25.Magnus, X and Neudecker, Heinz. Matrix differential calculus. New York, 1988.
  26. 26.Mairal, Julien, Bach, Francis, and Ponce, Jean. Task-driven dictionary learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(4):791–804, 2012.
  27. 27.Mattingley, Jacob and Boyd, Stephen. Cvxgen: A code generator for embedded convex optimization. Optimization and Engineering, 13(1):1–27, 2012.
  28. 28.Metz, Luke, Poole, Ben, Pfau, David, and Sohl-Dickstein, Jascha. Unrolled generative adversarial networks. arXiv preprint arXiv:1611.02163, 2016.
  29. 29.Morari, Manfred and Lee, Jay H. Model predictive control: past, present and future. Computers & Chemical Engineering, 23(4):667–682, 1999.
  30. 30.Peng, Jian, Bo, Liefeng, and Xu, Jinbo. Conditional neural fields. In Advances in neural information processing systems, pp. 1419–1427, 2009.
  31. 31.Sastry, Shankar and Bodson, Marc. Adaptive control: stability, convergence and robustness. Courier Corporation, 2011.
  32. 32.Schmidt, Uwe and Roth, Stefan. Shrinkage fields for effective image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2774–2781, 2014.
  33. 33.Sra, Suvrit, Nowozin, Sebastian, and Wright, Stephen J. Optimization for machine learning. Mit Press, 2012.
  34. 34.Stoyanov, Veselin, Ropson, Alexander, and Eisner, Jason. Empirical risk minimization of graphical model parameters given approximate inference, decoding, and model structure. In AISTATS, pp. 725–733, 2011.
  35. 35.Tappen, Marshall F, Liu, Ce, Adelson, Edward H, and Freeman, William T. Learning gaussian conditional random fields for low-level vision. In Computer Vision and Pattern Recognition, 2007. CVPR’07. IEEE Conference on, pp. 1–8. IEEE, 2007.
  36. 36.Zheng, Shuai, Jayasumana, Sadeep, Romera-Paredes, Bernardino, Vineet, Vibhav, Su, Zhizhong, Du, Dalong, Huang, Chang, and Torr, Philip HS. Conditional random fields as recurrent neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1529–1537, 2015.

Citation

MLA
Amos, B., and J. Z. Kolter. “OptNet: Differentiable Optimization as a Layer in Neural Networks”. International Conference on Machine Learning, vol. 70, 2017, pp. 136–45, https://proceedings.mlr.press/v70/amos17a.html.
APA
Amos, B., & Kolter, J. Z. (2017). OptNet: Differentiable Optimization as a Layer in Neural Networks. International Conference on Machine Learning, 70, 136–145. https://proceedings.mlr.press/v70/amos17a.html
Chicago
Amos, B., and J. Z. Kolter. 2017. “OptNet: Differentiable Optimization as a Layer in Neural Networks”. International Conference on Machine Learning 70: 136–45. https://proceedings.mlr.press/v70/amos17a.html.
Harvard
Amos, B. and Kolter, J.Z. (2017) “OptNet: Differentiable Optimization as a Layer in Neural Networks”, International Conference on Machine Learning. PMLR, pp. 136–145. Available at: https://proceedings.mlr.press/v70/amos17a.html.
Vancouver
1. Amos B, Kolter JZ (2017) OptNet: Differentiable Optimization as a Layer in Neural Networks. In: International Conference on Machine Learning. PMLR, pp 136–145

BibTeX

@InProceedings{pmlr-v70-amos17a,
  title = 	 {{O}pt{N}et: Differentiable Optimization as a Layer in Neural Networks},
  author =       {Brandon Amos and J. Zico Kolter},
  booktitle = 	 {Proceedings of the 34th International Conference on Machine Learning},
  pages = 	 {136--145},
  year = 	 {2017},
  editor = 	 {Precup, Doina and Teh, Yee Whye},
  volume = 	 {70},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {06--11 Aug},
  publisher =    {PMLR},
  pdf = 	 {http://proceedings.mlr.press/v70/amos17a/amos17a.pdf},
  url = 	 {https://proceedings.mlr.press/v70/amos17a.html},
  abstract = 	 {This paper presents OptNet, a network architecture that integrates optimization problems (here, specifically in the form of quadratic programs) as individual layers in larger end-to-end trainable deep networks. These layers encode constraints and complex dependencies between the hidden states that traditional convolutional and fully-connected layers often cannot capture. In this paper, we explore the foundations for such an architecture: we show how techniques from sensitivity analysis, bilevel optimization, and implicit differentiation can be used to exactly differentiate through these layers and with respect to layer parameters; we develop a highly efficient solver for these layers that exploits fast GPU-based batch solves within a primal-dual interior point method, and which provides backpropagation gradients with virtually no additional cost on top of the solve; and we highlight the application of these approaches in several problems. In one notable example, we show that the method is capable of learning to play mini-Sudoku (4x4) given just input and output games, with no a priori information about the rules of the game; this highlights the ability of our architecture to learn hard constraints better than other neural architectures.}
}
Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: https://creativecommons.org/licenses/by/4.0/