Back

Invariant Risk Minimization

Martin ArjovskyLéon BottouIshaan GulrajaniDavid Lopez-Paz

article2019arXiv2,976 citations

Introduces a learning paradigm that extracts causal, invariant representations across multiple training environments to achieve reliable out-of-distribution generalization.

Listen

Machine learning systems frequently fail to generalize when deployed beyond their training distributions because they latch onto spurious correlations created by selection biases, confounding factors, or environment-specific artifacts. Classic examples include image classifiers that rely on background scenery rather than object shape, or predictors that exploit varying noise levels across data sources. These failures matter now because real-world applications routinely encounter distribution shifts, yet standard training assumes training and test data come from the same distribution.

The paper introduces Invariant Risk Minimization (IRM) to learn predictors whose performance remains stable across multiple observed training environments and therefore extends reliably to unseen but related environments. The central claim is that a representation elicits an invariant predictor when the optimal classifier on top of that representation is identical for every training environment; enforcing this condition yields predictors that recover stable, causal relationships instead of transient correlations.

The authors first demonstrate the shortcomings of empirical risk minimization, robust optimization, domain adaptation, and prior invariant causal prediction methods on a simple linear structural equation model with spurious variables. They then formulate IRM as a constrained optimization problem that searches for a representation making the same classifier optimal everywhere, and derive a practical penalty-based algorithm (IRMv1) that replaces the inner optimization with a squared gradient-norm term. Theoretical analysis shows that, under linear models and a diversity condition called linear general position, invariance learned from sufficiently many training environments transfers to all possible interventions on the underlying system. Experiments compare IRM against baselines on eight synthetic variants that vary observability, noise structure, and feature scrambling, plus a Colored MNIST task where color provides a strong but unstable cue.

IRM recovers the true causal coefficients with errors often orders of magnitude smaller than competing methods and produces near-zero weight on non-causal variables. On Colored MNIST it reaches roughly 67 percent accuracy on a deliberately shifted test environment, compared with 17 percent for standard training and close to an oracle that ignores color. Two training environments suffice in these cases to identify useful invariances, and the method remains effective under moderate model misspecification.

These results imply that collecting data under distinct conditions and optimizing for invariance can produce predictors that extrapolate rather than merely interpolate, reducing the risk of silent failures when environments change. The approach therefore offers a concrete route to more reliable models in safety-critical or high-variability settings without requiring an explicit causal graph.

Next steps include extending the penalty to nonlinear classifiers, characterizing the minimal number of environments needed for nonlinear problems, and testing IRM in reinforcement learning, fairness, and self-supervised settings. Practitioners should treat the invariance penalty weight as a hyperparameter tuned on a held-out environment and verify that training environments exhibit genuine diversity.

The strongest guarantees apply to linear representations; the practical algorithm remains non-convex and can be sensitive to initialization and penalty strength. Results rest on the assumption that environments lie in linear general position, which may not hold for arbitrary data partitions.

distribution functionsout-of-distribution generalizationsvmInvariant Risk MinimizationSpurious CorrelationTraining Environment
arXiv: 1907.02893facebookresearch/InvariantRiskMinimization
  • Book: Domain-Adversarial Training of Neural Networks, Yaroslav Ganin et al. (2016). Understanding the foundational principles of domain-adversarial neural networks and domain adaptation provides the essential technical scaffolding required to grasp Invariant Risk Minimization's approach to out-of-distribution generalization.

No sufficiently relevant recommendations were found.

Cover for Invariant Risk Minimization

Abstract

We introduce Invariant Risk Minimization (IRM), a learning paradigm to estimate invariant correlations across multiple training distributions. To achieve this goal, IRM learns a data representation such that the optimal classifier, on top of that data representation, matches for all training distributions. Through theory and experiments, we show how the invariances learned by IRM relate to the causal structures governing the data and enable out-of-distribution generalization.

Topics

distribution functionsout-of-distribution generalizationsvmscale invariantintegrated dataoptimization

Keywords

Invariant Risk MinimizationSpurious CorrelationTraining EnvironmentOut‑of‑Distribution GeneralizationStructural Equation ModelInvariant PredictorRobust LearningDomain AdaptationCausal InferenceGradient Penalty

Table of Contents

  • 1 Introduction
  • 2 The many faces of generalization
  • 3 Algorithms for invariant risk minimization
  • 3.1 From () to ()
  • 3.1.1 Phrasing the constraints as a penalty
  • 3.1.2 Choosing a penalty 𝔻\mathbb{D} for linear classifiers ww
  • 3.1.3 Fixing the linear classifier ww
  • 3.1.4 Scalar fixed classifiers w~\tilde{w} are sufficient to monitor invariance
  • 3.1.5 Extending to general losses and multivariate outputs
  • 3.2 Implementation details
  • 3.3 About nonlinear invariances ww
  • 4 Invariance, causality and generalization
  • 4.1 Generalization theory for IRM
  • 4.2 On the nonlinear case and the number of environments
  • 4.3 Causation as invariance
  • 5 Experiments
  • 5.1 Synthetic data
  • 5.2 Colored MNIST
  • 6 Looking forward: a concluding dialogue
  • References
  • A Additional theorems
  • B Proofs
  • B.1 Proof of Proposition
  • B.2 Proof of Theorem
  • B.3 Proof of Theorem
  • B.4 Proof of Theorem
  • C Failure cases for Domain Adaptation
  • D Minimal implementation of IRM in PyTorch

Knowls

  1. Knowl 1 — Invariant Risk Minimization (IRM) Framework

    model/method

    Let Etr\mathcal{E}_\text{tr} denote a collection of distinct training environments where datasets De={(xie,yie)}i=1neD_e = \{(x_i^e, y_i^e)\}_{i=1}^{n_e} are collected under different experimental or observational conditions, with examples drawn i.i.d. from distribution P(Xe,Ye)P(X^e, Y^e). A data representation Φ:X→H\Phi: \mathcal{X} \to \mathcal{H} is defined to elicit an invariant predictor w∘Φw \circ \Phi across environments Etr\mathcal{E}_\text{tr} if there exists a single classifier w:H→Yw: \mathcal{H} \to \mathcal{Y} that is simultaneously optimal across all training environments:

    w∈arg⁡min⁡wˉ:H→YRe(wˉ∘Φ)for all e∈Etr,w \in \arg\min_{\bar{w}: \mathcal{H} \to \mathcal{Y}} R^e(\bar{w} \circ \Phi) \quad \text{for all } e \in \mathcal{E}_\text{tr},

    where Re(f):=EXe,Ye[ℓ(f(Xe),Ye)]R^e(f) := \mathbb{E}_{X^e, Y^e}[\ell(f(X^e), Y^e)] denotes the expected loss (risk) under environment ee for a loss function ℓ\ell.

    The Invariant Risk Minimization (IRM) principle seeks an invariant data representation Φ\Phi that minimizes empirical risk across all training environments via the constrained bilevel optimization problem:

    min⁡Φ:X→Hw:H→Y∑e∈EtrRe(w∘Φ)subject tow∈arg⁡min⁡wˉ:H→YRe(wˉ∘Φ)for all e∈Etr.\min_{\substack{\Phi: \mathcal{X} \to \mathcal{H} \\ w: \mathcal{H} \to \mathcal{Y}}} \sum_{e \in \mathcal{E}_\text{tr}} R^e(w \circ \Phi) \quad \text{subject to} \quad w \in \arg\min_{\bar{w}: \mathcal{H} \to \mathcal{Y}} R^e(\bar{w} \circ \Phi) \quad \text{for all } e \in \mathcal{E}_\text{tr}.

  2. Knowl 2 — Practical First-Order Invariant Risk Minimization (IRMv1)

    model/method

    Because the exact bilevel IRM objective is computationally intractable, the practical surrogate objective (IRMv1) fixes the classifier to a constant scalar identity w=1.0w = 1.0 (or a vector of ones for multivariate outputs) and penalizes the squared norm of the gradient of the environment risk with respect to this fixed classifier:

    min⁡Φ:X→Y∑e∈EtrRe(Φ)+λ⋅∥∇w∣w=1.0Re(w⋅Φ)∥2,\min_{\Phi: \mathcal{X} \to \mathcal{Y}} \sum_{e \in \mathcal{E}_\text{tr}} R^e(\Phi) + \lambda \cdot \left\| \nabla_{w \mid w = 1.0} R^e(w \cdot \Phi) \right\|^2,

    where Φ:X→Y\Phi: \mathcal{X} \to \mathcal{Y} represents the entire invariant predictor mapping inputs directly to the label space Y\mathcal{Y}, Re(w⋅Φ)=EXe,Ye[ℓ(w⋅Φ(Xe),Ye)]R^e(w \cdot \Phi) = \mathbb{E}_{X^e, Y^e}[\ell(w \cdot \Phi(X^e), Y^e)], and λ∈[0,∞)\lambda \in [0, \infty) is a regularization hyperparameter balancing overall predictive accuracy against the first-order stationarity condition of invariance across each environment e∈Etre \in \mathcal{E}_\text{tr}.

  3. Knowl 3 — Unbiased Stochastic Mini-Batch Estimation of the IRMv1 Invariance Penalty

    algorithm

    When optimizing the IRMv1 objective using stochastic gradient descent, evaluating the squared gradient norm ∥E[∇wℓ]∥2\|\mathbb{E}[\nabla_w \ell]\|^2 on a single mini-batch yields a biased estimator because E[∥g^∥2]=∥E[g^]∥2+Var(g^)\mathbb{E}[\|\hat{g}\|^2] = \|\mathbb{E}[\hat{g}]\|^2 + \text{Var}(\hat{g}). An unbiased estimate of the gradient penalty ∥∇w∣w=1.0Re(w⋅Φ)∥2\|\nabla_{w \mid w = 1.0} R^e(w \cdot \Phi)\|^2 is computed by drawing two independent random mini-batches of size bb, (Xe,i,Ye,i)(X^{e,i}, Y^{e,i}) and (Xe,j,Ye,j)(X^{e,j}, Y^{e,j}), from environment ee and calculating the inner product of their sample gradients:

    ∑k=1b[∇w∣w=1.0ℓ(w⋅Φ(Xke,i),Yke,i)⋅∇w∣w=1.0ℓ(w⋅Φ(Xke,j),Yke,j)].\sum_{k=1}^b \left[ \nabla_{w \mid w=1.0} \ell\left(w \cdot \Phi\left(X_k^{e,i}\right), Y_k^{e,i}\right) \cdot \nabla_{w \mid w=1.0} \ell\left(w \cdot \Phi\left(X_k^{e,j}\right), Y_k^{e,j}\right) \right].

  4. Knowl 4 — Geometric Characterization of Linear Invariant Predictors

    theoretical result

    Let E\mathcal{E} be a set of environments, and let Re:Rd→RR^e: \mathbb{R}^d \to \mathbb{R} be convex differentiable cost functions for all e∈Ee \in \mathcal{E}. A linear predictor v∈Rdv \in \mathbb{R}^d can be factored as v=Φ⊤wv = \Phi^\top w, where Φ∈Rp×d\Phi \in \mathbb{R}^{p \times d} and w∈Rpw \in \mathbb{R}^p satisfies w∈arg⁡min⁡wˉ∈RpRe(Φ⊤wˉ)w \in \arg\min_{\bar{w} \in \mathbb{R}^p} R^e(\Phi^\top \bar{w}) simultaneously for all e∈Ee \in \mathcal{E}, if and only if:

    v⊤∇Re(v)=0for all e∈E.v^\top \nabla R^e(v) = 0 \quad \text{for all } e \in \mathcal{E}.

    Furthermore, the representation matrices Φ\Phi for which such a decomposition exists are precisely those whose nullspace Ker(Φ)\text{Ker}(\Phi) is orthogonal to vv and contains all the gradient vectors ∇Re(v)\nabla R^e(v) for e∈Ee \in \mathcal{E}. As a consequence, any linear invariant predictor can be decomposed using a rank-1 representation matrix Φ∈R1×d\Phi \in \mathbb{R}^{1 \times d} with a fixed scalar classifier w~=1.0\tilde{w} = 1.0.

  5. Knowl 5 — Linear General Position of Training Environments

    assumption

    Let Xe∈RdX^e \in \mathbb{R}^d denote the observable input random vector and ϵe∈R\epsilon^e \in \mathbb{R} denote the target noise random variable in environment e∈Etre \in \mathcal{E}_\text{tr}. A set of training environments Etr\mathcal{E}_\text{tr} lies in linear general position of degree r∈Nr \in \mathbb{N} if ∣Etr∣>d−r+dr|\mathcal{E}_\text{tr}| > d - r + \frac{d}{r} and for every non-zero vector x∈Rdx \in \mathbb{R}^d:

    dim⁡(span({EXe[Xe(Xe)⊤]x−EXe,ϵe[Xeϵe]}e∈Etr))>d−r.\dim\left( \text{span}\left( \left\{ \mathbb{E}_{X^e}\left[ X^e (X^e)^\top \right] x - \mathbb{E}_{X^e, \epsilon^e}\left[ X^e \epsilon^e \right] \right\}_{e \in \mathcal{E}_\text{tr}} \right) \right) > d - r.

    This condition bounds the co-linearity of environment cross-moments, ensuring that each additional environment eliminates at least one degree of freedom in the search space of invariant linear representations.

  6. Knowl 6 — Out-of-Distribution Generalization of Linear Invariant Risk Minimization

    theoretical result

    Consider a linear data generating process where for each environment e∈Ealle \in \mathcal{E}_\text{all}, the scalar target YeY^e is generated from latent causal variables Z1e∈RcZ_1^e \in \mathbb{R}^c via:

    Ye=(Z1e)⊤γ+ϵe,Z1e⊥ϵe,E[ϵe]=0,Y^e = (Z_1^e)^\top \gamma + \epsilon^e, \quad Z_1^e \perp \epsilon^e, \quad \mathbb{E}[\epsilon^e] = 0,

    with invariant causal parameter vector γ∈Rc\gamma \in \mathbb{R}^c. The observed features are scrambled linear combinations Xe=S(Z1e,Z2e)∈RdX^e = S(Z_1^e, Z_2^e) \in \mathbb{R}^d, where Z2e∈RqZ_2^e \in \mathbb{R}^q represents non-causal features that may be arbitrarily correlated with ϵe\epsilon^e, and S∈Rd×(c+q)S \in \mathbb{R}^{d \times (c+q)} is a transformation whose Z1Z_1 component is left-invertible (i.e., there exists S~∈Rc×d\tilde{S} \in \mathbb{R}^{c \times d} such that S~S(z1,z2)=z1\tilde{S} S(z_1, z_2) = z_1 for all z1,z2z_1, z_2).

    Let Φ∈Rd×d\Phi \in \mathbb{R}^{d \times d} have rank r>0r > 0. If a training subset Etr⊆Eall\mathcal{E}_\text{tr} \subseteq \mathcal{E}_\text{all} containing at least d−r+drd - r + \frac{d}{r} environments lies in linear general position of degree rr, then the normal equations condition:

    ΦEXe[Xe(Xe)⊤]Φ⊤w=ΦEXe,Ye[XeYe]\Phi \mathbb{E}_{X^e}\left[ X^e (X^e)^\top \right] \Phi^\top w = \Phi \mathbb{E}_{X^e, Y^e}\left[ X^e Y^e \right]

    holds for all e∈Etre \in \mathcal{E}_\text{tr} if and only if Φ\Phi elicits the invariant causal predictor Φ⊤w=S~⊤γ\Phi^\top w = \tilde{S}^\top \gamma for all e∈Ealle \in \mathcal{E}_\text{all}.

  7. Knowl 7 — Generic Prevalence of the Linear General Position Condition

    theoretical result

    Let ΣX,Xe:=EXe[Xe(Xe)⊤]∈S+d×d\Sigma_{X,X}^e := \mathbb{E}_{X^e}[X^e (X^e)^\top] \in \mathbb{S}_+^{d \times d} denote the symmetric positive semi-definite second-moment matrix of inputs in environment ee, and let ΣX,ϵe:=EXe[Xeϵe]∈Rd\Sigma_{X,\epsilon}^e := \mathbb{E}_{X^e}[X^e \epsilon^e] \in \mathbb{R}^d.

    For any arbitrary fixed tuple of cross-covariance vectors (ΣX,ϵe)e∈Etr∈(Rd)∣Etr∣(\Sigma_{X,\epsilon}^e)_{e \in \mathcal{E}_\text{tr}} \in (\mathbb{R}^d)^{|\mathcal{E}_\text{tr}|}, the set of covariance tuples:

    {(ΣX,Xe)e∈Etr∈(S+d×d)∣Etr∣  |  Etr does not satisfy linear general position of degree r}\left\{ (\Sigma_{X,X}^e)_{e \in \mathcal{E}_\text{tr}} \in (\mathbb{S}_+^{d \times d})^{|\mathcal{E}_\text{tr}|} \;\middle|\; \mathcal{E}_\text{tr} \text{ does not satisfy linear general position of degree } r \right\}

    has Lebesgue measure zero in (S+d×d)∣Etr∣(\mathbb{S}_+^{d \times d})^{|\mathcal{E}_\text{tr}|} whenever ∣Etr∣>d−r+dr|\mathcal{E}_\text{tr}| > d - r + \frac{d}{r}.

  8. Knowl 8 — Equivalence of Robust Risk Minimization to Weighted Empirical Risk Minimization

    theoretical result

    Consider the robust learning / distributionally robust optimization objective over training environments Etr\mathcal{E}_\text{tr}:

    Rrob(f)=max⁡e∈Etr(Re(f)−re),R_\text{rob}(f) = \max_{e \in \mathcal{E}_\text{tr}} \left( R^e(f) - r_e \right),

    where re∈Rr_e \in \mathbb{R} are fixed environment baselines (e.g., re=0r_e = 0 for worst-case risk, or re=V[Ye]r_e = \mathbb{V}[Y^e] for minimax unexplained variance).

    Under Karush-Kuhn-Tucker (KKT) differentiability and qualification conditions, there exist environment weights λe≥0\lambda_e \ge 0 such that any minimizer f∗f^* of RrobR_\text{rob} is a first-order stationary point of the weighted mixture risk ∑e∈EtrλeRe(f)\sum_{e \in \mathcal{E}_\text{tr}} \lambda_e R^e(f). Consequently, robust risk minimization and Empirical Risk Minimization (the special case where λe=1/∣Etr∣\lambda_e = 1 / |\mathcal{E}_\text{tr}|) interpolate within the convex hull of training environments and cannot isolate invariant causal relationships when spurious correlations are present across all training environments, leading to unbounded worst-case risk ROOD(f)=max⁡e∈EallRe(f)R_\text{OOD}(f) = \max_{e \in \mathcal{E}_\text{all}} R^e(f) under out-of-distribution interventions.

  9. Knowl 9 — Out-of-Distribution Generalization Performance on Colored MNIST

    data/table

    Colored MNIST is a synthetic binary classification task where digits 0–4 are assigned preliminary label y~=0\tilde{y} = 0 and digits 5–9 are assigned y~=1\tilde{y} = 1. The true label yy is obtained by flipping y~\tilde{y} with probability 0.25. The image is colored red (z=1z = 1) or green (z=0z = 0) by flipping yy with environment-specific probability pep^e. In training environment 1, pe1=0.2p^{e_1} = 0.2; in training environment 2, pe2=0.1p^{e_2} = 0.1; in the unseen test environment, petest=0.9p^{e_\text{test}} = 0.9 (reversing the correlation between color and label).

    Multi-layer perceptrons (MLPs) were trained with Empirical Risk Minimization (ERM), Invariant Risk Minimization (IRMv1), and compared to a grayscale oracle MLP.

    Algorithm Acc. train envs. (%) Acc. test env. (%)
    ERM 87.4±0.287.4 \pm 0.2 17.1±0.617.1 \pm 0.6
    IRM (ours) 70.8±0.970.8 \pm 0.9 66.9±2.566.9 \pm 2.5
    Random guessing (hypothetical) 50.050.0 50.050.0
    Optimal invariant model (hypothetical) 75.075.0 75.075.0
    ERM, grayscale model (oracle) 73.5±0.273.5 \pm 0.2 73.0±0.473.0 \pm 0.4

    ERM relies predominantly on the spurious color feature due to its strong correlation with the label in both training environments (80%80\% and 90%90\%), resulting in failure below random chance on the test distribution (17.1%17.1\%) where the correlation reverses. IRM detects that the color correlation varies across the two training environments and discards it in favor of the invariant digit shape features, achieving 66.9%66.9\% test accuracy, near the grayscale oracle (73.0%73.0\%) and the theoretical invariant ceiling (75.0%75.0\%).

  10. Knowl 10 — Comparative Evaluation of IRM, ERM, and ICP on Scrambled and Confounded Linear SEMs

    empirical result

    In synthetic structural equation experiments predicting Ye=(Z1e)⊤W1→y+N(0,σy2)+Wh→yHeY^e = (Z_1^e)^\top W_{1 \to y} + \mathcal{N}(0, \sigma_y^2) + W_{h \to y} H^e from features Xe=S(Z1e,Z2e)X^e = S(Z_1^e, Z_2^e) with non-causal effect Z2e=Wy→2Ye+N(0,σ22)+Wh→2HeZ_2^e = W_{y \to 2} Y^e + \mathcal{N}(0, \sigma_2^2) + W_{h \to 2} H^e across training environments Etr={0.2,2,5}\mathcal{E}_\text{tr} = \{0.2, 2, 5\}, eight configurations were evaluated across fully-observed vs. partially-observed (hidden confounder HH), homoskedastic vs. heteroskedastic noise, and scrambled (SS orthogonal) vs. unscrambled (S=IS = I) observations.

    IRMv1 achieved orders-of-magnitude lower squared error on the recovered causal coefficients M^1→y=W^1→y⊤S⊤\hat{M}_{1 \to y} = \hat{W}_{1 \to y}^\top S^\top than ERM across all eight settings, and drove the estimated weights on the non-causal variables M^y→2\hat{M}_{y \to 2} close to zero. In contrast, ERM assigned large weights to non-causal variables Z2eZ_2^e due to spurious correlation with the noise of YeY^e. Invariant Causal Prediction (ICP) exhibited conservative behavior, rejecting causal covariates in the presence of confounding or heteroskedasticity and yielding large causal estimation errors.

  11. Knowl 11 — Incompatibility of Domain Adaptation Matching Objectives with Invariant Prediction

    limitation

    Domain adaptation objectives that match feature distributions across environments can conflict fundamentally with invariant causal prediction:

    1. Marginal Feature Matching (Adversarial Domain Adaptation): Enforcing P(Φ(Xes))=P(Φ(Xet))P(\Phi(X^{e_s})) = P(\Phi(X^{e_t})) between source ese_s and target ete_t forces identical output label marginals P(w(Φ(Xes)))=P(w(Φ(Xet)))P(w(\Phi(X^{e_s}))) = P(w(\Phi(X^{e_t}))). When the true label prior shifts across domains (e.g., P(Yes=1)=0.5P(Y^{e_s}=1) = 0.5 while P(Yet=1)=0.9P(Y^{e_t}=1) = 0.9), feature matching severely degrades target classification accuracy (bounded above by 20%20\% in this binary case), whereas invariant prediction without marginal matching generalizes correctly.

    2. Conditional Feature Matching (Conditional Domain Adaptation): Enforcing P(Φ(Xes)∣Yes)=P(Φ(Xet)∣Yet)P(\Phi(X^{e_s}) \mid Y^{e_s}) = P(\Phi(X^{e_t}) \mid Y^{e_t}) almost surely rules out the true invariant representation Φ(x)=x\Phi(x) = x when the marginal input distribution P(Xe)P(X^e) shifts across environments, because the set of environment feature distributions satisfying exact conditional equality has Lebesgue measure zero.

Coverage note — None was omitted; the theoretical framework (invariance definitions, linear characterization, generalization bounds, general position theorem), practical IRMv1 surrogate, optimization dynamics, domain adaptation failure analysis, and all empirical benchmarks were fully covered, while informal conversational dialogues in Section 6 were omitted.

References

  1. 1.John Aldrich. Autonomy. Oxford Economic Papers, 1989.
  2. 2.James Andrew Bagnell. Robust supervised learning. In AAAI, 2005.
  3. 3.Peter L. Bartlett, Philip M. Long, Gabor Lugosi, and Alexander Tsigler. Benign Overfitting in Linear Regression. arXiv, 2019.
  4. 4.Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In ECCV, 2018.
  5. 5.Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In NIPS. 2007.
  6. 6.Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust optimization. Princeton University Press, 2009.
  7. 7.Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sebastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to disentangle causal mechanisms. arXiv, 2019.
  8. 8.Leon Bottou, Corinna Cortes, John S. Denker, Harris Drucker, Isabelle Guyon, Lawrence D. Jackel, Yann Le Cun, Urs A. Muller, Eduard Sackinger, Patrice Simard, and Vladimir Vapnik. Comparison of classifier methods: a case study in handwritten digit recognition. In ICPR, 1994.
  9. 9.Wieland Brendel and Matthias Bethge. Approximating CNNs with bag-of-local-features models works surprisingly well on imagenet. In ICLR, 2019.
  10. 10.Joan Bruna and Stephane Mallat. Invariant scattering convolution networks. TPAMI, 2013.
  11. 11.Joan Bruna, Stephane Mallat, Emmanuel Bacry, and Jean-Franois Muzy. Intermittent process analysis with scattering moments. The Annals of Statistics, 2015.
  12. 12.Nancy Cartwright. Two theorems on invariance and causality. Philosophy of Science, 2003.
  13. 13.Patricia W. Cheng and Hongjing Lu. Causal invariance as an essential constraint for creating a causal representation of the world. The Oxford handbook of causal reasoning, 2017.
  14. 14.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.
  15. 15.John Duchi, Peter Glynn, and Hongseok Namkoong. Statistics of robust optimization: A generalized empirical likelihood approach. arXiv, 2016.
  16. 16.Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francois Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR, 2016.
  17. 17.Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. ICLR, 2019.
  18. 18.AmirEmad Ghassami, Saber Salehkaleybar, Negar Kiyavash, and Kun Zhang. Learning causal structures using regression invariance. In NIPS, 2017.
  19. 19.Patrick J. Grother. NIST Special Database 19: Handprinted forms and characters database. https://www.nist.gov/srd/nist-special-database-19, 1995. File doc/doc.ps in the 1995 NIST CD ROM NIST Special Database 19.
  20. 20.Trygve Haavelmo. The probability approach in econometrics. Econometrica: Journal of the Econometric Society, 1944.
  21. 21.Christina Heinze-Deml and Nicolai Meinshausen. Conditional variance penalties and domain shift robustness. arXiv, 2017.
  22. 22.Christina Heinze-Deml, Jonas Peters, and Nicolai Meinshausen. Invariant causal prediction for nonlinear models. Journal of Causal Inference, 2018.
  23. 23.Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. Revisiting visual question answering baselines. In ECCV, 2016.
  24. 24.Fredrik D. Johansson, David A. Sontag, and Rajesh Ranganath. Support and invertibility in domain-invariant representations. AISTATS, 2019.
  25. 25.Niki Kilbertus, Giambattista Parascandolo, and Bernhard Scholkopf. Generalization in anti-causal learning. arXiv, 2018.
  26. 26.Kun Kuang, Peng Cui, Susan Athey, Ruoxuan Xiong, and Bo Li. Stable prediction across unknown environments. In SIGKDD, 2018.
  27. 27.Brenden M. Lake, Tomer D. Ullman, Joshua B Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 2017.
  28. 28.James M. Lee. Introduction to Smooth Manifolds. Springer, 2003.
  29. 29.David Lewis. Counterfactuals. John Wiley & Sons, 2013.
  30. 30.Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In ECCV, 2018.
  31. 31.David Lopez-Paz. From dependence to causation. PhD thesis, University of Cambridge, 2016.
  32. 32.David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, and Leon Bottou. Discovering causal signals in images. In CVPR, 2017.
  33. 33.Gilles Louppe, Michael Kagan, and Kyle Cranmer. Learning to pivot with adversarial networks. In Advances in neural information processing systems, pages 981–990, 2017.
  34. 34.Sara Magliacane, Thijs van Ommen, Tom Claassen, Stephan Bongers, Philip Versteeg, and Joris M Mooij. Domain adaptation by using causal inference to predict invariant conditional distributions. In NIPS, 2018.
  35. 35.Gary Marcus. Deep learning: A critical appraisal. arXiv, 2018.
  36. 36.Nicolai Meinshausen. Causality from a distributional robustness point of view. In Data Science Workshop (DSW), 2018.
  37. 37.Nicolai Meinshausen and Peter Buhlmann. Maximin effects in inhomogeneous large-scale data. The Annals of Statistics, 2015.
  38. 38.Sandra D. Mitchell. Dimensions of scientific law. Philosophy of Science, 2000.
  39. 39.Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, 2nd edition, 2009.
  40. 40.Jonas Peters, Peter Buhlmann, and Nicolai Meinshausen. Causal inference using invariant prediction: identification and confidence intervals. JRSS B, 2016.
  41. 41.Jonas Peters, Dominik Janzing, and Bernhard Scholkopf. Elements of causal inference: foundations and learning algorithms. MIT press, 2017.
  42. 42.Michael Redhead. Incompleteness, non locality and realism. a prolegomenon to the philosophy of quantum mechanics. 1987.
  43. 43.Mateo Rojas-Carulla, Bernhard Scholkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. JMLR, 2018.
  44. 44.Donald B. Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 1974.
  45. 45.Bernhard Scholkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. In ICML, 2012.
  46. 46.Aman Sinha, Hongseok Namkoong, and John Duchi. Certifying some distributional robustness with principled adversarial training. ICLR, 2018.
  47. 47.Brian Skyrms. Causal necessity: a pragmatic investigation of the necessity of laws. Yale University Press, 1980.
  48. 48.Bob L. Sturm. A simple method to determine if a music information retrieval system is a ‐horse‑. IEEE Transactions on Multimedia, 2014.
  49. 49.Antonio Torralba and Alexei Efros. Unbiased look at dataset bias. In CVPR, 2011.
  50. 50.Vladimir Vapnik. Principles of risk minimization for learning theory. In NIPS. 1992.
  51. 51.Vladimir N. Vapnik. Statistical Learning Theory. John Wiley & Sons, 1998.
  52. 52.Max Welling. Do we still need models or just more data and compute?, 2019.
  53. 53.Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In NIPS. 2017.
  54. 54.James Woodward. Making things happen: A theory of causal explanation. Oxford university press, 2005.
  55. 55.Sewall Wright. Correlation and causation. Journal of agricultural research, 1921.
  56. 56.Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2016.

Citation

MLA
Arjovsky, M., et al. “Invariant Risk Minimization”. arXiv, 2019, https://doi.org/10.48550/arxiv.1907.02893.
APA
Arjovsky, M., Bottou, L., Gulrajani, I., & Lopez-Paz, D. (2019). Invariant Risk Minimization. arXiv. https://doi.org/10.48550/arxiv.1907.02893
Chicago
Arjovsky, M., L. Bottou, I. Gulrajani, and D. Lopez-Paz. 2019. “Invariant Risk Minimization”. Preprint, ArXiv. https://doi.org/10.48550/arxiv.1907.02893.
Harvard
Arjovsky, M. et al. (2019) “Invariant Risk Minimization”. arXiv. Available at: https://doi.org/10.48550/arxiv.1907.02893.
Vancouver
1. Arjovsky M, Bottou L, Gulrajani I, Lopez-Paz D (2019) Invariant Risk Minimization. https://doi.org/10.48550/arxiv.1907.02893

BibTeX

@misc{https://doi.org/10.48550/arxiv.1907.02893,
  doi = {10.48550/ARXIV.1907.02893},
  url = {https://arxiv.org/abs/1907.02893},
  author = {Arjovsky, Martin and Bottou, Léon and Gulrajani, Ishaan and Lopez-Paz, David},
  keywords = {Machine Learning (stat.ML), Artificial Intelligence (cs.AI), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {Invariant Risk Minimization},
  publisher = {arXiv},
  year = {2019},
  copyright = {arXiv.org perpetual, non-exclusive license}
}
Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors