All Models are Wrong, but Many are Useful: Learning a Variable’s Importance by Studying an Entire Class of Prediction Models Simultaneously

Machine learning models are increasingly deployed in high-stakes domains such as criminal sentencing and healthcare, yet conventional variable importance methods only describe how much a single, specific model depends on a given feature. In practice, many distinct models achieve nearly identical prediction accuracy while relying on completely different variables—a phenomenon known as the Rashomon effect. Evaluating variable importance using only one arbitrarily chosen model creates substantial risks for transparency, fairness, and compliance, especially when auditing proprietary or black-box algorithms.

This article introduces Model Class Reliance (MCR), a statistical framework designed to estimate the full upper and lower bounds of reliance placed on a variable across an entire class of well-performing prediction models (the Rashomon set). By evaluating the minimum and maximum degree to which any high-performing model relies on a feature, MCR provides a comprehensive and robust characterization that reflects the nature of the prediction problem rather than an individual analyst's modeling choices.

To establish this framework, the authors define Model Reliance (MR) using permutation-based loss ratios and draw novel theoretical connections to U-statistics, causal treatment effects, and additive model coefficients. They derive finite-sample probabilistic bounds to guarantee the accuracy of empirical MCR estimates and develop practical computational procedures. These optimization procedures formulate MCR estimation as quadratic programs with quadratic constraints (QP1QC) for regularized linear models and kernel regression in reproducing kernel Hilbert spaces (RKHS), and as convex relaxations or difference-of-convex programs for broader model classes. The methodology is evaluated using simulated datasets and applied to real-world criminal recidivism records from Broward County, Florida, to audit the reliance of the proprietary COMPAS risk tool on sensitive demographics.

Key findings include:

1. Permutation-based importance measures are formal U-statistics, enabling finite-sample probabilistic bounds and proving that variable importance can be reliably estimated in-sample without mandatory sample-splitting if model complexity is controlled.
2. In simulation benchmarks under increasing model misspecification, MCR bootstrap confidence intervals maintained nominal 95% coverage at moderate non-linearity levels where standard single-model bootstrap methods failed.
3. In the criminal recidivism study, the empirical MCR range for inadmissible demographic variables (race and sex) was [1.00, 1.56] with a 95% bootstrap confidence interval of [1.00, 1.73], meaning that no well-performing model in the class increases its loss by more than 56% to 73% when race and sex are scrambled.
4. In contrast, admissible criminal history features (age, prior convictions, felony status) exhibited an MCR range of [1.77, 3.61] with a 95% bootstrap confidence interval of [1.62, 3.96], demonstrating that criminal history and age are the primary predictive drivers of COMPAS scores rather than race and sex directly.

These findings provide significant implications for algorithmic auditing, regulatory compliance, and risk governance. MCR enables decision-makers to audit proprietary, black-box decision tools without knowing their exact internal code by checking whether well-performing proxy models must rely on sensitive attributes. Furthermore, a low upper bound (MCR+) proves that a feature can be safely eliminated from a production pipeline without sacrificing predictive accuracy, directly informing variable selection and reducing data collection costs.

Organizations evaluating high-stakes predictive models should adopt MCR to establish worst-case and best-case feature dependence before deployment. When features exhibit strong correlations, analysts should compute Conditional Model Class Reliance (CMCR) to isolate unique predictive value. Further research and development are recommended to design scalable, exact MCR computation algorithms for complex deep neural networks and non-convex model classes.

The findings carry high statistical confidence for linear, regularized, and kernel-based models, supported by non-asymptotic mathematical proofs. However, users should exercise caution regarding boundary conditions: MCR upper bounds can be constrained by regularization limits, and observational datasets may still contain unmeasured proxy variables that conflate admissible legal factors with systemic demographic disparities.