An empirical comparison of supervised learning algorithms
R. CaruanaAlexandru Niculescu-Mizil
Presents a comprehensive benchmark of ten supervised learning algorithms across nine performance metrics, revealing how post-training probability calibration enables boosted decision trees and random forests to outperform methods like neural networks and support vector machines.
This large-scale empirical study compares ten supervised learning algorithms—support vector machines, neural networks, logistic regression, naive Bayes, memory-based learning, random forests, decision trees, bagged trees, boosted trees, and boosted stumps—across nine performance metrics on eight binary classification problems drawn from standard repositories and real-world sources. The work addresses the scarcity of comprehensive benchmarks that include newer ensemble methods and metrics beyond simple accuracy, such as ranking measures used in information retrieval and probability calibration needed for risk assessment or decision thresholds. Such comparisons matter because algorithms optimize different internal criteria, and performance on one metric often fails to predict results on others.
The study trained roughly 2000 model variants per problem, selected the best parameter settings via cross-validation on held-out validation data, and evaluated final performance on large independent test sets. Results were normalized to a common scale to allow averaging across metrics and datasets, and predictions from models not designed to output probabilities were adjusted using Platt scaling and isotonic regression before re-evaluation.
Before calibration, neural networks, bagged trees, and random forests delivered the strongest average performance across all metrics. After calibration, boosted trees rose to the top overall, followed closely by random forests, while neural networks remained competitive without adjustment. Calibration produced large gains for boosted trees, support vector machines, and naive Bayes on squared error, cross-entropy, and calibration error, but offered little or no benefit to neural networks or bagged trees. Boosted stumps consistently underperformed boosted full trees, and logistic regression and naive Bayes ranked lowest on average.
These patterns indicate that modern ensemble methods can achieve excellent results on both ranking and probability metrics once their outputs are properly scaled, whereas neural networks provide robust performance with fewer post-processing steps. For applications that require well-calibrated probabilities, such as medical risk scoring or cost-sensitive decisions, calibration should be treated as a standard final step rather than an optional refinement. When ranking or threshold-based decisions dominate, uncalibrated boosted trees and random forests already perform strongly.
No algorithm dominated every problem or metric, confirming that selection must still be validated on the target data. Practitioners should therefore run modest calibration experiments on their own validation sets and compare a small number of top candidates rather than defaulting to any single method. The findings rest on eight datasets and five trials per problem; broader testing on additional domains, larger training sets, and newer algorithms would increase confidence in the rankings.
- Paper: A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection, Ron Kohavi (1995). Reading this foundational study on cross-validation and bootstrap accuracy estimation improves understanding of the evaluation protocols used to compare supervised algorithms.
- Paper: Statistical Comparisons of Classifiers over Multiple Data Sets, Janez Demšar (2006). Understanding these proper statistical tests for comparing classifiers across multiple datasets provides essential background for interpreting the performance differences reported in the source paper.
- Paper: Random Forests, Leo Breiman (2001). Reviewing Breiman's introduction to random forests is essential for grasping the mechanics of one of the top-performing ensemble methods evaluated in the source comparison.
- Paper: Greedy function approximation: A gradient boosting machine, Jerome H. Friedman (2001). Studying Friedman's gradient boosting machine provides the necessary algorithmic foundation for the boosted tree and stump methods benchmarked in the source study.
- Paper: Experiments with a New Boosting Algorithm, Yoav Freund et al. (1996). Reading the original work on AdaBoost clarifies the boosting algorithms whose predictive performance and calibration are examined across multiple criteria in the source evaluation.
- Paper: Bagging Predictors, L. Breiman (1996). Understanding bagging predictors provides critical context for the bagged tree methods whose accuracy and probability calibration are compared in the source paper.
- Paper: Support-vector networks, Corinna Cortes et al. (1995). Examining the foundational support-vector machine paper helps clarify the optimization and probability calibration challenges that motivate the scaling techniques discussed in the source study.
- Paper: Do we need hundreds of classifiers to solve real world classification problems?, Manuel Fernández Delgado et al. (2014). This later study naturally extends the source's empirical comparison by scaling up to 179 classifiers and 121 datasets to provide definitive guidance on top-performing algorithms.
- Paper: On Calibration of Modern Neural Networks, Chuan Guo et al. (2017). Building directly on the source's findings regarding model calibration, this paper investigates the miscalibration of modern neural networks and evaluates post-hoc scaling techniques.
- Paper: Extremely randomized trees, Pierre Geurts et al. (2006). This paper continues the exploration of tree-based ensembles evaluated in the source by introducing and benchmarking extremely randomized trees against standard bagging and random forests.
- Paper: XGBoost: A Scalable Tree Boosting System, Tianqi Chen et al. (2016). Extending the gradient boosting methods compared in the source, this work introduces a highly scalable tree boosting system optimized for speed and large datasets.
- Paper: CatBoost: unbiased boosting with categorical features, Liudmila Prokhorenkova et al. (2018). This paper advances gradient boosting technology beyond the algorithms evaluated in the source by introducing ordered boosting to eliminate target leakage with categorical features.
- Paper: LightGBM: A Highly Efficient Gradient Boosting Decision Tree, Guolin Ke et al. (2017). Building on the gradient boosting algorithms examined in the source, this chapter presents LightGBM to dramatically accelerate training times on massive datasets.
