Efficient and Robust Automated Machine Learning

Matthias FeurerAaron KleinKatharina EggenspergerJost Tobias SpringenbergManuel BlumFrank Hutter

article2015NeurIPS1,980 citations

Introduces Auto-sklearn, an automated machine learning framework that significantly accelerates algorithm selection and hyperparameter optimization across scikit-learn pipelines by combining meta-learning warmstarts with post-hoc ensemble construction.

Listen

Organizations increasingly seek to deploy machine learning solutions, but successful implementation typically requires specialized data science expertise to select algorithms, preprocess features, and fine-tune hyperparameters. Automated machine learning aims to remove this barrier by producing high-performing predictive models autonomously within a fixed compute budget. The article evaluates AUTO-SKLEARN, an automated machine learning system built on the scikit-learn framework, to demonstrate how incorporating past task experience and automated model ensembling can significantly boost efficiency and predictive accuracy.

The authors conducted a comprehensive empirical evaluation using 140 diverse classification datasets from the OpenML repository. To solve the combined algorithm selection and hyperparameter optimization problem across 15 classifiers, 14 feature preprocessors, and 4 data preprocessing methods (spanning 110 hyperparameters), the system combines tree-based Bayesian optimization with two key innovations. First, it uses meta-learning to identify similar past datasets based on 38 dataset characteristics and warmstarts the search with previously successful model configurations. Second, rather than discarding suboptimal configurations evaluated during the search, it uses an efficient greedy selection procedure to automatically build a 50-model ensemble from the collected models.

The findings demonstrate substantial performance improvements over existing automated tools. In baseline comparisons across 21 standard benchmark datasets, AUTO-SKLEARN matched or outperformed AUTO-WEKA in 18 instances and outperformed HYPEROPT-SKLEARN on all datasets where the latter functioned properly. On the broader collection of 140 datasets, meta-learning provided immediate accuracy gains from the very first evaluated configuration, while automated ensembling consistently improved performance across both short and extended time budgets. Tree-based ensemble classifiers such as random forests and gradient boosting proved the most robust across datasets, while simpler baselines like decision trees, k-nearest neighbors, and discriminant analysis routinely underperformed. Overall, searching the combined space of preprocessing methods and classifiers proved far more robust than optimizing any single model type in isolation.

These results show that automated machine learning can dramatically lower the barriers to entry and operational costs for deploying machine learning, reducing the reliance on scarce expert data scientists and manual trial-and-error workflows. By re-using evaluated models in an ensemble rather than relying on a single final point estimate, the system also reduces the operational risk of overfitting to validation data.

Organizations should consider adopting automated frameworks like AUTO-SKLEARN as open-source baselines to accelerate modeling pipelines and democratize machine learning across non-specialist teams. However, decision-makers should note that the current implementation is restricted to classification tasks on small-to-medium-sized datasets and does not support regression or semi-supervised tasks. Future work is needed to extend these meta-learning and ensembling methods to large-scale deep learning frameworks before they can be applied to large-scale unstructured data domains.

University of Freiburg bayesian methods optimization optimization problems automated machine learning auto-sklearn hyperparameter optimization

automl/auto-sklearn Feurer et al (2015).pdf

Paper: Scikit-learn: Machine Learning in Python, Fabian Pedregosa et al. (2011). This paper presents the underlying scikit-learn library, whose structured pipelines of classifiers and preprocessors form the concrete hypothesis space optimized by Auto-sklearn.
Paper: Algorithms for Hyper-Parameter Optimization, James Bergstra et al. (2011). This work establishes foundational sequential model-based Bayesian optimization algorithms for hyperparameter tuning in structured and conditional search spaces.
Paper: Practical Bayesian Optimization of Machine Learning Algorithms, Jasper Snoek et al. (2012). This study demonstrates practical Bayesian optimization techniques using Gaussian processes to automate hyperparameter selection for machine learning algorithms.
Paper: Random Search for Hyper-Parameter Optimization, James Bergstra et al. (2012). This foundational paper analyzes the efficiency of random hyperparameter search compared to grid search, establishing essential baselines and motivation for automated hyperparameter optimization.
Paper: A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning, Eric Brochu et al. (2010). This tutorial provides essential background on acquisition functions and surrogate modeling for optimizing expensive black-box machine learning objectives.
Paper: On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation, G. Cawley et al. (2010). This work details the statistical risks of overfitting and selection bias during hyperparameter tuning, motivating the post-hoc ensembling and cross-validation strategies used in Auto-sklearn.
Paper: Do we need hundreds of classifiers to solve real world classification problems?, Manuel Fernández Delgado et al. (2014). This extensive benchmark across 121 datasets demonstrates classifier performance variability, providing empirical justification for searching across diverse algorithm families.

Paper: Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization, Lisha Li et al. (2016). This paper introduces Hyperband to dramatically speed up hyperparameter search using multi-armed bandits and early stopping, addressing the computational bottlenecks of Bayesian optimization.
Paper: Optuna: A Next-generation Hyperparameter Optimization Framework, Takuya Akiba et al. (2019). This paper presents Optuna, a next-generation hyperparameter optimization framework featuring dynamic search space definition and trial pruning.
Paper: On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice, Li Yang et al. (2020). This survey provides an updated, comprehensive overview of modern hyperparameter optimization and AutoML software architectures developed since Auto-sklearn.
Paper: ModelDB: a system for machine learning model management, Manasi Vartak et al. (2016). This paper introduces ModelDB to systematically log, store, and manage the numerous complex pipelines generated during automated ML exploration.
Paper: Accelerating the Machine Learning Lifecycle with MLflow, Matei Zaharia et al. (2018). This work presents MLflow, extending lifecycle management to track, package, and deploy complex machine learning pipelines in production.
Paper: The ML test score: A rubric for ML production readiness and technical debt reduction, Eric Breck et al. (2017). This paper establishes a testing rubric to validate reliability and monitor technical debt when deploying complex, automated machine learning pipelines into production.

Cover for Efficient and Robust Automated Machine Learning

Abstract

The success of machine learning in a broad range of applications has led to an ever-growing demand for machine learning systems that can be used off the shelf by non-experts. To be effective in practice, such systems need to automatically choose a good algorithm and feature preprocessing steps for a new dataset at hand, and also set their respective hyperparameters. Recent work has started to tackle this automated machine learning (AutoML) problem with the help of efficient Bayesian optimization methods. Building on this, we introduce a robust new AutoML system based on scikit-learn (using 15 classifiers, 14 feature preprocessing methods, and 4 data preprocessing methods, giving rise to a structured hypothesis space with 110 hyperparameters). This system, which we dub AUTO-SKLEARN, improves on existing AutoML methods by automatically taking into account past performance on similar datasets, and by constructing ensembles from the models evaluated during the optimization. Our system won the first phase of the ongoing ChaLearn AutoML challenge, and our comprehensive analysis on over 100 diverse datasets shows that it substantially outperforms the previous state of the art in AutoML. We also demonstrate the performance gains due to each of our contributions and derive insights into the effectiveness of the individual components of AUTO-SKLEARN.

Topics

bayesian methods optimization optimization problems regularization parameters ensemble classifiers

Keywords

automated machine learning auto-sklearn hyperparameter optimization algorithm selection Bayesian optimization ensemble selection meta-learning ChaLearn AutoML challenge

1 Introduction
2 AutoML as a CASH problem
3 New methods for increasing efficiency and robustness of AutoML
3.1 Meta-learning for finding good instantiations of machine learning frameworks
3.2 Automated ensemble construction of models evaluated during optimization
4 A practical automated machine learning system
5 Comparing AUTO-SKLEARN to AUTO-WEKA and HYPEROPT-SKLEARN
6 Evaluation of the proposed AutoML improvements
7 Detailed analysis of AUTO-SKLEARN components
8 Discussion and Conclusion
Acknowledgments
References

Knowls

Knowl 1 — Combined Algorithm Selection and Hyperparameter Optimization Problem
definition
The Combined Algorithm Selection and Hyperparameter optimization (CASH) problem formalizes AutoML as a joint optimization over algorithm choice and algorithm-specific hyperparameters.

Let $\mathcal{A} = \{A^{(1)}, \dots, A^{(R)}\}$ denote a finite set of machine learning algorithms, where each algorithm $A^{(j)}$ possesses a hyperparameter space $\Lambda^{(j)}$ . Let $D_{\text{train}} = \{(x_1, y_1), \dots, (x_n, y_n)\}$ be a training dataset partitioned into $K$ cross-validation folds $\{D_{\text{valid}}^{(1)}, \dots, D_{\text{valid}}^{(K)}\}$ and $\{D_{\text{train}}^{(1)}, \dots, D_{\text{train}}^{(K)}\}$ such that $D_{\text{train}}^{(i)} = D_{\text{train}} \setminus D_{\text{valid}}^{(i)}$ for $i = 1, \dots, K$ .

Given a loss metric $\mathcal{L}(A_\lambda^{(j)}, D_{\text{train}}^{(i)}, D_{\text{valid}}^{(i)})$ representing the loss achieved on $D_{\text{valid}}^{(i)}$ by algorithm $A^{(j)}$ when trained on $D_{\text{train}}^{(i)}$ with hyperparameter configuration $\lambda \in \Lambda^{(j)}$ , the CASH problem is to find the optimal algorithm $A^*$ and hyperparameter configuration $\lambda^*$ that minimizes the cross-validation loss:

$A^*, \lambda^* \in \arg\min_{A^{(j)} \in \mathcal{A}, \lambda \in \Lambda^{(j)}} \frac{1}{K} \sum_{i=1}^K \mathcal{L}\left(A_\lambda^{(j)}, D_{\text{train}}^{(i)}, D_{\text{valid}}^{(i)}\right)$
Knowl 2 — Auto-sklearn System Architecture
model/method
AUTO-SKLEARN is an automated machine learning framework built on top of scikit-learn that solves the CASH problem using Sequential Model-based Algorithm Configuration (SMAC) based on random forests. It augments Bayesian hyperparameter optimization with two major components:
1. Meta-Learning for Warmstarting: Instead of initializing the Bayesian optimizer uniformly at random, AUTO-SKLEARN matches the meta-features of a new dataset against a precomputed meta-dataset to warmstart SMAC with configurations that succeeded on similar datasets.
2. Automated Ensemble Construction: Rather than retaining only the single best-performing pipeline configuration found during optimization, all evaluated models are stored and combined post-hoc into an ensemble using greedy ensemble selection.
The search space covers an entire machine learning pipeline comprising data preprocessing (imputation, scaling, one-hot encoding, and target balancing), feature preprocessing (14 optional methods), and supervised classification (15 classifiers), resulting in a hierarchical space of 110 hyperparameters.
Knowl 3 — Meta-Learning Warmstart Procedure for CASH
model/method
AUTO-SKLEARN employs meta-learning to identify promising machine learning pipeline instantiations for a target dataset prior to running Bayesian optimization:
- Offline Phase: For a collection of 140 historical datasets (e.g., from OpenML), 38 meta-features are extracted. These include simple dataset statistics (e.g., number of instances, features, classes), statistical meta-features (e.g., data skewness), and information-theoretic meta-features (e.g., target entropy). Landmarking meta-features are excluded due to high online computation cost. SMAC is executed on each dataset for 24 hours (with 10-fold cross-validation on two-thirds of the data) to discover and store the pipeline configuration achieving the best performance on the remaining third.
- Online Phase: When presented with a new dataset $D$ , AUTO-SKLEARN computes its 38 meta-features, calculates the $L_1$ distance in normalized meta-feature space between $D$ and all historical datasets, and retrieves the stored optimal pipeline configurations of the $k = 25$ nearest datasets. These $k$ configurations are evaluated first on $D$ to initialize the random forest surrogate model in SMAC before standard Bayesian exploration and exploitation proceed.
Knowl 4 — Post-Hoc Greedy Ensemble Selection
algorithm
To exploit the diverse collection of models trained during Bayesian hyperparameter optimization without discarding near-optimal solutions, AUTO-SKLEARN applies Caruana et al.'s greedy ensemble selection algorithm on validation set predictions to construct an ensemble (target size 50):
Input: Library of evaluated models $\mathcal{M} = \{M_1, \dots, M_N\}$ with validation predictions $\hat{Y}_{M_i}$ , true validation targets $Y_{\text{valid}}$ , target ensemble size $S = 50$ , validation loss function $\mathcal{L}$
Output: Ensemble $E$ of models with integer weights
Initialize $E \leftarrow \emptyset$
for $t = 1$ to $S$ do
$M^* \leftarrow \arg\min_{M \in \mathcal{M}} \mathcal{L}\left(\frac{1}{t}\left(\sum_{M' \in E} \hat{Y}_{M'} + \hat{Y}_M\right), Y_{\text{valid}}\right)$
$E \leftarrow E \cup \{M^*\}$
end for
return $E$
Starting from an empty ensemble, the algorithm greedily adds the model from library $\mathcal{M}$ that minimizes the combined ensemble loss on a validation hold-out set. Models can be selected multiple times, which corresponds to assigning positive integer weights. This approach avoids the overfitting and computational cost associated with stacking or gradient-free numerical weight optimization.
Knowl 5 — Structured Configuration Space of Auto-sklearn
model/method
AUTO-SKLEARN defines a parameterized machine learning pipeline comprising 110 hyperparameters (including conditional hyperparameters activated based on parent categorical choices):
1. Data Preprocessing: Up to four fixed operations executed when applicable:
  - Rescaling: Min/max, standard scaling, etc. (1 categorical hyperparameter).
  - Missing value imputation: Mean, median, etc. (1 categorical hyperparameter).
  - One-hot encoding of categorical features (2 hyperparameters: 1 categorical, 1 conditional continuous).
  - Class balancing / sample weighting (1 categorical hyperparameter).
2. Feature Preprocessing: Exactly one (or none) of 14 methods is chosen:
  - Dimensionality reduction / matrix decomposition: PCA (2 hyperparameters), Kernel PCA (5), FastICA (4), Truncated SVD.
  - Kernel approximations: Nystroem Sampler (5), Random Kitchen Sinks (2).
  - Feature clustering & embeddings: Feature Agglomeration (4), Random Trees Embedding (4).
  - Feature selection: Select Percentile (2), Select Rates (3), Extra Trees Preprocessor (5), Linear SVM Preprocessor (3).
  - Feature expansion: Polynomial expansion (3).
  - No feature preprocessing.
3. Classifiers: Exactly one of 15 classification algorithms is selected:
  - Linear models: SGD Classifier (10 hyperparameters: 4 categorical, 6 continuous with 3 conditional), Passive Aggressive (3).
  - Support Vector Machines: Linear SVM (4), Kernel SVM (7: 2 categorical, 5 continuous with 2 conditional).
  - Discriminant Analysis: LDA (4: 1 categorical, 3 continuous with 1 conditional), QDA (2).
  - Nearest Neighbors: $k$ -NN (3).
  - Naive Bayes: Gaussian NB (0), Bernoulli NB (2), Multinomial NB (2).
  - Decision Trees: Decision Tree (4).
  - Ensembles: Random Forest (5), Extremely Randomized Trees (5), AdaBoost (4), Gradient Boosting (6).
Knowl 6 — Balanced Classification Error Rate
definition
To handle class imbalance without biasing metric evaluations toward majority classes, the Balanced Error Rate (BER) is defined as the unweighted average of the class-specific misclassification error rates:

$\text{BER} = \frac{1}{|C|} \sum_{c \in C} e_c = \frac{1}{|C|} \sum_{c \in C} \left( \frac{1}{|D_c|} \sum_{i \in D_c} \mathbb{I}(\hat{y}_i \neq y_i) \right)$

where $C$ is the set of distinct class labels, $D_c = \{i \mid y_i = c\}$ is the subset of evaluation instances belonging to class $c$ , $|D_c|$ is the size of class $c$ , $y_i$ is the ground-truth target, $\hat{y}_i$ is the predicted label for instance $i$ , $\mathbb{I}(\cdot)$ is the indicator function, and $e_c$ is the misclassification rate within class $c$ .

Knowl 7 — Empirical Benchmark Against Auto-WEKA and Hyperopt-sklearn

data/table

Vanilla AUTO-SKLEARN (without meta-learning and ensemble building) was compared to AUTO-WEKA (AW) and HYPEROPT-SKLEARN (HS) on 21 benchmark datasets using the experimental protocol of Thornton et al. (10 runs per dataset simulating 4 parallel processes, reporting median percent test classification error over 100,000 bootstrap samples).

Dataset	Auto-sklearn (AS)	Auto-WEKA (AW)	Hyperopt-sklearn (HS)
Abalone	73.50	73.50	76.21
Amazon	16.00	30.00	16.22
Car	0.39	0.00	0.39
Cifar-10	51.70	56.95	–
Cifar-10 Small	54.81	56.20	57.95
Convex	17.53	21.80	19.18
Dexter	5.56	8.33	–
Dorothea	5.51	6.38	–
German Credit	27.00	28.33	27.67
Gisette	1.62	2.29	2.29
KDD09 Appetency	1.74	1.74	–
KR-vs-KP	0.42	0.31	0.42
Madelon	12.44	18.21	14.74
MNIST Basic	2.84	2.84	2.82
MRBI	46.92	60.34	55.79
Secom	7.87	8.09	–
Semeion	5.24	5.24	5.87
Shuttle	0.01	0.01	0.05
Waveform	14.93	14.13	14.07
Wine Quality	33.76	33.36	34.72
Yeast	40.67	37.75	38.45

Vanilla AUTO-SKLEARN achieved statistically significantly lower classification error than AUTO-WEKA in 6 of 21 datasets, tied in 12, and lost in 3 (on the 3 losses, AUTO-WEKA selected decision trees with pruning methods not implemented in scikit-learn). HYPEROPT-SKLEARN crashed on 5 datasets due to lack of support for sparse inputs or missing values and memory limits, tying the best method on 9 and losing on 7 of the remaining 16 datasets.

Knowl 8 — Ablation of Meta-Learning and Ensembling Across 140 OpenML Datasets
empirical result
In an ablation study across 140 OpenML classification datasets (each with at least 1,000 instances, evaluated across 10 independent runs under a 1-hour wallclock budget and a 6-minute single-model cutoff using leave-one-dataset-out validation):
1. Meta-Learning Impact: Initializing SMAC using $k = 25$ configurations from meta-learning yielded immediate, drastic improvements in average rank based on Balanced Error Rate (BER) from the very first time steps. Although vanilla AUTO-SKLEARN partially closed the performance gap as optimization time approached 3600 seconds, meta-learning maintained a superior average rank throughout the entire budget.
2. Ensemble Selection Impact: Post-hoc greedy ensemble construction (size 50) improved the average rank of both vanilla AUTO-SKLEARN and meta-learning-initialized AUTO-SKLEARN across all time steps.
3. Complementary Synergy: The combined system (AUTO-SKLEARN + meta-learning + ensemble) consistently achieved the lowest average rank across the 1-hour evaluation period. Ensembling produced performance gains earlier in the optimization timeline when combined with meta-learning because meta-learning rapidly provided high-quality, diverse individual models to populate the ensemble library.
Knowl 9 — Subspace Optimization Analysis of Classifiers and Preprocessors
empirical result
Evaluating individual classifier subspaces (each optimized across all 14 preprocessors for 24 hours) and individual preprocessor subspaces (each optimized across all 15 classifiers for 24 hours) against full AUTO-SKLEARN (optimized for 48 hours) across 13 diverse dataset clusters yielded key insights:
- Classifier Performance: Tree ensemble methods (Random Forest, Extremely Randomized Trees, AdaBoost, and Gradient Boosting) delivered the most consistently robust performance across diverse datasets. Support Vector Machines demonstrated strong peak performance on specific datasets. Simpler models—including single Decision Trees, Passive Aggressive classifiers, $k$ -NN, Gaussian Naive Bayes, LDA, and QDA—were statistically significantly inferior to the best model on most datasets.
- Joint Optimization vs. Individual Selection: No single classification algorithm or preprocessing method achieved superior performance across all datasets. AUTO-SKLEARN searching the joint 110-hyperparameter configuration space matched or exceeded the performance of the best specialized algorithm/preprocessor subspace on nearly every dataset.
Knowl 10 — Limitations and Scope of Auto-sklearn
limitation
The design and evaluation of AUTO-SKLEARN exhibit several explicit boundaries:
1. Problem Types: The framework is formulated and evaluated solely for supervised classification problems (binary and multiclass); regression, semi-supervised, and unsupervised settings are not supported.
2. Scale and Framework Bounds: By relying exclusively on scikit-learn algorithms and CPU-based execution, AUTO-SKLEARN is targeted at small to medium-sized datasets and lacks support for deep learning architectures or distributed frameworks for very large-scale unstructured data.
3. Exclusion of Landmarking Meta-Features: Although landmarking meta-features (which measure the empirical performance of fast base learners) are known to be informative for algorithm selection, they were omitted from meta-learning because their online computational overhead compromised optimization efficiency under rigid time constraints.

Coverage note — None was omitted; all primary methodological contributions, algorithmic formulations, configuration spaces, empirical benchmarks, component analyses, and stated limitations are fully covered.

References

1.I. Guyon, K. Bennett, G. Cawley, H. Escalante, S. Escalera, T. Ho, N.Maci`a, B. Ray, M. Saeed, A. Statnikov, and E. Viegas. Design of the 2015 ChaLearn AutoML Challenge. In Proc. of IJCNN’15, 2015.
2.C. Thornton, F. Hutter, H. Hoos, and K. Leyton-Brown. Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms. In Proc. of KDD’13, pages 847–855, 2013.
3.E. Brochu, V. Cora, and N. de Freitas. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. CoRR, abs/1012.2599, 2010.
4.M. Feurer, J. Springenberg, and F. Hutter. Initializing Bayesian hyperparameter optimization via metalearning. In Proc. of AAAI’15, pages 1128–1135, 2015.
5.Reif M, F. Shafait, and A. Dengel. Meta-learning for evolutionary parameter optimization of classifiers. Machine Learning, 87:357–380, 2012.
6.T. Gomes, R. Prudˆencio, C. Soares, A. Rossi, and A. Carvalho. Combining meta-learning and search techniques to select parameters for support vector machines. Neurocomputing, 75(1):3–13, 2012.
7.F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. JMLR, 12:2825–2830, 2011.
8.M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. Witten. The WEKA data mining software: An update. SIGKDD, 11(1):10–18, 2009.
9.F. Hutter, H. Hoos, and K. Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Proc. of LION’11, pages 507–523, 2011.
10.J. Bergstra, R. Bardenet, Y. Bengio, and B. K´egl. Algorithms for hyper-parameter optimization. In Proc. of NIPS’11, pages 2546–2554, 2011.
11.J. Snoek, H. Larochelle, and R. P. Adams. Practical Bayesian optimization of machine learning algorithms. In Proc. of NIPS’12, pages 2960–2968, 2012.
12.K. Eggensperger, M. Feurer, F. Hutter, J. Bergstra, J. Snoek, H. Hoos, and K. Leyton-Brown. Towards an empirical foundation for assessing Bayesian optimization of hyperparameters. In NIPS Workshop on Bayesian Optimization in Theory and Practice, 2013.
13.B. Komer, J. Bergstra, and C. Eliasmith. Hyperopt-sklearn: Automatic hyperparameter configuration for scikit-learn. In ICML workshop on AutoML, 2014.
14.L. Breiman. Random forests. MLJ, 45:5–32, 2001.
15.P. Brazdil, C. Giraud-Carrier, C. Soares, and R. Vilalta. Metalearning: Applications to Data Mining. Springer, 2009.
16.R. Bardenet, M. Brendel, B. K´egl, and M. Sebag. Collaborative hyperparameter tuning. In Proc. of ICML’13 [28], pages 199–207.
17.D. Yogatama and G. Mann. Efficient transfer learning method for automatic hyperparameter tuning. In Proc. of AISTATS’14, pages 1077–1085, 2014.
18.J. Vanschoren, J. van Rijn, B. Bischl, and L. Torgo. OpenML: Networked science in machine learning. SIGKDD Explorations, 15(2):49–60, 2013.
19.D. Michie, D. Spiegelhalter, C. Taylor, and J. Campbell. Machine Learning, Neural and Statistical Classification. Ellis Horwood, 1994.
20.A. Kalousis. Algorithm Selection via Meta-Learning. PhD thesis, University of Geneve, 2002.
21.B. Pfahringer, H. Bensusan, and C. Giraud-Carrier. Meta-learning by landmarking various learning algorithms. In Proc. of (ICML’00), pages 743–750, 2000.
22.I. Guyon, A. Saffari, G. Dror, and G. Cawley. Model selection: Beyond the Bayesian/Frequentist divide. JMLR, 11:61–87, 2010.
23.A. Lacoste, M. Marchand, F. Laviolette, and H. Larochelle. Agnostic Bayesian learning of ensembles. In Proc. of ICML’14, pages 611–619, 2014.
24.R. Caruana, A. Niculescu-Mizil, G. Crew, and A. Ksikes. Ensemble selection from libraries of models. In Proc. of ICML’04, page 18, 2004.
25.R. Caruana, A. Munson, and A. Niculescu-Mizil. Getting the most out of ensemble selection. In Proc. of ICDM’06, pages 828–833, 2006.
26.D. Wolpert. Stacked generalization. Neural Networks, 5:241–259, 1992.
27.G. Hamerly and C. Elkan. Learning the k in k-means. In Proc. of NIPS’04, pages 281–288, 2004.
28.Proc. of ICML’13, 2014.

Citation

MLA

Feurer, M., et al. “Efficient and Robust Automated Machine Learning”. Advances in Neural Information Processing Systems, vol. 28, 2015, https://proceedings.neurips.cc/paper_files/paper/2015/file/11d0e6287202fced83f79975ec59a3a6-Paper.pdf.

APA

Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., & Hutter, F. (2015). Efficient and Robust Automated Machine Learning. Advances in Neural Information Processing Systems, 28. https://proceedings.neurips.cc/paper_files/paper/2015/file/11d0e6287202fced83f79975ec59a3a6-Paper.pdf

Chicago

Feurer, M., A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and F. Hutter. 2015. “Efficient and Robust Automated Machine Learning”. Advances in Neural Information Processing Systems 28. https://proceedings.neurips.cc/paper_files/paper/2015/file/11d0e6287202fced83f79975ec59a3a6-Paper.pdf.

Harvard

Feurer, M. et al. (2015) “Efficient and Robust Automated Machine Learning”, Advances in Neural Information Processing Systems. Curran Associates, Inc. Available at: https://proceedings.neurips.cc/paper_files/paper/2015/file/11d0e6287202fced83f79975ec59a3a6-Paper.pdf.

Vancouver

1. Feurer M, Klein A, Eggensperger K, Springenberg J, Blum M, Hutter F (2015) Efficient and Robust Automated Machine Learning. Advances in Neural Information Processing Systems 28:

BibTeX

@inproceedings{feurer2015efficient,
  title = {Efficient and Robust Automated Machine Learning},
  author = {Feurer, Matthias and Klein, Aaron and Eggensperger, Katharina and Springenberg, Jost and Blum, Manuel and Hutter, Frank},
  year = {2015},
  booktitle = {Advances in Neural Information Processing Systems},
  publisher = {Curran Associates, Inc.},
  volume = {28},
  url = {https://proceedings.neurips.cc/paper_files/paper/2015/file/11d0e6287202fced83f79975ec59a3a6-Paper.pdf}
}

Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

Efficient and Robust Automated Machine Learning

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Combined Algorithm Selection and Hyperparameter Optimization Problem

Knowl 2 — Auto-sklearn System Architecture

Knowl 3 — Meta-Learning Warmstart Procedure for CASH

Knowl 4 — Post-Hoc Greedy Ensemble Selection

Knowl 5 — Structured Configuration Space of Auto-sklearn

Knowl 6 — Balanced Classification Error Rate

Knowl 7 — Empirical Benchmark Against Auto-WEKA and Hyperopt-sklearn

Knowl 8 — Ablation of Meta-Learning and Ensembling Across 140 OpenML Datasets

Knowl 9 — Subspace Optimization Analysis of Classifiers and Preprocessors

Knowl 10 — Limitations and Scope of Auto-sklearn

References

Citation

BibTeX

Source Code

Access the Paper