Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning
Matthias FeurerKatharina EggenspergerStefan FalknerMarius LindauerFrank Hutter
Presents Auto-Sklearn 2.0, a truly hands-free automated machine learning framework that uses meta-learning and bandit-based budget allocation to automatically configure its own optimization strategy, outperforming its predecessor in ten minutes compared to an hour of compute time across 39 benchmark datasets.
Automated Machine Learning systems help developers and non-specialists build predictive pipelines without requiring manual trial and error. However, existing tools face major operational hurdles: evaluating complex models on large datasets often exceeds strict time and memory limits, and the systems themselves introduce difficult high-level configuration choices, such as selecting validation methods or resource allocation strategies. The article evaluates a new generation of automated tools—culminating in the development of Auto-sklearn 2.0—to determine how to deliver high predictive accuracy rapidly under rigid computational constraints without requiring human intervention.
The authors designed and evaluated their framework using extensive empirical benchmarks. They trained their meta-learning components across 208 diverse open-source classification datasets and tested performance on an independent benchmark suite of 39 standard datasets. The approach introduces pre-computed portfolios of diverse, high-performing pipelines to eliminate the need for expensive real-time meta-feature calculations. It integrates a successive halving allocation strategy that early-stops poorly performing pipelines to allocate more compute time to promising models. To make the entire framework hands-free, the researchers built an automated policy selector that uses simple, linear-time metadata (the number of rows and columns) along with a fallback safety rule to pick the optimal validation and budget allocation strategy for any unseen dataset.
The experimental findings show substantial improvements in efficiency, accuracy, and robustness. Under strict 10-minute compute limits, Auto-sklearn 2.0 achieved a lower prediction error than its predecessor, Auto-sklearn 1.0, achieved after 60 minutes of computation. Against Auto-sklearn 1.0, the new framework reduced relative error by a factor of 4.5 at the 10-minute horizon and by a factor of 3 at the 60-minute mark. On large, complex datasets where previous tools frequently failed due to memory and time limits, the new pipeline retrieval and allocation mechanisms completed execution reliably. In head-to-head comparisons against other established automated machine learning platforms across 39 benchmark datasets, Auto-sklearn 2.0 achieved the lowest average rank and won the highest number of individual datasets.
These results demonstrate that automating higher-level system design decisions eliminates common operational bottlenecks and significantly lowers the compute cost and timeline required for rapid prototyping. Practitioners can obtain higher-quality models in a fraction of the time, reducing cloud hardware expenses and democratizing advanced predictive modeling for non-expert teams. The findings also reveal that complex dataset descriptions are unnecessary; simple metadata paired with disciplined warmstart portfolios is sufficient to guide algorithm selection effectively.
Organizations deploying automated machine learning should adopt portfolio warmstarts and multi-fidelity budget allocation to handle tight operational deadlines. For automated system configuration, the evidence supports using per-instance policy selection combined with aggressive fallback mechanisms to prevent timeouts on out-of-distribution datasets. Future initiatives should focus on extending these automated selection policies across varying compute budgets and metrics, dynamically adjusting portfolio sizes, and exploring adaptive fallbacks during execution.
The findings are supported by strong statistical testing and multi-dataset cross-validation; however, certain boundaries remain. The evaluation focused primarily on classification tasks with fixed tabular constraints, and the system relies on an offline training phase across historical datasets to construct its portfolios and selector models. Readers should exercise caution when applying these configurations directly to regression tasks, complex multi-modal inputs, or environments where the target evaluation metrics differ substantially from those modeled in the offline phase.
- Paper: Efficient and Robust Automated Machine Learning, Matthias Feurer et al. (2015). It introduces the original Auto-sklearn framework, establishing the foundational meta-learning and automated ensembling architecture that Auto-sklearn 2.0 directly builds upon and enhances.
- Paper: Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms, Chris J. Thornton et al. (2012). It formulates the combined algorithm selection and hyperparameter optimization (CASH) problem that underpins the entire line of Bayesian optimization-based AutoML systems.
- Paper: Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization, Lisha Li et al. (2016). It presents the Hyperband bandit-based resource allocation strategy that Auto-sklearn 2.0 adapts to handle large datasets efficiently under rigid time constraints.
- Paper: BOHB: Robust and Efficient Hyperparameter Optimization at Scale, Stefan Falkner et al. (2018). It integrates Bayesian optimization with bandit-based multi-fidelity evaluation, serving as a direct prerequisite for the budget allocation strategies used in modern AutoML pipelines.
- Paper: AutoML: A Survey of the State-of-the-Art, Xin He et al. (2019). It provides a comprehensive survey of AutoML components, search spaces, and optimization paradigms, contextualizing the design choices addressed in hands-free AutoML.
- Paper: API design for machine learning software: experiences from the scikit-learn project, Lars Buitinck et al. (2013). It details the API design and modular pipeline abstractions of scikit-learn, which form the core structural framework parameterized and optimized by Auto-sklearn.
- Paper: OpenML: networked science in machine learning, Joaquin Vanschoren et al. (2014). It outlines the OpenML benchmarking infrastructure and standardized dataset repositories that provide the foundational meta-data and evaluation suites used in Auto-sklearn 2.0.
- Paper: SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization, Marius Lindauer et al. (2022). It details the SMAC3 Bayesian optimization package and modular facades that execute the underlying hyperparameter optimization and algorithm configuration in modern AutoML pipelines.
- Paper: Pre-trained Gaussian Processes for Bayesian Optimization, Zi Wang et al. (2024). It advances data-driven prior specification for Bayesian optimization via pre-trained Gaussian processes, taking the meta-learning principles of Auto-sklearn further across diverse task spaces.
- Paper: AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML, Patara Trirat et al. (2025). It extends the goal of hands-free AutoML from fixed tabular pipeline search to conversational, multi-agent LLM systems that manage end-to-end machine learning workflows across diverse modalities.
- Paper: Automated Design of Agentic Systems, Shengran Hu et al. (2025). It generalizes automated pipeline design concepts to the meta-search and iterative discovery of entire agentic software architectures.
