Back

SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization

Marius LindauerKatharina EggenspergerMatthias FeurerAndré BiedenkappDifan DengCarolin BenjaminsTim RuhkopfRené SassFrank Hutter

article2022JMLR569 citations

Presents SMAC3, an open-source Bayesian optimization library featuring modular facades tailored for complex algorithm configuration, multi-fidelity tuning in deep learning, and continuous black-box problems.

Listen

Achieving high performance from machine learning models requires tuning their hyperparameters, which control key learning behaviors. While Bayesian optimization is widely considered an efficient approach to automate this process, existing techniques are often brittle and sensitive to their own internal configurations. Selecting and configuring the right optimization framework remains a complex, error-prone obstacle for practitioners deploying machine learning pipelines.

The article demonstrates SMAC3, an open-source Bayesian optimization software package designed to automate algorithm hyperparameter tuning across diverse tasks. The authors evaluate SMAC3's architecture, pre-configured operating modes, and performance compared to leading optimization frameworks on deep learning and neural architecture search benchmarks.

To evaluate the system, the authors conducted simulated sequential optimization benchmarks across standard deep neural network and neural architecture search datasets, tracking validation performance over evaluation budgets. SMAC3 incorporates random forest models alongside standard Gaussian processes, accommodates multi-fidelity evaluations where cheap partial training runs act as proxies for full model costs, and supports multi-instance algorithm configuration. The software provides modular, specialized pre-sets, termed facades, that streamline configuration across standard black-box problems, structured algorithm selection pipelines, expensive deep learning models, and general algorithm configuration tasks.

The findings show that SMAC3 offers distinct practical advantages over competing tools. First, SMAC3 consistently outperformed established frameworks like Dragonfly and Tree-structured Parzen Estimator-based tools like BOHB on the tested benchmarks. Second, SMAC3's multi-fidelity mode matches the rapid early-stage efficiency of Hyperband while achieving superior performance in mid-stage evaluations before its random-forest-based Bayesian optimization catches up in later stages. Third, the platform's random forest surrogate models effectively handle complex, hierarchical configuration spaces and scale to large algorithm configuration problems containing over 300 hyperparameters.

These results demonstrate that SMAC3 can significantly reduce the computational cost and time required to deploy high-performing machine learning systems. By providing pre-configured facades, the tool abstracts the internal complexities of Bayesian optimization, allowing organizations to avoid the overhead of manually designing optimization pipelines. The framework's flexibility makes it suitable for integration into automated machine learning infrastructure and general algorithm tuning workflows.

Organizations developing automated machine learning pipelines should consider deploying SMAC3 using its specialized facades to match their specific problem structures, such as using multi-fidelity settings for expensive deep learning tasks. Future development plans outlined in the article include incorporating local Bayesian optimization methods to better exploit optimization landscape structures and adding automated mechanisms, such as bandits or reinforcement learning, to adapt SMAC3's internal settings during runtime.

While SMAC3 provides robust pre-set configurations, the authors note that selecting its internal hyperparameters can still pose challenges when default settings do not fully align with unusual problem landscapes. Additionally, the empirical benchmarks presented in the article rely on simulated evaluations from standardized surrogate datasets, so real-world execution gains may vary depending on infrastructure and hardware constraints.

Bosch Center for AILeibniz Universität HannoverUniversity of Freiburgbayesian methodsoptimizationoptimization problemsSMAC3hyperparameter optimizationBayesian optimization
arXiv: 2109.09831automl/SMAC3
Cover for SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization

Abstract

Algorithm parameters, in particular hyperparameters of machine learning algorithms, can substantially impact their performance. To support users in determining well-performing hyperparameter configurations for their algorithms, datasets and applications at hand, SMAC3 offers a robust and flexible framework for Bayesian Optimization, which can improve performance within a few evaluations. It offers several facades and pre-sets for typical use cases, such as optimizing hyperparameters, solving low dimensional continuous (artificial) global optimization problems and configuring algorithms to perform well across multiple problem instances. The SMAC3 package is available under a permissive BSD-license at https://github.com/automl/SMAC3.

Topics

bayesian methodsoptimizationoptimization problemsregularization parameterskernel parameter

Keywords

SMAC3hyperparameter optimizationBayesian optimizationautomatic algorithm configurationmulti-fidelity optimizationblack-box optimizationsurrogate modelacquisition function

Table of Contents

  • 1. Introduction
  • 2. Different Use Cases and Modes of SMAC3
  • 2.1 SMAC4BB : SMAC for Low-dimensional and Continuous Black-Box Functions
  • 2.2 SMAC4HPO : SMAC for CASH and Structured Hyperparameter Optimization
  • 2.3 SMAC4MF : SMAC for Expensive Tasks and Automated Deep Learning
  • 2.4 SMAC4AC : SMAC for Algorithm Configuration
  • 3. Brief Empirical Comparison
  • 4. Related Work
  • 5. Outlook
  • Acknowledgements
  • References

Knowls

  1. Knowl 1 — SMAC3 Modular SMBO Architecture and Facade Interface

    model/method

    SMAC3 is a modular Bayesian optimization framework based on Sequential Model-Based Optimization (SMBO). The framework decomposes the optimization pipeline into several interchangeable components:

    • Scenario: Encapsulates configuration parameters, the configuration space Λ\Lambda, overall optimization budget, and optional problem instance sets I\mathcal{I}.
    • Target Algorithm Evaluator (TAE): Executes candidate evaluations via a Python callable function, a command-line interface (CLI) for arbitrary binaries, or distributed execution using Dask.
    • Initial Design: Generates initial candidate configurations prior to surrogate modeling, supporting strategies including default configurations λd\lambda_d, random sampling, Latin Hypercube Designs (LHD), and Sobol sequences.
    • Empirical Performance Model (EPM): Serves as the surrogate model, supporting Gaussian Processes (GPs) and Random Forests (RFs).
    • Acquisition Function: Guides exploration versus exploitation, supporting Expected Improvement (EI), logarithmic Expected Improvement (logEI\text{logEI}), Probability of Improvement (PI), Lower Confidence Bound (LCB), and EI per second (accounting for evaluation runtimes).
    • Intensification: Determines evaluation budgets and racing mechanisms across configurations, supporting Aggressive Racing, Successive Halving, and Hyperband.

    To simplify deployment, SMAC3 exposes pre-configured facades that bundle specialized defaults for common optimization paradigms (black-box continuous optimization, structured/CASH optimization, multi-fidelity optimization, and algorithm configuration).

  2. Knowl 2 — SMAC4BB Facade for Low-Dimensional Continuous Black-Box Optimization

    model/method

    The SMAC4BB facade addresses continuous black-box hyperparameter optimization (HPO) formulated as:

    λ∗∈arg⁡min⁡λ∈Λc(λ)=arg⁡min⁡λ∈ΛL(Dtrain,Dval;λ)\lambda^* \in \arg\min_{\lambda \in \Lambda} c(\lambda) = \arg\min_{\lambda \in \Lambda} \mathcal{L}(\mathcal{D}_{\text{train}}, \mathcal{D}_{\text{val}}; \lambda)

    where Λ\Lambda is a low-dimensional continuous configuration space, c(λ)c(\lambda) is the black-box cost function, and L(Dtrain,Dval;λ)\mathcal{L}(\mathcal{D}_{\text{train}}, \mathcal{D}_{\text{val}}; \lambda) represents the validation loss evaluated on dataset Dval\mathcal{D}_{\text{val}} for a model trained on Dtrain\mathcal{D}_{\text{train}} with hyperparameter configuration λ\lambda.

    SMAC4BB uses the following component configuration:

    • Initial Design: Sobol sequence sampling.
    • Surrogate Model: Gaussian Process (GP) with a Matérn 5/25/2 covariance kernel.
    • Acquisition Function: Expected Improvement (EI).

    Additional supported acquisition functions include Probability of Improvement (PI), Lower Confidence Bound (LCB), Thompson Sampling (TS), EI per second (for variable runtime costs), and logEI\text{logEI} (for heavy-tailed cost distributions).

  3. Knowl 3 — SMAC4HPO Facade for CASH and Hierarchically Structured Hyperparameter Spaces

    model/method

    The SMAC4HPO facade solves Combined Algorithm Selection and Hyperparameter Optimization (CASH) and structured hyperparameter optimization problems formulated as:

    (A∗,λ∗)∈arg⁡min⁡Ai∈A, λ∈Λic(Ai,λ)=arg⁡min⁡Ai∈A, λ∈ΛiL(Dtrain,Dval;Ai(λ))(A^*, \lambda^*) \in \arg\min_{A_i \in \mathcal{A}, \, \lambda \in \Lambda_i} c(A_i, \lambda) = \arg\min_{A_i \in \mathcal{A}, \, \lambda \in \Lambda_i} \mathcal{L}(\mathcal{D}_{\text{train}}, \mathcal{D}_{\text{val}}; A_i(\lambda))

    where A={A1,…,Am}\mathcal{A} = \{A_1, \dots, A_m\} is a discrete set of candidate machine learning algorithms, Λi\Lambda_i is the hyperparameter search subspace specific to algorithm AiA_i, and Ai(λ)A_i(\lambda) denotes algorithm AiA_i configured with parameters λ∈Λi\lambda \in \Lambda_i.

    Because hyperparameter subspace Λi\Lambda_i is active only when AiA_i is selected, the search space contains conditional hierarchical dependencies across multiple levels (such as choosing an algorithm, then choosing an optimizer, then choosing optimizer-specific parameters).

    SMAC4HPO handles these non-continuous, conditional spaces using the following configuration:

    • Initial Design: Sobol sequence.
    • Surrogate Model: Random Forest (RF) regression model capable of handling categorical, numerical, and inactive/conditional parameters.
    • Acquisition Function: Logarithmic Expected Improvement (logEI\text{logEI}).

  4. Knowl 4 — SMAC4MF Facade for Multi-Fidelity Hyperparameter Optimization

    model/method

    The SMAC4MF facade optimizes expensive black-box objectives, such as deep neural network hyperparameters and architectures, using multi-fidelity approximations formulated as:

    λ∗∈arg⁡min⁡λ∈Λc(λ,bmax⁡)=arg⁡min⁡λ∈ΛL(Dtrain,Dval;λ,bmax⁡)\lambda^* \in \arg\min_{\lambda \in \Lambda} c(\lambda, b_{\max}) = \arg\min_{\lambda \in \Lambda} \mathcal{L}(\mathcal{D}_{\text{train}}, \mathcal{D}_{\text{val}}; \lambda, b_{\max})

    where b≤bmax⁡b \le b_{\max} denotes an allocated fidelity budget (such as training epochs, dataset subsample size, or neural network channel capacity), and evaluating c(λ,b)c(\lambda, b) provides an inexpensive proxy for the full objective c(λ,bmax⁡)c(\lambda, b_{\max}).

    SMAC4MF integrates the Hyperband bandit strategy with Bayesian optimization (following the BOHB framework): it uses Hyperband for intensification to aggressively prune poorly performing configurations at small budgets, and fits a Random Forest surrogate model on evaluations at the highest budget level that has collected sufficient observations. The SMAC4MF default configuration consists of:

    • Initial Design: Uniform random sampling.
    • Surrogate Model: Random Forest (RF) regressor.
    • Intensification: Hyperband.

  5. Knowl 5 — SMAC4AC Facade for Algorithm Configuration Across Problem Instances

    model/method

    The SMAC4AC facade targets general algorithm configuration (AC), where an algorithm's parameters λ∈Λ\lambda \in \Lambda are optimized across a set of problem instances I\mathcal{I}:

    λ∗∈arg⁡min⁡λ∈Λc(λ)=arg⁡min⁡λ∈Λ∑i∈Ic′(λ,i)\lambda^* \in \arg\min_{\lambda \in \Lambda} c(\lambda) = \arg\min_{\lambda \in \Lambda} \sum_{i \in \mathcal{I}} c'(\lambda, i)

    where c′(λ,i)c'(\lambda, i) denotes the cost (such as execution runtime or solution quality) of running configuration λ\lambda on problem instance i∈Ii \in \mathcal{I}.

    SMAC4AC configures SMAC3 with mechanisms specialized for large-scale and noisy instance tuning:

    • Intensification: Aggressive racing, which assesses unpromising candidate configurations on a small subset of instances and progressively evaluates promising configurations across more instances to statistically validate incumbents.
    • Censored Data Handling: Imputation of right-censored runtime observations caused by algorithm execution cutoffs.
    • Surrogate Model: Random Forest (RF) customized with specific hyperparameters for algorithm configuration.
    • Acquisition Function: Logarithmic Expected Improvement (logEI\text{logEI}) to handle heavy-tailed runtime distributions.
    • Initial Design: A single default parameter configuration.

  6. Knowl 6 — Parallelization Strategies in SMAC3

    model/method

    SMAC3 supports two distinct parallelization paradigms to accelerate hyperparameter optimization and algorithm configuration workflows:

    1. Dask-based Task Parallelism: Uses the Dask distributed framework within the Target Algorithm Evaluator (TAE) to asynchronously evaluate multiple candidate configurations in parallel across local CPU cores or compute clusters.
    2. Shared-Filesystem Multi-Instance Parallelism: Executes an arbitrary number of independent SMAC3 processes without a centralized master node. The independent optimizer instances communicate asynchronously by reading and writing evaluation histories, performance metrics, and configuration data directly to a shared directory on the file system.

  7. Knowl 7 — Optimization Efficiency of SMAC3 on Tabular and Surrogate Benchmarks

    empirical result

    Empirical evaluations on three HPOBench benchmarks—NetLetter (a 6-dimensional deep neural network hyperparameter tuning task), NBHPONaval (a 9-dimensional joint hyperparameter and neural architecture search task), and Nas1Shot12 (a 9-dimensional neural architecture search task)—compare SMAC3 configurations against Random Search, Hyperband, Dragonfly, and BOHB:

    • Early Iterations: The multi-fidelity facade of SMAC3 (SMAC-HB) achieves optimization progress comparable to standard Hyperband.
    • Intermediate Iterations: SMAC-HB attains lower median optimized regret than Hyperband, BOHB, Dragonfly, and Random Search.
    • Late Iterations: Standard model-based SMAC with Random Forests (SMAC RF) catches up with and matches or exceeds the regret reduction of multi-fidelity methods.
    • Baseline Comparison: Across all benchmarks and evaluation budgets, SMAC3 variants consistently outperform Dragonfly and surpass BOHB in later optimization stages.

  8. Knowl 8 — Landscape Exploitation and Hyperparameter Adaptability Limitations in SMAC3

    limitation

    SMAC3 has two documented structural limitations:

    1. Lack of Local Landscape Exploitation: SMAC3 employs global surrogate models (Gaussian Processes and Random Forests) over the entire parameter space and does not explicitly leverage local objective landscape characteristics or trust regions (as done in local Bayesian optimization frameworks like TuRBO).
    2. Sensitivity to Internal Hyperparameters: Although facades provide standard component presets, SMAC3's optimization efficiency is sensitive to its own internal hyperparameters (such as surrogate tree hyperparameters and acquisition parameters). It currently lacks built-in dynamic adaptation mechanisms (such as multi-armed bandits or reinforcement learning controllers) to adaptively adjust its own settings during an active optimization run.

Coverage note — General background on Bayesian optimization, external benchmark implementations, and descriptions of third-party systems that utilize SMAC3 (such as Auto-sklearn and Auto-PyTorch) were omitted as they are context rather than direct contributions of SMAC3.

References

  1. 1.C. Ansotegui, M. Sellmann, and K. Tierney. A gender-based genetic algorithm for the automatic configuration of algorithms. In I. Gent, editor, Proceedings of the Fifteenth International Conference on Principles and Practice of Constraint Programming (CP’09), volume 5732 of Lecture Notes in Computer Science, pages 142–157. Springer, 2009.
  2. 2.N. Awad, G. Shala, D. Deng, N. Mallik, M. Feurer, K. Eggensperger, A. Biedenkapp, D. Vermetten, H. Wang, C. Doerr, M. Lindauer, and F. Hutter. Squirrel: A switching hyperparameter optimizer description of the entry by AutoML.org & IOHprofiler to the NeurIPS 2020 BBO challenge. arXiv:2012.08180 [cs.LG], 2020.
  3. 3.N. Awad, N. Mallik, and F. Hutter. DEHB: Evolutionary hyberband for scalable, robust and efficient hyperparameter optimization. In Z. Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 2147–2153. ijcai.org, 2021.
  4. 4.E. Bakshy, L. Dworkin, B. Karrer, K. Kashin, B. Letham, A. Murthy, and S. Singh. Ae: A domain-agnostic platform for adaptive experimentation. In Proceedings of the international conference on Neural Information Processing Systems, pages 1–8, 2018.
  5. 5.M. Balandat, B. Karrer, D. Jiang, S. Daulton, B. Letham, A. Wilson, and E. Bakshy. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Proceedings of the 33rd International Conference on Advances in Neural Information Processing Systems (NeurIPS’20). Curran Associates, 2020.
  6. 6.J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13:281–305, 2012.
  7. 7.J. Bergstra, D. Yamins, and D. Cox. Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms. In Proceedings of the 12th Python in science conference, volume 13, page 20, 2013.
  8. 8.A. Biedenkapp, H. F. Bozkurt, T. Eimer, F. Hutter, and M. Lindauer. Dynamic Algorithm Configuration: Foundation of a New Meta-Algorithmic Framework. In J. Lang, G. De Giacomo, B. Dilkina, and M. Milano, editors, Proceedings of the Twenty-fourth European Conference on Artificial Intelligence (ECAI’20), pages 427–434, June 2020.
  9. 9.L. Breimann. Random forests. Machine Learning Journal, 45:5–32, 2001.
  10. 10.K. Eggensperger, P. Muller, N. Malik, M. Feurer, R. Sass, A. Klein, N. Awad, Marius Lindauer, and Frank Hutter. HPOBench: A collection of reproducible multi-fidelity benchmark problems for hpo. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
  11. 11.D. Eriksson, M. Pearce, J. Gardner, R. Turner, and M. Poloczek. Scalable global optimization via local bayesian optimization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alche-Buc, E. Fox, and R. Garnett, editors, Proceedings of the 32nd International Conference on Advances in Neural Information Processing Systems (NeurIPS’19). Curran Associates, 2019.
  12. 12.S. Falkner, A. Klein, and F. Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning (ICML’18), volume 80, pages 1437–1446. Proceedings of Machine Learning Research, 2018.
  13. 13.M. Feurer and F. Hutter. Hyperparameter optimization. In F. Hutter, L. Kotthoff, and J. Vanschoren, editors, Automated Machine Learning: Methods, Systems, Challenges, volume 5 of The Springer Series on Challenges in Machine Learning, chapter 1, pages 3–38. Springer, 2019. Available for free at http://automl.org/book.
  14. 14.M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and F. Hutter. Efficient and robust automated machine learning. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Proceedings of the 28th International Conference on Advances in Neural Information Processing Systems (NeurIPS’15), pages 2962–2970. Curran Associates, 2015.
  15. 15.F. Fortin, F. De Rainville, M. Gardner, M. Parizeau, and C. Gagne. DEAP: evolutionary algorithms made easy. Journal of Machine Learning Research, 13:2171–2175, 2012.
  16. 16.T. Head, M. Kumar, H. Nahrstaedt, G. Louppe, and I. Shcherbatyi. scikit-optimize/scikit-optimize, 2021. URL https://doi.org/10.5281/zenodo.5565057.
  17. 17.M. Hoffman, E. Brochu, and N. de Freitas. Portfolio allocation for bayesian optimization. In Cozman F and A. Pfeffer, editors, Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence, pages 327–336. AUAI Press, 2011.
  18. 18.F. Hutter, H. Hoos, K. Leyton-Brown, and T. Stutzle. ParamILS: An automatic algorithm configuration framework. Journal of Artificial Intelligence Research, 36:267–306, 2009.
  19. 19.F. Hutter, H. Hoos, K. Leyton-Brown, and K. Murphy. Time-bounded sequential parameter optimization. In C. Blum, editor, Proceedings of the Fourth International Conference on Learning and Intelligent Optimization (LION’10), volume 6073 of Lecture Notes in Computer Science, pages 281–298. Springer, 2010.
  20. 20.F. Hutter, H. Hoos, and K. Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In C. Coello, editor, Proceedings of the Fifth International Conference on Learning and Intelligent Optimization (LION’11), volume 6683 of Lecture Notes in Computer Science, pages 507–523. Springer, 2011.
  21. 21.F. Hutter, M. Lindauer, A. Balint, S. Bayless, H. Hoos, and K. Leyton-Brown. The configurable SAT solver challenge (CSSC). Artificial Intelligence, 243:1–25, 2017.
  22. 22.D. Jones, M. Schonlau, and W. Welch. Efficient global optimization of expensive black box functions. Journal of Global Optimization, 13:455–492, 1998.
  23. 23.K. Kandasamy, K. Vysyaraju, W. Neiswanger, B. Paria, C. Collins, J. Schneider, B. Poczos, and E. Xing. Tuning hyperparameters without grad students: Scalable and robust Bayesian optimisation with Dragonfly. Journal of Machine Learning Research, 21(81): 1–27, 2020.
  24. 24.A. Klein and F. Hutter. Tabular benchmarks for joint architecture and hyperparameter optimization. arXiv:1905.04970 [cs.LG], 2019.
  25. 25.G. Lan, J. Tomczak, D. Roijers, and A. Eiben. Time efficiency in optimization with a bayesian-evolutionary algorithm. arXiv:2005.04166 [cs.NE], 2020.
  26. 26.L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18(185):1–52, 2018.
  27. 27.Y. Li, Y. Shen, W. Zhang, Y. Chen, H. Jiang, M. Liu, J. Jiang, J. Gao, W. Wu, Z. Yang, C. Zhang, and B Cui. Openbox: A generalized black-box optimization service. In Proceedings of the 27th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2021, 2021.
  28. 28.M. Lindauer, K. Eggensperger, M. Feurer, A. Biedenkapp, J. Marben, P. Muller, and F. Hutter. BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameters. arXiv:1908.06756 [cs.LG], 2019a.
  29. 29.M. Lindauer, M. Feurer, K. Eggensperger, A. Biedenkapp, and F. Hutter. Towards assessing the impact of bayesian optimization’s own hyperparameters. In P. De Causmaecker, M. Lombardi, and Y. Zhang, editors, IJCAI 2019 DSO Workshop, 2019b.
  30. 30.M. Lopez-Iba˙nez, J. Dubois-Lacoste, L. Perez Caceres, M. Birattari, and T. Stutzle. The irace package: Iterated racing for automatic algorithm configuration. Operations Research Perspectives, 3:43–58, 2016.
  31. 31.I. Loshchilov and F. Hutter. CMA-ES for hyperparameter optimization of deep neural networks. In International Conference on Learning Representations Workshop track, 2016. Published online: iclr.cc.
  32. 32.J. Mockus, V. Tiesis, and A. Zilinskas. The application of Bayesian methods for seeking the extremum. Towards Global Optimization, 2(117-129), 1978.
  33. 33.L. Nardi, D. Koeplinger, and K. Olukotun. Practical design space exploration. In 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS), pages 347–358. IEEE, 2019.
  34. 34.R. Olson, N. Bartley, R. Urbanowicz, and J. Moore. Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science. In T. Friedrich, editor, Proceedings of the Genetic and Evolutionary Computation Conference (GECCO’16), pages 485–492. ACM, 2016.
  35. 35.J. Rapin and O. Teytaud. Nevergrad - A gradient-free optimization platform. https://GitHub.com/FacebookResearch/Nevergrad, 2018.
  36. 36.M. Rocklin. Dask: Parallel computation with blocked algorithms and task scheduling. In K. Huff and J. Bergstra, editors, Proceedings of the 14th Python in Science Conference, pages 130 – 136, 2015.
  37. 37.B. Shahriari, K. Swersky, Z. Wang, R. Adams, and N. de Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1):148–175, 2016.
  38. 38.J. Snoek, H. Larochelle, and R. Adams. Practical Bayesian optimization of machine learning algorithms. In P. Bartlett, F. Pereira, C. Burges, L. Bottou, and K. Weinberger, editors, Proceedings of the 25th International Conference on Advances in Neural Information Processing Systems (NeurIPS’12), pages 2960–2968. Curran Associates, 2012.
  39. 39.N. Srinivas, A. Krause, S. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In J. Furnkranz and T. Joachims, editors, Proceedings of the 27th International Conference on Machine Learning (ICML’10), pages 1015–1022. Omnipress, 2010.
  40. 40.W. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285–294, 1933.
  41. 41.C. Thornton, F. Hutter, H. Hoos, and K. Leyton-Brown. Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms. In I. Dhillon, Y. Koren, R. Ghani, T. Senator, P. Bradley, R. Parekh, J. He, R. Grossman, and R. Uthurusamy, editors, The 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’13), pages 847–855. ACM Press, 2013.
  42. 42.R. Turner, D. Eriksson, M. McCourt, J. Kiili, E. Laaksonen, Z. Xu, and I. Guyon. Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020. In H. Escalante and K. Hofmann, editors, NeurIPS 2020 Competition and Demonstration Track, volume 133 of Proceedings of Machine Learning Research, pages 3–26. PMLR, 2020.
  43. 43.A. Zela, T. Elsken, T. Saikia, Y. Marrakchi, T. Brox, and F. Hutter. Understanding and robustifying differentiable architecture search. In Proceedings of the International Conference on Learning Representations (ICLR’20), 2020. Published online: iclr.cc.
  44. 44.L. Zimmer, M. Lindauer, and F. Hutter. Auto-Pytorch: Multi-fidelity metalearning for efficient and robust AutoDL. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 3079–3090, 2021.

Citation

MLA
Lindauer, M., et al. “SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization”. Journal of Machine Learning Research, vol. 23, no. 54, 2022, pp. 1–9, https://www.jmlr.org/papers/v23/21-0888.html.
APA
Lindauer, M., Eggensperger, K., Feurer, M., Biedenkapp, A., Deng, D., Benjamins, C., Ruhkopf, T., Sass, R., & Hutter, F. (2022). SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization. Journal of Machine Learning Research, 23(54), 1–9. https://www.jmlr.org/papers/v23/21-0888.html
Chicago
Lindauer, M., K. Eggensperger, M. Feurer, et al. 2022. “SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization”. Journal of Machine Learning Research 23 (54): 1–9. https://www.jmlr.org/papers/v23/21-0888.html.
Harvard
Lindauer, M. et al. (2022) “SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization”, Journal of Machine Learning Research, 23(54), pp. 1–9. Available at: https://www.jmlr.org/papers/v23/21-0888.html.
Vancouver
1. Lindauer M, Eggensperger K, Feurer M, Biedenkapp A, Deng D, Benjamins C, Ruhkopf T, Sass R, Hutter F (2022) SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization. Journal of Machine Learning Research 23:1–9

BibTeX

@article{JMLR:v23:21-0888,
  author  = {Marius Lindauer and Katharina Eggensperger and Matthias Feurer and André Biedenkapp and Difan Deng and Carolin Benjamins and Tim Ruhkopf and René Sass and Frank Hutter},
  title   = {SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization},
  journal = {Journal of Machine Learning Research},
  year    = {2022},
  volume  = {23},
  number  = {54},
  pages   = {1--9},
  url     = {http://jmlr.org/papers/v23/21-0888.html}
}
Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: https://creativecommons.org/licenses/by/4.0/