Back

Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces

Leonard PapenmeierLuigi NardiMatthias Poloczek

article2022NeurIPS75 citations

Proposes BAXUS, a high-dimensional Bayesian optimization algorithm that adaptively expands nested random subspaces to remove the need for prior assumptions about active subspace dimensionality while providing theoretical convergence guarantees and superior empirical performance.

Listen

High-dimensional black-box optimization is critical across engineering and science, including chemical engineering, drug discovery, vehicle design, and machine learning tuning. In these domains, evaluating candidate designs is expensive, and optimization problems often involve hundreds or thousands of input parameters. Traditional Bayesian optimization methods, which model unknown functions to guide sampling efficiently, struggle in high dimensions due to the exponential growth of the search space. Existing high-dimensional techniques either degrade rapidly as parameter counts grow or rely on unprovable assumptions, such as guessing the unknown size of an active lower-dimensional subspace beforehand.

The article develops and evaluates BAXUS (Bayesian optimization with adaptively expanding subspaces), an optimization algorithm designed to optimize high-dimensional black-box functions efficiently without requiring users to guess subspace dimensions. The primary objective is to demonstrate that adaptively expanding nested subspaces provides strong theoretical guarantees and outperforms existing state-of-the-art optimization methods on complex, high-dimensional benchmarks.

The authors designed a sparse random linear embedding that starts in a low-dimensional target space and systematically increases dimensionality over time as more data is evaluated. By splitting target dimensions and copying prior observations, the method preserves all previously collected data in newly expanded spaces. The framework also integrates an adaptive trust-region strategy to focus sampling around the best-known candidates while dynamically adjusting failure tolerances to ensure the full input space can be reached within a fixed budget. The authors evaluated the approach across six diverse benchmarks ranging from 124 to 1,000 dimensions—including vehicle design, hyperparameter tuning, synthetic test functions, and LASSO benchmarks with and without observational noise—running 20 repeated trials per method alongside established baselines like TURBO, SAASBO, ALEBO, HESBO, and CMA-ES.

The evaluation revealed several key findings. First, BAXUS achieved the best overall optimization performance across the benchmarks, consistently outperforming standard high-dimensional baselines on 1,000-dimensional and 300-dimensional LASSO tasks as well as on a 388-dimensional support vector machine tuning problem and a 124-dimensional vehicle design problem. Second, the proposed sparse embedding was mathematically proven to provide a larger worst-case guarantee of containing the true global optimum than existing hash-based embeddings like HESBO, achieving optimality among sparse linear embeddings. Third, BAXUS demonstrated strong robustness to observational noise, maintaining steady optimization progress past 1,000 evaluations where competing methods degraded significantly. Finally, while methods like SAASBO converged quickly on specific low-dimensional active spaces (such as the 500-dimensional Hartmann function), they faced severe computational scalability bottlenecks, whereas BAXUS scaled efficiently across large evaluation budgets.

These results show that engineering teams and researchers can optimize complex systems with hundreds of design parameters without manual, risky guesses about problem structure. By removing the risk of subspace misspecification and lowering evaluation waste, the approach reduces the time, computational overhead, and experimental costs required to find high-performing designs in sensitive applications such as drug discovery and manufacturing.

Organizations handling high-dimensional black-box optimization should consider deploying BAXUS as an out-of-the-box solver, utilizing its open-source implementation. In the near term, teams should validate performance on internal pilot problems, particularly those involving noisy experimental data. Future development should explore tailoring subspace expansion strategies using domain knowledge and extending the framework to structured combinatorial search spaces common in materials science.

Confidence in these findings is supported by rigorous convergence proofs and diverse experimental benchmarks with repeated trials. However, users should note that the current implementation focuses primarily on continuous parameter spaces and uses a fixed heuristic schedule for subspace growth, which may benefit from domain-specific tuning on highly specialized tasks.

AmazonDBtuneLund UniversityStanford Universitybayesian methodsoptimizationfeature spaceadaptive Bayesian optimizationhigh-dimensional Bayesian optimizationnested subspaces
arXiv: 2304.11468lpapenme/BAxUS
Cover for Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces

Abstract

Recent advances have extended the scope of Bayesian optimization (BO) to expensive-to-evaluate black-box functions with dozens of dimensions, aspiring to unlock impactful applications, for example, in the life sciences, neural architecture search, and robotics. However, a closer examination reveals that the state-of-the-art methods for high-dimensional Bayesian optimization (HDBO) suffer from degrading performance as the number of dimensions increases or even risk failure if certain unverifiable assumptions are not met. This paper proposes BAXUS that leverages a novel family of nested random subspaces to adapt the space it optimizes over to the problem. This ensures high performance while removing the risk of failure, which we assert via theoretical guarantees. A comprehensive evaluation demonstrates that BAXUS achieves better results than the state-of-the-art methods for a broad set of applications.

Topics

bayesian methodsoptimizationfeature spaceoptimization problems

Keywords

adaptive Bayesian optimizationhigh-dimensional Bayesian optimizationnested subspaceshyperparameter optimizationThompson samplingactive subspacesneural architecture searchpartial least squareseffective dimensionalitysparse projection

Table of Contents

  • 1 Introduction
  • 2 Background
  • 3 The BAXUS algorithm
  • 3.1 The sparse BAXUS subspace embedding
  • 3.2 Trust-region approach
  • 3.3 Splitting strategy
  • 3.4 Controlling the number of accepted failures
  • 4 Experimental evaluation
  • 4.1 Experimental results
  • 5 Discussion
  • Acknowledgments and Disclosure of Funding
  • References

Knowls

  1. Knowl 1 — BAXUS adaptively expands the Bayesian-optimization search space

    algorithm

    BAXUS (Bayesian optimization with adaptively expanding subspaces) optimizes an expensive black-box function f:[−1,1]D→Rf:[-1,1]^D\to\mathbb{R} without requiring the effective dimensionality of the problem to be known. It represents an input point as x=S⊤yx=S^\top y, where S⊤S^\top maps a lower-dimensional target vector y∈Rdy\in\mathbb{R}^d into the DD-dimensional input space.

    The algorithm starts with a small target dimension d0d_0, fits a Gaussian-process surrogate to initial observations, and optimizes the surrogate inside a trust region using Thompson sampling. When the trust region becomes too small, BAXUS increases the target dimension from dd to min⁡((b+1)d,D)\min((b+1)d,D) by splitting every target dimension into b+1b+1 target dimensions while preserving all existing observations. This continues until the target dimension reaches the input dimension DD. If the full dimension has been reached and the trust region terminates again, BAXUS resets the trust region and draws a new initial design rather than expanding further.

    The returned solution is the best observed point for noiseless evaluations, or the point with the smallest Gaussian-process posterior mean for noisy evaluations. In the reported experiments, BAXUS used b=3b=3, allocated mD=1000m_D=1000 evaluations to the expansion phase, and began each run with ten initial samples. Because the target space eventually reaches the full input dimension, the paper establishes a global-convergence guarantee in the asymptotic limit while retaining low-dimensional Bayesian optimization during the early part of the run.

  2. Knowl 2 — Balanced sparse nested embeddings preserve observations during expansion

    model/method

    For target dimension d≤Dd\le D, BAXUS constructs a sparse matrix S∈{0,±1}d×DS\in\{0,\pm1\}^{d\times D} such that every column has exactly one nonzero entry. Each input coordinate is therefore assigned to one target-coordinate bin and given an independent random sign. The embedding is x=S⊤yx=S^\top y, so all input coordinates assigned to the same bin share the corresponding target coordinate, up to their assigned signs.

    BAXUS first randomly permutes the DD input coordinates and partitions them into dd bins whose sizes differ by at most one: each bin has either ⌊D/d⌋\lfloor D/d\rfloor or ⌈D/d⌉\lceil D/d\rceil contributing input coordinates. This balanced assignment is the defining difference from the uniformly hashed sparse embedding used by HESBO.

    To expand the embedding, BAXUS splits each current bin into b+1b+1 bins and reassigns its contributing input coordinates among the resulting bins. For every previously observed target point, the old coordinate value is copied into all of its child coordinates. Consequently, the embedded input point S⊤yS^\top y is unchanged for every retained observation, even though the new target space has more coordinates. The construction produces a nested sequence of target spaces rather than discarding data whenever the optimization dimension increases.

  3. Knowl 3 — Finite-dimensional worst-case success probability of BAXUS

    theoretical result

    Let DD be the input dimension, let dd be the target dimension with de≤d≤Dd_e\le d\le D, and let ded_e be the number of active input dimensions. Define the success event Y∗Y^* as the event that all ded_e active input dimensions are assigned to distinct target bins. This event is sufficient, but not necessary, for the embedding to contain a global optimum; it is used as a worst-case guarantee.

    For the balanced BAXUS embedding, define the small and large bin sizes by

    βsmall=⌊Dd⌋,βlarge=⌈Dd⌉.\beta_{\mathrm{small}}=\left\lfloor\frac{D}{d}\right\rfloor, \qquad \beta_{\mathrm{large}}=\left\lceil\frac{D}{d}\right\rceil.

    The probability of the success event is

    pB(Y∗;D,d,de)=∑i=0de(d(1+βsmall)−Di)(D−dβsmallde−i)βsmalliβlargede−i(Dde).p_B(Y^*;D,d,d_e)= \frac{\displaystyle\sum_{i=0}^{d_e} \binom{d(1+\beta_{\mathrm{small}})-D}{i} \binom{D-d\beta_{\mathrm{small}}}{d_e-i} \beta_{\mathrm{small}}^{i} \beta_{\mathrm{large}}^{d_e-i}} {\displaystyle\binom{D}{d_e}}.

    The numerator counts selections of active coordinates from the small and large bins while requiring at most one active coordinate per bin. The formula shows that balancing the bins improves the worst-case probability that a random embedding can represent an optimum.

  4. Knowl 4 — BAXUS is optimal among sparse embeddings and approaches HESBO asymptotically

    theoretical result

    For the same input dimension DD, target dimension dd, and effective dimension ded_e, no sparse embedding matrix—meaning a matrix in {0,±1}d×D\{0,\pm1\}^{d\times D} with exactly one nonzero entry in every column—has a larger worst-case success probability than the balanced BAXUS embedding.

    The BAXUS success probability converges, as the input dimension tends to infinity, to

    lim⁡D→∞pB(Y∗;D,d,de)=d!(d−de)! dde.\lim_{D\to\infty}p_B(Y^*;D,d,d_e) = \frac{d!}{(d-d_e)!\,d^{d_e}}.

    This is the worst-case success probability of the HESBO sparse embedding. For finite DD, BAXUS has a strictly larger worst-case success probability than HESBO; when d=Dd=D, BAXUS has success probability one. Thus, BAXUS improves the finite-dimensional guarantee without changing the limiting guarantee of the competing sparse construction.

  5. Knowl 5 — Trust regions make optimization feasible as the target dimension grows

    model/method

    BAXUS performs Gaussian-process Bayesian optimization inside hyperrectangular trust regions in the current target space. If the Gaussian-process kernel has length scale ℓj\ell_j in target coordinate jj, the trust-region side length in that coordinate is scaled proportionally to ℓj\ell_j, so dimensions along which the surrogate varies rapidly receive narrower search intervals.

    The trust region has a base side length LL. It shrinks when the optimizer experiences τfail\tau_{\mathrm{fail}} consecutive evaluations without improving the incumbent and expands after τsuccess=3\tau_{\mathrm{success}}=3 consecutive improvements. A trust-region cycle terminates when LL falls below Lmin⁡=2−7L_{\min}=2^{-7}. The next candidate is selected by drawing a Gaussian-process function realization over candidate points in the trust region and evaluating the point with the smallest sampled value; this is Thompson sampling.

    Unlike the original trust-region schedule, BAXUS uses termination of a trust-region cycle to trigger an increase in target dimension rather than simply restarting at the same dimension. Once the target dimension has reached DD, it resets LL to its initial value and reinitializes from new random observations, allowing subsequent cycles to explore regions beyond a previously encountered local minimum.

  6. Knowl 6 — Exponential splitting and dimension-dependent failure budgets control the expansion

    algorithm

    If the initial target dimension is dinitd_{\mathrm{init}} and every target dimension is split into b+1b+1 children, the target dimension after ii expansions is

    di=dinit(b+1)i.d_i=d_{\mathrm{init}}(b+1)^i.

    The number of expansions needed to approach input dimension DD is selected as the nearest integer to

    n≈log⁡b+1(Ddinit).n\approx\log_{b+1}\left(\frac{D}{d_{\mathrm{init}}}\right).

    BAXUS allocates an evaluation budget mDm_D across target dimensions in proportion to their dimensions. For i=0,…,ni=0,\ldots,n, the rounded minimum budget is

    mis=round⁡(mDdi∑j=0ndj).m_i^s=\operatorname{round}\left(m_D\frac{d_i}{\sum_{j=0}^{n}d_j}\right).

    Let kk be the number of trust-region halvings required to reduce the base side length from its initial value LinitL_{\mathrm{init}} to Lmin⁡L_{\min}. The number of accepted failures at target dimension did_i is set to

    τfaili=max⁡(1,min⁡(round⁡(misk),di)).\tau_{\mathrm{fail}}^i =\max\left(1,\min\left(\operatorname{round}\left(\frac{m_i^s}{k}\right),d_i\right)\right).

    This schedule ensures that a run that makes no progress still receives approximately its allocated budget before expansion, while successful runs are not forcibly stopped after a fixed number of evaluations. To make the last target dimension as close as possible to DD, BAXUS chooses the initial dimension from {1,…,b}\{1,\ldots,b\} to minimize

    ∣dinit(b+1)n−D∣.\left|d_{\mathrm{init}}(b+1)^n-D\right|.

    The exponential growth requires only logarithmically many expansion events in DD, while the budget allocation prevents the fixed evaluation budget from trapping the algorithm at a low target dimension.

  7. Knowl 7 — Benchmark protocol for high-dimensional Bayesian optimization

    experimental setup

    BAXUS was compared with TURBO using one and five trust regions, SAASBO, ALEBO, HESBO, random search, and CMA-ES. The implementations and settings supplied by the respective authors were used unless otherwise stated; CMA-ES used PYCMA, while HESBO and ALEBO used the AX implementation. HESBO and ALEBO were each tested with target dimensions d=10d=10 and d=20d=20.

    The evaluation covered a 124124-dimensional MOPTA08 vehicle-design problem, a 388388-dimensional SVM hyperparameter problem, 500500-dimensional augmented BRANIN2 and HARTMANN6 functions, and the LASSOBENCH LASSO-HARD and LASSO-HIGH tasks with dimensions 10001000 and 300300, respectively. The LASSO tasks have effective dimensions equal to 5%5\% of their input dimensions: 5050 for LASSO-HARD and 1515 for LASSO-HIGH. Both noiseless and noisy LASSO variants were tested.

    Each optimizer received ten initial samples and was run for 20 repeated trials. BAXUS used b=3b=3 and mD=1000m_D=1000. The total budget was 1000 evaluations for MOPTA08, BRANIN2, and HARTMANN6, and 2000 evaluations for the other benchmarks; BRANIN2 and HARTMANN6 runs were also stopped once simple regret fell below 0.0010.001. SAASBO was limited to 100 evaluations and ALEBO to 500 evaluations per run because of runtime and memory limits. Reported curves show mean performance with one standard error.

  8. Knowl 8 — BAXUS is consistently strong on noiseless high-dimensional benchmarks

    empirical result

    On the noiseless benchmarks, BAXUS was the only method that consistently achieved high performance across the full collection. It obtained the best solutions on the 124124-dimensional MOPTA08 task, followed by TURBO and CMA-ES, although SAASBO initially progressed slightly faster. On the 388388-dimensional SVM task, BAXUS improved faster than TURBO and CMA-ES from the beginning and adapted to a target dimension that supported better solutions than the low-dimensional embeddings.

    On the 500500-dimensional BRANIN2 task, SAASBO, BAXUS, ALEBO, and HESBO all found excellent solutions, with SAASBO and ALEBO converging fastest. On the 500500-dimensional HARTMANN6 task, SAASBO performed best and BAXUS was close behind, while the other methods either progressed slowly or converged to suboptimal values.

    The largest advantage appeared on the active-subspace LASSO tasks: BAXUS found considerably better solutions than all state-of-the-art baselines on both 10001000-dimensional LASSO-HARD and 300300-dimensional LASSO-HIGH. TURBO and CMA-ES outperformed SAASBO and ALEBO there, but still underperformed BAXUS. BAXUS also showed only small variation across repeated runs despite its randomized embedding.

  9. Knowl 9 — BAXUS remains effective under observational noise and benefits from its embedding

    empirical result

    On noisy versions of the 10001000-dimensional LASSO-HARD and 300300-dimensional LASSO-HIGH tasks, BAXUS achieved substantially better solutions than the competing methods at every reported observation count. Its performance remained comparable to its noiseless performance and continued improving beyond 1000 evaluations.

    In contrast, SAASBO, CMA-ES, and HESBO with target dimension d=20d=20 degraded substantially on noisy LASSO-HIGH relative to their noiseless results, and CMA-ES performed much worse than on the corresponding noiseless tasks. The robustness of BAXUS was attributed to expanding a nested target space while retaining prior observations.

    An embedding ablation replaced the BAXUS nested embedding with a nested sequence of HESBO embeddings while leaving the broader optimization strategy unchanged. The ablation showed a significant performance loss, indicating that the balanced BAXUS embedding—not only adaptive dimension expansion or trust-region optimization—contributes materially to the method's performance.

  10. Knowl 10 — The expansion policy is simple and leaves application-specific improvements open

    limitation

    The authors describe the BAXUS expansion rule as deliberately simple and note substantial room for improvement through application-specific domain knowledge or a more sophisticated data-driven procedure for learning a suitable target space. The reported method is designed for continuous Euclidean domains and does not yet address structured domains such as the combinatorial spaces arising in materials science and drug discovery. Consequently, its demonstrated guarantees and empirical conclusions are limited to the continuous high-dimensional settings evaluated in the paper.

Coverage note — Detailed proofs, supplementary benchmark plots, compute-resource details, and additional appendix experiments were omitted because the knowls retain the main theoretical results, algorithmic mechanisms, and primary empirical conclusions.

References

  1. 1.E. Bakshy, L. Dworkin, B. Karrer, K. Kashin, B. Letham, A. Murthy, and S. Singh. Ae: A domain-agnostic platform for adaptive experimentation. In Conference on Neural Information Processing Systems, pages 1–8, 2018.
  2. 2.R. Baptista and M. Poloczek. Bayesian optimization of combinatorial structures. In International Conference on Machine Learning, pages 462–471. PMLR, 2018.
  3. 3.E. F. Beckenbach and R. Bellman. Inequalities, volume 30. Springer Science & Business Media, 2012.
  4. 4.J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of machine learning research, 13(2), 2012.
  5. 5.J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for Hyper-Parameter Optimization. In Advances in Neural Information Processing Systems (NeurIPS), volume 24. Curran Associates, Inc., 2011.
  6. 6.M. Binois. Uncertainty quantification on Pareto fronts and high-dimensional strategies in Bayesian optimization, with applications in multi-objective automotive design. PhD thesis, Ecole Nationale Supérieure des Mines de Saint-Etienne, 2015.
  7. 7.M. Binois and N. Wycoff. A survey on high-dimensional Gaussian process modeling with application to Bayesian optimization. ACM Transactions on Evolutionary Learning and Optimization, 2(2):1–26, 2022.
  8. 8.M. Binois, D. Ginsbourger, and O. Roustant. A warped kernel improving robustness in Bayesian optimization via random embeddings. In International Conference on Learning and Intelligent Optimization (LION), pages 281–286. Springer, 2015.
  9. 9.M. Binois, D. Ginsbourger, and O. Roustant. On the choice of the low-dimensional domain for global optimization via random embeddings. Journal of global optimization, 76(1):69–90, 2020.
  10. 10.M. A. Bouhlel, N. Bartoli, R. G. Regis, A. Otsmane, and J. Morlier. Efficient global optimization for high-dimensional constrained problems by using the Kriging models combined with the partial least squares method. Engineering Optimization, 50(12):2038–2053, 2018.
  11. 11.B. Burger, P. M. Maffettone, V. V. Gusev, C. M. Aitchison, Y. Bai, X. Wang, X. Li, B. M. Alston, B. Li, R. Clowes, et al. A mobile robotic chemist. Nature, 583(7815):237–241, 2020.
  12. 12.R. Calandra, N. Gopalan, A. Seyfarth, J. Peters, and M. P. Deisenroth. Bayesian Gait Optimization for Bipedal Locomotion. In P. M. Pardalos, M. G. Resende, C. Vogiatzis, and J. L. Walteros, editors, Learning and Intelligent Optimization, pages 274–290, Cham, 2014. Springer International Publishing.
  13. 13.R. Calandra, A. Seyfarth, J. Peters, and M. P. Deisenroth. Bayesian optimization for learning gaits under uncertainty. Annals of Mathematics and Artificial Intelligence, 76(1):5–23, 2016.
  14. 14.A. Candelieri, R. Perego, and F. Archetti. Bayesian Optimization of Pump Operations in Water Distribution Systems. Journal of Global Optimization, 71(1):213235, May 2018. ISSN 0925-5001.
  15. 15.M. Charikar, K. Chen, and M. Farach-Colton. Finding Frequent Items in Data Streams. In P. Widmayer, S. Eidenbenz, F. Triguero, R. Morales, R. Conejo, and M. Hennessy, editors, Automata, Languages and Programming, pages 693–703, Berlin, Heidelberg, 2002. Springer Berlin Heidelberg. ISBN 978-3-540-45465-6.
  16. 16.J. Chen, G. Zhu, C. Yuan, and Y. Huang. Semi-supervised Embedding Learning for High-dimensional Bayesian Optimization. arXiv preprint arXiv:2005.14601, 2020.
  17. 17.P. G. Constantine. Active Subspaces. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2015.
  18. 18.Z. Cosenza, R. Astudillo, P. Frazier, K. Baar, and D. E. Block. Multi-Information Source Bayesian Optimization of Culture Media for Cellular Agriculture. Biotechnology and Bioengineering, 2022.
  19. 19.A. Ejjeh, L. Medvinsky, A. Councilman, H. Nehra, S. Sharma, V. Adve, L. Nardi, E. Nurvitadhi, and R. A. Rutenbar. HPVM2FPGA: Enabling True Hardware-Agnostic FPGA Programming. In Proceedings of the 33rd IEEE International Conference on Application-specific Systems, Architectures, and Processors, 2022.
  20. 20.D. Eriksson and M. Jankowiak. High-dimensional Bayesian optimization with sparse axis-aligned subspaces. In C. de Campos and M. H. Maathuis, editors, Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pages 493–503. PMLR, 27–30 Jul 2021.
  21. 21.D. Eriksson and M. Poloczek. Scalable Constrained Bayesian Optimization. In A. Banerjee and K. Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 730–738. PMLR, 13–15 Apr 2021.
  22. 22.D. Eriksson, M. Pearce, J. Gardner, R. D. Turner, and M. Poloczek. Scalable Global Optimization via Local Bayesian Optimization. In Advances in Neural Information Processing Systems (NeurIPS), pages 5496–5507, 2019.
  23. 23.P. I. Frazier and J. Wang. Bayesian Optimization for Materials Design, pages 45–75. Springer International Publishing, Cham, 2016. ISBN 978-3-319-23871-5.
  24. 24.J. Gardner, C. Guo, K. Weinberger, R. Garnett, and R. Grosse. Discovering and exploiting additive structure for Bayesian optimization. In International Conference on Artificial Intelligence and Statistics, pages 1311–1319, 2017.
  25. 25.R. L. Graham, D. E. Knuth, O. Patashnik, and S. Liu. Concrete mathematics: a foundation for computer science. Computers in Physics, 3(5):106–107, 1989.
  26. 26.N. Hansen and A. Ostermeier. Adapting arbitrary normal mutation distributions in evolution strategies: the covariance matrix adaptation. In Proceedings of IEEE International Conference on Evolutionary Computation (ICEC), pages 312–317, 1996.
  27. 27.N. Hansen, Y. Akimoto, and P. Baudis. CMA-ES/pycma on Github. Zenodo, DOI:10.5281/zenodo.2559634, Feb. 2019. Last accessed: 05/09/2022. License: BSD-3-Clause.
  28. 28.F. Hase, L. M. Roch, C. Kreisbeck, and A. Aspuru-Guzik. Phoenics: a Bayesian optimizer for chemistry. ACS central science, 4(9):1134–1145, 2018.
  29. 29.F. Häse, M. Aldeghi, R. J. Hickman, L. M. Roch, and A. Aspuru-Guzik. Gryffin: An algorithm for Bayesian optimization of categorical variables informed by expert knowledge. Applied Physics Reviews, 8(3):031406, 2021.
  30. 30.H. C. Herbol, W. Hu, P. Frazier, P. Clancy, and M. Poloczek. Efficient search of compositional space for hybrid organic–inorganic perovskites via Bayesian optimization. npj Computational Materials, 4(1):1–7, 2018.
  31. 31.J. M. Hernández-Lobato, J. Requeima, E. O. Pyzer-Knapp, and A. Aspuru-Guzik. Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical Space. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1470–1479. PMLR, 06–11 Aug 2017.
  32. 32.Z. E. Hughes, M. A. Nguyen, J. Wang, Y. Liu, M. T. Swihart, M. Poloczek, P. I. Frazier, M. R. Knecht, and T. R. Walsh. Tuning materials-binding peptide sequences toward gold-and silver-binding selectivity with Bayesian optimization. ACS nano, 15(11):18260–18269, 2021.
  33. 33.C. Hvarfner, D. Stoll, A. Souza, L. Nardi, M. Lindauer, and F. Hutter. PiBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization. In International Conference on Learning Representations, 2022.
  34. 34.D. R. Jones. Large-scale multi-disciplinary mass optimization in the auto industry. In MOPTA 2008 Conference (20 August 2008), 2008.
  35. 35.K. Kandasamy, J. Schneider, and B. Póczos. High dimensional Bayesian optimisation and bandits via additive models. In International conference on machine learning (ICML), pages 295–304, 2015.
  36. 36.K. Kandasamy, W. Neiswanger, J. Schneider, B. Poczos, and E. P. Xing. Neural Architecture Search with Bayesian Optimisation and Optimal Transport. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems (NeurIPS), volume 31. Curran Associates, Inc., 2018.
  37. 37.D. P. Kingma and M. Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
  38. 38.A. Klein, S. Falkner, S. Bartels, P. Hennig, and F. Hutter. Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets. In A. Singh and J. Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 528–536. PMLR, 20–22 Apr 2017.
  39. 39.R. Lam, M. Poloczek, P. Frazier, and K. E. Willcox. Advances in Bayesian optimization with applications in aerospace engineering. In 2018 AIAA Non-Deterministic Approaches Conference, page 1656, 2018.
  40. 40.B. Letham, R. Calandra, A. Rai, and E. Bakshy. Re-Examining Linear Embeddings for High-Dimensional Bayesian Optimization. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 1546–1558. Curran Associates, Inc., 2020.
  41. 41.D. J. Lizotte, T. Wang, M. H. Bowling, D. Schuurmans, et al. Automatic Gait Optimization With Gaussian Process Regression. In IJCAI, volume 7, pages 944–949, 2007.
  42. 42.X. Lu, J. Gonzalez, Z. Dai, and N. D. Lawrence. Structured Variationally Auto-encoded Optimization. In J. G. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 3273–3281. PMLR, 2018.
  43. 43.T. W. Lukaczyk, P. Constantine, F. Palacios, and J. J. Alonso. Active subspaces for shape optimization. In 10th AIAA multidisciplinary design optimization conference, page 1171, 2014.
  44. 44.A. W. Marshall, I. Olkin, and B. C. Arnold. Inequalities: theory of majorization and its applications, volume 143. Springer, 1979.
  45. 45.N. Maus, H. T. Jones, J. S. Moore, M. J. Kusner, J. Bradshaw, and J. R. Gardner. Local latent space Bayesian optimization over structured inputs. arXiv, 2022.
  46. 46.M. Mayr, F. Ahmad, K. I. Chatzilygeroudis, L. Nardi, and V. Krüger. Skill-based Multi-objective Reinforcement Learning of Industrial Robot Tasks with Planning and Knowledge Integration. CoRR, abs/2203.10033, 2022.
  47. 47.R. Moriconi, M. P. Deisenroth, and K. S. Sesh Kumar. High-dimensional Bayesian optimization using low-dimensional feature spaces. Machine Learning, 109(9):1925–1943, Sep 2020. ISSN 1573-0565.
  48. 48.M. Mutny and A. Krause. Efficient high dimensional Bayesian optimization with additivity and quadrature Fourier features. Advances in Neural Information Processing Systems, 31, 2018.
  49. 49.L. Nardi, D. Koeplinger, and K. Olukotun. Practical design space exploration. In 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS), pages 347–358. IEEE, 2019.
  50. 50.A. Nayebi, A. Munteanu, and M. Poloczek. A framework for Bayesian Optimization in Embedded Subspaces. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research (PMLR), pages 4752–4761. PMLR, 09–15 Jun 2019.
  51. 51.D. M. Negoescu, P. I. Frazier, and W. B. Powell. The Knowledge-Gradient Algorithm for Sequencing Experiments in Drug Discovery. INFORMS Journal on Computing, 23(3):346–363, 2011.
  52. 52.D. Packwood. Bayesian optimization for materials science. Springer.
  53. 53.G. Pedrielli and S. H. Ng. G-STAR: A new kriging-based trust region method for global optimization. In 2016 Winter Simulation Conference (WSC), pages 803–814. IEEE, 2016.
  54. 54.A. Rai, R. Antonova, S. Song, W. Martin, H. Geyer, and C. Atkeson. Bayesian optimization using domain knowledge on the ATRIAS biped. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1771–1778. IEEE, 2018.
  55. 55.R. G. Regis. Trust regions in Kriging-based optimization with expected improvement. Engineering optimization, 48(6):1037–1059, 2016.
  56. 56.H. Robbins. A remark on stirling’s formula. The American mathematical monthly, 62(1):26–29, 1955.
  57. 57.B. Ru, X. Wan, X. Dong, and M. Osborne. Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels. In International Conference on Learning Representations, 2021.
  58. 58.A. M. Schweidtmann, A. D. Clayton, N. Holmes, E. Bradford, R. A. Bourne, and A. A. Lapkin. Machine learning meets continuous flow chemistry: Automated optimization towards the Pareto front of multiple objectives. Chemical Engineering Journal, 352:277–282, 2018.
  59. 59.K. Šehic, A. Gramfort, J. Salmon, and L. Nardi. LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lasso. In First Conference on Automated Machine Learning (Main Track), 2022.
  60. 60.B. J. Shields, J. Stevens, J. Li, M. Parasram, F. Damani, J. I. M. Alvarado, J. M. Janey, R. P. Adams, and A. G. Doyle. Bayesian reaction optimization as a tool for chemical synthesis. Nature, 590(7844):89–96, 2021.
  61. 61.J. Snoek, H. Larochelle, and R. P. Adams. Practical Bayesian Optimization of Machine Learning Algorithms. In F. Pereira, C. Burges, L. Bottou, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012.
  62. 62.H. H. Sohrab. Basic real analysis, volume 231. Springer, 2003.
  63. 63.F. J. Solis and R. J.-B. Wets. Minimization by Random Search Techniques. Mathematics of Operations Research, 6(1):19–30, 1981. ISSN 0364765X, 15265471.
  64. 64.A. Souza, L. B. Oliveira, S. Hollatz, M. Feldman, K. Olukotun, J. M. Holton, A. E. Cohen, and L. Nardi. DeepFreak: Learning crystallography diffraction patterns with automated machine learning. arXiv preprint arXiv:1904.11834, 2019.
  65. 65.L. Tallorin, J. Wang, W. E. Kim, S. Sahu, N. M. Kosa, P. Yang, M. Thompson, M. K. Gilson, P. I. Frazier, M. D. Burkart, et al. Discovering de novo peptide substrates for enzymes using machine learning. Nature communications, 9(1):1–10, 2018.
  66. 66.W. R. Thompson. On the Likelihood that One Unknown Probability Exceeds Another in View of the Evidence of Two Samples. Biometrika, 25(3/4):285–294, 1933. ISSN 00063444.
  67. 67.A. Tripp, E. Daxberger, and J. M. Hernández-Lobato. Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 11259–11272. Curran Associates, Inc., 2020.
  68. 68.T. Ueno, T. D. Rhone, Z. Hou, T. Mizoguchi, and K. Tsuda. COMBO: An efficient Bayesian optimization library for materials science. Materials Discovery, 4:18–21, 2016. ISSN 2352-9245.
  69. 69.X. Wan, V. Nguyen, H. Ha, B. Ru, C. Lu, and M. A. Osborne. Think Global and Act Local: Bayesian Optimisation over High-Dimensional Categorical and Mixed Search Spaces. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10663–10674. PMLR, 18–24 Jul 2021.
  70. 70.L. Wang, R. Fonseca, and Y. Tian. Learning search space partition for black-box optimization using monte carlo tree search. Advances in Neural Information Processing Systems, 33:19511–19522, 2020.
  71. 71.Z. Wang, F. Hutter, M. Zoghi, D. Matheson, and N. de Feitas. Bayesian optimization in a billion dimensions via random embeddings. Journal of Artificial Intelligence Research (JAIR), 55:361–387, 2016.
  72. 72.Z. Wang, C. Gehring, P. Kohli, and S. Jegelka. Batched large-scale Bayesian optimization in high-dimensional spaces. In International Conference on Artificial Intelligence and Statistics, pages 745–754, 2018.
  73. 73.C. K. Williams and C. E. Rasmussen. Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA, 2006.
  74. 74.D. P. Woodruff et al. Sketching as a tool for numerical linear algebra. Foundations and Trends® in Theoretical Computer Science, 10(1–2):1–157, 2014.
  75. 75.J. Zhou, Z. Yang, Y. Si, L. Kang, H. Li, M. Wang, and Z. Zhang. A trust-region parallel Bayesian optimization method for simulation-driven antenna design. IEEE Transactions on Antennas and Propagation, 69(7):3966–3981, 2020.

Citation

MLA
Papenmeier, L., et al. “Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces”. Advances in Neural Information Processing Systems, vol. 35, 2022, pp. 11586–601, https://proceedings.neurips.cc/paper_files/paper/2022/file/4b7439a4ab0b8e4bcb4e2412c6a10a58-Paper-Conference.pdf.
APA
Papenmeier, L., Nardi, L., & Poloczek, M. (2022). Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces. Advances in Neural Information Processing Systems, 35, 11586–11601. https://proceedings.neurips.cc/paper_files/paper/2022/file/4b7439a4ab0b8e4bcb4e2412c6a10a58-Paper-Conference.pdf
Chicago
Papenmeier, L., L. Nardi, and M. Poloczek. 2022. “Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces”. Advances in Neural Information Processing Systems 35: 11586–601. https://proceedings.neurips.cc/paper_files/paper/2022/file/4b7439a4ab0b8e4bcb4e2412c6a10a58-Paper-Conference.pdf.
Harvard
Papenmeier, L., Nardi, L. and Poloczek, M. (2022) “Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces”, Advances in Neural Information Processing Systems. Curran Associates, Inc., pp. 11586–11601. Available at: https://proceedings.neurips.cc/paper_files/paper/2022/file/4b7439a4ab0b8e4bcb4e2412c6a10a58-Paper-Conference.pdf.
Vancouver
1. Papenmeier L, Nardi L, Poloczek M (2022) Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces. In: Advances in Neural Information Processing Systems. Curran Associates, Inc., pp 11586–11601

BibTeX

@inproceedings{papenmeier2022increasing,
  title = {Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces},
  author = {Papenmeier, Leonard and Nardi, Luigi and Poloczek, Matthias},
  year = {2022},
  booktitle = {Advances in Neural Information Processing Systems},
  publisher = {Curran Associates, Inc.},
  volume = {35},
  pages = {11586-11601},
  url = {https://proceedings.neurips.cc/paper_files/paper/2022/file/4b7439a4ab0b8e4bcb4e2412c6a10a58-Paper-Conference.pdf}
}
Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: Authors