Bayesian optimization is an iterative, probabilistic optimization approach for minimizing an unknown function f: X → ℝ that is expensive to evaluate.
The method models f using a Gaussian Process (GP) as a surrogate model, which provides a normal distribution with mean μ(x) and variance σ²(x) for each point x ∈ X.
The selection of the next evaluation point x_{n+1} is done by maximizing an acquisition function α(x; D_n), which is based on the previous data D_n = {(x_i, y_i)}.
For example, the Expected Improvement (EI) is defined as:
EI(x) = 𝔼[max(0, f_{best} - f(x))]
where f_{best} is the best observed value so far.
By iteratively updating the GP model and selectively choosing x_{n+1}, evaluations are used efficiently to find the optimum of f.
Definition:
Bayesian optimization is a probabilistic optimization method that efficiently optimizes expensive black-box functions globally using a surrogate model and an acquisition function.
Source:
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & de Freitas, N. (2016). Taking the Human Out of the Loop: A Review of Bayesian Optimization. Proceedings of the IEEE, 104(1), 148-175.