Hyperparameter Optimization Explained for Experts

Hyperparameter optimization (HPO) is the process of searching for a set of hyperparameters \( \lambda \in \Lambda \) that minimize the performance of a learning algorithm \( A_{\lambda} \) on a validation function \( L_{val} \):

\[ \lambda^* = \arg\min_{\lambda \in \Lambda} L_{val}(A_{\lambda}(D_{train})) \]

Here, \( D_{train} \) denotes the training dataset and \( \Lambda \) the hyperparameter space, which can include discrete and continuous parameters.

Common optimization methods include:

• Grid Search: systematic but computationally intensive exhaustive search.
• Random Search: sampling-based search, often more efficient in high dimensionality.
• Bayesian Optimization: model-based search that explores the search space using probabilistic surrogate models (e.g., Gaussian Processes).
• Gradient-based methods and meta-learning: newer approaches for adaptive optimization.

Hyperparameter optimization is essential for improving the generalization and efficiency of complex models, especially in deep learning.

Definition:
“Hyperparameter optimization is the process of systematically searching for optimal settings of parameters that are not learned during training, in order to maximize the performance of a learning algorithm on validation data.”

Source:
Feurer, M., & Hutter, F. (2019). Hyperparameter Optimization. In Automated Machine Learning (pp. 3-33). Springer.