Neural Architecture Search (NAS) refers to the automated process of optimizing neural network architectures within a defined search space using algorithmic search methods.
Formally, NAS can be described as a bi-level optimization problem:
\[ \min_{\alpha \in \mathcal{A}} \mathcal{L}_{val}(w^{*}(\alpha), \alpha) \quad \text{with} \quad w^{*}(\alpha) = \arg\min_{w} \mathcal{L}_{train}(w, \alpha) \]
Here, \( \alpha \) denotes the architecture parameters in the search space \( \mathcal{A} \) and \( w \) the weights of the network. The goal is to find the architecture \( \alpha \) that minimizes the validation loss \( \mathcal{L}_{val} \), where the weights \( w^{*} \) are optimally trained on the training data \( \mathcal{L}_{train} \).
NAS methods include reinforcement learning-based search (e.g., Zoph & Le, 2017), evolutionary algorithms (Real et al., 2019), and differentiable NAS such as DARTS (Liu et al., 2019), which make the search process differentiable.
These approaches have led to powerful architectures applied in areas such as image classification, speech processing, and more.
Definition:
“Neural Architecture Search is an automated optimization process aimed at finding the optimal structure of a neural network within a search space to maximize the model’s performance and efficiency.”
Source:
Zoph, B., & Le, Q. V. (2017). Neural Architecture Search with Reinforcement Learning. arXiv preprint arXiv:1611.01578.