Neural style transfer is a method for synthesizing an image \( \hat{I} \) that combines the content of a source image \( I_c \) and the style of a target image \( I_s \), based on deep convolutional neural networks (CNNs).
Formally, the optimization is performed by minimizing the total loss function
\[ \mathcal{L}_{total}(\hat{I}) = \alpha \mathcal{L}_{content}(\hat{I}, I_c) + eta \mathcal{L}_{style}(\hat{I}, I_s) \]
where \( \alpha \) and \( eta \) are weighting factors.
The content loss is often defined as the squared error of the feature maps of a specific CNN layer:
\[ \mathcal{L}_{content}(\hat{I}, I_c) = rac{1}{2} \sum_{i,j} (F_{ij}^l(\hat{I}) - F_{ij}^l(I_c))^2 \]
where \( F^l \) are the activations in layer \( l \).
The style loss is based on the difference of the Gram matrices of the feature maps across multiple layers, capturing texture and style information:
\[ \mathcal{L}_{style}(\hat{I}, I_s) = \sum_{l} w_l \| G^l(\hat{I}) - G^l(I_s) \|_F^2 \]
with the Gram matrix \( G^l \) and weights \( w_l \).
The optimization is usually done with gradient-based methods to generate an image that unites content and style.
Definition:
“Neural style transfer is a method that, using deep neural networks, combines the content of one image with the style of another image by extracting separate representations for content and style in the network layers and merging them through optimization.”
Source:
Gatys, L. A., Ecker, A. S., & Bethge, M. (2016). Image Style Transfer Using Convolutional Neural Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).