Neural Activation Function Explained for Experts

Neural activation functions are mathematical functions applied to the weighted sum of the inputs of an artificial neuron to determine its output signal.

Formally, the activation a of a neuron is calculated as a = f(z), where z = Σ w_i x_i + b is the weighted sum of the inputs plus bias and f represents the activation function.

The activation function introduces nonlinearity into the model, which is necessary to map complex, nonlinear relationships in data.

Common functions include:

• Sigmoid: f(z) = 1 / (1 + e^{-z}), range (0,1), differentiable, but with vanishing gradient problem.
• Hyperbolic Tangent (tanh): f(z) = (e^{z} - e^{-z}) / (e^{z} + e^{-z}), range (-1,1), centered around 0.
• ReLU (Rectified Linear Unit): f(z) = max(0,z), sparse and effective, but not differentiable at 0.

The choice of activation function influences convergence speed, model accuracy, and training stability.

Definition:
“A neural activation function is a nonlinear function applied to the weighted sum of the inputs of an artificial neuron to determine its output signal and thus enable the network’s ability to model complex relationships.”

Source:
Nwankpa, C., Ijomah, W., Gachagan, A., & Marshall, S. (2018). Activation Functions: Comparison of Trends in Practice and Research for Deep Learning. arXiv preprint arXiv:1811.03378.