Neural differential equations are a class of models that combine differential equations with neural networks to describe continuous dynamic systems.
Formally, a differential equation of the form
\[ \frac{d\mathbf{z}(t)}{dt} = f(\mathbf{z}(t), t, \theta) \]
is defined, where \( f \) is a neural network with parameters \( \theta \) that describes the time derivative of the state vector \( \mathbf{z}(t) \).
The solution of the equation is obtained through numerical integration, e.g., using Runge-Kutta methods, which allows neural networks to be interpreted as continuous depth models.
These models are particularly suitable for problems with irregular time stamps or physics-based simulations and offer advantages over discrete architectures.
Definition:
“Neural differential equations are models in which the temporal evolution of a system is described by a differential equation whose right-hand side is parameterized by a neural network.”
Source:
Chen, R. T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D. (2018). Neural Ordinary Differential Equations. Advances in Neural Information Processing Systems, 31.