Multimodal AI refers to a paradigm of machine learning in which models simultaneously process multiple data modalities \( M = \{m_1, m_2, ..., m_k\} \) and map them into a common representation space \( \mathcal{Z} \).
Formally, for each modality \( m_i \), encoding functions \( f_{m_i}: X_{m_i} ightarrow \mathcal{Z}_{m_i} \) are trained, which transform the input data into latent vectors. Subsequently, a fusion or alignment of these representations takes place, e.g., via cross-modality attention or joint embedding.
The goal is to learn a model \( F: \prod_{i=1}^k X_{m_i} ightarrow Y \) that makes robust and semantically rich predictions from the combined modalities.
Multimodal AI addresses challenges such as heterogeneous data characteristics, missing modalities, and different scales of data.
Definition:
“Multimodal AI encompasses methods and models that simultaneously process and integrate multiple distinct data modalities to enable a more comprehensive understanding and better predictions.”
Source:
Baltrusaitis, T., Ahuja, C., & Morency, L. P. (2019). Multimodal Machine Learning: A Survey and Taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2), 423-443.