Multimodal Foundation Models

A multimodal foundation model is a large model, pretrained on broad data, that processes or generates more than one kind of data (modality), such as text, images, audio and video, within a single system. The term covers models that understand images and answer in text, models that generate images from text, and models that handle several input and output modalities.

Language models learn from text alone, which limits their access to visual and auditory information. Multimodal models extend the same large-scale pretraining approach to other signals and, in most current designs, reuse the Transformer architecture (see [TECH-ARCH-001]) by converting each modality to a sequence of vectors. They are the basis for image captioning, visual question answering, document and chart understanding, speech recognition, and text-to-image or text-to-video generation.

Three design questions organise the field: how each modality is turned into tokens or embeddings (see [TECH-TOK-001]); how those representations are connected to a language model or to each other; and whether the model understands, generates, or both. Generation of images and video has largely followed a separate line of diffusion and flow-based models (see [TECH-DIFF-001]), so "multimodal" currently covers two partly distinct families (Section 5).

This article reports only what developers have disclosed for closed systems. Architecture, training data, and modality coverage of such systems beyond developer statements are undisclosed.

Vision and language modelling were largely separate research lines, with earlier captioning and visual question answering systems pairing image encoders with language models; pre-2020 vision-language systems are not surveyed here. The Vision Transformer (Dosovitskiy et al., 2021) showed that a pure Transformer applied to sequences of image patches can perform very well on image classification when pre-trained on large amounts of data, providing a common architecture for both modalities.

Contrastive language-image pre-training changed the supervision signal. CLIP (Radford et al., 2021) trained on 400 million image-text pairs from the internet with the task of predicting which caption goes with which image, and used natural language to reference visual concepts for zero-shot transfer, benchmarked on over 30 computer vision datasets. SigLIP (Zhai et al., 2023) replaced the softmax-normalised contrastive loss with a pairwise sigmoid loss that does not need a global view of pairwise similarities.

The connection of vision to pretrained language models then followed. Frozen (Tsimpoukelli et al., 2021) trained a vision encoder to produce a sequence of embeddings acting as a prefix to a frozen language model, enabling few-shot multimodal learning. Flamingo (Alayrac et al., 2022) proposed architectural changes to bridge pretrained vision-only and language-only models and to handle arbitrarily interleaved images and text, trained on large-scale multimodal web data. BLIP-2 (Li et al., 2023) bridged frozen image encoders and frozen language models with a lightweight Querying Transformer and reported outperforming Flamingo80B on zero-shot VQAv2 by 8.7% with 54 times fewer trainable parameters. LLaVA (Liu et al., 2023) connected a vision encoder to a language model and trained it on instruction-following data generated by language-only GPT-4.

In generation, DALL-E (Ramesh et al., 2021) modelled text and image tokens as a single stream with an autoregressive Transformer; latent diffusion models (Rombach et al., 2022) ran diffusion in the latent space of pretrained autoencoders with cross-attention for conditioning; Diffusion Transformers (Peebles & Xie, 2023) replaced the U-Net backbone with a Transformer; and Esser et al. (2024) combined rectified flow with a Transformer using separate weights for the two modalities. Speech recognition was scaled by Whisper (Radford et al., 2023), trained on 680,000 hours of multilingual and multitask supervision from the internet.

Closed frontier developers describe multimodal systems. The GPT-4 report describes a model that "can accept image and text inputs and produce text outputs" (OpenAI et al., 2023), and the Gemini report describes a family handling image, audio, video and text understanding (Gemini Team, 2023). Chameleon (Chameleon Team, 2024) reported an early-fusion, token-based model that understands and generates images and text in arbitrary sequence.

#4.1 Encoding modalities

Each non-text modality is mapped to a sequence of vectors. For images, a Vision Transformer splits an image into patches $p_1,\dots,p_m$ and projects each to $x_j = W_p\, \mathrm{flatten}(p_j) + e_j$, with positional embedding $e_j$ (Dosovitskiy et al., 2021). Audio is typically converted to a spectrogram and processed analogously (Radford et al., 2023). The result is a sequence of continuous embeddings, or, in token-based approaches, a sequence of discrete codes from a learned image tokenizer.

#4.2 Contrastive alignment

CLIP-style training embeds an image and a caption into a shared space and trains with a symmetric cross-entropy over a batch of $B$ pairs. With normalised image embeddings $u_i$, text embeddings $v_i$ and a temperature $\tau$,

and the text-to-image loss $\mathcal{L}_{t \to i}$ is defined symmetrically, the total being their average. SigLIP replaces the softmax over the batch with an independent binary (sigmoid) classification for each image-text pair (Zhai et al., 2023). Matching pairs are positives and all other pairings in the batch are negatives.

#4.3 Connecting vision to a language model

Three connection patterns dominate.

1. Prefix or projection: the vision encoder output is mapped by a linear layer or small network into the language model's embedding space and inserted as extra input tokens (Frozen; LLaVA).
2. Query-based bottleneck: a small Transformer with learned queries (the Q-Former in BLIP-2) compresses image features to a fixed number of vectors for a frozen language model.
3. Cross-attention layers: new layers inside a frozen language model attend to visual features, with support for interleaved image and text input (Flamingo).

Training commonly proceeds in stages: align the connector on image-caption data, then instruction-tune on multimodal dialogue. LLaVA used instruction data generated by GPT-4 from text descriptions and reported 85.1% relative score against GPT-4 on a synthetic instruction-following dataset (Liu et al., 2023), a figure whose meaning depends on that synthetic benchmark and GPT-4 as a reference.

#4.4 Early-fusion token models

In early fusion, all modalities are converted to discrete tokens from the start and a single Transformer models the interleaved sequence with a next-token objective. Chameleon is a published example for text and images, with an alignment recipe and training approach tailored to this setting; its authors report competitive text performance and human-judged long-form mixed-modal generation results compared with several named models (Chameleon Team, 2024).

#4.5 Generative image and video models

Latent diffusion trains a denoiser in an autoencoder's latent space, using cross-attention to condition on text (Rombach et al., 2022). A diffusion Transformer treats latent patches as tokens (Peebles & Xie, 2023). In rectified-flow models a straight-line path connects data to noise and a network predicts the velocity along it; Esser et al. (2024) report that, in a large-scale study, their noise-sampling improvements outperform established diffusion formulations for high-resolution text-to-image synthesis, and that their architecture with separate weights per modality improves text comprehension and typography (see [TECH-DIFF-001]).

For closed models the boundary between families is a matter of undisclosed implementation: whether an image generator in a given product is the same network as the language model, or a separate model called by it, is undisclosed unless the developer states it.

• Image and document understanding, including captioning, visual question answering and chart reading, via VLMs such as LLaVA and BLIP-2.
• Retrieval across modalities, where CLIP-style embeddings allow text queries against image collections (see [TECH-RAG-001]).
• Speech recognition and transcription, with Whisper as an openly released example (Radford et al., 2023).
• Image generation, deployed through latent diffusion and related systems (Rombach et al., 2022; Esser et al., 2024).
• Robotics and agents, where vision-language models serve as perception and planning components (see [TECH-ROBOT-001]).
• Evaluation. MMMU (Yue et al., 2024) contains 11.5K multimodal questions from college exams, quizzes and textbooks across 30 subjects, and reported at publication that GPT-4V and Gemini Ultra achieved 56% and 59%. Benchmark scores change quickly and are tracked in MODEL-001; this figure is a dated result from the benchmark paper.

#Advantages

Transfer from large-scale pretraining to many perception tasks without task-specific training (CLIP); reuse of strong pretrained language models, reducing trainable parameters (BLIP-2); a common interface through language.

#Limitations

• Object hallucination. Li et al. (2023) report that evaluated large vision-language models mostly suffer from severe object hallucination, generating objects not present in the image, with objects that are frequent in visual instructions or co-occur with image objects more prone to be hallucinated, and that existing evaluation methods can be affected by instructions and generation style (see [CAP-LIMITS-001]).
• Weak visual representations. Tong et al. (2024) identify "CLIP-blind pairs", images that CLIP perceives as similar despite clear visual differences, build the MMVP benchmark from them, and report that state-of-the-art systems including GPT-4V struggle with straightforward questions across nine basic visual patterns, sometimes giving hallucinated explanations; they find a correlation between patterns that challenge CLIP and model failures.
• Benchmark difficulty at expert level. The MMMU authors report that GPT-4V and Gemini Ultra reached 56% and 59% at publication and describe significant room for improvement; the benchmark paper does not state a human-expert score in its abstract.
• Data and rights issues. Web-scale image-text training data raises provenance and copyright questions (see [TECH-DATA-001]).
• Cost. Images, audio and video produce many tokens per input, increasing compute and latency.

#Unresolved questions

1. Whether early-fusion, single-model designs outperform modular systems at the same compute; the evidence comes from few comparisons.
2. Whether multimodal training improves language-only reasoning or grounds meaning in the sense debated in [AGI-WM-001]; this encyclopedia treats the question as open.
3. How to evaluate perception without contamination from benchmark leakage or language priors.

As of 2026-10-19, contrastive image-text encoders, projection or query-based connectors to language models, and diffusion or flow-based generators are well documented in the open literature. Claims that a particular closed model is "natively multimodal" or generates images within the same network should be sourced to the developer; beyond the statements cited here (GPT-4 image and text inputs with text outputs; Gemini's image, audio, video and text understanding), architectures, data and training procedures are undisclosed. Reported benchmark scores for named models are dated and held in MODEL-001. This article does not survey multimodal systems released after the cited 2023-2024 reports, so developer-reported capabilities of 2025-2026 multimodal models are not covered here. Evaluation of perception remains an area where independent studies find systematic failures (Tong et al., 2024; Li et al., 2023).

Observed trends: unified models that both understand and generate several modalities (Chameleon), larger use of video and audio, and better evaluation of perceptual faults. Whether single-network multimodal models will displace composed systems is an open research question; no forecast record is registered in this article.

• Foundational Concepts:
  • TECH-LLM-001 Large Language Models: The language backbone.
  • TECH-ARCH-001 The Transformer Architecture: Vision Transformer and attention.
  • TECH-DIFF-001 Diffusion Models: Generative image and video modelling.
• Related Methods & Technologies:
  • TECH-TOK-001 Tokenization and Embeddings: Modality tokenization and shared embedding spaces.
  • TECH-RAG-001 Retrieval-Augmented Generation: Multimodal retrieval.
  • TECH-ROBOT-001 Robotics and Embodied AI: Vision-language models in embodied settings.
  • CAP-GENAI-001 Generative Artificial Intelligence: Generative capabilities.
• Advanced Implications & Future Trajectories:
  • CAP-BENCH-001 AI Evaluation: Multimodal benchmarks.
  • AGI-WM-001 World Models: Grounding and world modelling.