---
id: TECH-MM-001
title: "Multimodal Foundation Models"
alternative_names:
  - "Vision-Language Models"
  - "Large Multimodal Models"
  - "LMMs"
  - "Multimodal LLMs"
category: "Technical → Foundation Models"
volume: III
chapter: 19
entry_type: "Technical Article"
level: 3
status: "Draft"
last_reviewed: 2026-10-15
version: "0.1.0"
prerequisites:
  - "TECH-LLM-001"
  - "TECH-DIFF-001"
  - "TECH-ARCH-001"
related_entries:
  - "TECH-TOK-001"
  - "TECH-RAG-001"
  - "TECH-ROBOT-001"
  - "CAP-GENAI-001"
  - "CAP-BENCH-001"
  - "AGI-WM-001"
---

# Multimodal Foundation Models

## 1. Concise Definition
A **multimodal foundation model** is a large model, pretrained on broad data, that processes or generates more than one kind of data (modality), such as text, images, audio and video, within a single system. The term covers models that understand images and answer in text, models that generate images from text, and models that handle several input and output modalities.

## 2. Overview & Significance
Language models learn from text alone, which limits their access to visual and auditory information. Multimodal models extend the same large-scale pretraining approach to other signals and, in most current designs, reuse the Transformer architecture (see [TECH-ARCH-001]) by converting each modality to a sequence of vectors. They are the basis for image captioning, visual question answering, document and chart understanding, speech recognition, and text-to-image or text-to-video generation.

Three design questions organise the field: how each modality is turned into tokens or embeddings (see [TECH-TOK-001]); how those representations are connected to a language model or to each other; and whether the model understands, generates, or both. Generation of images and video has largely followed a separate line of diffusion and flow-based models (see [TECH-DIFF-001]), so "multimodal" currently covers two partly distinct families (Section 5).

This article reports only what developers have disclosed for closed systems. Architecture, training data, and modality coverage of such systems beyond developer statements are **undisclosed**.

## 3. Historical Development
Vision models and language models developed separately until the early 2020s. The Vision Transformer (Dosovitskiy et al., 2021) showed that a pure Transformer applied to sequences of image patches can perform very well on image classification when pre-trained on large amounts of data, providing a common architecture for both modalities.

Contrastive language-image pre-training changed the supervision signal. CLIP (Radford et al., 2021) trained on 400 million image-text pairs from the internet with the task of predicting which caption goes with which image, and used natural language to reference visual concepts for zero-shot transfer, benchmarked on over 30 computer vision datasets. SigLIP (Zhai et al., 2023) replaced the softmax-normalised contrastive loss with a pairwise sigmoid loss that does not need a global view of pairwise similarities.

The connection of vision to pretrained language models then followed. Frozen (Tsimpoukelli et al., 2021) trained a vision encoder to produce a sequence of embeddings acting as a prefix to a frozen language model, enabling few-shot multimodal learning. Flamingo (Alayrac et al., 2022) proposed architectural changes to bridge pretrained vision-only and language-only models and to handle arbitrarily interleaved images and text, trained on large-scale multimodal web data. BLIP-2 (Li et al., 2023) bridged frozen image encoders and frozen language models with a lightweight Querying Transformer and reported outperforming Flamingo80B on zero-shot VQAv2 by 8.7% with 54 times fewer trainable parameters. LLaVA (Liu et al., 2023) connected a vision encoder to a language model and trained it on instruction-following data generated by language-only GPT-4.

In generation, DALL-E (Ramesh et al., 2021) modelled text and image tokens as a single stream with an autoregressive Transformer; latent diffusion models (Rombach et al., 2022) ran diffusion in the latent space of pretrained autoencoders with cross-attention for conditioning; Diffusion Transformers (Peebles & Xie, 2023) replaced the U-Net backbone with a Transformer; and Esser et al. (2024) combined rectified flow with a Transformer using separate weights for the two modalities. Speech recognition was scaled by Whisper (Radford et al., 2023), trained on 680,000 hours of multilingual and multitask supervision from the internet.

Closed frontier developers describe multimodal systems. The GPT-4 report describes a model that "can accept image and text inputs and produce text outputs" (OpenAI et al., 2023), and the Gemini report describes a family handling image, audio, video and text understanding (Gemini Team, 2023). Chameleon (Chameleon Team, 2024) reported an early-fusion, token-based model that understands and generates images and text in arbitrary sequence.

## 4. Technical / Conceptual Explanation

### 4.1 Encoding modalities
Each non-text modality is mapped to a sequence of vectors. For images, a Vision Transformer splits an image into patches $p_1,\dots,p_m$ and projects each to $x_j = W_p\, \mathrm{flatten}(p_j) + e_j$, with positional embedding $e_j$ (Dosovitskiy et al., 2021). Audio is typically converted to a spectrogram and processed analogously (Radford et al., 2023). The result is a sequence of continuous embeddings, or, in token-based approaches, a sequence of discrete codes from a learned image tokenizer.

### 4.2 Contrastive alignment
CLIP-style training embeds an image and a caption into a shared space and trains with a symmetric cross-entropy over a batch of $B$ pairs. With normalised image embeddings $u_i$, text embeddings $v_i$ and a temperature $\tau$,
$$ \mathcal{L}_{i \to t} = -\frac{1}{B}\sum_{i=1}^{B} \log \frac{\exp(u_i^\top v_i / \tau)}{\sum_{j=1}^{B}\exp(u_i^\top v_j / \tau)}, $$
and the text-to-image loss $\mathcal{L}_{t \to i}$ is defined symmetrically, the total being their average. SigLIP replaces the softmax over the batch with an independent binary (sigmoid) classification for each image-text pair (Zhai et al., 2023). Matching pairs are positives and all other pairings in the batch are negatives.

### 4.3 Connecting vision to a language model
Three connection patterns dominate.
1. **Prefix or projection**: the vision encoder output is mapped by a linear layer or small network into the language model's embedding space and inserted as extra input tokens (Frozen; LLaVA).
2. **Query-based bottleneck**: a small Transformer with learned queries (the Q-Former in BLIP-2) compresses image features to a fixed number of vectors for a frozen language model.
3. **Cross-attention layers**: new layers inside a frozen language model attend to visual features, with support for interleaved image and text input (Flamingo).

Training commonly proceeds in stages: align the connector on image-caption data, then instruction-tune on multimodal dialogue. LLaVA used instruction data generated by GPT-4 from text descriptions and reported 85.1% relative score against GPT-4 on a synthetic instruction-following dataset (Liu et al., 2023), a figure whose meaning depends on that synthetic benchmark and GPT-4 as a reference.

### 4.4 Early-fusion token models
In early fusion, all modalities are converted to discrete tokens from the start and a single Transformer models the interleaved sequence with a next-token objective. Chameleon is a published example for text and images, with an alignment recipe and training approach tailored to this setting; its authors report competitive text performance and human-judged long-form mixed-modal generation results compared with several named models (Chameleon Team, 2024).

### 4.5 Generative image and video models
Latent diffusion trains a denoiser in an autoencoder's latent space, using cross-attention to condition on text (Rombach et al., 2022). A diffusion Transformer treats latent patches as tokens (Peebles & Xie, 2023). In rectified-flow models a straight-line path connects data to noise and a network predicts the velocity along it; Esser et al. (2024) report that, in a large-scale study, their noise-sampling improvements outperform established diffusion formulations for high-resolution text-to-image synthesis, and that their architecture with separate weights per modality improves text comprehension and typography (see [TECH-DIFF-001]).

## 5. Types, Components & Classifications

| Family | Modalities | Representative designs | Output |
| :--- | :--- | :--- | :--- |
| Contrastive dual encoders | Image–text | CLIP; SigLIP | Embeddings, zero-shot classification, retrieval |
| Vision-language models (understanding) | Image/video in, text out | Frozen; Flamingo; BLIP-2; LLaVA | Text |
| Native multimodal LLMs | Text, images, audio, video | Gemini, as described by its developer; GPT-4 (image and text in, text out) | Text (and in some products other modalities; per developer documentation) |
| Early-fusion token models | Interleaved text and image | Chameleon | Text and images |
| Diffusion / flow generators | Text to image or video | Latent diffusion; DiT; rectified-flow Transformers | Image or video |
| Speech models | Audio to text | Whisper | Transcripts |

For closed models the boundary between families is a matter of undisclosed implementation: whether an image generator in a given product is the same network as the language model, or a separate model called by it, is **undisclosed** unless the developer states it.

## 6. Applications & Concrete Examples
- **Image and document understanding**, including captioning, visual question answering and chart reading, via VLMs such as LLaVA and BLIP-2.
- **Retrieval across modalities**, where CLIP-style embeddings allow text queries against image collections (see [TECH-RAG-001]).
- **Speech recognition and transcription**, with Whisper as an openly released example (Radford et al., 2023).
- **Image generation**, deployed through latent diffusion and related systems (Rombach et al., 2022; Esser et al., 2024).
- **Robotics and agents**, where vision-language models serve as perception and planning components (see [TECH-ROBOT-001]).
- **Evaluation.** MMMU (Yue et al., 2024) contains 11.5K multimodal questions from college exams, quizzes and textbooks across 30 subjects, and reported at publication that GPT-4V and Gemini Ultra achieved 56% and 59%. Benchmark scores change quickly and are tracked in `MODEL-001`; this figure is a dated result from the benchmark paper.

## 7. Advantages, Limitations & Unresolved Questions

### Advantages
Transfer from large-scale pretraining to many perception tasks without task-specific training (CLIP); reuse of strong pretrained language models, reducing trainable parameters (BLIP-2); a common interface through language.

### Limitations
- **Object hallucination.** Li et al. (2023) report that evaluated large vision-language models mostly suffer from severe object hallucination, generating objects not present in the image, with objects that are frequent in visual instructions or co-occur with image objects more prone to be hallucinated, and that existing evaluation methods can be affected by instructions and generation style (see [CAP-LIMITS-001]).
- **Weak visual representations.** Tong et al. (2024) identify "CLIP-blind pairs", images that CLIP perceives as similar despite clear visual differences, build the MMVP benchmark from them, and report that state-of-the-art systems including GPT-4V struggle with straightforward questions across nine basic visual patterns, sometimes giving hallucinated explanations; they find a correlation between patterns that challenge CLIP and model failures.
- **Benchmark difficulty at expert level.** The MMMU authors report that the strongest models tested at publication fell well short of expert-level accuracy.
- **Data and rights issues.** Web-scale image-text training data raises provenance and copyright questions (see [TECH-DATA-001]).
- **Cost.** Images, audio and video produce many tokens per input, increasing compute and latency.

### Unresolved questions
1. Whether early-fusion, single-model designs outperform modular systems at the same compute; the evidence comes from few comparisons.
2. Whether multimodal training improves language-only reasoning or grounds meaning in the sense debated in [AGI-WM-001]; this encyclopedia treats the question as open.
3. How to evaluate perception without contamination from benchmark leakage or language priors.

## 8. Current State of Knowledge
As of 2026-10-15, contrastive image-text encoders, projection or query-based connectors to language models, and diffusion or flow-based generators are well documented in the open literature. Claims that a particular closed model is "natively multimodal" or generates images within the same network should be sourced to the developer; beyond the statements cited here (GPT-4 image and text inputs with text outputs; Gemini's image, audio, video and text understanding), architectures, data and training procedures are **undisclosed**. Reported benchmark scores for named models are dated and held in `MODEL-001`. Evaluation of perception remains an area where independent studies find systematic failures (Tong et al., 2024; Li et al., 2023).

## 9. Future Trajectory & Strategic Outlook
Observed trends: unified models that both understand and generate several modalities (Chameleon), larger use of video and audio, and better evaluation of perceptual faults. Whether single-network multimodal models will displace composed systems is an open research question; no forecast record is registered in this article.

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[TECH-LLM-001] Large Language Models](./TECH-LLM-001_large_language_models.md): The language backbone.
  - [[TECH-ARCH-001] The Transformer Architecture](./TECH-ARCH-001_transformer_architecture.md): Vision Transformer and attention.
  - [[TECH-DIFF-001] Diffusion Models](./TECH-DIFF-001_diffusion_models.md): Generative image and video modelling.
- **Related Methods & Technologies**:
  - [[TECH-TOK-001] Tokenization and Embeddings](./TECH-TOK-001_tokenization_and_embeddings.md): Modality tokenization and shared embedding spaces.
  - [[TECH-RAG-001] Retrieval-Augmented Generation](./TECH-RAG-001_retrieval_augmented_generation.md): Multimodal retrieval.
  - [[TECH-ROBOT-001] Robotics and Embodied AI](./TECH-ROBOT-001_robotics_and_embodied_ai.md): Vision-language models in embodied settings.
  - [[CAP-GENAI-001] Generative Artificial Intelligence](../vol04_capabilities/CAP-GENAI-001_generative_ai.md): Generative capabilities.
- **Advanced Implications & Future Trajectories**:
  - [[CAP-BENCH-001] AI Evaluation](../vol04_capabilities/CAP-BENCH-001_ai_evaluation.md): Multimodal benchmarks.
  - [[AGI-WM-001] World Models](../vol05_agi/AGI-WM-001_world_models.md): Grounding and world modelling.

## 11. References & Foundational Literature
1. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2021). *An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale*. ICLR 2021. arXiv:2010.11929. [Vision Transformer on image patches].
2. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). *Learning Transferable Visual Models From Natural Language Supervision*. ICML 2021. arXiv:2103.00020. [CLIP; 400M image-text pairs; zero-shot transfer on 30+ datasets].
3. Zhai, X., Mustafa, B., Kolesnikov, A., & Beyer, L. (2023). *Sigmoid Loss for Language Image Pre-Training*. ICCV 2023. arXiv:2303.15343. [Pairwise sigmoid loss for language-image pre-training].
4. Tsimpoukelli, M., Menick, J., Cabi, S., Eslami, S. M. A., Vinyals, O., & Hill, F. (2021). *Multimodal Few-Shot Learning with Frozen Language Models*. NeurIPS 2021. arXiv:2106.13884. [Image embeddings as a prefix to a frozen language model].
5. Alayrac, J.-B., Donahue, J., Luc, P., et al. (2022). *Flamingo: a Visual Language Model for Few-Shot Learning*. NeurIPS 2022. arXiv:2204.14198. [Interleaved image-text few-shot visual language model].
6. Li, J., Li, D., Savarese, S., & Hoi, S. (2023). *BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models*. ICML 2023. arXiv:2301.12597. [Querying Transformer bridging frozen encoders and LLMs].
7. Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023). *Visual Instruction Tuning*. NeurIPS 2023. arXiv:2304.08485. [LLaVA and GPT-4-generated multimodal instruction data].
8. Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021). *Zero-Shot Text-to-Image Generation*. ICML 2021. arXiv:2102.12092. [Autoregressive Transformer over text and image tokens].
9. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). *High-Resolution Image Synthesis with Latent Diffusion Models*. CVPR 2022. arXiv:2112.10752. [Latent-space diffusion with cross-attention conditioning].
10. Peebles, W., & Xie, S. (2023). *Scalable Diffusion Models with Transformers*. ICCV 2023. arXiv:2212.09748. [Diffusion Transformers].
11. Esser, P., Kulal, S., Blattmann, A., et al. (2024). *Scaling Rectified Flow Transformers for High-Resolution Image Synthesis*. ICML 2024. arXiv:2403.03206. [Rectified flow with a Transformer using separate weights per modality].
12. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023). *Robust Speech Recognition via Large-Scale Weak Supervision*. ICML 2023. arXiv:2212.04356. [Whisper; 680,000 hours of supervision].
13. Chameleon Team. (2024). *Chameleon: Mixed-Modal Early-Fusion Foundation Models*. arXiv:2405.09818. [Early-fusion token-based image and text model].
14. OpenAI, Achiam, J., Adler, S., Agarwal, S., et al. (2023). *GPT-4 Technical Report*. arXiv:2303.08774. [Image and text inputs, text outputs; architecture and training details withheld].
15. Gemini Team, Anil, R., Borgeaud, S., et al. (2023). *Gemini: A Family of Highly Capable Multimodal Models*. arXiv:2312.11805. [Developer description of a multimodal model family covering image, audio, video and text understanding].
16. Yue, X., Ni, Y., Zhang, K., et al. (2024). *MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI*. CVPR 2024. arXiv:2311.16502. [11.5K-question expert-level multimodal benchmark].
17. Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., & Wen, J.-R. (2023). *Evaluating Object Hallucination in Large Vision-Language Models*. Proceedings of EMNLP 2023. arXiv:2305.10355. [Systematic study of object hallucination in LVLMs].
18. Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., & Xie, S. (2024). *Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs*. CVPR 2024. arXiv:2401.06209. [CLIP-blind pairs and the MMVP benchmark].

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Transformers can process image patches as sequences | Established | Dosovitskiy et al. (2021) | High |
| Contrastive image-text pretraining enables zero-shot transfer | Well Supported | Radford et al. (2021); Zhai et al. (2023) | High |
| Frozen-LLM connectors reach strong results with few trainable parameters | Well Supported | Li et al. (2023); Alayrac et al. (2022); Tsimpoukelli et al. (2021) | Moderate-High |
| Latent diffusion and rectified-flow Transformers for image synthesis | Well Supported | Rombach et al. (2022); Peebles & Xie (2023); Esser et al. (2024) | High |
| Early-fusion token models are competitive | Preliminary | Chameleon Team (2024) | Moderate (single-lab report) |
| LVLMs show object hallucination | Well Supported | Li et al. (2023); Tong et al. (2024) | Moderate-High |
| Early-fusion single-model designs outperform modular designs | Contested / open | n/a | Not established |
| Architecture of closed multimodal models beyond developer statements | Undisclosed | OpenAI et al. (2023); Gemini Team (2023) | Not inferred |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.