---
id: TECH-MOE-001
title: "Mixture-of-Experts Models"
alternative_names:
  - "MoE"
  - "Sparse Mixture of Experts"
  - "Sparsely-Gated Mixture-of-Experts"
  - "Conditional Computation Models"
category: "Technical → Architectures"
volume: III
chapter: 18
entry_type: "Technical Article"
level: 3
status: "Draft"
last_reviewed: 2026-10-15
version: "0.1.0"
prerequisites:
  - "TECH-ARCH-001"
  - "TECH-DL-001"
related_entries:
  - "TECH-LLM-001"
  - "TECH-SCALE-001"
  - "TECH-HW-001"
  - "TECH-INFER-001"
  - "TECH-DIST-001"
  - "CAP-OPEN-001"
---

# Mixture-of-Experts Models

## 1. Concise Definition
A **Mixture-of-Experts (MoE)** model is a neural network in which part of the computation is split among several parallel sub-networks ("experts"), and a learned routing function (the "gate" or "router") selects which experts process each input. In the sparse variant used in modern large language models, only a small subset of experts is active per token, so the total parameter count can grow much faster than the compute spent per token.

## 2. Overview & Significance
In a dense network every parameter participates in every forward pass, so parameters and per-token compute rise together. Sparse MoE decouples the two: capacity (total parameters) and cost (active parameters) become separate quantities. This matters because empirical scaling studies relate model quality to both parameters and training compute (see [TECH-SCALE-001]), and MoE offers a way to add parameters without a proportional increase in floating-point operations per token.

The decoupling has costs. All experts must be stored in accelerator memory even though few are used per token; routing introduces load-balancing and stability problems; and distributed training requires heavy all-to-all communication. MoE is therefore best understood as a trade of memory and engineering complexity for lower compute per token, not as a free increase in capability.

Among public frontier-model disclosures, several developers state that their models use MoE layers. Others do not describe their architecture. This article restricts claims about specific systems to what the developers have published and marks the remainder as undisclosed (Section 8).

## 3. Historical Development
The idea predates deep learning. Jacobs, Jordan, Nowlan and Hinton (1991) proposed an "adaptive mixture of local experts": several expert networks and a gating network trained jointly, so that experts specialise on different regions of the input space. In that formulation every expert was evaluated and outputs were combined by the gate's soft weights, so there was no compute saving.

Shazeer et al. (2017) introduced the sparsely-gated MoE layer, with up to thousands of feed-forward experts and a trainable gate that selects a sparse combination per example; they applied it between stacked LSTM layers, with up to 137 billion parameters in the MoE layer, for language modelling and machine translation. GShard (Lepikhin et al., 2020) moved the idea into Transformers and added compiler-level automatic sharding, scaling a multilingual translation model beyond 600 billion parameters. Switch Transformers (Fedus, Zoph & Shazeer, 2022; arXiv 2021) simplified routing to a single expert per token and reported up to 7x pre-training speed-ups over dense T5 baselines at matched compute. ST-MoE (Zoph et al., 2022) concentrated on training instability and transfer to downstream tasks. GLaM (Du et al., 2022) reported a 1.2-trillion-parameter sparse model whose training used about one third of the energy of GPT-3 according to its authors.

From 2024, open-weight MoE models made the architecture inspectable: Mixtral 8x7B (Jiang et al., 2024), DeepSeekMoE (Dai et al., 2024) and DeepSeek-V3 (DeepSeek-AI, 2024). Gemini 1.5 Pro is described by its developer as a sparse MoE Transformer (Gemini Team, 2024).

## 4. Technical / Conceptual Explanation

### 4.1 The sparse MoE layer
In a Transformer (Vaswani et al., 2017; see [TECH-ARCH-001]) the position-wise feed-forward network (FFN) in a block is replaced by $N$ expert FFNs $E_1,\dots,E_N$ and a router. For a token representation $x \in \mathbb{R}^d$ the router computes scores $s = x W_r$ with $W_r \in \mathbb{R}^{d \times N}$ and keeps the $k$ largest:

$$ g_i(x) = \begin{cases} \dfrac{\exp(s_i)}{\sum_{j \in \mathcal{T}(x)} \exp(s_j)} & i \in \mathcal{T}(x) \\ 0 & \text{otherwise} \end{cases} \qquad y = \sum_{i \in \mathcal{T}(x)} g_i(x)\, E_i(x) $$

where $\mathcal{T}(x)$ is the set of the top-$k$ indices. Normalising only over the selected experts is one common choice; other designs apply the softmax over all $N$ scores before selection. With $k \ll N$ the per-token compute is roughly $k$ expert FFNs rather than $N$. Attention layers are normally dense and shared.

### 4.2 Load balancing
A router trained by gradient descent tends to favour a few experts, which then receive more updates and become more favoured ("routing collapse"). Shazeer et al. (2017) added auxiliary losses on expert importance and load. The Switch Transformer uses a loss of the form

$$ \mathcal{L}_{\text{aux}} = \alpha\, N \sum_{i=1}^{N} f_i\, P_i $$

where $f_i$ is the fraction of tokens dispatched to expert $i$, $P_i$ is the mean router probability assigned to expert $i$, and $\alpha$ is a small coefficient (Fedus et al., 2022). The loss is minimised by uniform dispatch. Auxiliary losses inject gradients unrelated to the language-modelling objective; Wang et al. (2024) proposed a bias-based, loss-free alternative in which a per-expert bias is added to routing scores before top-$k$ selection and updated according to recent load, and DeepSeek-V3 reports adopting an auxiliary-loss-free strategy (DeepSeek-AI, 2024).

### 4.3 Capacity and token dropping
For efficient batched execution on accelerators each expert is given a fixed buffer ("capacity"), commonly set as a multiple (the capacity factor) of the even-split share of tokens. Tokens routed to a full expert are dropped from that layer and pass through the residual connection unchanged (GShard; Switch). Capacity limits make shapes static but add a quality-versus-efficiency hyperparameter.

### 4.4 Routing variants
- **Token choice** (top-1 in Switch, top-2 in GShard and Mixtral): each token picks experts.
- **Expert choice** (Zhou et al., 2022): each expert picks its top tokens, giving perfect balance and a variable number of experts per token; the authors report more than 2x faster training convergence than Switch top-1 and GShard top-2 at matched compute. Because an expert's selection depends on other tokens in the batch, the scheme is awkward for autoregressive decoding.
- **Fine-grained and shared experts** (DeepSeekMoE): split experts into $mN$ smaller ones and activate $mK$, and reserve $K_s$ always-active shared experts for common knowledge (Dai et al., 2024).

### 4.5 Stabilisation
ST-MoE (Zoph et al., 2022) studies instabilities and transfer behaviour, and Switch Transformers show training sparse models in bfloat16 with selective precision measures. Details are in those papers rather than reproduced here.

## 5. Types, Components & Classifications

| Axis | Options | Representative examples |
| :--- | :--- | :--- |
| Gating | Soft (dense mixture); sparse top-$k$; expert-choice | Jacobs et al. 1991; Shazeer et al. 2017; Zhou et al. 2022 |
| Expert granularity | Few large experts; many fine-grained experts plus shared experts | Mixtral (8 experts, 2 active per token); DeepSeekMoE |
| Placement | Every FFN; alternating layers | GShard and GLaM alternate MoE and dense layers (see papers) |
| Balancing | Auxiliary loss; loss-free bias | Switch; Wang et al. 2024 |

Mixtral 8x7B has 8 feed-forward experts per layer with 2 selected per token, giving access to 47 billion parameters while using 13 billion active parameters per token (Jiang et al., 2024). DeepSeek-V3 reports 671 billion total and 37 billion activated parameters per token, pre-trained on 14.8 trillion tokens (DeepSeek-AI, 2024). These are figures from the respective reports as of their publication dates; current model tables belong in the Living Reference (`MODEL-001`).

## 6. Applications & Concrete Examples
- **Open-weight language models.** Mixtral 8x7B was released with open weights under the Apache 2.0 licence; DeepSeek-V3 checkpoints were also released publicly (primary reports above). Both can be inspected and fine-tuned, which makes them the main empirical basis for MoE research outside industrial labs.
- **Machine translation.** GShard's multilingual translation model was an early demonstration of MoE at scale.
- **Closed models.** Google's Gemini 1.5 report states that Gemini 1.5 Pro is a sparse MoE Transformer; expert count, routing design and parameter counts are undisclosed. OpenAI's GPT-4 technical report states it contains no further details about architecture, including model size, so whether GPT-4 uses MoE is **undisclosed** by the developer. Public figures for either system that do not come from the developer are not repeated here.

## 7. Advantages, Limitations & Unresolved Questions

### Advantages
- Lower training and inference FLOPs per token for a given total parameter count; the Switch, GLaM and DeepSeekMoE papers each report efficiency gains over dense baselines at matched quality or compute, measured on their own setups.
- Scaling-law analyses indicate a systematic advantage. Clark et al. (2022) derive scaling laws in which parameter count and per-token compute are separate axes for routed models. Krajewski et al. (2024) report that MoE models outperform dense Transformers across the budgets they studied and that the gap widens with scale, and that setting expert size equal to the dense FFN size is rarely optimal. Both are single-group studies with fitted laws and rest on the compute regimes tested.

### Limitations
- **Memory.** All experts must be resident. Throughput at low batch sizes is bounded by memory bandwidth for the experts actually touched, and larger batches touch more experts, eroding the sparsity advantage on the inference side (see [TECH-INFER-001]).
- **Communication.** Expert parallelism requires all-to-all token exchange between devices; this shapes cluster design (see [TECH-DIST-001], [TECH-HW-001]).
- **Training stability and balance.** Router collapse, token dropping and loss spikes require the mitigations above.
- **Fine-tuning.** Early sparse models transferred less reliably than dense ones; ST-MoE was written specifically to address this.

### Unresolved questions
1. **Expert specialisation.** Intuition suggests experts specialise by topic. In Mixtral's analysis on The Pile the authors "do not observe obvious patterns in the assignment of experts based on the topic", with specialisation appearing more syntactic (e.g. consistent expert use for certain tokens such as code indentation) and with some positional locality across consecutive tokens (Jiang et al., 2024). This is a finding about one model; how general it is remains open.
2. **Best balancing method.** Auxiliary-loss and loss-free approaches have been compared mainly at small scale (Wang et al. test up to 3 billion parameters and 200 billion tokens); evidence at frontier scale is limited to developer reports.
3. **Interpretability.** Whether experts are meaningful units for analysis is unsettled (see [SAFE-INTERP-001]).

## 8. Current State of Knowledge
As of 2026-10-15, sparse MoE is a mainstream design for large language models in published open-weight work, and at least one closed developer (Google, for Gemini 1.5 Pro) states it uses MoE. The architectural components described in Section 4 (top-$k$ routing, capacity limits, load balancing, fine-grained plus shared experts) are well documented in primary papers. Claims about which proprietary models use MoE, how many experts they have, or their active-parameter counts are **undisclosed** unless a developer report says otherwise, and this encyclopedia does not infer them. Current model details should be read from `MODEL-001`.

The comparative claim "MoE is more compute-efficient than dense at the same quality" is supported by multiple papers from different groups using different setups, but the margin depends on the budget, the comparison baseline (equal total parameters, equal active parameters, or equal training FLOPs), and the serving constraints. It is rated Well Supported with those qualifications.

## 9. Future Trajectory & Strategic Outlook
Observed trends, not forecasts:
- Finer-grained experts with shared experts (DeepSeekMoE), and loss-free balancing, have appeared in recent open reports.
- Research on scaling laws for MoE (Clark et al.; Krajewski et al.) is moving toward treating granularity, expert count and active parameters as joint design variables.
- Serving MoE efficiently is an active systems problem (see [TECH-INFER-001]).

No quantitative forecast is registered in this article. Whether MoE remains dominant depends on hardware memory and interconnect characteristics, and on whether alternatives to the dense FFN (for example hybrid or recurrent state-space designs, see [TECH-ARCH-001]) reduce its advantage; both are open.

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[TECH-ARCH-001] The Transformer Architecture](./TECH-ARCH-001_transformer_architecture.md): The dense FFN block that MoE replaces.
  - [[TECH-DL-001] Deep Learning](./TECH-DL-001_deep_learning.md): General background on training deep networks.
- **Related Methods & Technologies**:
  - [[TECH-LLM-001] Large Language Models](./TECH-LLM-001_large_language_models.md): The main setting where sparse MoE is deployed.
  - [[TECH-SCALE-001] Scaling Laws](./TECH-SCALE-001_scaling_laws.md): Dense and routed scaling relationships.
  - [[TECH-HW-001] AI Hardware](./TECH-HW-001_ai_hardware.md): Memory and interconnect constraints on expert parallelism.
  - [[TECH-DIST-001] Distributed Training](./TECH-DIST-001_distributed_training.md): All-to-all communication and expert parallelism.
  - [[TECH-INFER-001] Inference, Quantization and Efficiency](./TECH-INFER-001_inference_quantization_and_efficiency.md): Serving cost of sparse models.
- **Advanced Implications & Future Trajectories**:
  - [[CAP-OPEN-001] Open-Weight versus Closed Models](../vol04_capabilities/CAP-OPEN-001_open_weight_versus_closed_models.md): Disclosure differences across MoE releases.
  - [[SAFE-INTERP-001] Interpretability and Mechanistic Interpretability](../vol08_safety_governance/SAFE-INTERP-001_interpretability_and_mechanistic_interpretability.md): Open question of expert interpretability.

## 11. References & Foundational Literature
1. Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991). *Adaptive Mixtures of Local Experts*. Neural Computation, 3(1), 79–87. https://doi.org/10.1162/neco.1991.3.1.79. [Original mixture-of-experts formulation with a jointly trained gating network].
2. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017). *Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer*. ICLR 2017. arXiv:1701.06538. [Sparsely-gated MoE layer with thousands of experts and up to 137B parameters in the MoE layer].
3. Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., & Chen, Z. (2020). *GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding*. arXiv:2006.16668 (ICLR 2021). [Sparsely-gated MoE Transformer scaled beyond 600B parameters with automatic sharding].
4. Fedus, W., Zoph, B., & Shazeer, N. (2022). *Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity*. Journal of Machine Learning Research, 23(120). arXiv:2101.03961. [Single-expert routing, auxiliary balancing loss, up to 7x pre-training speed-up].
5. Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., & Fedus, W. (2022). *ST-MoE: Designing Stable and Transferable Sparse Expert Models*. arXiv:2202.08906. [Training-instability and transfer design guide; 269B-parameter sparse model].
6. Du, N., Huang, Y., Dai, A. M., et al. (2022). *GLaM: Efficient Scaling of Language Models with Mixture-of-Experts*. ICML 2022. arXiv:2112.06905. [1.2T-parameter sparse model with reported training-energy and inference-FLOP reductions versus GPT-3].
7. Zhou, Y., Lei, T., Liu, H., Du, N., Huang, Y., Zhao, V., Dai, A., Chen, Z., Le, Q., & Laudon, J. (2022). *Mixture-of-Experts with Expert Choice Routing*. NeurIPS 2022. arXiv:2202.09368. [Experts select tokens; reported >2x faster convergence versus token-choice baselines].
8. Clark, A., de las Casas, D., Guy, A., et al. (2022). *Unified Scaling Laws for Routed Language Models*. ICML 2022. arXiv:2202.01169. [Scaling laws with parameter count and compute as separate axes for routed models].
9. Jiang, A. Q., Sablayrolles, A., Roux, A., et al. (2024). *Mixtral of Experts*. arXiv:2401.04088. [8 experts, top-2 routing, 47B total/13B active parameters, Apache 2.0, routing analysis on The Pile].
10. Dai, D., Deng, C., Zhao, C., et al. (2024). *DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models*. arXiv:2401.06066. [Fine-grained expert segmentation and shared-expert isolation].
11. Wang, L., Gao, H., Zhao, C., Sun, X., & Dai, D. (2024). *Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts*. arXiv:2408.15664. [Bias-based balancing without an auxiliary loss, tested up to 3B parameters and 200B tokens].
12. DeepSeek-AI, Liu, A., Feng, B., et al. (2024). *DeepSeek-V3 Technical Report*. arXiv:2412.19437. [671B total/37B active parameters; auxiliary-loss-free balancing; 14.8T training tokens].
13. Krajewski, J., Ludziejewski, J., Adamczewski, K., et al. (2024). *Scaling Laws for Fine-Grained Mixture of Experts*. arXiv:2402.07871. [Granularity as a scaling variable; reported widening MoE-versus-dense gap].
14. Gemini Team, Georgiev, P., Lei, V. I., Burnell, R., et al. (2024). *Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context*. arXiv:2403.05530. [States Gemini 1.5 Pro is a sparse MoE Transformer; other architectural details undisclosed].
15. OpenAI, Achiam, J., Adler, S., Agarwal, S., et al. (2023). *GPT-4 Technical Report*. arXiv:2303.08774. [States the report contains no further details on architecture, model size, hardware, training compute, dataset construction or training method].
16. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). *Attention Is All You Need*. NeurIPS 2017. arXiv:1706.03762. [Transformer block whose FFN MoE replaces].

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Sparse top-$k$ gating mechanism and its compute/parameter decoupling | Established | Shazeer et al. (2017); Fedus et al. (2022) | High |
| Router collapse and need for load balancing | Well Supported | Shazeer et al. (2017); Fedus et al. (2022); Wang et al. (2024) | High |
| Auxiliary-loss-free balancing performs better than auxiliary-loss balancing | Preliminary | Wang et al. (2024); DeepSeek-AI (2024) | Moderate (small-scale controlled study plus developer report) |
| MoE more compute-efficient than dense at matched quality | Well Supported | Fedus et al. (2022); Du et al. (2022); Clark et al. (2022); Krajewski et al. (2024) | Moderate-High (setup dependent) |
| Expert specialisation by topic | Contested | Jiang et al. (2024) reports no obvious topic pattern in one model | Low-Moderate |
| Gemini 1.5 Pro uses sparse MoE | Established (developer statement) | Gemini Team (2024) | High; details undisclosed |
| GPT-4 architecture | Undisclosed | OpenAI et al. (2023) | Not stated by developer |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.