AI Infrastructure and Hardware

Distributed training is the practice of training a single neural network across many accelerators by partitioning the data, the model, the optimizer state, or the computation schedule, with the devices coordinating through communication collectives. It is required when a model or its training time exceeds what one device can provide.

Frontier models are trained on clusters of thousands of accelerators because neither their memory footprint nor their compute requirement fits on one chip (see TECH-HW-001 AI Hardware & Accelerator Systems and TECH-SCALE-001 Scaling Laws). The engineering problem is not only to split work but to keep devices busy: communication, idle time in pipelines, and hardware failures all reduce the fraction of peak throughput actually achieved. The standard summary metric is model FLOPs utilisation (MFU), the ratio of the model's useful floating-point operations per second to the hardware's theoretical peak.

Distributed training is a principal determinant of who can build frontier systems, because it couples algorithm design, interconnect topology and operations reliability (see ECON-COMP-001 Compute Economics). This article covers the techniques and their published evidence; it does not tabulate current cluster sizes or chip specifications, which are in the Living Reference.

Early large-scale deep learning used data parallelism across commodity machines. Goyal et al. (2017) showed that ResNet-50 could be trained on ImageNet in one hour on 256 GPUs with minibatches of up to 8,192 images and no loss of accuracy relative to small-batch baselines, using a linear learning-rate scaling rule with warmup.

As language models outgrew single-device memory, three model-partitioning lines developed. Huang et al. (2018) introduced GPipe for pipeline parallelism, splitting layers across devices and splitting each batch into micro-batches. Shoeybi et al. (2019) introduced Megatron-LM's intra-layer (tensor) parallelism for transformers, training an 8.3-billion-parameter model on 512 GPUs. Rajbhandari et al. (2019) introduced ZeRO, which partitions optimizer state, gradients and parameters across data-parallel workers to remove redundancy. Narayanan et al. (2021) combined tensor, pipeline and data parallelism and introduced an interleaved pipeline schedule. Zhao et al. (2023) described PyTorch's fully sharded data-parallel implementation (FSDP), a production system based on the same sharding idea. Jiang et al. (2024) and Grattafiori et al. (2024) documented the operational problems of clusters beyond ten thousand GPUs.

#4.1 Memory budget

With mixed-precision training (Micikevicius et al., 2018) and an Adam-type optimizer, each parameter requires storage not only for weights and gradients but also for optimizer state (fp32 master weights and two moments). The ZeRO analysis (Rajbhandari et al., 2019) counts this as roughly 16 bytes per parameter in total, before activations; a model with $\Psi$ parameters therefore needs about $16\Psi$ bytes for these states. Activations add memory proportional to batch size, sequence length and depth. Activation (gradient) checkpointing recomputes selected activations during the backward pass, trading extra computation for memory; Chen et al. (2016) show memory cost sublinear in network depth with roughly one additional forward pass.

#4.2 Data parallelism

Each of $N$ workers holds a full model replica, processes a different shard of the batch, and averages gradients with an all-reduce collective before the optimizer step. For a ring all-reduce over a message of $M$ bytes, each worker sends and receives about $2M(N-1)/N$ bytes, nearly independent of $N$. Data parallelism increases the global batch size, and large batches need learning-rate adjustment; Goyal et al. (2017) give the linear scaling rule: when the minibatch is multiplied by $k$, multiply the learning rate by $k$, with warmup. There is a limit beyond which larger batches stop reducing steps proportionally; that limit depends on the task and is a topic of scaling-law research.

#4.3 Sharded data parallelism (ZeRO and FSDP)

ZeRO partitions the optimizer state (stage 1), gradients (stage 2) and parameters (stage 3) across the data-parallel group, gathering parameters just in time for each layer. Rajbhandari et al. (2019) report training models above 100 billion parameters on 400 GPUs at about 15 petaflops, and, without model parallelism, models up to 13 billion parameters; these are 2019 results on the authors' hardware. FSDP implements fully sharded data parallelism in PyTorch; its authors report it matches standard data-parallel throughput while supporting much larger models, with near-linear scaling in TFLOPS in their experiments (Zhao et al., 2023).

#4.4 Tensor (intra-layer) parallelism

Matrix multiplications inside a layer are split across devices. In a transformer MLP block $Y=\mathrm{GeLU}(XA)B$, splitting $A$ by columns and $B$ by rows lets each device compute independently and require a single all-reduce per block in the forward pass (Shoeybi et al., 2019). The communication volume per layer is high, so tensor parallelism is usually confined to devices connected by the fastest links within a node.

#4.5 Pipeline parallelism

Layers are divided into $p$ sequential stages on different devices. A batch is split into $m$ micro-batches that flow through the stages; devices are idle during pipeline fill and drain, the bubble, whose fraction in the simple GPipe schedule is $(p-1)/(m+p-1)$ (Huang et al., 2018). Interleaved schedules assign several non-contiguous layer chunks to each device, and Narayanan et al. (2021) report that their interleaved schedule improves throughput by more than 10% at comparable memory.

#4.6 Combining the dimensions

Frontier-scale runs combine several parallelisms ("3D" or "4D" parallelism, the latter adding context or sequence parallelism). Narayanan et al. (2021) report a 1-trillion-parameter model on 3,072 GPUs at 502 petaFLOP/s, 52% of theoretical peak per-GPU throughput. Grattafiori et al. (2024) describe 4D parallelism (tensor, context, pipeline, data via FSDP) for their Llama 3 models, and Jiang et al. (2024) report 55.2% MFU for a 175-billion-parameter model on 12,288 GPUs, a 1.34 times improvement over Megatron-LM in their comparison. These are results from 2021–2024 on specific hardware and software and should not be read as current limits.

#4.7 Reliability and stragglers

At the scale of thousands of devices, failures are routine. Grattafiori et al. (2024) report 466 job interruptions during a 54-day pre-training snapshot of Llama 3 training, 419 of them unexpected, with hardware faults (GPU and associated components) the largest category, while maintaining effective training time above 90%. Jiang et al. (2024) describe diagnostic tooling, fault tolerance and straggler mitigation as core components of their system. Frequent checkpointing and automated restart are therefore part of training design.

• Data parallelism: replicate model, split data; all-reduce on gradients.
• Sharded data parallelism: ZeRO stages 1–3 and FSDP; split states, not computation.
• Tensor parallelism: split matrix operations within layers.
• Pipeline parallelism: split by depth; schedules include GPipe-style and interleaved.
• Sequence/context parallelism: split the sequence dimension for long contexts (named in Grattafiori et al., 2024; not further detailed here).
• Expert parallelism: place different experts of a mixture-of-experts model on different devices (see TECH-MOE-001 Mixture-of-Experts Models).
• Synchronous versus asynchronous updates: the large-scale results cited here use synchronous training.
• Memory techniques: mixed precision, activation checkpointing, offloading.

• Open documentation of production runs: the Llama 3 technical report (Grattafiori et al., 2024) is a primary-source description of parallelism configuration and failure statistics for a large run; the reproduced numbers are limited to what the report states.
• System co-design: MegaScale (Jiang et al., 2024) is a published production system by an industrial team.
• Library support: PyTorch FSDP (Zhao et al., 2023) and the Megatron-LM and ZeRO implementations are the principal open implementations discussed in the cited papers.
• Training runs of closed frontier models are generally undisclosed in cluster size, parallelism layout and utilisation.

Advantages. Enables models and training durations impossible on one device, and shortens wall-clock time (Goyal et al., 2017; Narayanan et al., 2021).

Limitations. (1) Communication overhead and pipeline bubbles keep utilisation well below 100% (the published figures above are 30%–55%, depending on setup). (2) Sub-linear returns from large batches. (3) Numerical and hyperparameter fragility at scale; loss spikes and silent hardware errors are operational hazards (Grattafiori et al., 2024; Jiang et al., 2024). (4) Capital and energy requirements concentrate frontier training in few organisations. (5) Reported MFU values from different papers are not directly comparable because model shapes, sequence lengths and accounting conventions differ.

Unresolved. How far synchronous training scales on a single site versus geographically distributed or asynchronous training; how far algorithmic changes (for example communication-reduced optimizers) can lower interconnect requirements. These are active areas; this article cites no claim on either.

As of 2026-10-15, the four parallelism families and their trade-offs are established and widely implemented (section 12). Utilisation of roughly 40–55% of peak at ten-thousand-GPU scale is documented in the cited 2021–2024 papers; later improvements and current cluster scales are not tracked here, and the published numbers should be checked against newer technical reports. Hardware specifications, interconnect bandwidths and cluster sizes are in FACT-001 and MODEL-001 and the hardware principles in TECH-HW-001 AI Hardware & Accelerator Systems.

Observed trends. Published systems have moved from hundreds to over ten thousand devices (Shoeybi et al., 2019: 512 GPUs; Jiang et al., 2024: 12,288 GPUs), with reliability engineering becoming a first-class concern.

Near term and beyond. Extrapolating cluster size requires assumptions about power, capital and supply that are outside this article. No forecast is asserted and no FCST- record is attached.

• Foundational Concepts:
  • TECH-HW-001 AI Hardware & Accelerator Systems: Accelerators, memory and interconnects.
  • TECH-ARCH-001 Transformer Architecture: The layer structure being partitioned.
• Related Methods & Technologies:
  • TECH-LLM-001 Large Language Models: The main workload.
  • TECH-MOE-001 Mixture-of-Experts Models: Expert parallelism.
  • TECH-INFER-001 Inference, Quantization and Efficiency: Serving counterpart.
  • TECH-SCALE-001 Scaling Laws: Why compute is scaled.
• Economics:
  • ECON-COMP-001 Compute Economics: Cost of training clusters.

AI Hardware & Accelerator Systems refer to the specialized physical computing architectures, memory hierarchies, and interconnect fabrics engineered to execute the dense, highly parallel matrix and tensor operations fundamental to artificial neural networks. Unlike general-purpose central processing units (CPUs) designed for sequential logic, AI accelerators—such as Graphics Processing Units (GPUs), Tensor Processing Units (TPUs), and Neural Processing Units (NPUs)—trade general-purpose flexibility for massive deterministic parallelism and extreme memory bandwidth, enabling the training and inference of large-scale foundation models. (The first-generation TPU, an inference accelerator, is documented in Jouppi et al., 2017; the characterization of GPUs and NPUs here is a general description without a single supporting source.)

The history of artificial intelligence is fundamentally bound to the physical constraints of its underlying hardware. The "Bitter Lesson" (Sutton, 2019) argues that, over 70 years of AI research, general methods that leverage increasing computation (search and learning) have ultimately outperformed methods built on hand-encoded human knowledge. It is a claim about which class of method wins, not that algorithms matter less than hardware.

The significance of specialized AI hardware rests on three pillars:

1. The Parallel Nature of Deep Learning: Deep neural networks compute vast arrays of linear algebraic operations (dot products, convolutions). CPUs execute these sequentially; AI accelerators execute very large numbers of these operations in parallel (the exact count depends on the chip and numeric precision).
2. The Memory Bandwidth Bottleneck: Modern Large Language Models (LLMs) are rarely bounded by raw calculation speed (FLOPs); they are bounded by how quickly data can be transferred from memory to the compute cores (the Von Neumann bottleneck). Modern AI hardware relies heavily on High-Bandwidth Memory (HBM) physically co-packaged with the processor.
3. Geopolitical and Economic Moats: The physical manufacturing of frontier AI chips requires extreme-ultraviolet (EUV) lithography and advanced 3D packaging (e.g., TSMC's CoWoS), creating fragile global supply chains and establishing compute infrastructure as a primary lever of national security.

In the 1980s, the expert systems boom was accompanied by Lisp Machines, specialized hardware designed to execute symbolic programming languages, whose market later declined as general-purpose processors improved [VERIFICATION NEEDED: no source cited for Lisp Machine history].

In the late 2000s, researchers began applying commodity GPUs, designed for graphics rendering, to the dense linear algebra of neural networks; Raina et al. (2009) reported large speedups for unsupervised deep learning on graphics processors. In 2012, the AlexNet authors reported training their network in five to six days on two GTX 580 3GB GPUs (Krizhevsky et al., 2012), and the network won the 2012 ImageNet competition.

As models expanded from millions to hundreds of billions of parameters, hardware evolved from commodity graphics cards to dedicated datacenter AI accelerators. Google deployed the first Tensor Processing Unit (TPU), an inference-only design built around a 65,536-unit 8-bit matrix multiply array, in its datacenters from 2015 (Jouppi et al., 2017). NVIDIA introduced Tensor Cores, specialized mixed-precision matrix units, in its datacenter GPUs starting with the Volta architecture [VERIFICATION NEEDED: cite NVIDIA Volta whitepaper for year]. Today, hardware development is largely characterized by cluster-level engineering—interconnecting tens of thousands of chips via specialized networking fabrics (NVLink, InfiniBand).

#4.1 Matrix Multiplication and Parallelism

The core operation in a neural network layer is $Y = WX + B$, where $W$ is a massive weight matrix and $X$ is the input vector. A modern GPU splits this massive matrix multiplication into thousands of smaller tiles, distributing them across thousands of Streaming Multiprocessors (SMs). Systolic Arrays (used in TPUs) take this further: data flows rhythmically through a grid of Arithmetic Logic Units (ALUs) without needing to write intermediate results back to registers, drastically improving energy efficiency.

#4.2 Numerical Precision and Quantization

Classical scientific computing demands 64-bit double-precision floating-point (FP64) accuracy. Deep learning is remarkably robust to noise and does not require such precision. Reducing precision allows hardware to pack more ALUs into the same silicon area and halves memory bandwidth requirements. Hardware has progressed from FP32 (single precision) to FP16 and BF16 (bfloat16, which preserves the exponent range of FP32 while truncating the mantissa). Recent hardware supports FP8 and, for inference, lower-precision formats such as 4-bit integers, which can increase throughput; the accuracy cost depends on the model and method (see TECH-INFER-001).

#4.3 Memory Hierarchies and Bandwidth

A processor's theoretical peak performance (TFLOPs) is rarely achieved. Real-world performance is dictated by Arithmetic Intensity—the ratio of computations performed per byte of data loaded from memory. Because LLM inference relies heavily on reading the weight matrices and the KV cache (past token states) for every generated token, the operation is memory-bandwidth bound. To alleviate this, modern AI chips utilize High-Bandwidth Memory (HBM). Unlike standard DDR memory installed on a motherboard, HBM is stacked vertically and connected directly to the GPU die via an ultra-dense silicon interposer, delivering multiple terabytes per second per accelerator. As of NVIDIA product pages accessed 2026-10-15: H100 SXM, 80 GB at 3.35 TB/s; H100 NVL, 94 GB at 3.9 TB/s (NVIDIA, H100 page); DGX B200 system, 1,440 GB total and 64 TB/s across eight GPUs, i.e. roughly 180 GB and 8 TB/s per GPU (derived by division; NVIDIA, DGX B200 page). Newer parts may exceed these figures; consult vendor datasheets and FACT-001 for current values.

#4.4 Interconnects and Distributed Training

Training a 100-billion parameter model requires far more memory than a single accelerator provides (for example 80 GB on an H100 SXM, as of 2026-10-15), because weights, gradients and optimizer states must be stored. Training requires distributed parallelization across thousands of chips:

• Data Parallelism: Copies of the model are stored on different GPUs; each processes a different subset of the training data, and gradients are synchronized.
• Tensor/Model Parallelism: A single massive matrix multiplication is mathematically split across multiple GPUs (e.g., Megatron-LM; Shoeybi et al., 2019). Because these chips must synchronize constantly, the networking speed between them is as critical as the chips themselves. Technologies like NVLink provide ultra-high-speed point-to-point connections within a server, while InfiniBand provides low-latency optical networking across the datacenter cluster.

1. Graphics Processing Units (GPUs): The dominant architecture (e.g., NVIDIA H100 and B200 and AMD Instinct MI300-series, products of the 2022-2025 period; see vendor pages for current lines). Highly flexible, supporting thousands of concurrent threads.
2. Tensor Processing Units (TPUs): Application-Specific Integrated Circuits (ASICs) designed by Google. Less flexible than GPUs but highly optimized for matrix math and interconnected at massive scale via optical circuit switches.
3. Field-Programmable Gate Arrays (FPGAs): Reconfigurable hardware used where custom low-latency logic is needed, in both edge and datacenter settings [VERIFICATION NEEDED: no source cited for specific FPGA uses].
4. Neural Processing Units (NPUs): Low-power, highly specialized co-processors embedded directly into mobile phone SoCs and laptop CPUs (e.g., Apple Neural Engine) to execute lightweight inference locally.

• Training Clusters: In March 2024 Meta described two clusters of 24,576 H100 GPUs each, and stated that it used this cluster design for Llama 3 training (Meta Engineering, 2024). The Llama 3 paper reports that the 405B model was trained on up to 16K H100 GPUs (Llama Team, 2024). Large clusters require handling of hardware failures and gradient synchronization across many devices.
• Inference Scaling: Serving systems commonly batch many user requests together so that each load of model weights from memory is amortized over more computation, raising arithmetic intensity. Serving architectures of specific commercial products are undisclosed.

#Demonstrated Advantages

• Lower-precision data formats (BF16, FP8) and HBM have allowed accelerator throughput to keep increasing even as general-purpose CPU scaling slowed (see Dally et al., 2020 on domain-specific accelerators).

#Core Limitations

• The Reticle Limit: The size of a single silicon chip is physically bounded by the photolithography reticle size (approximately $800\text{ mm}^2$ [VERIFICATION NEEDED: cite lithography vendor source]). Larger systems are built by combining multiple dies via advanced packaging (chiplets), creating thermal and latency challenges.
• Energy and Thermal Constraints: Rack-scale AI systems such as the liquid-cooled GB200 NVL72 (72 GPUs and 13.4 TB of HBM3E per rack, per the NVIDIA product page accessed 2026-10-15) draw very high power per rack; the NVIDIA page does not state a figure [VERIFICATION NEEDED: cite rack power from a datasheet]. Grid power availability and cooling capacity are widely discussed constraints on datacenter growth [VERIFICATION NEEDED: cite e.g. IEA data-centre energy analysis].
• Theoretical Peak vs. MFU: Hardware is often marketed by its theoretical peak FLOPs. In practice, due to memory latency and network communication bottlenecks, Model FLOPs Utilization (MFU) is well below 100% in large distributed training; for example, the Llama 3 paper reports 38-43% MFU for 405B pre-training configurations on 8,192-16,384 H100 GPUs (Llama Team, 2024).

#Unresolved Scientific Questions

1. Neuromorphic Computing: Can hardware that mimics the continuous, spike-timing dynamics of biological brains drastically reduce the megawatt power consumption of current digital accelerators?
2. Analog and Optical Compute: Can matrix multiplication be performed at the speed of light using optical photonics without the thermal heat dissipation of silicon electrons?

NVIDIA is widely regarded as the leading supplier of AI training accelerators, with a mature software ecosystem (CUDA), and is challenged by Google's TPUs, AMD and custom accelerators [VERIFICATION NEEDED: market-share estimate with as-of date and Tier 4 source, e.g. company filings or analyst data]. Engineering work on memory bottlenecks includes advanced 2.5D/3D packaging; wafer-scale designs (e.g., Cerebras) are an alternative approach [VERIFICATION NEEDED: no source cited].

• Near-Term (0–3 Years) [speculative forecast, no forecast record assigned]: Training and inference hardware may diverge further, with liquid-cooled GPU/TPU clusters used for training and specialized NPUs and low-latency inference chips used for serving.
• Medium-Term (3–10 Years) [speculative]: Chiplet and 3D-packaged designs, already in use, may become more prevalent as reticle size and yield limit monolithic dies. Some AI datacenters may be co-located with dedicated power generation [VERIFICATION NEEDED: cite energy outlook].
• Long-Term (10+ Years) [speculative]: Possible commercialization of photonic computing and reversible computing architectures, fundamentally breaking the current thermal density barriers limiting silicon.

• Foundational Concepts:
  • FOUND-004 Computational Complexity in AI: The mathematical limits bounding hardware scaling.
• Related Methods & Technologies:
  • TECH-ARCH-001 The Transformer Architecture: The specific matrix operations dictating modern hardware design.
  • TECH-LLM-001 Large Language Models: The primary driver of the datacenter accelerator boom.
• Advanced Implications & Future Trajectories:
  • ECON-COMP-001 The Economics of Compute & Scaling: The capital expenditure models governing hardware procurement.

Inference is the use of a trained model to produce outputs for new inputs. Inference efficiency is the set of techniques that reduce the latency, memory, energy or monetary cost of that process, including numerical quantization, compression, attention and cache optimisations, batching and scheduling, and speculative decoding.

Training happens once per model version; inference happens for every request, so for widely used systems the cumulative inference workload can exceed training cost (the balance varies by deployment and is not asserted here; see ECON-COMP-001 Compute Economics). For autoregressive language models, inference has a distinctive structure: tokens are generated one at a time, and each step must read the model weights and the accumulated attention cache from memory, so decoding is typically limited by memory bandwidth rather than arithmetic throughput (Pope et al., 2022; see TECH-HW-001 AI Hardware & Accelerator Systems). Most efficiency techniques therefore target data movement and memory footprint.

Efficiency also determines who can run a model: quantization methods have allowed models with tens or hundreds of billions of parameters to run on a single accelerator or consumer-grade GPU (Frantar et al., 2022; Dettmers et al., 2022). Reasoning models that generate long intermediate outputs make inference cost a first-order capability question (see TECH-REASON-001 Reasoning Models and Test-Time Compute).

Model compression predates large language models. Hinton, Vinyals and Dean (2015) described distillation, training a small model to match the output distribution of a larger model or ensemble. Jacob et al. (2018) described integer-arithmetic-only quantization and a training scheme that simulates quantization, targeted at mobile inference.

For transformers, a series of 2022–2023 papers addressed LLM-specific obstacles. Dettmers et al. (2022) introduced LLM.int8(), identifying systematic outlier feature dimensions that harm naive 8-bit quantization. Frantar et al. (2022) introduced GPTQ for post-training 3- to 4-bit weight quantization. Lin et al. (2023) proposed activation-aware weight quantization (AWQ). Micikevicius et al. (2022) proposed 8-bit floating-point (FP8) formats. On the systems side, Dao et al. (2022) introduced FlashAttention, Leviathan et al. (2022) speculative decoding, Ainslie et al. (2023) grouped-query attention, and Kwon et al. (2023) PagedAttention in the vLLM serving system.

#4.1 Prefill and decode

Serving an autoregressive transformer has two phases. Prefill processes the whole prompt in parallel and is generally compute-bound. Decode generates one token per step per sequence, repeatedly reading all weights; with small batches its arithmetic intensity (operations per byte moved) is low, so throughput is bounded by memory bandwidth. Batching many requests amortises weight reads across sequences and raises utilisation, at the price of added latency and cache memory. Pope et al. (2022) analyse partitioning strategies on TPU v4 for models with more than 500 billion parameters and report, for PaLM 540B with int8 weights, about 29 ms per token at low batch size and 76% model FLOPs utilisation for large-batch input processing at 2,048-token context (figures from the authors' 2022 setup, not current hardware).

#4.2 The key-value cache

To avoid recomputation, the keys and values of previous tokens are stored per layer. For a model with $L$ layers, $H_{kv}$ key-value heads of dimension $d_h$, sequence length $T$ and element size $b$ bytes, the per-sequence cache size is

It grows linearly with context length and batch size, and can dominate memory. Two responses are architectural and systems-level. Multi-query attention uses a single key-value head, and grouped-query attention (GQA) an intermediate number, reducing $H_{kv}$; Ainslie et al. (2023) report an uptraining recipe that converts multi-head checkpoints with about 5% of original pre-training compute and reaches quality close to multi-head attention with speed comparable to multi-query. Kwon et al. (2023) observe that conventional serving wastes cache memory through fragmentation and duplication, and PagedAttention, modelled on operating-system paging, reduces that waste; they report 2–4 times higher throughput than prior systems at the same latency, with larger gains for longer sequences and larger models.

#4.3 Quantization

Uniform affine quantization maps a real value $x$ to an integer

with scale $s$ and zero point $z$. Granularity (per tensor, per channel, per group) controls the error. Two broad regimes exist:

• Post-training quantization (PTQ): weights (and optionally activations) are quantized after training, using a small calibration set. GPTQ quantizes weights to 3–4 bits; the authors report quantizing a 175-billion-parameter model in about four GPU hours with little accuracy loss and end-to-end speedups of about 3.25 times on an A100 and 4.5 times on an A6000 relative to FP16 (Frantar et al., 2022). AWQ observes that a small set of weight channels matters more, identified through activation statistics, and scales them to reduce error; its authors report over 3 times speedup relative to a Hugging Face FP16 baseline in their TinyChat framework (Lin et al., 2023).
• Quantization-aware training (QAT): quantization error is simulated during training (Jacob et al., 2018).

LLM.int8() performs 8-bit matrix multiplication while isolating systematic outlier feature dimensions into a 16-bit multiplication, so that over 99.9% of values are multiplied in 8-bit, roughly halving inference memory without reported degradation up to 175B parameters (Dettmers et al., 2022). Micikevicius et al. (2022) define two FP8 encodings (E4M3 and E5M2) and report training results matching 16-bit across CNNs, RNNs and transformers, including language models up to 175B parameters.

#4.4 Distillation and pruning

Distillation trains a student on the teacher's soft output distribution, often with temperature scaling (Hinton et al., 2015). Pruning and sparsity remove weights; they require hardware or kernel support to yield wall-clock gains and are not covered in depth here.

#4.5 Exact-attention kernels

Standard attention materialises an $N\times N$ matrix, with memory traffic that dominates runtime. FlashAttention computes exact attention with tiling to reduce reads and writes between high-bandwidth memory and on-chip memory; its authors report, in 2022, 15% end-to-end speedup on BERT-large, 3 times on GPT-2 at 1K context, and 2.4 times on Long Range Arena (Dao et al., 2022). It changes cost, not results.

#4.6 Speculative decoding

A small draft model proposes several tokens, and the large model verifies them in one parallel pass; accepted tokens are kept and the first rejection is resampled. Leviathan et al. (2022) show this leaves the output distribution unchanged and report 2–3 times acceleration on T5-XXL with no retraining. The gain depends on how often the draft agrees with the target.

• Numerical precision: FP16/BF16, FP8, INT8, 4-bit and lower weight formats. Weight-only quantization lowers memory and bandwidth; weight-and-activation quantization can additionally use low-precision matrix units.
• Compression: quantization, distillation, pruning, low-rank factorisation.
• Architectural: multi-query and grouped-query attention, sparse or mixture-of-experts layers (see TECH-MOE-001 Mixture-of-Experts Models), which reduce active parameters per token while leaving total memory large.
• Systems: continuous batching, paged cache management, kernel fusion, tensor/pipeline parallel serving across devices (see TECH-DIST-001 Distributed Training for the shared parallelism vocabulary).
• Algorithmic: speculative decoding; caching of shared prompt prefixes.
• Deployment target: datacenter, edge or on-device, each with different constraints on memory and power.

• Hosted model serving: production serving stacks combine batching, cache paging and quantization; PagedAttention/vLLM is an openly documented example (Kwon et al., 2023). The configurations of commercial services are generally undisclosed.
• Running large models on small hardware: GPTQ and LLM.int8() were demonstrated with 175B-class models on a single GPU or server (Frantar et al., 2022; Dettmers et al., 2022); AWQ reported deploying a 70B Llama-2 model on mobile GPUs (Lin et al., 2023).
• Training-time use of low precision: FP8 formats were evaluated for training as well as inference (Micikevicius et al., 2022).
• Distilled small models for latency-sensitive uses follow Hinton et al. (2015); individual commercial products are not enumerated here.

Advantages. Large reductions in memory and cost at small measured quality loss for the methods and models tested; some techniques (FlashAttention, speculative decoding) are exact, with identical output distributions (Dao et al., 2022; Leviathan et al., 2022).

Limitations. (1) Quantization error is model- and task-dependent; benchmark averages can hide degradation on long-context, multilingual or reasoning tasks, which the cited papers do not exhaustively test. (2) Outlier activations make activation quantization harder than weight quantization (Dettmers et al., 2022). (3) Reported speedups are specific to the authors' hardware and software versions and generally do not transfer. (4) Cache memory scales with context length and batch size, limiting long-context serving. (5) Efficiency gains often reduce unit cost but may increase total consumption by enabling more use; that effect is an economic question not settled by these sources.

Unresolved. How far weight precision can be reduced without loss for the largest models; the interaction of quantization with safety behaviours and calibration; how test-time compute growth interacts with per-token cost reductions.

As of 2026-10-15, 8-bit weight quantization, 4-bit weight-only PTQ, grouped-query attention, FlashAttention-style kernels, paged cache management and speculative decoding are established techniques in the published literature with open implementations (see section 12 for per-claim ratings). Specific accelerators' throughput, per-token prices, energy per query, and the precision formats supported by current chips change frequently and are deliberately not listed here; see MODEL-001 and FACT-001 for dated records, and TECH-HW-001 AI Hardware & Accelerator Systems for hardware principles. The techniques used inside closed commercial models are undisclosed.

Observed trends. Over 2022–2024 the cited literature moved from 8-bit to 3–4-bit weights, from training at 16-bit to 8-bit formats, and from per-request to continuously batched, paged serving.

Near term (0–3 years). Further lowering of precision, cache compression and wider adoption of speculative methods are natural extensions of the existing record; no quantified forecast is made.

Medium and long term. Inference cost relative to capability depends on algorithmic progress, hardware cycles and demand, none of which this article forecasts. No FCST- record is registered.

• Foundational Concepts:
  • TECH-HW-001 AI Hardware & Accelerator Systems: Memory hierarchy and numeric formats in silicon.
  • TECH-ARCH-001 Transformer Architecture: Attention and the KV cache.
• Related Methods & Technologies:
  • TECH-LLM-001 Large Language Models: The main serving workload.
  • TECH-DIST-001 Distributed Training: Shared parallelism concepts.
  • TECH-MOE-001 Mixture-of-Experts Models: Sparse activation as an efficiency design.
  • TECH-REASON-001 Reasoning Models and Test-Time Compute: Long generations raise inference cost.
• Economics:
  • ECON-COMP-001 Compute Economics: Cost per token and demand effects.