---
id: TECH-INFER-001
title: "Inference, Quantization and Efficiency"
alternative_names:
  - "Model Serving"
  - "Efficient Inference"
  - "Model Compression"
category: "Technical → Infrastructure"
volume: III
chapter: 21
entry_type: "Technical Article"
level: 3
status: "Draft"
last_reviewed: 2026-10-15
version: "0.1.0"
prerequisites:
  - "TECH-HW-001"
related_entries:
  - "TECH-ARCH-001"
  - "TECH-LLM-001"
  - "TECH-DIST-001"
  - "TECH-MOE-001"
  - "TECH-REASON-001"
  - "ECON-COMP-001"
---

# Inference, Quantization and Efficiency

## 1. Concise Definition
**Inference** is the use of a trained model to produce outputs for new inputs. **Inference efficiency** is the set of techniques that reduce the latency, memory, energy or monetary cost of that process, including numerical quantization, compression, attention and cache optimisations, batching and scheduling, and speculative decoding.

## 2. Overview & Significance
Training happens once per model version; inference happens for every request, so for widely used systems the cumulative inference workload can exceed training cost (the balance varies by deployment and is not asserted here; see [[ECON-COMP-001] Compute Economics](../vol09_economy/ECON-COMP-001_compute_economics.md)). For autoregressive language models, inference has a distinctive structure: tokens are generated one at a time, and each step must read the model weights and the accumulated attention cache from memory, so decoding is typically limited by memory bandwidth rather than arithmetic throughput (Pope et al., 2022; see [[TECH-HW-001] AI Hardware & Accelerator Systems](./TECH-HW-001_ai_hardware.md)). Most efficiency techniques therefore target data movement and memory footprint.

Efficiency also determines who can run a model: quantization methods have allowed models with tens or hundreds of billions of parameters to run on a single accelerator or consumer-grade GPU (Frantar et al., 2022; Dettmers et al., 2022). Reasoning models that generate long intermediate outputs make inference cost a first-order capability question (see [[TECH-REASON-001] Reasoning Models and Test-Time Compute](./TECH-REASON-001_reasoning_models_and_test_time_compute.md)).

## 3. Historical Development
Model compression predates large language models. Hinton, Vinyals and Dean (2015) described **distillation**, training a small model to match the output distribution of a larger model or ensemble. Jacob et al. (2018) described integer-arithmetic-only quantization and a training scheme that simulates quantization, targeted at mobile inference.

For transformers, a series of 2022–2023 papers addressed LLM-specific obstacles. Dettmers et al. (2022) introduced LLM.int8(), identifying systematic outlier feature dimensions that harm naive 8-bit quantization. Frantar et al. (2022) introduced GPTQ for post-training 3- to 4-bit weight quantization. Lin et al. (2023) proposed activation-aware weight quantization (AWQ). Micikevicius et al. (2022) proposed 8-bit floating-point (FP8) formats. On the systems side, Dao et al. (2022) introduced FlashAttention, Leviathan et al. (2022) speculative decoding, Ainslie et al. (2023) grouped-query attention, and Kwon et al. (2023) PagedAttention in the vLLM serving system.

## 4. Technical / Conceptual Explanation

### 4.1 Prefill and decode
Serving an autoregressive transformer has two phases. **Prefill** processes the whole prompt in parallel and is generally compute-bound. **Decode** generates one token per step per sequence, repeatedly reading all weights; with small batches its arithmetic intensity (operations per byte moved) is low, so throughput is bounded by memory bandwidth. Batching many requests amortises weight reads across sequences and raises utilisation, at the price of added latency and cache memory. Pope et al. (2022) analyse partitioning strategies on TPU v4 for models with more than 500 billion parameters and report, for PaLM 540B with int8 weights, about 29 ms per token at low batch size and 76% model FLOPs utilisation for large-batch input processing at 2,048-token context (figures from the authors' 2022 setup, not current hardware).

### 4.2 The key-value cache
To avoid recomputation, the keys and values of previous tokens are stored per layer. For a model with $L$ layers, $H_{kv}$ key-value heads of dimension $d_h$, sequence length $T$ and element size $b$ bytes, the per-sequence cache size is
$$\text{KV bytes} = 2 \, L \, H_{kv}\, d_h\, T\, b.$$
It grows linearly with context length and batch size, and can dominate memory. Two responses are architectural and systems-level. Multi-query attention uses a single key-value head, and grouped-query attention (GQA) an intermediate number, reducing $H_{kv}$; Ainslie et al. (2023) report an uptraining recipe that converts multi-head checkpoints with about 5% of original pre-training compute and reaches quality close to multi-head attention with speed comparable to multi-query. Kwon et al. (2023) observe that conventional serving wastes cache memory through fragmentation and duplication, and PagedAttention, modelled on operating-system paging, reduces that waste; they report 2–4 times higher throughput than prior systems at the same latency, with larger gains for longer sequences and larger models.

### 4.3 Quantization
Uniform affine quantization maps a real value $x$ to an integer
$$q = \mathrm{clip}\!\Big(\mathrm{round}\big(x/s\big) + z,\; q_{\min},\; q_{\max}\Big),\qquad \hat{x} = s\,(q - z),$$
with scale $s$ and zero point $z$. Granularity (per tensor, per channel, per group) controls the error. Two broad regimes exist:
- **Post-training quantization (PTQ)**: weights (and optionally activations) are quantized after training, using a small calibration set. GPTQ quantizes weights to 3–4 bits; the authors report quantizing a 175-billion-parameter model in about four GPU hours with little accuracy loss and end-to-end speedups of about 3.25 times on an A100 and 4.5 times on an A6000 relative to FP16 (Frantar et al., 2022). AWQ observes that a small set of weight channels matters more, identified through activation statistics, and scales them to reduce error; its authors report over 3 times speedup relative to a Hugging Face FP16 baseline in their TinyChat framework (Lin et al., 2023).
- **Quantization-aware training (QAT)**: quantization error is simulated during training (Jacob et al., 2018).

LLM.int8() performs 8-bit matrix multiplication while isolating systematic outlier feature dimensions into a 16-bit multiplication, so that over 99.9% of values are multiplied in 8-bit, roughly halving inference memory without reported degradation up to 175B parameters (Dettmers et al., 2022). Micikevicius et al. (2022) define two FP8 encodings (E4M3 and E5M2) and report training results matching 16-bit across CNNs, RNNs and transformers, including language models up to 175B parameters.

### 4.4 Distillation and pruning
Distillation trains a student on the teacher's soft output distribution, often with temperature scaling (Hinton et al., 2015). Pruning and sparsity remove weights; they require hardware or kernel support to yield wall-clock gains and are not covered in depth here.

### 4.5 Exact-attention kernels
Standard attention materialises an $N\times N$ matrix, with memory traffic that dominates runtime. FlashAttention computes exact attention with tiling to reduce reads and writes between high-bandwidth memory and on-chip memory; its authors report, in 2022, 15% end-to-end speedup on BERT-large, 3 times on GPT-2 at 1K context, and 2.4 times on Long Range Arena (Dao et al., 2022). It changes cost, not results.

### 4.6 Speculative decoding
A small draft model proposes several tokens, and the large model verifies them in one parallel pass; accepted tokens are kept and the first rejection is resampled. Leviathan et al. (2022) show this leaves the output distribution unchanged and report 2–3 times acceleration on T5-XXL with no retraining. The gain depends on how often the draft agrees with the target.

## 5. Types, Components & Classifications
- **Numerical precision**: FP16/BF16, FP8, INT8, 4-bit and lower weight formats. Weight-only quantization lowers memory and bandwidth; weight-and-activation quantization can additionally use low-precision matrix units.
- **Compression**: quantization, distillation, pruning, low-rank factorisation.
- **Architectural**: multi-query and grouped-query attention, sparse or mixture-of-experts layers (see [[TECH-MOE-001] Mixture-of-Experts Models](./TECH-MOE-001_mixture_of_experts_models.md)), which reduce active parameters per token while leaving total memory large.
- **Systems**: continuous batching, paged cache management, kernel fusion, tensor/pipeline parallel serving across devices (see [[TECH-DIST-001] Distributed Training](./TECH-DIST-001_distributed_training.md) for the shared parallelism vocabulary).
- **Algorithmic**: speculative decoding; caching of shared prompt prefixes.
- **Deployment target**: datacenter, edge or on-device, each with different constraints on memory and power.

## 6. Applications & Concrete Examples
- **Hosted model serving**: production serving stacks combine batching, cache paging and quantization; PagedAttention/vLLM is an openly documented example (Kwon et al., 2023). The configurations of commercial services are generally undisclosed.
- **Running large models on small hardware**: GPTQ and LLM.int8() were demonstrated with 175B-class models on a single GPU or server (Frantar et al., 2022; Dettmers et al., 2022); AWQ reported deploying a 70B Llama-2 model on mobile GPUs (Lin et al., 2023).
- **Training-time use of low precision**: FP8 formats were evaluated for training as well as inference (Micikevicius et al., 2022).
- **Distilled small models** for latency-sensitive uses follow Hinton et al. (2015); individual commercial products are not enumerated here.

## 7. Advantages, Limitations & Unresolved Questions
**Advantages.** Large reductions in memory and cost at small measured quality loss for the methods and models tested; some techniques (FlashAttention, speculative decoding) are exact, with identical output distributions (Dao et al., 2022; Leviathan et al., 2022).

**Limitations.** (1) Quantization error is model- and task-dependent; benchmark averages can hide degradation on long-context, multilingual or reasoning tasks, which the cited papers do not exhaustively test. (2) Outlier activations make activation quantization harder than weight quantization (Dettmers et al., 2022). (3) Reported speedups are specific to the authors' hardware and software versions and generally do not transfer. (4) Cache memory scales with context length and batch size, limiting long-context serving. (5) Efficiency gains often reduce unit cost but may increase total consumption by enabling more use; that effect is an economic question not settled by these sources.

**Unresolved.** How far weight precision can be reduced without loss for the largest models; the interaction of quantization with safety behaviours and calibration; how test-time compute growth interacts with per-token cost reductions.

## 8. Current State of Knowledge
As of 2026-10-15, 8-bit weight quantization, 4-bit weight-only PTQ, grouped-query attention, FlashAttention-style kernels, paged cache management and speculative decoding are established techniques in the published literature with open implementations (see section 12 for per-claim ratings). Specific accelerators' throughput, per-token prices, energy per query, and the precision formats supported by current chips change frequently and are deliberately not listed here; see `MODEL-001` and `FACT-001` for dated records, and [[TECH-HW-001] AI Hardware & Accelerator Systems](./TECH-HW-001_ai_hardware.md) for hardware principles. The techniques used inside closed commercial models are undisclosed.

## 9. Future Trajectory & Strategic Outlook
**Observed trends.** Over 2022–2024 the cited literature moved from 8-bit to 3–4-bit weights, from training at 16-bit to 8-bit formats, and from per-request to continuously batched, paged serving.

**Near term (0–3 years).** Further lowering of precision, cache compression and wider adoption of speculative methods are natural extensions of the existing record; no quantified forecast is made.

**Medium and long term.** Inference cost relative to capability depends on algorithmic progress, hardware cycles and demand, none of which this article forecasts. No `FCST-` record is registered.

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[TECH-HW-001] AI Hardware & Accelerator Systems](./TECH-HW-001_ai_hardware.md): Memory hierarchy and numeric formats in silicon.
  - [[TECH-ARCH-001] Transformer Architecture](./TECH-ARCH-001_transformer_architecture.md): Attention and the KV cache.
- **Related Methods & Technologies**:
  - [[TECH-LLM-001] Large Language Models](./TECH-LLM-001_large_language_models.md): The main serving workload.
  - [[TECH-DIST-001] Distributed Training](./TECH-DIST-001_distributed_training.md): Shared parallelism concepts.
  - [[TECH-MOE-001] Mixture-of-Experts Models](./TECH-MOE-001_mixture_of_experts_models.md): Sparse activation as an efficiency design.
  - [[TECH-REASON-001] Reasoning Models and Test-Time Compute](./TECH-REASON-001_reasoning_models_and_test_time_compute.md): Long generations raise inference cost.
- **Economics**:
  - [[ECON-COMP-001] Compute Economics](../vol09_economy/ECON-COMP-001_compute_economics.md): Cost per token and demand effects.

## 11. References & Foundational Literature
1. Hinton, G., Vinyals, O., & Dean, J. (2015). *Distilling the Knowledge in a Neural Network*. arXiv:1503.02531. [Defines distillation of ensembles or large models into a smaller model.]
2. Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., & Kalenichenko, D. (2018). *Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference*. CVPR 2018. https://arxiv.org/abs/1712.05877. [Integer-only inference and quantization-aware training.]
3. Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022). *LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale*. NeurIPS 2022. https://arxiv.org/abs/2208.07339. [Outlier-aware 8-bit inference up to 175B parameters.]
4. Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2022). *GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers*. ICLR 2023. https://arxiv.org/abs/2210.17323. [3–4-bit post-training weight quantization and reported speedups.]
5. Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., & Han, S. (2023). *AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration*. arXiv:2306.00978 (MLSys 2024). [Activation-aware weight scaling and on-device deployment.]
6. Micikevicius, P., Stosic, D., Burgess, N., Cornea, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., Mellempudi, N., Oberman, S., Shoeybi, M., Siu, M., & Wu, H. (2022). *FP8 Formats for Deep Learning*. arXiv:2209.05433. [E4M3/E5M2 formats and training and quantization results.]
7. Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Levskaya, A., Heek, J., Xiao, K., Agrawal, S., & Dean, J. (2022). *Efficiently Scaling Transformer Inference*. arXiv:2211.05102 (MLSys 2023). [Partitioning analysis and reported PaLM 540B latency and utilisation.]
8. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., & Sanghai, S. (2023). *GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints*. EMNLP 2023. https://arxiv.org/abs/2305.13245. [Grouped-query attention and the uptraining recipe.]
9. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022). *FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness*. NeurIPS 2022. https://arxiv.org/abs/2205.14135. [IO-aware exact attention and reported speedups.]
10. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). *Efficient Memory Management for Large Language Model Serving with PagedAttention*. SOSP 2023. https://arxiv.org/abs/2309.06180. [Paged KV-cache management and reported 2–4 times throughput.]
11. Leviathan, Y., Kalman, M., & Matias, Y. (2022). *Fast Inference from Transformers via Speculative Decoding*. ICML 2023. https://arxiv.org/abs/2211.17192. [Speculative decoding with unchanged output distribution; 2–3 times speedup on T5-XXL.]

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Decode is typically memory-bandwidth-bound; KV cache grows linearly with context | Established (analysis) | Pope et al. (2022); formula in section 4.2 | High |
| Affine quantization definition | Established | Jacob et al. (2018) | High |
| 8-bit weight inference without reported degradation to 175B | Well Supported | Dettmers et al. (2022) | Moderate to High |
| 3–4-bit post-training quantization with small loss | Well Supported | Frantar et al. (2022); Lin et al. (2023) | Moderate |
| FlashAttention and speculative decoding preserve exact results/distributions | Established (algorithmic property) | Dao et al. (2022); Leviathan et al. (2022) | High |
| Specific speedup factors | Well Supported for stated hardware/date; not transferable | Sections 4.2-4.6 sources | Moderate |
| GQA near multi-head quality at multi-query speed | Well Supported | Ainslie et al. (2023) | Moderate |
| FP8 training parity with 16-bit | Preliminary to Well Supported | Micikevicius et al. (2022) | Moderate |
| Closed-model serving internals | Undisclosed | n/a | n/a |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.