---
title: "AI Inference Economics: Falling Prices, Rising Spend"
slug: inference-economics-crisis
description: "Per-token inference prices keep falling, yet total spend can rise as tasks use more tokens. What the data shows and how cost-per-outcome helps."
date: 2026-06-15
last_reviewed: 2026-11-02
status: published
type: knowledge-base
author: [liyam-flexer]
image: inference-economics-crisis
image_alt: "Cover graphic: AI Inference Economics: Falling Prices, Rising Spend, over a dark blue network of connected points."
tags: [ai economics, inference compute, hardware accelerators, llm deployment, quantization, capital allocation]
---

# AI Inference Economics: Falling Prices, Rising Spend

**In short:** Per-token inference prices at a fixed capability level have fallen steeply since 2022, but total inference spend can still rise because applications now consume far more tokens per task (long contexts, multi-turn agents, reasoning). The practical response is cost-per-outcome engineering: route work to the smallest adequate model, cache and batch where possible, compress models, and price by outcome rather than by token. The "plateau" in the headline is about bills, not list prices; the evidence below does not show list prices flattening.

The early investment narrative for generative AI was that a model is trained once at high fixed cost and then served cheaply to a growing number of users, so margins should expand with volume. Whether that holds depends on how the cost of serving each request compares with what the customer pays, and that comparison is shifting as workloads change.

This analysis separates what is measured (falling price per unit of capability, rising token demand per task) from what is asserted without public data (any particular company's gross margin). It then covers the engineering levers teams use and their documented limits.

---

## What the Data Shows: Cheaper Tokens, More Tokens

The simple amortization view treats unit cost as training cost spread over queries plus marginal serving cost:

$$\text{Unit Cost per Query} = \frac{\text{Amortized Training Run}}{\text{Total Inference Queries}} + \text{Marginal Compute Cost}$$

As an illustration only (invented numbers): a \$20M training run amortized over 100 billion served tokens adds \$0.0002 per 1,000 tokens, which is small next to typical API list prices. In this framing, serving cost, not training cost, is what a high-volume operator has to manage.

Measured trends on price per unit of capability point down, not flat:

- The Stanford AI Index 2025 reports that the inference cost for a system performing at GPT-3.5 level [dropped over 280-fold between November 2022 and October 2024](https://hai.stanford.edu/ai-index/2025-ai-index-report), and that hardware costs have declined by about 30% annually with energy efficiency improving by about 40% a year.
- Epoch AI (12 March 2025) found that the [price to reach GPT-4-level performance on GPQA Diamond fell about 40x per year](https://epoch.ai/data-insights/llm-inference-price-trends), with declines ranging from 9x to 900x per year depending on the milestone. Epoch cautions that it is less clear the fastest declines will persist.

The cost problem is therefore not that a given capability stopped getting cheaper. It is that frontier-tier models keep being priced high, and that usage per task grows. As of 2 November 2026, [Anthropic's price list](https://platform.claude.com/docs/en/about-claude/pricing) spans \$0.10/\$0.50 per million input/output tokens for a small Haiku 5.5 tier up to \$10/\$50 for Claude Fable 5.1, a 100x spread. An agent that makes many calls, each re-sending a growing context, multiplies whatever the per-token price is; the mechanics are in [How LLM API Pricing Works: Tokens, Asymmetry & Waste](./how-llm-api-pricing-works.html).

Why serving is expensive per token: NVIDIA's engineering blog describes the [decode phase as memory-bound](https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/), with the speed of moving weights, keys and values from memory dominating latency rather than arithmetic. That is why batching and quantization (which move less data) help, and why low-latency serving that cannot batch deeply is costlier per token.

For deeper analysis of compute scaling, see [AI Scaling Laws Explained](./ai-scaling-laws-explained.html) and [Hardware Acceleration](../chapters/ch-19.html).

---

## Three Workload Profiles

Workloads differ in how much latency they tolerate, which determines how well a provider can pack hardware. This is a framework for reasoning, not a measured market split.

| Profile | Latency tolerance | Economic implication |
|---|---|---|
| Asynchronous batch jobs | Minutes to hours | Can be packed efficiently. Providers price it accordingly: Anthropic's [Message Batches API](https://platform.claude.com/docs/en/build-with-claude/batch-processing) cuts cost by 50 percent, with most batches finishing in under an hour. |
| Real-time interactive agents | Sub-second to seconds | Deep batching is limited by latency targets, so serving capacity is used less efficiently. Cost per user depends heavily on how many tokens each session consumes. |
| High-value specialized work | Seconds to minutes | When the task replaces expensive human time, compute is a small share of the value. Illustration: a task that saves three hours of a \$850/hour professional can justify a \$10 model run (invented numbers). |

I could not find a public source for utilization rates or per-user serving losses in the interactive segment, so this post does not give numbers for them. Treat any claim that interactive products are "underwater" as company-specific and check the filings or reporting behind it.

---

## Algorithmic Levers and Their Documented Limits

Teams reduce serving cost by compressing or speeding up models. Published results show real gains, with trade-offs that depend on model and task:

1. **Quantization.** [LLM.int8()](https://arxiv.org/abs/2208.07339) (Dettmers et al., NeurIPS 2022) reports halving inference memory with no measured degradation on models up to 175B parameters. [GPTQ](https://arxiv.org/abs/2210.17323) (Frantar et al., ICLR 2023) quantizes 175B-parameter models to 3 or 4 bits in about four GPU hours with negligible accuracy degradation, and reports end-to-end speedups of about 3.25x on an A100 and 4.5x on an A6000. Results below 3 bits are weaker and task-dependent; the GPTQ authors report reasonable accuracy even at 2-bit and ternary settings, so there is no universal precision floor.
2. **Distillation.** Training a smaller model to imitate a larger one can cut serving cost for narrow tasks. The size of the gain and of any quality loss varies by task and has to be measured on your own evaluation set; no general percentage is reliable.
3. **Speculative decoding.** A small draft model proposes tokens that a larger model verifies in parallel. [Leviathan, Kalman and Matias](https://arxiv.org/abs/2211.17192) (ICML 2023) report a 2x to 3x speedup on T5-XXL with identical outputs. The speedup depends on how often the draft is accepted, so it varies by workload.

---

## The Capital-Efficient Strategy: Outcome vs. Token Billing

Teams that manage serving cost well tend to optimize cost per completed task rather than cost per token:

```
User request ──► [ Small classifier / router ]
                     │
       ┌─────────────┴─────────────┐
       ▼                           ▼
[ Simple: small model ]    [ Complex: larger model, tools, retries ]
```

1. **Routing and cascading.** Send easy requests to a small, cheap model and escalate hard ones. The share you can keep on the small model must be measured on your own traffic; there is no standard figure.
2. **Context hygiene.** Passing curated context instead of dumping a whole corpus cuts input tokens directly. For example (invented sizes), replacing a 200,000-token dump with 4,000 retrieved tokens cuts prompt tokens by 98% on that call, though retrieval quality must be checked.
3. **Caching and batching.** Reused prefixes can be cached at a fraction of the input price and non-urgent work can be batched (see the pricing post and sources below).
4. **Outcome-based pricing.** Charging per document verified or per claim processed decouples revenue from token consumption, so a spike in tokens per task does not erase margin.

---

## Frequently asked questions

**Why is AI inference spending a concern if token prices are falling?**
Price per unit of capability has fallen sharply, with the Stanford AI Index reporting a 280-fold drop for GPT-3.5-level systems between November 2022 and October 2024. But agents, long contexts and reasoning increase tokens per task, and frontier models are priced much higher than small ones, so total bills can still grow even as unit prices fall.

**What is the difference between training economics and inference economics?**
Training is a large, mostly one-off cost for each model version. Inference is an ongoing cost that scales with usage, so it is paid per request and per token. A model with heavy usage can spend more on serving over its life than on training, but the split depends on the product and is rarely disclosed.

**Can open-source models solve the inference cost problem?**
Open-weight models remove the API vendor's margin and give control over quantization and routing, but they do not remove hosting costs. You still pay for GPUs, memory, electricity and engineering time, and low utilization can make self-hosting more expensive per token than a shared API for small workloads.

**What is speculative decoding?**
Speculative decoding is an inference technique in which a small draft model proposes several tokens and a larger model checks them in one parallel pass. Leviathan, Kalman and Matias reported a 2x to 3x speedup on T5-XXL with identical outputs. The gain depends on how often the larger model accepts the draft.

**What is cost-per-outcome engineering?**
It means optimizing the total cost of completing a user task, such as one verified document, instead of the price of each token. Tools include model routing, prompt caching, batching, context pruning and smaller or quantized models, and it often pairs with outcome-based pricing to the customer.

---

## Sources

1. [Stanford HAI: 2025 AI Index Report](https://hai.stanford.edu/ai-index/2025-ai-index-report).
2. [Epoch AI: LLM inference prices have fallen rapidly but unequally across tasks](https://epoch.ai/data-insights/llm-inference-price-trends) (12 March 2025).
3. [Anthropic: Pricing](https://platform.claude.com/docs/en/about-claude/pricing) and [Batch processing](https://platform.claude.com/docs/en/build-with-claude/batch-processing) (checked 2 November 2026).
4. [NVIDIA: Mastering LLM Techniques: Inference Optimization](https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/).
5. Dettmers, Lewis, Belkada, Zettlemoyer, [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://arxiv.org/abs/2208.07339) (NeurIPS 2022).
6. Frantar, Ashkboos, Hoefler, Alistarh, [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://arxiv.org/abs/2210.17323) (ICLR 2023).
7. Leviathan, Kalman, Matias, [Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192) (ICML 2023).