How LLM API Pricing Works: Tokens, Asymmetry & Waste

Engineering teams frequently encounter LLM invoice shock after deploying production applications. Two applications with the same number of daily users can see very different bills. The gap comes from how token accounting, context window accumulation, and pricing asymmetry compound across multi-turn interactions.

This architectural guide explains token conversion mechanics, why output tokens are priced higher, the four primary drivers of invisible token waste, and a formula for running an Asymmetry Audit.

---

#What Is a Token? The Core Unit of AI Accounting

A token is the fundamental atomic unit of text representation in transformer-based language models. Rather than operating on raw Unicode characters or whole grammatical words, models process chunks of text generated by subword tokenizers (such as Byte-Pair Encoding or SentencePiece):

• OpenAI's rule of thumb for English text is that 1 token is about 4 characters, or about three-quarters of a word, so 100 tokens is about 75 words and 1,000 tokens is about 750 words.
• The reverse mistake matters: 1,000 words is about 1,333 tokens, so budgeting 750 tokens for 1,000 words understates volume by about 44%.
• Tokenizers differ by provider and model, so the same text gives different counts. Code, JSON and non-English text usually need more tokens per character than English prose; measure your own payloads with the provider's counter (for example Anthropic's token counting) instead of trusting a rule of thumb.

For details on how token embeddings pass through transformer self-attention layers, see How Transformers Work and Transformer Architectures.

---

#The Anatomy of an API Call: The Output Premium

Every standard LLM API call invoices across two separate pricing tiers:

INPUT TOKENS (Processed in Parallel · Base Rate)
┌────────────────────────────────────────────────────────┐
│ System Prompt (Guidelines, Personas, Constraints)     │
├────────────────────────────────────────────────────────┤
│ Tool & Function Schemas (JSON-RPC Definitions)         │
├────────────────────────────────────────────────────────┤
│ Conversation History (Turns 1...N-1)                   │
├────────────────────────────────────────────────────────┤
│ Active User Message & RAG Context                      │
└────────────────────────────────────────────────────────┘
                            │
                            ▼ [ Forward Transformer Ingestion ]
                            │
OUTPUT TOKENS (Autoregressive Generation · ~5x Base Rate)
┌────────────────────────────────────────────────────────┐
│ [Token 1] ──► [Token 2] ──► [Token 3] ... ──► [Token N] │
│ (Each token incurs a discrete GPU memory bandwidth hop)│
└────────────────────────────────────────────────────────┘

Published list prices as of 2 November 2026 (standard tier, short context), from the vendors' own pages:

Sources: Anthropic pricing, OpenAI pricing, Gemini API pricing. Prices change often (some Gemini prices are scheduled to change on 1 December 2026 and 1 January 2027), so check the page before budgeting.

Providers do not publish the reasoning behind the ratio, but a common technical explanation is the difference between the two phases of inference. NVIDIA's engineering blog describes prefill as a highly parallel matrix-matrix operation that saturates the GPU, and decode as memory-bound: the speed at which weights and cached keys and values move from memory dominates latency. Treat this as the likely cost driver, not a disclosed pricing formula.

---

#The Compounding History Trap in Multi-Turn Systems

In interactive applications (chatbots, coding agents, support bots), the cumulative input billed over a session grows roughly quadratically with the number of turns, not linearly:

Turn 1:  [System] + [User 1]                    ──► [Output 1]
Turn 2:  [System] + [User 1] + [Output 1] + [User 2] ──► [Output 2]
Turn 3:  [System] + [User 1] + [Output 1] + [User 2] + [Output 2] + [User 3] ──► [Output 3]

On every conversational turn, the entire conversation history—including prior model outputs—must be re-sent and re-billed as input. This creates four common sources of silent waste:

1. Bloated system prompts. Re-sending a 3,000-token system prompt on every turn bills 60,000 input tokens for the prompt alone in a 20-turn chat.
2. Verbose outputs. Uncapped generation at roughly 5x input pricing. The verbosity also enters history and is re-billed as input on every later turn.
3. Missing cache layers. Re-processing identical prefixes at full price instead of using prompt caching (cache reads cost 0.05x to 0.1x base input on Anthropic models).
4. Model over-sizing. Sending routing, classification or JSON parsing to a frontier model instead of a small, cheap one.

For broader macroeconomic implications of this infrastructure cost curve, see The Inference Economics Crisis.

---

#The Asymmetry Audit: Where to Focus Optimization

Before starting prompt optimization or building RAG compression pipelines, teams must calculate their Output Spend Ratio ($S$). This formula prevents teams from spending weeks micro-optimizing input prompts when verbose outputs represent 80% of their actual invoice:

Where:

• $T_{\text{in}}$ and $T_{\text{out}}$ represent total monthly input and output token counts from API telemetry logs.
• $P_{\text{in}}$ and $P_{\text{out}}$ represent published provider prices per token.

                  ┌──────────────────────┐
                  │ Compute Output Ratio │
                  │      Score (S)       │
                  └──────────┬───────────┘
                             │
            ┌────────────────┴────────────────┐
            ▼                                 ▼
   [ S > 0.5: Output-Dominated ]     [ S < 0.5: Input-Dominated ]
   • Enforce strict max_tokens       • Enable Prompt Caching
   • Require terse JSON responses    • Prune static system prompts
   • Strip conversational filler     • Implement sliding RAG windows
            │                                 │
            └───────────────┬─────────────────┘
                            ▼
           [ Recompute S After Optimization ]
           (Reducing one side shifts balance)

#Worked examples (illustrative workloads, sourced prices)

The token volumes below are invented for illustration. The prices are Claude Sonnet 5.5 list prices as of 2 November 2026 ($2.00/M input, $10.00/M output; source), ignoring caching and batch discounts.

Example A: a support chatbot logging 50 million input tokens and 5 million output tokens a month:

• Input spend: $50 \times $2.00 = $100.00$
• Output spend: $5 \times $10.00 = $50.00$
• Total spend: $150.00
• Output ratio ($S$): $$50 / $150 = 0.33$

Although each output token costs 5× more, input volume dominates ($S = 0.33 < 0.5$). Trimming output saves little; caching the static prefix and compressing history is the higher-return fix.

Example B: a report-generation pipeline logging 2 million input tokens and 4 million output tokens:

• Input spend: $2 \times $2.00 = $4.00$
• Output spend: $4 \times $10.00 = $40.00$
• Total spend: $44.00
• Output ratio ($S$): $$40 / $44 = 0.91$

Here spend is output-dominated ($S > 0.5$). Compressing the prompt saves little; the priority is setting max_tokens and tightening format instructions. The 0.5 threshold is a rule of thumb for where to look first, not a law.

---

#Architectural Strategies for Token Optimization

1. Use prompt caching. Put static instructions and tool schemas at the start of the prompt so repeated calls can hit the cache. On Anthropic, cache reads cost 0.05x base input on Sonnet 5.5 and 0.1x on most other models, cache writes cost 1.25x (5-minute TTL) or 2x (1-hour TTL), and the minimum cacheable prefix is 512 to 4,096 tokens depending on the model. OpenAI caches automatically; on GPT-5.6 and later the prefix must be at least 1,024 visible tokens and reads cost 0.1x (0.05x on GPT-6.1 Sol and Terra).
2. Summarize old context. Rather than keeping raw turns indefinitely, periodically have a small model compress earlier exchanges into a short state summary. The interval and summary length are design choices to tune against quality.
3. Route by difficulty. Send simple classification or lookup requests to a cheap model and escalate only reasoning-heavy tasks. See Claude Code Review & Benchmarks.
4. Batch non-urgent work. Anthropic's Message Batches API cuts cost by 50 percent for asynchronous jobs, with most batches finishing in under an hour.

---

#Frequently asked questions

#Why do LLM providers charge more for output tokens than input tokens?

Providers do not publish a pricing rationale. The usual technical explanation is that input is processed in parallel during prefill, while output is generated one token at a time in a memory-bound decode phase, so each output token costs more serving capacity. On current pricing pages the premium is about 5× for Anthropic and OpenAI flagships, and 6× for some others.

#How many words are in 1,000 LLM tokens?

For English prose, OpenAI's rule of thumb is about 750 words per 1,000 tokens, or roughly 4 characters per token. Tokenizers differ between providers, and code, JSON and non-English text usually take more tokens, so use the provider's token counter for real budgeting rather than relying on the heuristic.

#What is prompt caching, and how does it reduce API bills?

Prompt caching lets a provider reuse the processed form of an identical prompt prefix. Cached reads are billed at a fraction of the input price: 0.05x to 0.1x on Anthropic models and 0.1x on recent OpenAI models, as of 2 November 2026. Anthropic also charges more to write the cache (1.25x or 2x), so it pays off only when the prefix is reused.

#Does conversation history get billed on every message?

Yes. Chat APIs are stateless, so the client resends the earlier messages with each request, and they are billed as input each time unless cached. The total input billed across a session therefore grows roughly quadratically with the number of turns. Caching and summarizing history are the standard ways to limit it.

#What is the Asymmetry Audit?

It is this site's name for a simple calculation, S = output spend divided by total spend, using your own token counts and the provider's list prices. A value above 0.5 means output dominates your bill; below 0.5 means input does. It tells you which side to optimize first.

---

#Sources

1. Anthropic: Pricing (checked 2 November 2026).
2. Anthropic: Prompt caching (checked 2 November 2026).
3. Anthropic: Batch processing and Token counting.
4. OpenAI: API pricing and Prompt caching guide (checked 2 November 2026).
5. OpenAI Help: What are tokens and how to count them.
6. Google: Gemini API pricing (checked 2 November 2026).
7. NVIDIA: Mastering LLM Techniques: Inference Optimization.