LLM Inference Cost Optimization: Where the Money Actually Goes
- Output tokens cost five times input tokens across every tier of Anthropic's published price list as of August 2026: $5 in and $25 out per million for Claude Opus 5, $2 and $10 for Sonnet 5, $1 and $5 for Haiku 4.5. Output discipline is therefore a five-times-weighted lever.
- Prompt caching is priced as a multiplier, not a discount: a cache read costs one tenth of standard input, a five-minute cache write costs 1.25 times, and a one-hour write costs twice. It pays back after one read on the short cache and two on the long one.
- Most teams optimise the model when context assembly is the real driver. If you have not measured how many tokens your retrieval layer injects per request, you do not yet know what your bill is made of.
- Retrieved context chunks46%
- Conversation history replayed22%
- System prompt and instructions13%
- Tool definitions and schemas9%
- The user's actual message4%
- Model output6%
Where does LLM inference money actually go?
Into context assembly and output length, in that order, not into the model name. On most production systems the retrieval layer, the replayed conversation history, the system prompt and the tool schemas together account for the large majority of tokens billed, and the user's actual message is a rounding error next to them.
This matters because it inverts the usual optimisation instinct. When a bill is too high the first move is almost always to swap to a cheaper model, which is a single-digit-hour change with a real quality cost. The larger and safer move is usually to send fewer tokens: retrieve five chunks instead of twenty, summarise conversation history past a threshold instead of replaying it verbatim, trim tool schemas that are attached to every request whether or not the model can use them.
The arithmetic makes the point quickly. Take a request that assembles 7,000 input tokens and produces 400 output tokens on Claude Sonnet 5, priced by Anthropic at $2 per million input tokens and $10 per million output tokens as of August 2026. That is $0.014 of input and $0.004 of output, so $0.018 per request, or $1,800 per hundred thousand requests. Halving the retrieved context saves about $6.40 per hundred thousand requests. Switching from Sonnet 5 to Haiku 4.5 at $1 and $5 per million saves about $9, but changes the quality of every answer, whereas the retrieval change may cost nothing at all.
There is a second cost dimension people forget: the per-call fees attached to server-side tooling. Anthropic prices web search at $10 per 1,000 searches on top of token costs, and the search results are then billed as input tokens on that turn and every subsequent turn they remain in. An agent that searches on every turn has a cost curve token accounting alone will not explain.
The rest of this post works through the levers in order of dollars saved per engineering hour, which is the same discipline that governs the AWS cost optimization checklist. If you want the same treatment applied to your own inference bill alongside your infrastructure bill, that is what a cloud cost optimization engagement covers.
Why do output tokens cost more than input tokens?
Because generating a token requires a full forward pass through the model for that token alone, while input tokens are processed in parallel in a single prefill pass. The pricing reflects the compute. Across Anthropic's published August 2026 price list the ratio is exactly five to one at every tier, and the practical consequence is that a word you ask the model not to write is worth five words you stop sending.
Look at the consistency. Claude Opus 5 is $5 per million input tokens and $25 per million output. Claude Sonnet 5 is $2 and $10. Claude Haiku 4.5 is $1 and $5. The absolute prices differ by a factor of five across the range but the input-to-output ratio does not move at all. OpenAI's published pricing table has the same column structure (input, cached input, cache writes, output) with output priced at several times input. This is not a vendor quirk; it is the economics of autoregressive decoding showing up on an invoice.
So output discipline is a five-times-weighted lever, and it is almost free. Cap max_tokens at a value derived from what your product actually renders, not the model default. Ask for structured output where you are going to parse it anyway, because JSON with short keys is far cheaper than prose that says the same thing. Instruct the model to answer directly rather than restate the question, list its reasoning, and summarise what it just said, a habit that can easily double an output.
The one place to be careful is extended reasoning. On models that produce internal reasoning tokens, those tokens are billed as output. Capping them aggressively to save money can cost you accuracy on exactly the requests where accuracy matters, and a wrong answer that triggers a retry costs more than the reasoning would have. Route reasoning-heavy request classes to a model configured for them and keep the cap tight everywhere else, rather than applying one global limit.
Measure this properly before you tune it. Log input and output token counts per request class, not per request, and rank the classes by total output tokens per day. The result is almost always concentrated: two or three request classes produce most of the output volume in the system, and those are the only prompts worth rewriting.
What is model routing and how much does it actually save?
Model routing means classifying each request and sending it to the cheapest model that can handle that class, rather than sending everything to the model your hardest request needs. On a realistic traffic mix it typically removes half to three quarters of the model line, because most production traffic is not hard.
The mistake is routing on the wrong axis. Teams try to route on predicted difficulty, which requires a classifier that is itself an inference call and is wrong often enough to be dangerous. Route on request class instead, a structural property you already know at the call site. Classification, extraction, short-form summarisation, formatting, moderation, routing decisions and intent detection are all cheap-model work. Multi-step reasoning, code generation, ambiguous synthesis and anything customer-visible and irreversible are expensive-model work. You know which is which when you write the call.
Do the arithmetic on a concrete mix. Say sixty percent of your traffic is extraction and classification, thirty percent is standard conversational work and ten percent is hard reasoning. Sending all of it to Claude Opus 5 at $5 and $25 per million costs a certain amount; sending the first sixty percent to Haiku 4.5 at $1 and $5, the next thirty to Sonnet 5 at $2 and $10, and only the last ten to Opus 5 cuts the blended rate by roughly seventy percent. No prompt was rewritten and no quality was lost on the requests that needed the big model.
Two implementation details decide whether this survives contact with production. First, put the routing decision behind an interface, so changing a class-to-model mapping is a config change, not a deployment. Model prices and capabilities move quarterly; a hard-coded model name is technical debt on a three-month clock. Second, build a fallback path: if the cheap model returns something that fails validation, retry once on the stronger model. A retry rate of a few percent barely dents the saving and removes the quality risk that stops teams adopting routing at all.
The failure mode to watch for is retry-storm economics. If your cheap tier fails validation on twenty percent of requests, you are paying for the cheap call plus the expensive call on a fifth of your traffic and you have made things worse while adding latency. Instrument the fallback rate per class from day one and demote any class above roughly five percent back to the stronger model.
Claude Haiku 4.5 is $1 per million input and $5 output. These classes have machine-checkable outputs, so a validation failure is a cheap, automatic signal to retry higher.
Claude Sonnet 5 at $2 and $10 per million, with cache reads at $0.20. For a stable system prompt and a shared document context, the cached input cost dominates and is one tenth of list.
Claude Opus 5 at $5 and $25 per million. Do not economise on the model here; economise on how much it writes and how much context you assemble for it.
The expected cost of one bad output exceeds the token saving by orders of magnitude. Routing is a cost optimisation, and cost optimisations do not belong on the path where a mistake is unrecoverable.
Anthropic publishes a 50% discount on both input and output tokens for batch processing, and states it stacks with prompt caching multipliers. This is the largest single discount available and most teams never enable it.
How much does prompt caching actually save?
A cache read costs one tenth of the standard input price. Anthropic publishes the multipliers directly: a five-minute cache write is 1.25 times the base input rate, a one-hour write is twice it, and a cache hit is 0.1 times. So the short cache pays for itself after a single read and the long cache after two.
Those break-even points are the whole decision, and they are far lower than people assume. If any two requests within five minutes share a prefix (the same system prompt, the same tool definitions, the same document, the same few-shot examples) caching is already profitable. On any system with sustained traffic that condition is met continuously, so the correct default for a stable prompt prefix is caching on, not caching considered.
The design constraint is that caching works on prefixes. Everything before your cache breakpoint must be byte-identical between requests, which has an architectural implication most teams discover the expensive way: put the stable material first. System prompt, tool schemas, few-shot examples and shared documents at the top; user-specific and turn-specific material at the bottom. Injecting a timestamp, a request ID or a personalised greeting into the system prompt invalidates the cache on every single request and silently converts your caching strategy into a 1.25x surcharge.
Concretely, on Claude Sonnet 5 at $2 per million input, a 20,000-token cached prefix costs $0.04 uncached, $0.05 to write into the five-minute cache, and $0.004 on every subsequent read. Across a hundred thousand requests where ninety-five percent hit the cache, that is roughly $4,000 uncached against roughly $600 cached. This is usually the single largest saving on an inference bill and it needs no quality trade-off, which makes it strictly better than model downgrading as a first move.
Anthropic also notes that caching multipliers stack with the batch discount and that on recent models the full million-token context window is billed at standard rates, so a very large cached prefix does not attract a long-context surcharge. That combination (a large stable prefix, cached, submitted in batch) is the cheapest way to run high-volume document work that exists on a commercial API today.
When should you use batching instead of real-time calls?
Whenever nothing is waiting on the answer. Anthropic publishes a 50% discount on both input and output tokens for batch processing, and states it stacks with prompt caching. Half your bill on the asynchronous share of your workload, for an architectural change and no quality change at all, is the best-value hour on this entire list.
Teams skip it because batching is a systems change, not a parameter change. You submit a job, you poll or receive a callback, you handle partial completion, you handle failures per item rather than per request. That is real work, but a day or two of it, and it applies to a category of workload that is often much larger than teams realise once they go looking.
Go looking in the obvious places. Nightly enrichment and classification of records. Backfills after a prompt change. Evaluation suites, which for a serious team are run on every prompt commit and can be the single largest token consumer in the company. Document ingestion pipelines. Report generation. Content moderation queues where a few minutes of latency is irrelevant. Embedding refreshes. Every one of these is billed at full price on most systems purely because the code path was written synchronously first and never revisited.
One accounting subtlety worth stating. The batch discount applies to token pricing, but not to everything. Anthropic notes, for example, that its managed agent sessions are stateful and interactive and therefore have no batch mode, and that some feature-specific charges sit outside the token line. Check what fraction of the workload you are moving is actually token-priced before you promise finance a fifty percent cut.
The strategic version of this argument is that latency should be a per-request-class product decision, not a global default. Most teams have exactly one code path, the synchronous one, and therefore pay real-time prices for work nobody is waiting on. Splitting into a fast path and a cheap path is the same insight that drives model routing, applied to time rather than to capability.
- 1. Log input and output tokens per request class, and rank by daily spendYou cannot route or trim what you have not attributed
- 2. Turn on prompt caching for every stable prefixCache reads are one tenth of input; the five-minute cache pays back after a single read
- 3. Move the stable material to the front of the prompt and remove anything volatile from itA timestamp in the system prompt turns caching into a 1.25x surcharge
- 4. Move every workload nobody is waiting on to the batch API50% off both input and output, and it stacks with caching
- 5. Cap max_tokens per request class based on what you actually renderOutput is billed at five times input at every published tier
- 6. Route request classes to model tiers behind a config-driven interfaceModel names hard-coded in application code are debt on a quarterly clock
- 7. Cut retrieved context to the smallest k that passes your evalsRetrieval is usually the single largest token contributor
- 8. Summarise conversation history past a turn threshold instead of replaying itReplayed history grows quadratically with conversation length
- 9. Audit tool schemas attached to every requestTool definitions and the tool-use system prompt are billed as input on every call
- 10. Instrument per-call server tool fees separately from tokensWeb search is priced per search on top of tokens, and results are billed as input thereafter
- 11. Measure the fallback rate on every routed class and demote anything above ~5%Retry storms turn a cost optimisation into a cost increase plus latency
- 12. Re-check published model prices quarterly and re-run the routing decisionEvery price in this post moved within the last year
When is self-hosting an open model cheaper than an API?
When you have sustained, predictable, high-volume traffic on a task an open model handles well, and someone who can own GPU operations. Below continuous utilisation the arithmetic almost never works, because you pay for the accelerator by the hour whether or not tokens are flowing through it.
The structural difference decides it. An API bills per token, so an idle system costs nothing. A self-hosted deployment bills per GPU-hour, so an idle system costs full price. That makes it a utilisation question, not a price question. A GPU running at fifteen percent utilisation has an effective per-token cost roughly seven times its cost at full utilisation, and most self-hosting business cases are built on the full-utilisation number and then operated at the low one.
Three cost categories get left out of every self-hosting spreadsheet I have reviewed. Redundancy: one instance is not a production deployment, so the real comparison is at least two, plus headroom for the traffic peak you cannot burst through. Operations: model updates, inference server upgrades, quantisation decisions, batching configuration, evaluation after every change. This is a recurring engineering load, not a one-off setup. And quality drift: an open model that matches a frontier model on your evals today may not after the next frontier release, and the migration back is not free.
So do the honest version of the calculation. Take your current API spend at published rates. Take a real quoted hourly rate for the accelerator you would actually reserve, on the commitment term you would actually sign. I am deliberately not quoting a GPU list price here, because they move faster than anything else in this post and a stale number would be worse than none. Multiply by the number of instances redundancy requires. Add a fraction of an engineer's loaded cost for operations. Then compare, applying a realistic utilisation factor to the self-hosted side rather than assuming full load.
Where self-hosting genuinely wins is narrower than the internet suggests: a single high-volume task with a stable prompt shape, an open model measurably good enough on your evals, traffic steady enough to keep accelerators busy, and a hard requirement like data residency, a latency floor, or a per-token price at a volume where nothing else closes. Meeting three of those four is not enough. This is the same build-versus-buy structure that governs voice infrastructure, worked through in how I cut a $200K/year cloud bill by more than 70%.
- Billed per token, so an idle system costs zero
- Prompt caching at one tenth of input and batch at 50% are available immediately
- Model upgrades are a string change, not a migration
- No accelerator capacity to reserve, monitor or fail over
- Per-token price is fixed and public, so forecasting is trivial
- You inherit the vendor's rate limits and their roadmap
- Billed per GPU-hour, so utilisation is the entire economic question
- Redundancy means at least two instances plus peak headroom
- Ongoing operations load: serving stack, quantisation, batching, evals
- Full control over data residency, latency floor and model version pinning
- Quality drift risk against each new frontier release
- Marginal token cost approaches zero once accelerators are saturated
Why do teams optimise the model first when context is the real driver?
Because the model name is the only variable that is visible without instrumentation. It appears in the code, in the vendor's pricing page and in every conversation about AI costs. Context assembly is distributed across a retrieval layer, a memory layer and a prompt template, and nobody sees its token contribution until somebody logs it.
There is also a cognitive pull. Swapping a model feels like a decision: discrete, reversible, attributable. Auditing why your retriever returns twenty chunks when five would do is diffuse work with no obvious owner, and it needs an evaluation harness to do safely. So the discrete decision gets made and the diffuse one gets deferred, even when the diffuse one is worth several times more.
The consequence is a specific and very common failure. A team downgrades from a frontier model to a mid tier, saves perhaps thirty percent of the model line, and takes a quality regression that shows up two weeks later as increased retries, longer conversations and more human escalations. Retries and longer conversations are additional inference calls. It is entirely possible to downgrade your model, degrade your product, and end the quarter with a higher bill.
The correct sequence: attribute, then cache, then trim, then batch, then route. Attribution tells you which request classes matter. Caching cuts the cost of the context you are already sending with no quality effect. Trimming reduces how much you send, guarded by evals. Batching halves the cost of everything nobody is waiting on. Routing, the move everyone starts with, comes fifth, after the four levers that carry no quality risk.
One organisational note that outperforms every technical lever here. Put a per-request cost number on the same dashboard as your latency and quality metrics, visible to the people writing the prompts. Prompts are written by engineers who currently have no cost signal, and the moment they get one, prompt bloat stops accumulating on its own. That is the same conclusion I reach about infrastructure spend generally, and the reason the cloud cost calculator exists as a self-serve tool rather than a lead form.
LLM inference costs: common questions
→Why are output tokens more expensive than input tokens?
Because each output token requires its own forward pass through the model, while input tokens are processed in parallel during a single prefill. Anthropic's August 2026 published pricing puts output at exactly five times input at every tier: $5 and $25 per million for Claude Opus 5, $2 and $10 for Sonnet 5, $1 and $5 for Haiku 4.5. OpenAI's published table has the same shape with output priced at several times input.
→How much does prompt caching save on LLM costs?
Anthropic publishes cache reads at 0.1 times the base input price, five-minute cache writes at 1.25 times and one-hour writes at 2 times. So the short cache pays back after a single read and the long one after two. On a workload where a large stable prefix is reused across most requests, that is close to a tenfold reduction on the input line with no effect on output quality.
→Does model routing actually reduce LLM costs?
Yes, substantially, if you route on request class rather than predicted difficulty. On a mix of roughly sixty percent extraction and classification, thirty percent conversational and ten percent hard reasoning, routing across published Claude tiers cuts the blended rate by roughly seventy percent. Watch the fallback rate: a routed class that fails validation more than about five percent of the time costs more than it saves once you add the retry.
→Is self-hosting an LLM cheaper than using an API?
Only with sustained, predictable, high-volume traffic on a task an open model handles well, plus someone to own GPU operations. APIs bill per token so idle costs nothing; self-hosting bills per GPU-hour so idle costs full price. Build the comparison on a real quoted accelerator rate at your actual commitment term, multiplied by redundancy, divided by your realistic utilisation, not on the full-utilisation number.
→What is the fastest way to cut an LLM bill?
Turn on prompt caching for every stable prefix and move every workload nobody is waiting on to the batch API. Those two changes are roughly two engineer-days, are backed by published multipliers of 0.1x on cache reads and 50% off batch tokens, and carry no quality risk at all. Do them before you consider changing models.
→Should you cut context to save on inference costs?
Yes, but only behind an evaluation harness. Retrieved context and replayed conversation history are usually the largest token contributors, so trimming them has real leverage. The risk is that you cut recall and pay it back in retries, longer conversations and human escalations, all additional inference calls. Cache first, since caching cuts the cost of context without removing any of it.