How to Cut LLM API Costs by 80%: Caching, Routing and Context Budgets

Most teams discover their inference bill the way you discover a leak: through the invoice. A prototype that costs a fraction of a cent per call feels free, right up to the point where it is running a hundred thousand times a day and someone in finance wants a word.

The good news is that the three biggest levers are structural, they compound, and none of them require a worse product. This article works one realistic scenario end to end with current published prices, so you can follow the arithmetic and substitute your own numbers.

The scenario

A customer support assistant. Every request carries a 12,000-token system prompt plus retrieved product documentation, the user asks a 300-token question, and the assistant returns roughly 400 tokens. It handles 100,000 requests a day.

The prices below are Anthropic’s published rates as of September 2026, taken from the official pricing page. The technique matters more than the specific figures — every major provider prices along the same axes.

Baseline: the naive implementation

Everything goes to Sonnet 5, full context sent fresh every time.

• Input: 12,300 tokens × $2/MTok = $0.0246
• Output: 400 tokens × $10/MTok = $0.0040
• Per request: $0.0286

At 100,000 requests a day that is $2,860 per day, or roughly $85,800 a month. Note where the money goes: 86% of it is input, and almost all of that input is the same 12,000 tokens re-sent on every single call.

Lever 1: prompt caching

If a large block of context is identical across requests, you can cache it. Subsequent reads of that block are charged at a fraction of the input rate — for Sonnet 5, $0.20 per million instead of $2, a tenfold reduction.

• Cached read: 12,000 × $0.20/MTok = $0.0024
• Fresh input: 300 × $2/MTok = $0.0006
• Output: 400 × $10/MTok = $0.0040
• Per request: $0.0070

That is $700 per day, about $21,000 a month. A 75% reduction from one change that alters nothing the user sees.

Writing to the cache costs more than a normal input token, which is the part people miss. At $2.50 per million for Sonnet 5, each write of our 12,000-token block costs $0.03. With a five-minute cache lifetime and steady traffic you would rewrite it roughly 288 times a day: about $8.64 daily, against $2,160 saved. The write cost only matters if your traffic is so sparse that the cache expires between requests — below roughly one request per cache lifetime, caching costs you money instead of saving it.

The structural requirement: the cached block must be byte-identical and must sit at the front of your prompt. Injecting a timestamp or the user’s name above the system prompt silently invalidates the cache on every request, and the bill looks exactly like the uncached case with no error to tell you.

Lever 2: model routing

Not every request needs your best model. In a support workload, a large share is order status, password resets, opening hours — retrieval plus light paraphrasing. Sending those to Haiku 4.5 rather than Sonnet 5 halves output cost and cuts cached input cost in half again.

Same request on Haiku 4.5 with caching:

• Cached read: 12,000 × $0.10/MTok = $0.0012
• Fresh input: 300 × $1/MTok = $0.0003
• Output: 400 × $5/MTok = $0.0020
• Per request: $0.0035

Assume a classifier routes 60% of traffic to Haiku and keeps 40% on Sonnet:

(0.6 × $0.0035) + (0.4 × $0.0070) = $0.0049 per request — $490 per day, roughly $14,700 a month.

The routing classifier itself costs something, but a small model deciding between two labels on a 300-token input is measured in hundredths of a cent, so it disappears into the rounding.

The real cost of routing is not money, it is the misroute. Send a hard question to the small model and the user gets a worse answer. Which means routing is only safe if you can measure quality per route — a point that comes back below.

Lever 3: the context budget

The third lever is the one teams reach for last and should reach for first: send less.

That 12,000-token block deserves an audit. In most systems it accumulated rather than being designed — a system prompt that grew by a paragraph per incident, retrieval configured to return the top twenty chunks because someone set it during testing and never revisited it, few-shot examples that predate the current model.

Cutting retrieval from twenty chunks to eight often improves answers, because relevant material stops competing with marginally related text. If that reduction takes the cached block from 12,000 tokens to 6,000, every cached-read figure above halves again.

Before you trim, though, you need to know what a good answer looks like, which is the thread running through all three levers.

Why none of this works without evaluation

Every optimisation here trades some quality risk for money. Routing can misroute. Trimming context can remove the passage that mattered. Caching is the only genuinely free one, and even that constrains how you structure prompts.

Without a test set you cannot tell a successful optimisation from a quiet regression, because both look identical on the invoice. Teams that cut costs and keep quality all have the same thing in common: they measured before they changed anything. Fifty labelled examples is enough to start.

It is also worth checking that retrieval is sound before you optimise around it. If the pipeline is returning the wrong passages, you are paying to send the wrong context — the eight failure modes we measured covers how to find that out.

A sensible order of operations

1. Measure first. Cost per request broken down into input, cached input and output. You cannot optimise a number you do not have.
2. Build a small evaluation set so you can detect quality changes.
3. Add prompt caching. Biggest win, lowest risk. Restructure prompts so the stable block sits first and never varies.
4. Audit the context. Free, and often improves quality.
5. Introduce routing once you can measure per-route quality. Start conservative and widen as the data supports it.
6. Batch anything not interactive. Batch processing is 50% off, and reports, backfills and bulk classification rarely need to be synchronous.

Frequently asked questions

Does prompt caching change the model’s output?

No. It is a billing and latency optimisation on identical input. The model sees the same tokens either way.

When is caching not worth it?

When traffic is sparse enough that the cache expires between requests, or when the shared block is small. Writes cost more than ordinary input tokens, so a cache that rarely gets read is a net loss.

Is a fine-tuned small model cheaper than routing?

Sometimes, at high and stable volume. But it adds a training and maintenance burden, and it locks you to a model version. Exhaust caching, context trimming and routing first — they are reversible in an afternoon.

How do I estimate before building?

Count tokens in a representative prompt, multiply by expected volume, then double it. Real systems retry, carry conversation history, and get used more than planned.

The short version

Caching, routing and a context audit took a hypothetical $85,800 monthly bill to $14,700 without touching the product. The arithmetic is simple enough to do on your own numbers in ten minutes, and it is worth doing before you need to.

Prices verified against Anthropic’s published rates in September 2026 and subject to change. More in System Design.