LLM Cost Optimization – Seven Levers That Actually Move the Bill

Practical ways to cut LLM API spend without downgrading quality, ordered by how much they typically save. Caching, routing, batching, and the mistakes that inflate bills quietly.

September 7, 2026 • 10 min read

costcachingarchitecture

Most advice about LLM costs starts and ends with “use a cheaper model.” That is the least interesting lever available, and often the one that costs you the most in quality. Below are seven that work, roughly ordered by how much they typically save on a real production workload.

A note on why this ordering matters: token prices have fallen dramatically and keep falling — one budget tier dropped 80% in a single day in July 2026. Architectural savings compound with those cuts. Model-swapping savings get erased by them.

1. Prompt caching

Typical saving: 40–80% of input cost.

Cached input bills at roughly a tenth of the base input rate. If your application resends a large stable prompt on every turn — a system prompt, tool definitions, retrieved documents, a codebase snapshot — you are currently paying full price to send identical bytes repeatedly.

The requirement is that the cached prefix must be byte-identical. This has a practical consequence people miss: anything dynamic must go at the end. A timestamp, a session ID, or a user name interpolated near the top of your system prompt invalidates the entire cache on every request.

Restructure so the order is: stable system prompt → tool definitions → stable context → dynamic user input. Providers also offer different cache durations at different write costs, so match the window to your traffic pattern.

2. Route by difficulty

Typical saving: 50–90% on mixed workloads.

The spread between tiers within one provider’s family is frequently 20x or more. Most production traffic is easy. Sending all of it to a flagship model is the most common and most expensive mistake in the field.

A workable pattern:

Classify difficulty cheaply — heuristics on input length and task type get you surprisingly far, and a small model classifier handles the rest.
Send the easy majority to a budget tier.
Verify cheaply — a schema check, a test run, a confidence threshold.
Escalate only on failure.

Because the escalation path only fires on the hard minority, your blended cost lands near the cheap tier’s price while your quality ceiling stays near the flagship’s. Model the numbers for your own traffic mix in the cost calculator.

3. Cut output tokens, not input tokens

Typical saving: 20–50%.

Output costs five to six times what input costs. Teams reflexively trim prompts and ignore responses, which is backwards.

Concrete moves: request structured output instead of prose explanations, set explicit length limits, stop asking the model to restate the question, and — for reasoning models — lower the effort setting on easy requests. That last one is the highest-leverage version of this lever, because hidden thinking tokens bill as output.

4. Batch anything asynchronous

Typical saving: 50% on eligible traffic.

Batch APIs are consistently half price across major providers, in exchange for a delayed, non-guaranteed completion window. Anything not blocking a user should be there: nightly enrichment, embeddings backfill, evals, bulk classification, report generation.

The mistake is treating batch as a fallback for cost emergencies rather than the default for background work.

5. Watch the long-context cliffs

Typical saving: avoids a silent 2x.

Several providers reprice the entire request once a prompt crosses a threshold. One current flagship charges $2 input per million tokens under 200K and $4 above it — and it applies to the whole prompt, not just the excess. Another applies higher rates past 272K tokens.

A prompt that drifts from 195K to 205K tokens can therefore double in cost with no visible change in behaviour. If you operate near a threshold, monitor prompt length as a first-class metric and compact aggressively before the boundary rather than after.

This is also why “we’ll just use the million-token window” is rarely the cheap answer. Retrieval and compaction remain cheaper than brute-force context stuffing, and long-context recall degrades enough that they are often more accurate too.

6. Count agent loop calls

Typical saving: highly variable, occasionally enormous.

A single agent task may make hundreds of model calls. Teams budget for the calls they wrote and get billed for the loop’s actual behaviour, which is usually several times higher.

Instrument call count per completed task before you optimise anything else about an agent. It is common to find retry logic firing far more often than expected, or a verification step that re-reads the entire context each time. Fixing loop structure often beats every other lever combined. See the agents guide for the architectural side of this.

7. Reconsider open weights at volume

Typical saving: depends entirely on utilisation.

Self-hosting trades per-token fees for GPU time, which means it wins only at sustained high volume — you pay for the GPU whether or not you use it. Open coding and general models have closed enough of the capability gap that this is now a real option rather than a compromise, and some run usefully on consumer hardware.

Run the arithmetic honestly: include engineering time, idle capacity, and the ceiling on quality. It is a genuine win for steady high-throughput workloads and a poor trade for spiky traffic. Compare the open-weight options before committing.

Two things that quietly inflate bills

Retries without backoff. A failed call that retries three times costs four times as much. Under provider rate limiting, naive retry logic can multiply spend during exactly the periods you can least afford it.

Forgotten evals. Running a full eval suite against a flagship model on every commit is a real recurring cost that appears in no one’s budget. Use a cheap tier for smoke tests and reserve the expensive suite for release candidates.

Where to start

If you do only one thing: instrument cost per completed task, broken down by model and by request type. Nearly every team that measures this finds at least one surprise — an endpoint using a flagship model for a trivial task, or an agent loop running three times more often than anyone thought. You cannot optimise a number you are not looking at.