Cost & economics

How can we reduce our LLM API costs?

Short answer

The four techniques that move LLM costs most are caching, model routing, context trimming and prompt-prefix reuse. Exact-match plus semantic caching typically removes 25 to 45 percent of calls in assistant workloads. Routing simple requests to a smaller model and reserving the largest model for hard synthesis often halves the remaining spend. Trimming retrieved context from ten passages to four usually improves answer quality while cutting input tokens. Applied together these commonly reduce inference bills by 60 to 80 percent, but each change has to be verified against an evaluation set, because cheaper routing quietly degrades some task types.

Last reviewed 10 June 2026

Measure before optimising

Almost every team I audit finds that a small fraction of requests accounts for most of the spend. Common culprits: a background job re-summarising documents that have not changed, a retrieval step returning twenty passages when four would do, a debug code path left in production, or one internal power user running bulk queries through an interactive endpoint.

Attribute cost per request, per feature and per user before changing anything. The first week of visibility usually pays for the work.

Caching

Exact-match caching on identical prompts. Trivially cheap, and in assistant-style products the repeat rate is higher than anyone expects, because the same twelve questions dominate.

Semantic caching on near-identical questions, using an embedding similarity threshold. Effective, but set the threshold carefully and evaluate it: too loose and you serve the answer to a subtly different question, which is a correctness bug rather than a saving.

Prompt-prefix caching where the provider supports it. Your system prompt and instruction block are identical on every call; providers charge substantially less for cached prefix tokens. This is usually a configuration change with no quality risk.

Typical combined effect: 25–45% fewer billed calls.

Model routing

Not every request needs the largest model. A classifier, or often just a heuristic on query type and length, routes straightforward lookups to a small model and reserves the expensive one for genuine synthesis.

This is the largest single lever, frequently halving remaining spend. It is also the one that most needs evaluation: quality degradation from routing is invisible in aggregate metrics and shows up only in the task types you did not check. Run the full evaluation suite per route.

Context trimming

Retrieval systems tend to over-fetch. Sending ten passages when four contain the answer costs you on input tokens and usually produces a worse answer, because relevant material competes with noise.

Better reranking lets you send fewer passages with higher confidence. This is one of the rare changes that improves quality and cost simultaneously.

Also check: are you resending the full conversation history every turn? Summarise older turns instead.

Output length

Output tokens usually cost several times more than input tokens. Instructing for concision, and setting sensible max_tokens, is free money on verbose systems.

Structural options

Batch APIs for anything not interactive, typically around half price at the cost of latency.
Self-hosting a small model for high-volume narrow tasks. Only worth it at sustained volume; below roughly €8,000 a month of inference the engineering and operational cost usually exceeds the saving.
Provider negotiation. At meaningful volume, committed-use pricing is available and rarely offered unprompted.

The rule

Every cost optimisation is a quality experiment. Run it against your evaluation set, compare the numbers, and keep the change only if quality holds. A 70% saving that costs five points of groundedness is not a saving. It is a decision to ship a worse product, and it should be made deliberately if at all.