---
title: How can we reduce our LLM API costs?
source: https://rubradigital.com/answers/how-to-reduce-llm-api-costs
site: Rubra Digital
description: Concrete techniques for cutting LLM inference costs: caching, model routing, and context trimming, with realistic savings for each.
updated: 2026-06-10
topic: cost
also_asked: Why is our OpenAI bill so high? | Does caching work for LLM applications? | Should we use a smaller model?
---

# How can we reduce our LLM API costs?

**Summary:** The four techniques that move LLM costs most are caching, model routing, context trimming and prompt-prefix reuse. Exact-match plus semantic caching typically removes 25 to 45 percent of calls in assistant workloads. Routing simple requests to a smaller model and reserving the largest model for hard synthesis often halves the remaining spend. Trimming retrieved context from ten passages to four usually improves answer quality while cutting input tokens. Applied together these commonly reduce inference bills by 60 to 80 percent, but each change has to be verified against an evaluation set, because cheaper routing quietly degrades some task types.

## Measure before optimising

Almost every team I audit finds that a small fraction of requests accounts for
most of the spend. Common culprits: a background job re-summarising documents
that have not changed, a retrieval step returning twenty passages when four
would do, a debug code path left in production, or one internal power user
running bulk queries through an interactive endpoint.

Attribute cost per request, per feature and per user before changing anything.
The first week of visibility usually pays for the work.

## Caching

**Exact-match caching** on identical prompts. Trivially cheap, and in
assistant-style products the repeat rate is higher than anyone expects, because
the same twelve questions dominate.

**Semantic caching** on near-identical questions, using an embedding similarity
threshold. Effective, but set the threshold carefully and evaluate it: too loose
and you serve the answer to a subtly different question, which is a correctness
bug rather than a saving.

**Prompt-prefix caching** where the provider supports it. Your system prompt and
instruction block are identical on every call; providers charge substantially
less for cached prefix tokens. This is usually a configuration change with no
quality risk.

Typical combined effect: **25–45% fewer billed calls.**

## Model routing

Not every request needs the largest model. A classifier, or often just a
heuristic on query type and length, routes straightforward lookups to a small
model and reserves the expensive one for genuine synthesis.

This is the largest single lever, frequently halving remaining spend. It is also
the one that most needs evaluation: quality degradation from routing is
invisible in aggregate metrics and shows up only in the task types you did not
check. Run the full evaluation suite per route.

## Context trimming

Retrieval systems tend to over-fetch. Sending ten passages when four contain the
answer costs you on input tokens and usually produces a _worse_ answer, because
relevant material competes with noise.

Better reranking lets you send fewer passages with higher confidence. This is
one of the rare changes that improves quality and cost simultaneously.

Also check: are you resending the full conversation history every turn?
Summarise older turns instead.

## Output length

Output tokens usually cost several times more than input tokens. Instructing for
concision, and setting sensible `max_tokens`, is free money on verbose systems.

## Structural options

- **Batch APIs** for anything not interactive, typically around half price at
  the cost of latency.
- **Self-hosting a small model** for high-volume narrow tasks. Only worth it at
  sustained volume; below roughly €8,000 a month of inference the engineering
  and operational cost usually exceeds the saving.
- **Provider negotiation.** At meaningful volume, committed-use pricing is
  available and rarely offered unprompted.

## The rule

Every cost optimisation is a quality experiment. Run it against your evaluation
set, compare the numbers, and keep the change only if quality holds. A 70%
saving that costs five points of groundedness is not a saving. It is a decision
to ship a worse product, and it should be made deliberately if at all.

---

Source: https://rubradigital.com/answers/how-to-reduce-llm-api-costs
Rubra Digital. Independent LLM and RAG consulting for regulated and document-heavy organisations in Europe and North America.
Contact: hello@rubradigital.com