Choose a unit and a ceiling on day one
For a deflection assistant at Tinsmith Retail the unit was a resolved contact and the benchmark was the fully loaded human cost of 4.20 pounds. We set the ceiling at 12 pence per resolved contact, including inference, retrieval, logging and the share of human review the design implied. Anything above that and the business case rested on headcount reduction nobody had actually agreed to.
The ceiling then constrains the architecture rather than the other way round. At 12 pence you can afford a retrieval step, a mid-sized model, and roughly one human review in twenty. You cannot afford three sequential model calls over 30,000 tokens of context per contact, so that design never gets prototyped. Deciding this on a whiteboard costs an afternoon; discovering it in month five costs the project.
Model the tokens explicitly, with units. Our first estimate for the assistant was 6,000 input and 800 output tokens per turn, at 2.4 turns per contact. Those three numbers are worth arguing over for an hour, because every later decision is downstream of them.
Where the money actually goes
Context bloat, almost always. Teams pass 20 retrieved chunks because retrieval returns 20, when precision at 5 was already 0.91 and the extra 15 chunks add 9,000 input tokens to every single call. Trimming that one setting cut per-contact cost by 38 per cent at Tinsmith and, unusually for a cost optimisation, improved answer quality — the model stopped being distracted by near-miss passages.
Prompt prefix caching is the next largest lever where the platform supports it. A 2,400-token system prompt with policy text and tool definitions is identical on every call; cached, it costs a fraction of the uncached rate. The engineering discipline is to keep the volatile parts — the date, the customer name — at the end of the prompt rather than sprinkled through it, which is a five-minute change most codebases never make.
Anything that does not need to be interactive should not be. Overnight document classification, back-catalogue enrichment and re-summarisation all run as batch jobs at materially lower rates and with no latency requirement to defend. We have moved 60 to 70 per cent of total token volume off the interactive path on two engagements simply by asking which requests a human is actually waiting for.
Routing, and knowing what the small model costs you
Tiered routing works, with a caveat. A cheap classifier sends the straightforward 70 per cent of traffic to a small model and escalates the rest on low confidence, long context or a sensitive intent. The saving is real — typically 45 to 60 per cent of inference spend.
The caveat is that you must evaluate each tier separately against the same set, or the saving is being funded by quality nobody is measuring. On one build the small model scored within a point of the large one on routine queries and eleven points worse on multi-document questions, which is exactly what the router was meant to catch. Without per-tier numbers, the router looked like free money.
Fallbacks also need a cost model. An escalation path that retries a failed small-model call on the large model doubles the cost of every failure. If failures are two per cent, that is noise; if a bad deployment pushes them to 30 per cent, the bill moves before the alert does.
A ceiling nobody enforces is a wish
Cost per request belongs on the same dashboard as p95 latency and error rate, refreshed hourly, broken down by tenant and by feature. Two of the three worst overspends we have been called in to fix were single tenants running automated loops nobody had rate-limited, and both were visible in the data for weeks before anyone looked.
Enforce with per-tenant token budgets and a hard monthly cap. When a tenant exceeds their allowance, the correct response is a 429 with a documented retry window and a queue for non-urgent work — not a silent 40,000-pound overage discovered by finance. Alert the delivery team at 70 per cent of monthly budget, which is early enough to change something.
Reprice quarterly. Model prices have fallen substantially and unevenly; a routing decision that was correct in January can be leaving money on the table by June, and a ceiling set against last year's rates may be far more generous than it needs to be.
The bill that is not tokens
Evaluation runs, human review time, log storage, vector index hosting and embedding regeneration are all real and all routinely omitted from the business case. Re-embedding a 1.2 million document corpus after an embedding model upgrade took eleven hours and cost around 900 pounds at Vantara Metals — once a year, forecastable, but not zero, and it needs to be in the plan before the finance director finds it.
Our rule of thumb is to add 15 to 20 per cent on top of modelled inference for everything that is not a token, and to name each line rather than hiding it in a contingency. A sponsor will forgive a cost they were shown in month one. They will not forgive the same number arriving as a surprise in month nine.