How a 30K-user B2B SaaS cut LLM costs 30% and p95 latency 60%
Eight weeks. No model retraining. No platform migration. Just deliberate engineering on a stack that had grown faster than its design assumptions.
LLM spend ($80k → $56k / mo)
p95 latency (4.2s → 1.7s)
audit to fully shipped
THE SITUATION
A core feature was running a $80k/mo OpenAI bill — and growing.
The client is a B2B SaaS company serving 30,000+ daily active users. Their flagship product had an LLM-powered feature that — over six months — had become the thing their customers came back for.
By the time we got involved, that feature had three problems:
- The OpenAI bill had crossed $80k/month and was growing 12% month-over-month while user growth was 4%.
- p95 latency on the critical user-facing call was 4.2 seconds. Customer complaints were concentrated in the slowest decile.
- Engineering was spending most of every sprint patching prompts and chasing edge cases instead of shipping new features.
The CTO didn't want a platform. He wanted the bill cut, the latency cut, and his engineers back.
WHAT THE AUDIT FOUND
Three issues. None of them were the model.
We instrumented end-to-end tracing on the feature — token-level cost attribution per prompt, per-call latency profiling, and a sample-based quality eval. Within a week we had a baseline and three clear bottlenecks:
Every request was hitting GPT-4
Even queries that a smaller, cheaper model would have answered correctly. There was no router — every call went to the most expensive model by default.
~40% of completions were re-computations
The same or near-identical prompts were being computed dozens of times per hour. Nothing was cached. The cache hit rate on a workload that should have been ~50% was effectively zero.
Three sequential LLM calls were running in serial
The agent loop made three model calls per request. The first call's output didn't feed the next two — they could have run in parallel. Sequential execution was adding ~2.5s to every request.
WHAT WE BUILT
Three changes. Shipped behind flags. Measured against the audit baseline.
1. Model router with quality gating
We classified each request type by complexity and routed simple queries to GPT-4o-mini, medium queries to GPT-4o, and only the hardest queries to GPT-4. We backed every routing decision with a quality eval that ran on every deploy — if the smaller model regressed quality on a class of queries, the router would shift them back. The eval harness now runs on every PR.
2. Prompt-aware semantic cache
We added a cache keyed on a hash of the prompt template plus a semantic embedding of the variable inputs. Identical and near-identical requests returned cached responses. Cache hit rate climbed from <1% to 38% by week four. Each hit avoided a $0.04 LLM call and saved ~1.2s of latency.
3. Parallelized agent calls
The three-step agent flow had two calls that didn't depend on each other's output. We refactored the orchestration to run them concurrently. Time-to-first-byte on the user-facing response dropped by 1.8s on the median path and 2.5s on p95.
THE RESULT
Numbers held two weeks after rollout. Held again at twelve.
We measured against the audit baseline twice after rollout: at +2 weeks and at +12 weeks (regression check). The numbers held both times.
LLM spend
p95 latency
cache hit rate
cost growth (now flat)
The team got their sprint capacity back. In the quarter that followed, they shipped two new features that had been blocked behind AI-firefighting work. Both used the same model router and cache we'd built — at no marginal cost increase.
Your stack probably has the same three issues.
Most production AI systems we scope have at least two of: no model routing, no caching, and serial calls that could be parallel. A scoping call will tell you which.