Blog · 2026-07-23 · Vynaris Team
Confidence-gated cascade routing: the double-spend behind the 65-85% savings claim
A confidence-gated cascade pays the cheap model on every call plus the premium on escalations. The real savings formula and break-even escalation rate, on live 2026 prices.
A confidence-gated cascade runs a cheap model on every call, scores how sure it is, and escalates only the low-confidence answers to a premium model. On the escalated fraction you pay twice: the cheap attempt plus the premium re-run. That double-spend means your savings are not 1 minus escalation rate; they are (1 minus the cheap/premium cost ratio) minus the escalation rate. On live prices verified 2026-07-23, the popular 65-85% cost-cut claim only holds when the cheap tier costs near zero. Swap in a cloud cheap model and the same escalation rate saves far less.
TL;DR
- Cascade savings vs always-premium follow one formula:
S = (1 - r) - e, whereris the cheap model's per-call cost divided by the premium's, andeis the escalation rate. The ceiling is1 - r; every point of escalation subtracts a point. - The Hacker News claim making the rounds (route 15-35% of queries up, cut API cost 65-85%) checks out only for an on-device cheap tier where
ris ~0. On our task shape it computes to 85% at 15% escalation and 65% at 35% escalation. With a cloud Haiku 4.5 cheap tier (r = 0.20), the same 15-35% escalation saves 65% down to 45%, not 65-85%. - A separate judge gate is paid on every call and can push the cascade underwater. Cheap Haiku 4.5 into premium Sonnet 5 with a Haiku judge loses 3.1% versus just always running Sonnet 5 at a 25% escalation rate. Prices verified 2026-07-23.
The finding, before the mechanics
A hybrid on-device project on Hacker News this week (75 points, 12 comments, read 2026-07-23) claims it "taught a small model to know when it's wrong," routes only 15-35% of queries to a premium cloud model, and cuts API cost 65-85%. The number is real. What the headline hides is why it is real, and the reason does not transfer to a pure-cloud cascade.
A cascade is not picking a cheaper model per request. It is sequential: the cheap model answers first, on every call. A confidence gate then accepts that answer or escalates it to a premium model, which reprocesses the input from scratch. Every escalated call carries two bills, not one. This is the model-routing pattern with a hidden tax, and the tax is the whole story below. Every dollar figure comes from a provider's live pricing page (captured 2026-07-23) or the arithmetic shown.
What a cascade actually pays
Define the per-call cost of the cheap model as c_cheap and the premium as c_prem. With escalation rate e (the fraction of calls the gate sends up) and a per-call gate overhead g:
cascade cost per call = c_cheap + g + e * c_prem
always-premium per call = c_premThe cheap model and the gate are paid on every call. The premium is paid on the escalated fraction e. Divide through by c_prem and the savings versus always-premium collapse to a two-term formula:
S = 1 - (c_cheap + g + e*c_prem) / c_prem
= (1 - r - g/c_prem) - ewhere r = c_cheap / c_prem. With a self-scored gate (the cheap model reports its own confidence, so g is ~0), this simplifies to S = (1 - r) - e. Two levers, and both cut the same direction:
1 - ris the ceiling: the most you can save, reached only if you escalate nothing.eis the haircut: each point of escalation is one point off the ceiling, because you double-pay those calls.
The break-even escalation rate, where the cascade stops beating always-premium, is e* = 1 - r - g/c_prem. Above it, you are paying two bills to do one job.
The workload (edit these)
Assumption Value Note
------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------ -------------------------------------
Representative task 1,500 [input tokens](https://vynaris.com/glossary/input-tokens) / 500 [output tokens](https://vynaris.com/glossary/output-tokens) one request in and out
Premium model [Opus 4.8](https://vynaris.com/models#claude-opus-4-8) at $5 / $25 per 1M the model you would run on every call
Cheap candidates Haiku 4.5, [Gemini 3.5 Flash-Lite](https://vynaris.com/models#gemini-3-5-flash-lite), [DeepSeek v4-flash](https://vynaris.com/models#deepseek-v4-flash), on-device the first-pass model
Escalation rate `e` the lever fraction the gate sends to premium
Gate overhead `g` $0 self-scored, else a judge call paid on every callAt $5 per 1M input and $25 per 1M output, Opus 4.8 costs (1,500 x $5 + 500 x $25) / 1,000,000 = $0.02 per call. That is the always-premium baseline every cascade below is measured against. Prices verified 2026-07-23.
The cheap-to-premium ratio decides the ceiling
r is the only thing that sets how much a cascade can save. Here it is for four cheap tiers against an Opus 4.8 premium, on the task shape above.
Cheap tier Cost per call `r` vs Opus 4.8 Savings ceiling `1 - r` Break-even escalation `e*`
----------------------------- ------------- --------------- ----------------------- --------------------------
On-device (local small model) ~$0.00000 0.000 100.0% 100.0%
DeepSeek v4-flash $0.00035 0.018 98.2% 98.2%
Gemini 3.5 Flash-Lite $0.00170 0.085 91.5% 91.5%
Haiku 4.5 $0.00400 0.200 80.0% 80.0%An on-device cheap tier has no marginal API cost, so its ceiling is 100% and it never loses to always-premium until you escalate every call. A cloud Haiku 4.5 tier costs a fifth of Opus 4.8 per call, so its ceiling is 80%: even escalating nothing, you still pay the Haiku bill on the whole volume. That gap between a 100% ceiling and an 80% ceiling is exactly what separates the on-device demo from your cloud rebuild.
Stress-testing the 65-85% claim
Now put the escalation rate back in and check the headline directly. The claim is 15-35% escalation yields a 65-85% cut. Using S = (1 - r) - e:
Cheap tier Savings at 15% escalation Savings at 35% escalation
------------------------------- ------------------------- -------------------------
On-device (`r` ~ 0) 85.0% 65.0%
DeepSeek v4-flash (`r` = 0.018) 83.2% 63.3%
Haiku 4.5 (`r` = 0.20) 65.0% 45.0%The on-device row reproduces the claim to the point: 85% at 15%, 65% at 35%. It works because r is ~0, so savings are just 1 - e. The moment the cheap tier is a billed cloud API, the ceiling drops by r and the whole band slides down. A Haiku 4.5 cascade at the same 15-35% escalation saves 65% to 45% — the top of its range equals the bottom of the on-device range. The double-spend on the cheap tier ate 20 points.
DeepSeek v4-flash is the interesting middle: at r = 0.018 it is cheap enough that its cascade behaves almost like on-device (83.2% down to 63.3%). If your cheap tier has to be a cloud model and quality allows it, the cheapest cloud model preserves the most of the ceiling.
The escalation-rate sensitivity
Savings decay linearly in the escalation rate, so the gate's calibration is the entire game. Here is a Haiku 4.5 to Opus 4.8 cascade across the range, self-scored gate.
Escalation rate `e` Cascade $/1,000 calls Savings vs always-Opus
------------------- --------------------- ----------------------
0% $4.00 80.0%
10% $6.00 70.0%
25% $9.00 55.0%
50% $14.00 30.0%
80% $20.00 0.0%
100% $24.00 -20.0%At 100% escalation you pay Haiku on all 1,000 calls plus Opus on all 1,000, so the cascade costs 20% more than just calling Opus directly. The break-even is 80%: below it the cascade wins, above it you would have been cheaper going straight to premium. A gate that escalates aggressively to protect quality can quietly walk you past that line. Drop your own token shape and escalation rate into the calculator to find your version of this table.
When the gate costs more than it saves
The self-scored gate assumption (g = 0) is the best case. Many teams run a separate judge: a small model reads the cheap answer and decides whether to escalate. That judge is a call on every request. Price it as a Haiku 4.5 LLM-as-judge reading the 1,500-token prompt plus the 500-token draft and emitting a 50-token verdict: (2,000 x $1 + 50 x $5) / 1,000,000 = $0.00225 per call. Against an Opus 4.8 premium that is g/c_prem = 0.1125, so it shaves 11.3 points off every break-even.
The judge is survivable when the premium-to-cheap gap is wide. It is fatal when the gap is narrow. Run a Haiku 4.5 cheap tier into a Sonnet 5 premium (introductory $2/$10, so $0.008 per call) instead of Opus 4.8:
Cascade (Haiku 4.5 cheap -> Sonnet 5 premium) Break-even escalation `e*` Savings at 25% escalation
--------------------------------------------- -------------------------- -------------------------
Self-scored gate (`g` = 0) 50.0% 25.0%
Separate Haiku judge (`g` = $0.00225) 21.9% -3.1%With r = 0.5 the ceiling is only 50% before you escalate anything. Add the judge and the break-even falls to 21.9%: escalate more than about one call in five and the cascade loses money. At a 25% escalation rate it is already 3.1% underwater versus just always running Sonnet 5. The rule: a cascade needs both a wide premium-to-cheap cost gap and a near-free gate. Kill either and the double-spend wins.
The accuracy floor you cannot escape
The escalation rate is not a free dial. Lowering e to save more ships more low-confidence answers from the cheap model. The gate's job is to escalate exactly the calls the cheap model would get wrong. A gate with poor recall either escalates too much (you pay the double-spend and lose savings) or too little (you ship worse answers and call it a saving). The on-device pitch "know when it's wrong" is a bet on high gate recall, and that calibration, not the routing structure, earns the savings. Measure the gate's escalate-when-wrong rate on your own traffic before trusting any headline percentage.
How this differs from the other routing posts
The double-spend is what makes runtime cascade math distinct. It is not offline downroute detection: replaying logs to find which share a cheap model could have handled has no double-spend, because each call goes to exactly one model — the recipe in which LLM calls a cheaper model could handle. It is not upfront task-type routing, where a classifier sends each request to one model by type, as in the coding-agent model-routers savings math. And it is a different axis from the labor break-even in when token optimization is not worth it: that asks whether the engineering pays back at your volume; this asks whether the routing structure saves once the double-spend is counted. We audited inflated vendor router claims the same way in the router savings claims audit.
When NOT to run a cascade
- Narrow premium-to-cheap gap. If
ris above ~0.4, the ceiling is already thin and a modest escalation rate erases it. Just run the premium model, or pick a cheaper cheap tier to widen the gap. - You need a separate judge and the gap is not huge. A billed judge on every call is a flat tax. If it pushes the break-even below your real escalation rate, the cascade is a net loss. Prefer a self-scored gate (logprobs, the model's own confidence) whenever the quality bar allows.
- Escalation you cannot hold down. If protecting quality forces you past
e* = 1 - r - g/c_prem, the cascade costs more than always-premium. Confirm your gate stays under it. - Low volume. The gate and escalation logic are engineering. On a few thousand calls a month the build cost swamps any per-call saving; pay the premium bill and skip the cascade.
Doing this on Vynaris
The formula above is all you need to model a cascade yourself; this section is the paid shortcut, clearly labeled. Vynaris is an OpenAI-compatible gateway that runs the cascade for you: cheap-first attempt, a self-scored confidence gate (no separate judge call), escalation only above your confidence threshold, and per-call reporting of the realized escalation rate and savings against the always-premium baseline. You see the measured e and S, not a demo number. See the docs and pricing.
FAQ
What is confidence-gated cascade routing? A pattern where a cheap model answers every request first, a confidence gate scores that answer, and only low-confidence ones escalate to a premium model, which reprocesses the input. It trades a double-spend on escalated calls for a cheap first pass on everything else.
Why don't cascade savings equal `1 minus the escalation rate`? The cheap model is paid on every call, not just the non-escalated ones. Savings are (1 - r) - e, where r is the cheap-to-premium per-call cost ratio. Only when the cheap tier costs ~0 (on-device) does r vanish and savings approach 1 - e.
Does the 65-85% cost-cut claim hold? Only for an on-device cheap tier where r is ~0. On our 1,500-in/500-out task that is 85% at 15% escalation and 65% at 35%. A cloud Haiku 4.5 cheap tier (r = 0.20) saves 65% down to 45% over the same band. Prices verified 2026-07-23.
When does a cascade lose money? Above the break-even escalation rate e* = 1 - r - g/c_prem. A Haiku 4.5 to Opus 4.8 cascade breaks even at 80% escalation; a Haiku 4.5 to Sonnet 5 cascade with a separate Haiku judge breaks even at 21.9% and is already 3.1% underwater at 25%.
Self-scored gate or separate judge? A self-scored gate (the model's own confidence or logprobs) adds no per-call cost. A separate judge is a billed call on every request and lowers every break-even by g/c_prem, 11.3 points against an Opus 4.8 premium here. Use a judge only when the premium-to-cheap gap is wide enough to absorb it.
Sources
- Hacker News, on-device hybrid cascade routing thread (75 points / 12 comments, read 2026-07-23): https://news.ycombinator.com/item?id=49010782
- Hacker News, "Smart model routing in Claude, Codex, Cursor" (216 points / 113 comments, read 2026-07-23): https://news.ycombinator.com/item?id=48688700
- Anthropic pricing (Opus 4.8 $5/$25, Haiku 4.5 $1/$5, Sonnet 5 introductory $2/$10), captured 2026-07-23: https://platform.claude.com/docs/en/about-claude/pricing
- DeepSeek API pricing (v4-flash $0.14/$0.28), captured 2026-07-23: https://api-docs.deepseek.com/quick_start/pricing
- Google Gemini API pricing (3.5 Flash-Lite $0.30/$2.50), captured 2026-07-23: https://ai.google.dev/gemini-api/docs/pricing
- Cost model script and all per-call arithmetic: derived from the assumptions table above.
Prices change. We re-verify every figure in this post monthly and stamp updates. Numbers here are current as of 2026-07-23.
The lesson: a cascade is not free money because it is cheap-first. Savings are (1 - r) - e. Widen the premium-to-cheap gap, keep the gate self-scored, and hold the escalation rate under the break-even, or skip the cascade and pay the premium bill.