VynarisEarly betaGet your API key

Blog · 2026-07-18 · Vynaris Team

Model fallback chains: designing for provider outages without doubling your bill

Firing two models on every request buys resilience you need on under 1% of calls and pays for it on 100%, adding 16% to 36% forever. A sequential fallback that only calls the backup on real failure adds under half a percent. We price both patterns, plus the timeout and cache traps. Verified 2026-07-

The expensive way to survive a provider outage is to fire two models on every request and take whichever answers first. That buys resilience you need on under 1% of calls and pays for it on 100% of them, adding 16% to 36% to the bill forever. A sequential fallback that only calls the backup when the primary actually fails adds under half a percent. Same uptime, a fraction of the cost. Prices verified 2026-07-18.

A fallback chain is the routing you reach for when a provider returns a 503, rate-limits you, or simply stops answering. Primary fails, try the secondary; secondary fails, try the tertiary. Built well, it is nearly free insurance. Built as a parallel hedge, it is one of the quietest ways to double an inference bill. This guide prices both patterns from live rates, shows the timeout and cache traps that turn cheap insurance expensive, and gives working config you can paste in. Every number comes from a public pricing page and an editable script; nothing is measured from inside any product.

Two patterns, two very different bills

There are two ways to build failover, and they cost nothing alike.

Sequential. Send the request to the primary. If it errors or times out, send it to the secondary. You pay for a wasted primary attempt only on the small fraction of requests that fail.

Hedged (parallel). Send the request to the primary and the secondary at the same time, return the first success, and cancel or ignore the loser. You get the lowest possible latency, and you pay for a second model on every single request, whether the primary was healthy or not.

Hedging is the pattern people copy from low-latency web systems, where a duplicate database read is cheap. A duplicate LLM call is not cheap. It is the single most expensive resilience decision most teams make without noticing, because the premium hides in aggregate token spend rather than in a line labeled "failover."

The chain we price

One representative production generation call, and a three-link chain across three providers so a single vendor outage cannot take out the whole chain.

Link          Model                                                              Cost per 1,000 calls
------------  -----------------------------------------------------------------  --------------------
Primary       [gpt-5.6-sol](https://vynaris.com/models#gpt-5-6-sol)              $25.00
Secondary     [claude-sonnet-5](https://vynaris.com/models#claude-sonnet-5)      $9.00
Tertiary      [claude-haiku-4.5](https://vynaris.com/models#claude-haiku-4-5)    $4.50
Alt tertiary  [deepseek-v4-flash](https://vynaris.com/models#deepseek-v4-flash)  $0.42

Assumptions: 2,000 input tokens, 500 output tokens per call. Prices per million tokens, verified 2026-07-18: Sol $5/$30, Sonnet 5 $2/$10 (introductory, through 2026-08-31), Haiku 4.5 $1/$5, DeepSeek v4-flash $0.14/$0.28. Note the shape of the chain: every link down is cheaper than the one above it, so a failover does not just keep you up, it often costs less per call than the primary would have.

What a failover attempt actually costs

When the primary fails, you have usually already paid for something. The amount depends on how it fails.

That gap matters. A provider that fails fast and loud is cheap to fall back from. A provider that hangs is expensive, because you pay for a dead request and then pay again for the retry. This is why detection speed, not just the fallback itself, drives the cost.

Steady state: sequential is near-free, hedging is not

Take a 0.5% primary failure rate, a realistic figure for a healthy provider. Price both patterns per 1,000 requests.

pattern                         cost/1k    vs primary-only
----------------------------------------------------------
no fallback (primary only)      $25.00        -
sequential (fail -> secondary)  $25.01        +0.03%
hedged, take-first              $29.00        +16%
hedged, no cancel               $34.00        +36%
Steady-state cost of four resilience patterns per 1,000 requests: sequential fallback sits on the baseline while hedging adds 16 to 36 percent
Cost per 1,000 requests at a 0.5% primary failure rate. Sequential fallback adds 0.03%; hedging adds 16% (take-first) to 36% (no cancel). Prices verified 2026-07-18.

The sequential chain barely moves the baseline. On the 0.5% of requests that fail, you eat a wasted primary attempt but then run the cheaper secondary, and because Sonnet 5 at $9 is far below Sol at $25, the two nearly cancel. The wasted attempts themselves cost about $0.09 per 1,000 requests. That is the true price of the insurance.

Hedging pays the second model on every request. Take-first with cancellation still bills both inputs plus the winner's output, +16%. Without cancellation, both calls run to completion, +36%. There is no failure-rate assumption that rescues hedging, because the premium is charged on the healthy 99.5%, not the failing 0.5%. If you do not have a hard sub-second latency SLA that a sequential retry would blow, hedging is paying a permanent tax to insure against a rare event.

If you want to plug your own failure rate and model prices into this, run the two patterns through the calculator before committing to a hedge. The crossover where hedging is worth it is a latency-SLA decision, not a cost one.

The timeout budget decides everything

Sequential fallback only works if you can tell a dead primary from a slow one, and that judgment lives in the timeout. Set it wrong in either direction and the cheap pattern gets expensive.

During a real outage, a clean 503 lets you fail over instantly, and every request runs on the $9 secondary, so your bill during the outage is actually lower than normal. A gray failure, where the primary hangs until your timeout fires, costs the wasted attempt plus the secondary: $26.50 per 1,000, about 6% above normal. The timeout length is the difference between failover being free and being a surcharge.

Set the timeout too tight and you fail over on healthy-but-slow responses. If a 4-second timeout trips 5% of requests that would have succeeded at 6 seconds, you needlessly run the fallback on all of them:

event                                   added cost/1k
------------------------------------------------------
real outages at 0.5% failure            +$0.09  (+0.03%)
too-tight timeout tripping 5% healthy   +$1.33  (+5.3%)

A mis-tuned timeout costs an order of magnitude more than the outages it was meant to protect against, and it also ships the fallback model's output to users who did not need it. Set the timeout near the primary's p99 latency, not its median. Slow-but-fine is not an outage.

Set the timeout too loose and every request during an outage burns the full timeout before failing over, wrecking tail latency for the whole window. The budget wants to sit just above p99: tight enough to fail over quickly when the provider is truly down, loose enough not to abandon healthy requests.

Failover throws away your cache

The trap that surprises teams: a fallback resets your prompt cache. If the primary serves a 15,000-token stable prefix from cache at the 0.1x read rate, its warm cost is low. The secondary has never seen that prefix, so it reprocesses everything at full input price, and if you want the secondary warm for the rest of the outage you pay a cache write on top.

request                                  cost/1k   vs warm primary
------------------------------------------------------------------
normal primary (warm cache)              $32.50       -
1st failover request (cold secondary)    $39.00      1.2x
1st failover + warm the secondary cache  $76.50      2.4x
later failover requests (warm secondary) $12.00      0.4x

Failover cost is front-loaded. The first requests after you switch providers pay 1.2x to 2.4x while the secondary cache is cold, then settle below normal once it warms. For a long outage this amortizes fine. For a 30-second blip, you may pay every cache-write premium and never collect the read discount. The lesson: do not warm the secondary's cache reflexively on the first failed request. Warm it only once the outage looks sustained, or you buy an expensive cache you throw away seconds later. We worked the caching side of this in detail in prompt caching in production.

Fail over on the right errors only

Half of a cheap fallback is not calling the backup when the backup cannot help either. A 400 from a malformed request will fail on every provider; failing over just doubles the cost of a request that was never going to work.

# Retry on the secondary ONLY for provider-side failures.
FAILOVER_STATUS = {408, 429, 500, 502, 503, 504, 529}  # timeouts, rate limits, overload

def should_failover(exc):
    if isinstance(exc, (TimeoutError, ConnectionError)):
        return True
    code = getattr(exc, "status_code", None)
    if code in FAILOVER_STATUS:
        return True
    # 400/401/403/404/422 are YOUR bug or auth; the secondary will fail too.
    return False

A minimal sequential chain with a p99-tuned timeout and no failover on client errors:

CHAIN = ["gpt-5.6-sol", "claude-sonnet-5", "claude-haiku-4.5"]

def generate(prompt, timeout_s=15):          # ~p99 of the primary, not the median
    last = None
    for model in CHAIN:
        try:
            return call(model, prompt, timeout=timeout_s)
        except Exception as exc:
            last = exc
            if not should_failover(exc):
                raise                          # don't burn the chain on a 400
    raise last                                 # whole chain down: fail loud

That is the whole pattern. No hedge, no duplicate spend, one timeout, and a hard stop that surfaces a real outage instead of hiding it behind a third-rate answer.

Where a gateway fits

You can run this chain yourself. The DIY version above works, and for a single provider pair it is a few dozen lines. The cost shows up when the chain grows: each provider speaks a different error format, a different streaming protocol, and a different cache API, so a portable prompt and a unified retry layer become their own maintenance job. An OpenAI-compatible gateway collapses that to one base URL and one request shape, with the fallback order, timeout budget, and error rules configured rather than coded per provider. Vynaris does this and prices each attempt transparently, so a failover shows up as a line you can read instead of a mystery bump in next month's invoice. Whether you buy it or build it, the design rules in this post are the same; we walked through the build-versus-buy math in the LiteLLM vs OpenRouter vs managed gateways TCO breakdown.

Do not build a fallback chain if

The honest section. Failover is not free insurance in every case.

For the raw rates behind every model here, see the July 2026 LLM price list; for how model routing decisions play out on agent workloads, the coding-agent router savings math.

FAQ

Does a fallback chain double my LLM bill? Only if you build it as a parallel hedge that fires two models on every request, which adds 16% to 36%. A sequential chain that calls the backup only on failure adds under half a percent at a 0.5% failure rate.

Sequential or hedged fallback, which should I use? Sequential, unless you have a hard latency SLA that a retry would breach. Hedging buys lower tail latency at a permanent 16% to 36% premium charged on every healthy request, not just the failing ones.

What does a failed primary attempt cost? About $0 if the provider fails fast with a 5xx or 429 before inference, and roughly the input plus partial output if it times out mid-generation, about $17.50 per 1,000 calls on gpt-5.6-sol.

How should I set the failover timeout? Near the primary's p99 latency, not its median. Too tight and you fail over on healthy-but-slow responses, which in one model cost 5.3% versus 0.03% for real outages. Too loose and every request during an outage burns the full timeout.

Which errors should trigger failover? Provider-side failures only: timeouts, connection errors, 429, and 5xx. Never fail over on 400, 401, 403, or 422, because those are client-side and the secondary will reject the request too, doubling the cost of a call that was always going to fail.

Do I lose prompt caching when I fail over? Yes. The secondary has a cold cache, so the first requests after failover cost 1.2x to 2.4x normal while it warms. Warm the secondary only once an outage looks sustained, or a brief blip makes you pay cache-write premiums you never recover.

Sources

Related: LiteLLM vs OpenRouter vs managed gateways TCO · Prompt caching in production · July 2026 LLM price list