Blog · 2026-11-08 · By Dhrumil Barot, Founder

LLM Routing Examples: 7 Rules With Code and Cost Math

LLM routing examples with code and per-1,000-request costs: rule-based, cascade, classifier, fallback, context-length, price-tier and RouteLLM.

TL;DR: An LLM router sends each request to the cheapest model that can handle it. On a 1,500-token-in, 400-token-out workload, Claude Sonnet 5.5 costs $7.00 per 1,000 requests and Claude Haiku 5.5 costs $0.35, so every request you move down a tier saves 95% of its cost. The seven LLM routing examples below cut that $7.00 bill to between $2.51 and $3.71, with code for each.

These LLM routing examples cover the patterns teams actually ship: rule-based routing, cascades, classifier routers, fallback chains, context-length routing, cross-provider price routing and a learned router (RouteLLM). Every cost uses list prices captured on 2026-11-08 from the Anthropic pricing page and the OpenAI pricing page. We show the arithmetic so you can swap in your own token counts. If you want the concept first, read what an LLM gateway is; this page is the copy-paste version.

What does the baseline workload cost?

The baseline is 1,000 requests, each with 1,500 input tokens and 400 output tokens. That is a typical support answer or RAG reply. Prices are per million tokens (MTok), standard tier, prices verified 2026-11-08.

Derivation for Sonnet 5.5: 1,500 x $2 / 1M = $0.0030 input, plus 400 x $10 / 1M = $0.0040 output, equals $0.0070. Sonnet 5.5 is 20x the price of Haiku 5.5 on this shape. That gap is the whole reason routing exists. Plug your own shape into the LLM cost calculator before trusting any savings figure here.

What are common LLM routing strategies?

The common LLM routing strategies are rule-based routing, cascades, classifier routing, fallback routing, context-length routing, price-tier routing and learned routers. They differ in when the decision happens. Rules and classifiers decide before the call. Cascades decide after a cheap attempt. Fallbacks react to errors. The RouteLLM paper formalizes the two-model case: a strong, expensive model and a weak, cheap one, with a threshold that sets how often the strong model is used.

1. Rule-based routing by task type

Use rule-based routing when your app already knows the task type. A tagging endpoint never needs the same model as a contract review endpoint.

CHEAP = "claude-haiku-5-5"
STRONG = "claude-sonnet-5-5"

def route(task: str, prompt: str) -> str:
    # Known-easy tasks always go cheap
    if task in {"classify", "extract", "tag", "summarize_short"}:
        return CHEAP
    # Code and long reasoning go strong
    if "```" in prompt or task in {"code_review", "contract_review"}:
        return STRONG
    # Default: short prompts cheap, everything else strong
    return CHEAP if len(prompt) < 4000 else STRONG

Cost math: if 60% of traffic matches the cheap rules, the bill is 0.6 x $0.35 + 0.4 x $7.00 = $0.21 + $2.80 = $3.01 per 1,000 requests, 57% less than always-Sonnet.

How to adapt it: log the task label and the model for a week, then move one label at a time to the cheap model and compare outputs. Rules are free to run and easy to debug. They fail on traffic that does not carry a clean label, such as an open chat box.

2. Cascade: cheap model first, escalate on failure

A cascade sends every request to the cheap model, checks the answer, and only escalates the failures. FrugalGPT (Stanford, 2023-05-09) reported that a learned cascade matched GPT-4 with up to 98% cost reduction on its benchmarks.

import anthropic, json

client = anthropic.Anthropic()

def ask(model, prompt, max_tokens=600):
    r = client.messages.create(model=model, max_tokens=max_tokens,
                               messages=[{"role": "user", "content": prompt}])
    return r.content[0].text

def verify(question, answer) -> bool:
    # A cheap judge call: about 500 tokens in, 20 out
    verdict = ask("claude-haiku-5-5",
        f"Question: {question}\nAnswer: {answer}\n"
        'Reply JSON {"ok": true|false}. ok=false if the answer is wrong, '
        "incomplete or says it cannot help.", max_tokens=20)
    try:
        return json.loads(verdict)["ok"]
    except (ValueError, KeyError):
        return False  # unparseable verdict means escalate

def cascade(prompt):
    draft = ask("claude-haiku-5-5", prompt)
    if verify(prompt, draft):
        return draft, "haiku"
    return ask("claude-sonnet-5-5", prompt), "sonnet"

How much does a cascade save versus always using the big model?

A cascade with a 30% escalation rate costs $2.51 per 1,000 requests, 64% less than always using Sonnet 5.5 at $7.00. The math: every request pays the Haiku attempt ($0.35 per 1,000) and the judge call (500 x $0.10 / 1M + 20 x $0.50 / 1M = $0.00006, so $0.06 per 1,000). The 30% that fail also pay Sonnet: 0.3 x $7.00 = $2.10. Total $0.35 + $0.06 + $2.10 = $2.51.

The break-even escalation rate is high. The cascade costs $0.41 + $7.00 x e per 1,000, which stays under $7.00 until e reaches 94%. On cost alone, a cascade almost always wins at a 20x price gap. The real price is latency: escalated requests wait for two answers plus a check.

3. Classifier router: a small model picks the model

A classifier router makes one cheap call to decide, then sends the request once. Anthropic lists Haiku 5.5 for "classification, extraction, and routing" on its pricing page, which is exactly this job.

ROUTER_PROMPT = """Rate how hard this request is for an AI assistant.
Reply with one word: EASY or HARD.
EASY: lookup, rewrite, short summary, simple extraction.
HARD: multi-step reasoning, code, math, legal or medical nuance.

Request:
{prompt}"""

def classify_and_route(prompt):
    label = ask("claude-haiku-5-5", ROUTER_PROMPT.format(prompt=prompt[:1200]),
                max_tokens=5).strip().upper()
    model = "claude-haiku-5-5" if label.startswith("EASY") else "claude-sonnet-5-5"
    return ask(model, prompt), model

Cost math: the classifier reads about 300 tokens and writes about 5: 300 x $0.10 / 1M + 5 x $0.50 / 1M = $0.0000325, or $0.03 per 1,000. With a 50/50 split the total is $0.175 + $3.50 + $0.0325 = $3.71 per 1,000, 47% less than always-Sonnet. Truncating the prompt to 1,200 characters keeps the router call cheap on long inputs.

How to adapt it: add a third label (MEDIUM) and a mid-priced model such as gpt-6.1-terra. Measure the classifier against a few hundred hand-labeled requests before trusting it.

4. Fallback chain for rate limits and outages

How do fallback routes work?

A fallback route catches an error from the primary model (HTTP 429 rate limit, 5xx, timeout) and retries the same request on the next model in a list. It is an availability rule, not a cost rule. Pick fallbacks at the same price tier so an outage does not change the bill: Claude Sonnet 5.5 and gpt-6.1-sol both cost $2 / $10 per MTok, so the 1,000-request cost stays $7.00 whichever one answers.

import time
from openai import OpenAI

openai_client = OpenAI()

CHAIN = [
    ("anthropic", "claude-sonnet-5-5"),
    ("openai", "gpt-6.1-sol"),
    ("anthropic", "claude-haiku-5-5"),  # last resort: degraded but alive
]

def call(provider, model, prompt):
    if provider == "anthropic":
        return ask(model, prompt)
    r = openai_client.responses.create(model=model, input=prompt)
    return r.output_text

def with_fallback(prompt, retries_per_model=2):
    last_err = None
    for provider, model in CHAIN:
        for attempt in range(retries_per_model):
            try:
                return call(provider, model, prompt), model
            except Exception as err:  # narrow this to 429/5xx/timeouts in production
                last_err = err
                time.sleep(2 ** attempt)
    raise last_err

What to watch: the two providers count tokens differently. Anthropic notes that Claude 4.7 and later models use a tokenizer that produces about 30% more tokens for the same text, so equal list prices do not guarantee an equal bill. Keep the system prompt tested on both models, or the fallback answers will drift in format.

5. Context-length routing around a price step

Use context-length routing when one model's price changes with prompt size. Claude Haiku 5.5 charges $0.10 / $0.50 per MTok for prompts up to 100,000 tokens and $0.50 / $2.50 above that, a 5x step.

def count_tokens(model, prompt):
    return client.messages.count_tokens(
        model=model, messages=[{"role": "user", "content": prompt}]
    ).input_tokens

def route_by_length(prompt, trim):
    n = count_tokens("claude-haiku-5-5", prompt)
    if n <= 100_000:
        return "claude-haiku-5-5", prompt
    trimmed = trim(prompt, target_tokens=90_000)  # your retrieval or summarizer
    return "claude-haiku-5-5", trimmed

Trimming from 150k to 90k tokens cuts the request from $0.0775 to $0.0095, an 88% drop, because it lands under the price step. It only works when the dropped 60k tokens were not needed; check answer quality on trimmed prompts first.

6. Price-tier routing across providers

Price-tier routing picks the cheapest model in a capability tier across providers. On 2026-11-08, gpt-6.1-terra lists at $1 / $5 per MTok, half of Sonnet 5.5's $2 / $10. On the baseline workload that is $3.50 per 1,000 instead of $7.00.

TIERS = {
    # tier: [(model, input $/MTok, output $/MTok)], prices verified 2026-11-08
    "small": [("claude-haiku-5-5", 0.10, 0.50), ("gpt-6-luna", 0.10, 0.50)],
    "mid":   [("gpt-6.1-terra", 1.00, 5.00)],
    "large": [("claude-sonnet-5-5", 2.00, 10.00), ("gpt-6.1-sol", 2.00, 10.00)],
}

def cheapest(tier, in_tok, out_tok):
    return min(TIERS[tier], key=lambda m: in_tok * m[1] + out_tok * m[2])[0]

How to adapt it: prices move, so load the table from config, not code. OpenAI's pricing page notes that GPT-5.6 Sol's promotional pricing ends on 2026-11-30; a hard-coded table would keep routing to a price that no longer exists. Our Claude API pricing guide tracks the Anthropic side.

7. Learned router with RouteLLM

A learned router predicts, per prompt, whether the strong model will do meaningfully better. RouteLLM from LMSYS ships four trained routers; the matrix factorization (mf) router performed best in the authors' tests.

pip install "routellm[serve,eval]"
# Pick a threshold so about 50% of calls go to the strong model
python -m routellm.calibrate_threshold --routers mf --strong-model-pct 0.5 --config config.example.yaml
# -> For 50.0% strong model calls for mf, threshold = 0.11593

from routellm.controller import Controller

client = Controller(
    routers=["mf"],
    strong_model="anthropic/claude-opus-5-5",
    weak_model="anthropic/claude-haiku-5-5",
)
response = client.chat.completions.create(
    model="router-mf-0.11593",  # router name + threshold
    messages=[{"role": "user", "content": "Hello!"}],
)

What the published numbers say: on MT Bench the mf router reached 95% of GPT-4 quality with 26% of calls going to GPT-4, and with 14% after LLM-judge data augmentation (LMSYS blog, 2024-07-01). Applied to today's Opus 5.5 / Haiku 5.5 pair, a 26% strong share costs 0.26 x $14.00 + 0.74 x $0.35 = $3.90 per 1,000 versus $14.00 for always-Opus, 72% less. Treat that as a model, not a result: the 26% was measured on GPT-4 and Mixtral 8x7B, and the README notes the mf and sw_ranking routers still need an OPENAI_API_KEY for embeddings.

How do you fine-tune a router model or customize the router?

You fine-tune a router model by labeling which of two models answered each past prompt better, then training a small classifier on those labels. The Anyscale and LMSYS tutorial fine-tunes Llama3-8B as a causal LLM classifier, scoring the cheap model's answers on a 1-to-5 scale against the strong model, and reports about 120 minutes end to end on 8xA10 GPUs. To customize the router without training, change three things: the model pair (strong_model, weak_model), the threshold (recalibrate with --strong-model-pct on your own traffic), and the router type (mf, sw_ranking, bert or causal_llm). Customize on your own logs: a router calibrated on Chatbot Arena prompts will route a legal or coding workload differently.

Which routing example should you use?

Most production setups combine two: a fallback chain for uptime plus one cost rule. Running these rules yourself means maintaining price tables, provider SDKs and retry logic in every service. Vynaris puts routing and fallbacks behind one OpenAI-compatible endpoint, if you would rather configure the rules than maintain the plumbing.

When does routing not pay off?

Routing does not pay off when the cheap model rarely succeeds, when volume is small, or when it breaks prompt caching. Three cases from the numbers above:

• High escalation. A cascade above about 94% escalation costs more than always-Sonnet, and well before that the latency is worse.
• Caching. Sonnet 5.5 cache hits cost $0.10 per MTok, the same as uncached Haiku 5.5 input. With 1,200 of 1,500 input tokens cached, Sonnet costs 1,200 x $0.10 / 1M + 300 x $2 / 1M + 400 x $10 / 1M = $0.00472 per request ($4.72 per 1,000, ignoring the one-time cache write). Caches are per model, so switching models per request throws that discount away.
• Small bills. At 10,000 requests a month the full Sonnet bill is $70. A router that saves 50% saves $35 a month, less than an hour of engineering time to maintain it.

We worked through the break-even in more detail in when not to use an LLM router and in the value of information math for router estimation.

FAQ

What is an LLM router?

An LLM router is a layer that picks which model answers each request, based on rules, a classifier, or a trained model. Its goal is to send each request to the cheapest model that answers it well enough, and to switch providers when one fails.

What is the difference between LLM routing and a fallback?

Routing decides which model to call before or after a cheap attempt, to save cost. A fallback only acts on errors such as HTTP 429 or 5xx, to keep the app answering. Most production setups use both.

Is a cascade or a classifier router cheaper?

On the baseline workload a cascade with 30% escalation costs $2.51 per 1,000 and a 50/50 classifier costs $3.71. The cascade is cheaper when most requests succeed on the cheap model; the classifier is faster because each request gets one answer call.

How much can LLM routing save?

Published results range widely. LMSYS reported more than 85% cost reduction on MT Bench, 45% on MMLU and 35% on GSM8K against GPT-4 alone, at 95% of GPT-4 quality. The examples on this page save 47% to 64% at current Anthropic prices.

Do learned routers work on any dataset?

Not automatically. LMSYS found its routers performed near random on MMLU when trained only on Chatbot Arena data, because those questions were out of distribution. Adding labeled data from the target domain fixed it.

Which model should act as the router?

A small, fast model. Anthropic positions Claude Haiku 5.5 for routing and classification; a 300-token routing call costs about $0.03 per 1,000 requests at its list price.