VynarisEarly betaGet your API key

Blog · 2026-07-21 · Vynaris Team

Which LLM calls a cheaper model could handle: a $6 detection recipe

Find the share of LLM traffic a cheaper model handles just as well for ~$6 in detection. Route it and cut a 100k-call bill 45-54%. Prices verified 2026-07-21.

You can find the share of your LLM traffic that a cheaper model would answer just as well in an afternoon, using local, open-source tools and your own request logs, for about $6 in detection cost. On a 100,000-call-per-month workload, routing the downroutable share off a frontier model cuts the bill 45% to 54% with prices verified 2026-07-21. The catch nobody mentions: the free offline estimate is a population prior, not your quality bar, and it can overstate your real savings by 38%. This is the recipe to measure it honestly.

The finding, before the tooling

The most-discussed cost thread on Hacker News this week is titled "I burned all my tokens researching how to save tokens" (171 points, 207 comments, read 2026-07-21). The joke lands because detection feels expensive: to know which calls a cheap model handles, it seems you must re-run everything on every model. You do not. You sample. Replay a few hundred logged prompts on a candidate cheap model, judge whether the answers match, and extrapolate. That costs a few dollars once, against savings that recur every month. Below we run the whole loop on live prices and show where the free estimate lies to you.

We use public sources only: your own OpenAI-style logs, the MIT-licensed detector Frugon, and provider pricing captured 2026-07-21. Every number is either from a live pricing page or derived in the arithmetic shown.

The workload (edit these)

Assumption          Value                                                            Note
------------------  ---------------------------------------------------------------  --------------------------------
Calls per month     100,000                                                          your production API volume
Input per call      1,500 [input tokens](https://vynaris.com/glossary/input-tokens)  prompt plus context
Output per call     500 [output tokens](https://vynaris.com/glossary/output-tokens)  the completion
Baseline model      [gpt-5.6-sol](https://vynaris.com/models#gpt-5-6-sol)            the frontier model you run today
Downroutable share  to be measured                                                   the whole point of the recipe

At $5 per million input tokens and $30 per million output, the baseline runs $0.0225 per call, or $2,250 per month. The recipe finds how much of that you can move to a cheaper model without users noticing.

Step 1: capture your calls as JSONL

Frugon reads one JSON object per line in OpenAI request/response shape, using the usage token counts when present and falling back to its own tokenizer when absent:

{"model": "gpt-5.6-sol", "request": {"messages": [...]}, "response": {"choices": [{"message": {"content": "..."}}]}, "usage": {"prompt_tokens": 1500, "completion_tokens": 500}, "timestamp": "2026-07-21T14:22:01Z"}

If you already log requests and responses, export a month to logs.jsonl. If you do not, Frugon can sit in front of your API as a logging proxy and write the file for you:

frugon capture --out ./logs.jsonl --port 8787 --upstream https://api.openai.com

Point your client's base URL at the proxy for a representative window, then stop. You now have a real sample of your traffic, not a synthetic benchmark.

Step 2: get the offline estimate (free)

The analyze command applies an offline easy-versus-hard split heuristic and recommends routing the easy calls to cheaper candidates, without executing any model:

frugon analyze ./logs.jsonl --candidates gpt-5.4-mini,haiku-4.5,deepseek-v4-flash --report out.html

This is fast and free, and it is where most people stop. They should not. Frugon's quality tiers come from the LMArena leaderboard and its savings bands from RouteLLM research, which the tool itself describes as "population priors drawn from public leaderboards". A prior is an average over everyone else's traffic. Your distribution of prompts is not the average, so treat this number as a hypothesis to test, not a result.

Step 3: validate on a judged sample (a few dollars)

Turn the estimate into a measurement. Replay a sample of the logged prompts on the candidate cheap model, then judge whether each cheap answer matches the baseline answer you already have in your logs. Frugon's --measure and --judge flags do this; you can also script it directly:

frugon analyze ./logs.jsonl --candidates gpt-5.4-mini --measure --judge

The judge is an LLM-as-judge: feed it the prompt, the baseline answer, and the cheap answer, and ask which is acceptable. This is the "effective cost per judged success" the top HN commenter argued for, rather than routing blindly to the cheapest model. Here is what that detection run costs on 1,000 sampled calls, priced 2026-07-21.

Detection step                                                                   Tokens                   Cost
-------------------------------------------------------------------------------  -----------------------  ---------
Replay 1,000 prompts on [gpt-5.4-mini](https://vynaris.com/models#gpt-5-4-mini)  1,500 in / 500 out each  $3.38
Judge 1,000 pairs on [Haiku 4.5](https://vynaris.com/models#claude-haiku-4-5)    2,700 in / 50 out each   $2.95
Baseline answers (already in your logs)                                          —                        free
**Total, one-time**                                                                                       **$6.32**

The baseline answers are historical, so you pay nothing to obtain them. The whole measurement is $6.32. Against the savings below, it pays back in about six hours of routed traffic. This is the answer to "I burned all my tokens researching how to save tokens": sampling turns detection from an open-ended token furnace into a fixed $6 line item.

Step 4: re-price the savings on live rates

Now compute what the measured downroutable share actually saves, on today's prices. Route the easy calls to a cheap model and keep the hard ones on the baseline. Prices verified 2026-07-21.

Cheap target                                                       Per call  Routed $/mo at 55%  Saved vs $2,250
-----------------------------------------------------------------  --------  ------------------  ---------------
[Haiku 4.5](https://vynaris.com/models#claude-haiku-4-5)           $0.00400  $1,232.50           45.2%
[gpt-5.4-mini](https://vynaris.com/models#gpt-5-4-mini)            $0.00337  $1,198.12           46.8%
[deepseek-v4-flash](https://vynaris.com/models#deepseek-v4-flash)  $0.00035  $1,031.75           54.1%

deepseek-v4-flash is 64x cheaper per call than the baseline, so even downrouting a bit past half your traffic to it lands a 54% cut. The savings scale with the share you can move, which is exactly why measuring that share honestly matters more than picking the cheap model.

Horizontal bar chart of monthly LLM spend, single blue hue: baseline all gpt-5.6-sol $2,250, route 55% to Haiku 4.5 $1,232, route 55% to gpt-5.4-mini $1,198, route 55% to deepseek-v4-flash $1,032, at 100,000 calls per month
Monthly spend at 100k calls: baseline frontier vs routing the 55% downroutable share to a cheaper model. Prices verified 2026-07-21.

Why the free estimate lies, quantified

The offline heuristic and the judged sample often disagree, and the gap is money. Suppose analyze flags 55% of your traffic as downroutable, but your judged sample confirms only 40% actually clear your quality bar.

Source                     Downroutable share  Monthly saving (route to gpt-5.4-mini)
-------------------------  ------------------  --------------------------------------
Offline heuristic (prior)  55%                 $1,051.88
Judged sample (measured)   40%                 $765.00

Believing the prior overstates your savings by $286.88 a month, 38% too high. It also overstates quality: the 15 points of traffic the heuristic called easy and the judge rejected are the calls that would have shipped worse answers to users. The judged number is both the honest savings and the safe routing set. Run your own token shape and downroutable share through the calculator to see your version of this table.

The sensitivity that decides everything

Savings track the downroutable share almost linearly, so the measurement in Step 3 is the whole ballgame.

Downroutable share  Routed $/mo (gpt-5.4-mini)  Saved
------------------  --------------------------  -----
30%                 $1,676.25                   25.5%
45%                 $1,389.38                   38.2%
55%                 $1,198.12                   46.8%
70%                 $911.25                     59.5%

A workload where 70% of calls are routine FAQ answers saves 59.5%; one where only 30% are simple saves 25.5%. Same tools, same prices, very different economics, and you cannot know which you have without measuring. This is the model-routing discipline we applied to real coding traffic in the coding-agent model-routers savings post and stress-tested against inflated vendor claims in the router savings claims audit.

The honest tradeoffs

When chasing this is not worth it

The break-even is a volume question. Routing 40% of calls to gpt-5.4-mini saves about $0.00765 per call. A single engineer-hour to wire and test the routing, at a $100 blended rate, pays back at roughly 13,000 calls. Below that lifetime volume, the labor to build the router costs more than the tokens it saves, and the right move is to pay the bill and spend the hour elsewhere. This is the real lesson under the "burned all my tokens" joke: detection is cheap, but acting on it is engineering, and engineering only pays back above a volume you should compute first. If you run millions of calls a month, this recipe is close to free money. If you run a few thousand, it is a hobby.

Doing this on Vynaris

The DIY path above genuinely works; this section is the paid shortcut, clearly labeled. Vynaris is an OpenAI-compatible gateway that runs the detection loop continuously instead of once: it scores each call's downroutability, routes the easy share to a cheaper model at live prices, keeps a judged sample running so the split stays honest as your traffic drifts, and reports the per-call cost and the realized savings. You get the measured number, not the prior, without maintaining the harness. One base URL swap. See the docs and pricing.

FAQ

How do I know which LLM calls a cheaper model could handle? Log your calls in OpenAI JSONL format, run an offline analyzer like the MIT-licensed Frugon to get a candidate easy/hard split, then validate it by replaying a sample on the cheap model and judging the answers against your logged baseline responses. The judged share is your true downroutable fraction.

How much can routing to a cheaper model save? On a 100,000-call month at 1,500 input and 500 output tokens, moving 55% of traffic off gpt-5.6-sol saves 45% to 54% depending on the cheap target, from $2,250 to between $1,032 and $1,233 per month. Prices verified 2026-07-21.

What does the detection itself cost? About $6.32 one-time to validate on a 1,000-call sample, because your baseline answers are already logged; you only pay to replay the sample on the cheap model ($3.38) and judge the pairs ($2.95). It pays back in about six hours of routed traffic.

Why not just trust the free offline estimate? Because it is a population prior from public leaderboards, not a measurement of your prompts. In the worked example it claimed 55% downroutable where a judged sample confirmed 40%, overstating monthly savings by 38% and mislabeling quality regressions as wins.

When is it not worth routing? Below roughly 13,000 lifetime calls, where a single $100 engineer-hour to build the router costs more than the tokens it saves. Low-volume workloads should pay the bill and skip the harness.

Sources

Prices change. We re-verify every figure in this post monthly and stamp updates. Numbers here are current as of 2026-07-21.

The lesson from the recipe: finding which calls a cheaper model could handle is a $6 measurement, not a research project, as long as you sample instead of exhaustively replay, judge instead of trust a prior, and re-price on live rates instead of a dated table.