Blog · 2026-09-05 · Vynaris Team
Pandora's Router cut normalized regret plus cost 84%: when estimator inference pays
Pandora's Router cut EmbedLLM's normalized regret plus inspection cost 83.6%. A live-price model puts estimator break-even at 11.15%.
Pandora's Router reduced EmbedLLM's normalized regret plus inspection cost from 2.356 to 0.386, an 83.6% cut versus always buying the expensive estimate. That is not an 83.6% invoice saving. In our disclosed production model, an extra $0.00017 inspection breaks even when it correctly demotes 11.15% of inspected requests.
Prices verified 2026-09-05 from Google's live Gemini API pricing. Paper results come from Pandora's AI Model Routing Box, revised 2026-08-21.
TL;DR
- Selective inspection beat always using the costly estimator on the paper's combined objective by 18.0% on MATH, 16.3% on RAG, and 83.6% on EmbedLLM.
- The costly estimator's price was 5.8x the cheap estimate on MATH, more than 7,000x on RAG, and 1.6x on EmbedLLM. These ratios used public Gemini rates.
- In our editable scenario, Gemini 3.5 Flash-Lite refinements cost $3.40 per 100,000 requests. Just 2,230 correct demotions recover that bill.
Verdict table
Decision Numeric threshold Verdict
------------------------------- -------------------------------------------- ---------------------------------------------------------------------
Always buy the refined estimate $0.170 per 1,000 inspections Wasteful unless nearly every request is ambiguous
Use only the embedding estimate $0.100 per 1,000 requests Best when the cheap score already separates candidates
Refine 20% of requests $0.134 per 1,000 routed requests Pays if 11.15% of inspected requests are correctly demoted
Run a router at all $13.40 per 100,000 requests in this scenario Skip it if routing cannot recover more than $13.40 or protect qualityThe paper's verdict is selective: buy more information only when its expected improvement exceeds its inspection cost. Our dollar model reaches the same conclusion. A refiner does not need perfect predictions, but it needs enough decisions that change the final route.
What the paper measured
Model routing normally treats the routing decision as free. It is not. A router may embed a prompt, search examples, run retrieval, generate a partial solution, or call another model before selecting the target.
The paper frames this as Pandora's Box. Every candidate model is a box. A cheap estimate is available first. The router can pay to inspect selected boxes, receive a better estimate, and then choose a target.
The cheap estimator embeds the prompt and retrieves nearby calibration examples. Its small prediction network is treated as free. The costly estimator uses a fine-tuned model with more information.
The three test domains make that extra information concrete:
- MATH reveals the first 20 reasoning tokens before judging a candidate.
- RAG runs retrieval before predicting whether a specialist should answer.
- EmbedLLM spends more compute to distinguish among more than 100 open-weight models.
The costly estimates lowered mean squared error from 0.154 to 0.096 on MATH, 0.175 to 0.109 on RAG, and 0.266 to 0.198 on EmbedLLM. Better estimates were real. The question was whether they were worth buying for every candidate.
The paper's 0.105, 0.118, and 0.386 results
Table 2 averages each method across a sweep of inspection-cost settings. Lower is better. Regret measures the routing decision's shortfall. Cost is a calibrated penalty for buying refined estimates.
Method and domain Normalized regret Inspection cost Regret + cost
-------------------------- ----------------- --------------- -------------
MATH, cheap only 0.117 0.000 0.117
MATH, costly only 0.090 0.038 0.128
MATH, Pandora's Router 0.094 0.011 0.105
RAG, cheap only 0.150 0.000 0.150
RAG, costly only 0.084 0.057 0.141
RAG, Pandora's Router 0.091 0.027 0.118
EmbedLLM, cheap only 0.393 0.000 0.393
EmbedLLM, costly only 0.370 1.986 2.356
EmbedLLM, Pandora's Router 0.311 0.075 0.386Pandora cut the combined objective versus costly-only by:
MATH: (0.128 - 0.105) / 0.128 = 17.97%
RAG: (0.141 - 0.118) / 0.141 = 16.31%
EmbedLLM: (2.356 - 0.386) / 2.356 = 83.62%
The inspection-cost column also shows how selective the policy was. Pandora spent 28.9% of the costly-only inspection amount on MATH, 47.4% on RAG, and 3.8% on EmbedLLM. Those are cost shares derived from Table 2, not reported production query rates.
Why the published cost ratios span more than 4,375x
The paper prices its cheap path with a prompt embedding model. Google now lists Gemini Embedding 2 text input at $0.20 per MTok. The paper's Appendix D.3 used that same $0.20 rate.
Its costly path changes by domain:
Domain Cheap estimator cost basis Costly estimator basis Published costly / cheap ratio
-------- ------------------------------------------- ------------------------------------------ ------------------------------
MATH 43 tokens x $0.20/MTok = $0.0000086 20 output tokens x $2.50/MTok = $0.0000500 5.8x
RAG 10.1 tokens x $0.20/MTok = about $0.0000020 Retrieval at $0.014 per evaluation More than 7,000x
EmbedLLM 126 tokens x $0.20/MTok = $0.0000252 126 input tokens plus one output token About 1.6xThe span is greater than 4,375x because RAG's ratio exceeds 7,000x while EmbedLLM's is about 1.6x: 7,000 / 1.6 = 4,375.
Google's current Gemini 3.5 Flash-Lite price is $0.30 per MTok input and $2.50 output. The paper captured $0.25 input and $2.50 output on 2026-08-01. MATH's displayed ratio is unchanged because its costly-path approximation uses output tokens. Repricing EmbedLLM at today's $0.30 input still gives 1.60x after rounding.
RAG is the warning flare. At $14 per 1,000 retrieval requests, the information-gathering tool can dwarf a tiny text embedding. An estimator can be more accurate and still destroy the cost per task.
An editable production break-even
Normalized regret is not currency. We therefore built a separate production scenario from live rates. These are modeling inputs, not Vynaris traffic or customer data.
Input Value Status
-------------------------- ------------------------------------------------------------------------------ -----------------------------------------
Monthly requests 100,000 Editable scale assumption
Prompt size 500 input tokens Editable workload assumption
Refiner response 8 output tokens Editable JSON-score assumption
Selective inspection share 20% Editable policy assumption
Target call shape 2,000 input / 500 output tokens Editable workload assumption
Cheap estimate Gemini Embedding 2 at $0.20/MTok Live Google price
Refiner Gemini 3.5 Flash-Lite at $0.30/$2.50 Live Google price
Cheap target Gemini 3.5 Flash-Lite at $0.30/$2.50 Live Google price
Expensive target [Gemini 3.8 Flash](https://vynaris.com/models#gemini-3-8-flash) at $0.75/$3.75 Live promotional price through 2026-12-31The always-on embedding costs $0.000100 per request:
500 input tokens x $0.20 / 1,000,000 = $0.000100
= $0.100 per 1,000 routed requestsOne refined score costs $0.000170:
(500 x $0.30 + 8 x $2.50) / 1,000,000
= $0.000170 per inspection
= $0.170 per 1,000 inspectionsAt a 20% inspection share, 100,000 requests need 20,000 refinements. The monthly estimate bill is $10.00 for embeddings plus $3.40 for refinements, or $13.40.
Use the LLM cost calculator to replace the target call's input tokens, output tokens, and rates. Keep the estimator bill separate so it cannot disappear inside target-model spend.
The refiner must be right 11.15% of the time
For the disclosed 2,000-input, 500-output target shape, Flash-Lite costs $0.001850 per task. Gemini 3.8 Flash costs $0.003375. A correct demotion saves $0.001525.
Flash-Lite target = (2,000 x $0.30 + 500 x $2.50) / 1M = $0.001850
3.8 Flash target = (2,000 x $0.75 + 500 x $3.75) / 1M = $0.003375
saving per correct demotion = $0.003375 - $0.001850 = $0.001525
break-even success among inspected requests
= $0.000170 / $0.001525
= 11.1475%The 20,000 refinements cost $3.40. Dividing by $0.001525 requires 2,229.5 correct demotions, so the first whole-request threshold is 2,230. That is 2.23% of all 100,000 requests.
This threshold covers only the incremental refiner. The $10.00 embedding bill remains. To justify the entire $13.40 router, the policy needs 8,787 correct demotions if savings are its only benefit. That is $13.40 divided by $0.001525, rounded up.
Our earlier upfront versus mid-task routing analysis explains why the savings may not survive a late decision. By then, the system has already paid for context, tools, or partial generation. Our when not to use a router guide covers the fixed-cost boundary.
What this means for a production router
Start with a cheap score. Log which decisions would change under a refined score. That counterfactual is the denominator.
Inspect near the decision boundary, not at random. A model cascade with a wide confidence gap does not need another inference call. Ambiguous requests do.
Measure three quantities during an evaluation: refiner cost, changed-route rate, and accepted-outcome delta. A cheaper target that creates retries is not a correct demotion.
The token cost threshold is only one value channel. A refined estimate can also avoid a failed cheap attempt or protect a high-value request. Assign those outcomes a monetary value before deploying the policy.
Honest tradeoff
Do not build this when the cheap estimate already separates candidates. MATH's cheap-only total was 0.117, below costly-only at 0.128. EmbedLLM's cheap-only total was 0.393, only 0.007 above Pandora's 0.386.
Pandora's gain also depends on calibrated uncertainty. The paper models refined values with a conditional Gaussian. Its appendix reports imperfect calibration and finds that a KNN alternative did not improve the combined objective. A production score can look precise while being wrong near the threshold.
The opposite tradeoff matters too. Our dollar model values only target-model savings. If a correct escalation prevents an expensive failure, the refiner may pay with far fewer changed routes. A savings-only model is deliberately conservative.
Where this model can be wrong
- The paper's regret and inspection penalties are calibrated research values. They are not dollars and should not be multiplied by request volume.
- The Google prices can change. Gemini 3.8 Flash promotional rates are scheduled to double on 2027-01-01.
- Our 500-token prompt, 8-token score, 20% inspection share, and 2,000/500 target shape are disclosed scenarios. Replace all four.
- A refiner may add latency even when its API bill clears the threshold.
- Correct demotion means equal accepted quality. A lower invoice with more failures is not a saving.
FAQ
What did Pandora's Router save?
It reduced the paper's normalized regret plus inspection cost by 18.0% on MATH, 16.3% on RAG, and 83.6% on EmbedLLM versus always using the costly estimator. Those percentages are not invoice savings.
How much does the example router cost?
At 100,000 monthly requests, the always-on embeddings cost $10.00. Refining 20% costs another $3.40. The total estimation bill is $13.40.
What is the break-even accuracy?
Accuracy alone is the wrong denominator. In our scenario, 11.15% of inspected requests must become correct $0.001525 demotions to recover the incremental refiner bill.
Should every request get an LLM-based routing score?
No. If the embedding score already produces a clear winner, the extra call buys little information. Reserve refinement for close or high-value decisions.
Sources
- Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimation, Tables 1 and 2 plus Appendix D.3; arXiv v2 captured 2026-09-05.
- Google Gemini API pricing, Gemini 3.5 Flash-Lite, Gemini 3.8 Flash, Gemini Embedding 2, and retrieval rates verified 2026-09-05.
All article arithmetic and the chart are reproducible in artifacts/llm-router-estimation-cost-value-of-information-break-even-math.py.