Blog · 2026-08-05 · Vynaris Team

The LLM-judge bill at 800k evaluations/week: $112 to $1,040 before you cut volume

800k judgments/week costs $112-$1,040 on an 800/100 shape. Sampling, nano judges, and rubrics beat full Sonnet Batch. No deprecated mini prices. Prices verified 2026-08-05.

Judging 800,000 model outputs a week costs $112 to $1,040 on the stated 800-input / 100-output shape, depending on whether you run deepseek-v4-flash, GPT-5.4 nano Batch, or Claude Sonnet 5 Batch on every item. Sampling, small judges, and deterministic rubrics cut that line without inventing quality scores. Prices verified 2026-08-05.

TL;DR

One judgment uses 800 input tokens and 100 output tokens. At 800k/week that is 640M input and 80M output tokens per week. Counts are editable assumptions.
Full Sonnet 5 Batch on 100% costs $1,040/week ($4,507/month). GPT-5.4 nano Batch on 100% costs $114/week ($494/month). deepseek-v4-flash costs $112/week with no Batch row.
A 10% Sonnet sample costs $104/week. An 80% deterministic rubric with Sonnet on the remaining 20% costs $208/week. Nano on 100% plus a 10% Sonnet escalate costs $218/week.
A Sonnet-only sample stays at or below the nano-100% budget only up to about 87,700 judgments/week (11.0% of volume). Beyond that, sampling alone is not the cheap path.

What we computed and why

Eval platforms burn tokens twice: once to generate candidates, again to score them. The second bill is easy to miss because it sits in a nightly job. We price only the judge calls.

Community threads trade anecdotes about monthly judge spend. We do not treat those anecdotes as receipts. We rebuild the arithmetic on live 2026-08-05 prices for current small and mid models. Deprecated gpt-4o-mini stickers do not appear.

Assumptions table

Assumption         Value               Why it is here
-----------------  ------------------  -----------------------------------------
Judgments / week   800,000             Workload in the topic brief
Input / judgment   800 tokens          Candidate snippet + rubric + instructions
Output / judgment  100 tokens          Score, rationale, short rationale
Weeks / month      52/12 ≈ 4.333       Calendar conversion
Batch default      On where published  Offline eval tolerates delay
Agreement rates    Not assumed         Measure on your labeled set

Replace the 800/100 shape if your judge prompt ships full transcripts or long chain-of-thought. Reasoning tokens, when billed as output, can dominate this line.

Results: cost per judgment and at 800k/week

Model                                                                      $/judgment std  $/judgment Batch  $/week preferred  $/month preferred
-------------------------------------------------------------------------  --------------  ----------------  ----------------  -----------------
[deepseek-v4-flash](https://vynaris.com/models#deepseek-v4-flash)          $0.000140       n/a               $112              $485
[gpt-5.4-nano](https://vynaris.com/models#gpt-5-4-nano)                    $0.000285       $0.000143         $114              $494
[Gemini 3.5 Flash-Lite](https://vynaris.com/models#gemini-3-5-flash-lite)  $0.000490       $0.000245         $196              $849
[gpt-5.4-mini](https://vynaris.com/models#gpt-5-4-mini)                    $0.001050       $0.000525         $420              $1,820
[Claude Haiku 4.5](https://vynaris.com/models#claude-haiku-4-5)            $0.001300       $0.000650         $520              $2,253
Gemini 3.6 Flash                                                           $0.001950       $0.000975         $780              $3,380
[Claude Sonnet 5](https://vynaris.com/models#claude-sonnet-5) (intro)      $0.002600       $0.001300         $1,040            $4,507

Preferred means Batch at 50% when the provider publishes it. DeepSeek stays on standard rates. The preferred weekly spread is $112 to $1,040, or 9.3x.

Horizontal bar chart on a log scale showing weekly LLM-judge cost at 800k evaluations: deepseek-v4-flash $112, gpt-5.4-nano Batch $114, Gemini 3.5 Flash-Lite Batch $196, gpt-5.4-mini Batch $420, Claude Haiku 4.5 Batch $520, Gemini 3.6 Flash Batch $780, Claude Sonnet 5 Batch $1,040. — Weekly cost for 800k judgments across current small and mid judges. Source: OpenAI, Anthropic, Google, DeepSeek pricing, verified 2026-08-05. 800/100 tokens per judgment are editable assumptions.

Strategy comparison on the same volume

Strategy                            LLM judgments  Weekly model cost  vs full Sonnet Batch
----------------------------------  -------------  -----------------  --------------------
Full Sonnet 5 Batch                 800,000        $1,040             100%
Haiku 4.5 Batch on 100%             800,000        $520               50%
Nano Batch on 100%                  800,000        $114               11.0%
deepseek-v4-flash on 100%           800,000        $112               10.8%
Cascade: nano 100% + Sonnet on 10%  880,000 calls  $218               21.0%
Rubric on 80%, Sonnet on 20%        160,000        $208               20.0%
Uniform Sonnet sample 10%           80,000         $104               10.0%
Uniform Sonnet sample 1%            8,000          $10.40             1.0%

Use the calculator on a single judgment, then multiply. Do not average a 200-token rubric with a 4,000-token transcript judge into one fake unit cost.

What the strategies actually buy

Uniform sampling

Sampling is the blunt lever. A 10% Sonnet sample is $104/week. That undercuts nano-on-100% by $10/week. The tradeoff is coverage, not price. Sampling answers "what is the distribution this week" only if the 10% is stratified by task type, model under test, and failure mode. A random 10% of an imbalanced eval set will miss rare regressions.

Sonnet Batch stays at or under the $114 nano budget only through about 87,700 judgments/week. That is 11.0% of 800k. If you need wider coverage than that at Sonnet quality, sampling alone loses to a small judge on 100%.

Small-model judges

Nano Batch and DeepSeek land near $112–$114/week for full coverage. That is the price of looking at every item with a cheap model. Haiku Batch is $520/week: still half of Sonnet, still 4.6x nano.

Small judges only pay if you measure them. Run a fixed labeled set. Report agreement with your trusted grader as an evaluation metric you own. We do not invent an agreement percentage here. A cheap judge that systematically misses your expensive failure mode is a discount on a useless signal.

Deterministic rubrics

If code can score 80% of cases (exact match, schema validity, unit-test pass, allowed-string checks), Sonnet only sees 160k items and costs $208/week. Rubric coverage must clear about 89% before a Sonnet-on-remainder design undercuts nano-on-100% ($114/week). Below that, the small judge on everything is cheaper than Sonnet on the leftovers.

Rubrics are not free to build. Price eng hours separately. The model table above is only the inference line.

Cascade

Nano on 100% plus Sonnet on a 10% escalate path costs $218/week. That is almost identical to the 80/20 rubric split, with a different failure mode: the escalate rate is a product decision, not a code coverage number. Calibrate escalate on confidence, disagreement with a second cheap judge, or rule flags. Cap it. An uncalibrated 40% escalate puts you back near $530/week (nano $114 + Sonnet on 320k = $416).

This is the same escalate math as our UGC moderation cascade playbook, applied to offline eval instead of trust-and-safety terminate paths. For parallel multi-judge labeling with an arbiter, see the LLM-jury labeling playbook. A jury pays N cheap calls on every item; this article prices one judge call and volume levers.

Arithmetic you can recompute

Sonnet 5 introductory Batch unit cost:

800 x $2.00 / 1,000,000 x 0.5 = $0.00080
100 x $10.00 / 1,000,000 x 0.5 = $0.00050
unit                              = $0.00130
x 800,000                         = $1,040 / week
x 52/12                           = $4,506.67 / month

gpt-5.4-nano Batch:

800 x $0.20 / 1,000,000 x 0.5 = $0.000080
100 x $1.25 / 1,000,000 x 0.5 = $0.0000625
unit                            = $0.0001425
x 800,000                       = $114 / week

deepseek-v4-flash standard (no Batch discount used):

800 x $0.14 / 1,000,000 = $0.000112
100 x $0.28 / 1,000,000 = $0.000028
unit                      = $0.000140
x 800,000                 = $112 / week

What it means for routing

Offline eval is batch processing territory. Turn Batch on before you invent a router. The 50% cut is free latency you already tolerate overnight.

After Batch, the decision is coverage versus model strength:

Need a weekly distribution estimate, not per-item labels: sample.
Need per-item labels and a measured-small judge clears your bar: nano or DeepSeek on 100%.
Need Sonnet-quality on hard cases only: rubric or cascade, with an escalate cap.
Need Sonnet on everything: pay $1,040/week and stop pretending sampling will save you.

Model routing helps when judgment difficulty is visible before the call. If every item looks the same until a human reads it, a router has nothing to key on. Sticker ranks for these judges also sit in the August 2026 pricing table and the budget-tier floor comparison.

Track LLM cost attribution as judge-tokens separate from candidate-tokens. Mixing them hides the second bill.

Honest tradeoff

Do not cut judge spend in a way that blinds the metric you ship on. A $112/week DeepSeek judge that disagrees with your production grader on the failures that matter is more expensive than a $1,040/week Sonnet judge you actually trust. Cost tables do not replace a labeled eval set. Build the labeled set first, then pick the row.

Also: Sonnet 5 introductory pricing ends 2026-08-31. After that the Batch unit becomes $0.00195 and full-volume Sonnet jumps from $1,040 to $1,560/week on this shape if you change nothing else.

Caveats

Token counts are assumptions. A judge that dumps full multi-turn transcripts will blow past 800 input tokens.
We ignore candidate-generation cost, embedding prefilters, and human audit hours.
DeepSeek peak/off-peak 2x pricing is announced but not dated live; we use the regular rates on the pricing page.
No agreement percentages are claimed. Any "87% agreement" figure from community threads is anecdote, not a receipt.

Sources

OpenAI API pricing, captured 2026-08-05: gpt-5.4-nano $0.20/$1.25, gpt-5.4-mini $0.75/$4.50; Batch 50% discount.
Anthropic API pricing, captured 2026-08-05: Haiku 4.5 $1/$5, Sonnet 5 intro $2/$10 through 2026-08-31, Batch 50%.
Gemini API pricing, captured 2026-08-05: Gemini 3.5 Flash-Lite $0.30/$2.50, Gemini 3.6 Flash $1.50/$7.50, Batch half of paid tier.
DeepSeek API pricing, captured 2026-08-05: deepseek-v4-flash $0.14/$0.28 cache miss.

All judgment volumes and per-judgment token counts are shown assumptions. Recompute before using the weekly totals in a budget.