VynarisEarly beta Kimi K3Get your API key

Blog · 2026-08-01 · Vynaris Team

What multi-model LLM-jury data labeling costs per 1M items: the arbiter, not the jury, sets the bill

A 3-model LLM jury labels 1M items for $304-$534. The cheap jurors are $151; the arbiter on 15% disagreement sets the bill. Prices verified 2026-08-01.

A 3-model LLM jury labels 1,000,000 items for $304 to $534 in model fees, but the three cheap jurors are only $151 of that. The rest is one frontier model arbiter firing on the 15% of items where the jury splits. Your labeling bill is set by the disagreement rate times the arbiter price, not by the jury. This is a reproducible cost model you can re-run with your own numbers. Prices verified 2026-08-01.

TL;DR

The finding, before the model

An LLM jury labels each data item with several cheap models in parallel, takes the majority vote, and sends only the disagreements to a stronger arbiter. It has a cost shape that looks like two other patterns and behaves like neither.

The label cost per item is N x juror_call + d x arbiter_call. You pay N cheap juror calls on every single item, and one arbiter call on the fraction d where the jury does not reach a clean majority. The majority vote itself is free arithmetic. There is no aggregation model reading the juror outputs on every item.

That last sentence is what separates a jury from an ensemble. An ensemble pays a combiner model on 100% of items to merge the panel's answers, so its break-even is the participant count. A jury pays no combiner and its second stage is conditional, so its break-even is the disagreement rate.

It is also not a cascade. A confidence-gated cascade runs one cheap model on every item and escalates a fraction to a frontier model, so it pays for one small call plus escalation. A jury pays for N small calls in parallel on every item, then escalates on inter-juror disagreement rather than a single model's self-reported confidence. The disagreement signal is the whole point: the jury spends more up front to get a better trigger for when the expensive model is worth calling.

We built this from public pricing only. No product data. Every token count below is an assumption you can edit, and every price is from a provider's live page, captured 2026-08-01.

The workload (edit these)

One labeled item is the outcome unit: one record scored, one final label written.

Assumption           Value                                                           Note
-------------------  --------------------------------------------------------------  -----------------------------------------------------------
Record + rubric in   300 [input tokens](https://vynaris.com/glossary/input-tokens)   the item plus the labeling instructions each juror reads
Label out            30 [output tokens](https://vynaris.com/glossary/output-tokens)  the class plus a one-line reason
Jury size N          3                                                               odd, for a clean majority vote
Disagreement rate d  15%                                                             share of items with no clean majority; measure on your data
Arbiter call         360 in / 30 out                                                 reads the record plus the 3 candidate labels
Vote                 free                                                            majority of N labels, no model call

Two numbers drive the bill. The jury size N multiplies the cheap call on every item. The disagreement rate d gates the expensive call. Everything else is cost-per-token arithmetic. The number people guess wrong is d, because it is a property of your data and your jurors, not a sticker price you can look up.

The bill, split: jurors versus arbiter

Same workload, three cheap jurors, one arbiter firing on 15% of items. Prices verified 2026-08-01.

Configuration                           Jurors (100%)  Arbiter (15%)  Total /1M  Arbiter share
--------------------------------------  -------------  -------------  ---------  -------------
3x deepseek-v4-flash + Claude Sonnet 5  $151.20        $153.00        $304.20    50.3%
3x deepseek-v4-flash + gpt-5.6-terra    $151.20        $162.00        $313.20    51.7%
3x deepseek-v4-flash + gemini-3.1-pro   $151.20        $162.00        $313.20    51.7%
3x deepseek-v4-flash + Claude Opus 5    $151.20        $382.50        $533.70    71.7%

Read the arbiter-share column. Even though the arbiter runs on only 15% of items, it is at least half the bill, and with Claude Opus 5 it is nearly three-quarters. The jury you designed so carefully is the cheap part. The line that moves your invoice is a model you call one time in seven. Price your own record and label sizes before you commit to an arbiter.

The two levers: jury size and disagreement rate

Hold the jurors at deepseek-v4-flash and the arbiter at Claude Opus 5, then sweep both levers. The formula is cost/1M = N x $50.40 + d x $2,550.00.

N jurors     d = 10%  d = 15%  d = 20%
-----------  -------  -------  -------
1 (no jury)  $305.40  $432.90  $560.40
3            $406.20  $533.70  $661.20
5            $507.00  $634.50  $762.00

Two things fall out. Moving from a 3-juror to a 5-juror panel adds $100.80 per 1M, a flat cost on every item. Moving the disagreement rate from 10% to 20% adds $255.00 per 1M, because each extra point of disagreement buys another slice of the $2,550 arbiter. And the two levers interact against you: a bigger panel is more likely to contain one dissenter, so raising N tends to raise d as well. Size the panel for the accuracy you need, not for a feeling that more voters is safer.

Horizontal bar chart, single blue hue, log x-axis, of model fees per 1,000,000 items across five configurations: lone deepseek-v4-flash $50.40, 3-juror deepseek plus Sonnet 5 arbiter $304.20, 3-juror deepseek plus Opus 5 arbiter $533.70, lone Claude Sonnet 5 $900.00, lone Claude Opus 5 $2,250.00
Cost per 1M items: a cheap jury plus a targeted arbiter sits between a lone budget labeler and a lone frontier labeler. Prices verified 2026-08-01.

Break-even: when a jury beats one model

The honest comparison is not jury versus jury. It is a jury versus the two simple things you could do instead: label everything with one cheap model, or label everything with one frontier model.

Approach                                 Cost /1M   Versus 3-juror + Opus 5
---------------------------------------  ---------  ------------------------------
Lone deepseek-v4-flash on 100%           $50.40     jury is 10.6x more
3-juror deepseek + Claude Sonnet 5 @15%  $304.20    43% cheaper than the Opus jury
3-juror deepseek + Claude Opus 5 @15%    $533.70    baseline
Lone Claude Sonnet 5 on 100%             $900.00    jury is 66% less
Lone Claude Opus 5 on 100%               $2,250.00  jury is 76% less

The jury with a frontier arbiter saves $1,716.30 per 1M against running that frontier model on every item, because it only pays frontier rates on the 15% the panel could not settle. That is the case for the pattern. The case against it is the top row: the same jury costs 10.6x a lone cheap labeler. The entire $483.30 premium buys redundancy plus arbitration. It is worth paying only if that redundancy catches errors a single cheap model would have shipped.

The arbiter is the lever you pull

Because the arbiter is most of the bill, it is also where your savings are. Two moves matter, in this order.

Swap the arbiter down. Moving the arbiter from Claude Opus 5 to Claude Sonnet 5, jurors untouched, drops the total from $533.70 to $304.20, a 43% cut. The arbiter only has to break ties on items where cheap models already narrowed the choices, so a mid-tier model is often enough. By contrast, halving the juror price with the arbiter unchanged drops the total to $458.10, only 14%. Same effort, a third of the payoff.

Tighten what counts as disagreement. Every point you shave off d is a slice of the arbiter you stop buying. If your jury escalates on any non-unanimous vote, a 3-model panel escalates a lot. Escalating only when the majority is a bare 2-1 split, or only when a confidence score is also low, pulls d down and takes the arbiter with it. This is a quality decision with a direct price tag, so measure the accuracy cost before you tighten.

Where routing changes the unit economics, and where it does not

Model routing has two real jobs on this workload.

Route the arbiter by item type. A short, unambiguous record and a long, contested one do not need the same referee. You can send bare 2-1 splits to Claude Sonnet 5 and reserve Claude Opus 5 for items your jurors flagged as genuinely hard. That is a weighted average on a signal the jury already produced, not a second cascade.

Route the jurors for diversity, not price. The cheapest three-model panel is tempting, but a jury of three near-identical models is barely a jury. More on that below. Routing here should optimize for jurors that fail differently, then take the cheapest models that satisfy that constraint.

Where routing does not help: the vote. The majority vote is free and deterministic, which is exactly why a jury is cheaper to run than an ensemble that pays a model to combine.

When this workload does not need a jury

Here is the honest tradeoff. A jury only pays off when disagreement correlates with actual error. That is not automatic, and it is the failure mode that wastes the most money.

If your three jurors share a base model or a training lineage, their mistakes are correlated. They agree confidently on the same wrong answers, so d collapses onto the genuinely ambiguous items and the arbiter never sees the confident-but-wrong ones. You pay 3x for a panel that rubber-stamps its own blind spots. A jury earns its premium only when the jurors fail independently, so that disagreement is a real signal that a human or a stronger model should look.

Two cases where you should skip the jury entirely. First, if a single mid-tier model already clears your accuracy bar on a spot-checked sample, run it flat and audit a random slice. One labeler plus a 2% audit is far cheaper than a standing three-model panel. Second, if wrong labels are cheap to fix downstream, redundancy is not worth 10x. Reserve the jury for high-stakes labels where a wrong one is expensive and where you can show, on a gold set, that inter-model disagreement predicts error. Confirm that on a labeled sample using an eval harness before you turn the panel on.

A note on the small stuff. Prompt caching helps only the static part of the 300-token input, the rubric, not the unique record, so it shaves a little and changes nothing about the levers above. And if labels are not needed in real time, the Batch API halves both juror and arbiter rates at once, which is the single biggest discount available on a labeling run that can wait overnight.

FAQ

What does LLM-jury data labeling cost per 1M items? With three deepseek-v4-flash jurors on every item and an arbiter on the 15% that disagree, $304.20 per 1M with a Claude Sonnet 5 arbiter and $533.70 with Claude Opus 5, on a 300-input, 30-output workload. The three jurors alone are $151.20. Prices verified 2026-08-01.

Why is the arbiter most of the bill if it runs on only 15% of items? Because it is a frontier model. At Claude Opus 5 rates the arbiter call costs $2,550 per 1M against $50.40 for a cheap juror call, a 50x gap. Multiply by 15% and the arbiter still outweighs three cheap jurors combined.

Does adding more jurors make labels cheaper? No. Each added juror is a flat cost on every item ($50.40 per 1M for deepseek-v4-flash), and a larger panel disagrees more often, which raises the arbiter bill too. Size the jury for accuracy, not for a sense of safety.

When is a jury cheaper than just using a frontier model? Whenever your disagreement rate is well below 100%. The 3-juror plus Claude Opus 5 setup is 76% cheaper than running Claude Opus 5 on every item, because it pays frontier rates only on the fraction the cheap panel cannot settle.

When should I not use a jury at all? When the jurors fail the same way, so disagreement does not track error, or when a single model already clears your accuracy bar. In both cases one labeler plus a random audit beats a standing panel on cost.

Sources

Prices change. We re-verify every figure in this post monthly and stamp updates. Numbers here are current as of 2026-08-01.

The lesson from the model: an LLM jury's cost is not the jury. It is the disagreement rate times the arbiter price. Pick jurors that fail differently so disagreement means something, set the escalation rule so only real ties reach the arbiter, and put your cheapest defensible model in the referee seat, because that seat is most of the bill. Vynaris is an OpenAI-compatible gateway that routes each request to the cheapest right-sized model and shows the per-item cost so you watch the number that matters. One base URL swap. Get an API key at vynaris.com.