VynarisEarly betaGet your API key

Blog · 2026-07-20 · Vynaris Team

Benchmarking LLMs on your own workload: a minimal eval harness that measures cost per correct answer

A minimal, provider-agnostic eval harness that measures cost per correct answer on your own data. Runs 500 examples through six models for about $10. Prices verified 2026-07-20.

Running 500 of your own examples through six models costs about $10.27 total, less than a coffee, and takes an afternoon to wire up. That is the whole argument against trusting a vendor leaderboard: measuring on your own task is cheap, and the number that should decide your model, cost per correct answer, can rank the models in the opposite order from cost per API call. Prices verified 2026-07-20. Here is the harness, the math, and the trap.

Why vendor benchmarks lie to your budget

A leaderboard tells you a model scores 94.2% on some public test set. Your workload is not that test set. The distribution is different, the prompt is different, the grading is different, and the 94.2% was tuned to be impressive. None of it tells you what you actually need: how much it costs to get a correct answer on your task.

The honest alternative is to measure. Take a few hundred examples from your own data, run them through the candidate models, grade the outputs against your ground truth, and count two things per model: the dollars it spent and the answers it got right. Divide. That ratio, cost per correct answer, is the only benchmark that maps to your bill. This post builds the harness that produces it, and shows why the ranking it gives can invert the ranking you would get from cost-per-token alone.

What the harness measures

Two numbers per model, both pulled from the API response, not estimated.

From those, cost per correct answer is cost_per_run / (N * accuracy). The point of measuring on your data is that both terms are yours: your token shape sets the cost, your ground truth sets the accuracy.

The harness (this runs)

A minimal, provider-agnostic loop against any OpenAI-compatible endpoint. It reads examples, calls the model, grades, and tallies real token usage. No framework.

import json, time
from openai import OpenAI

# Prices USD per 1M tokens (input, output), verified 2026-07-20.
PRICES = {
    "deepseek-v4-flash":     (0.14, 0.28),
    "gemini-3.1-flash-lite": (0.25, 1.50),
    "gpt-5.4-mini":          (0.75, 4.50),
    "claude-haiku-4-5":      (1.00, 5.00),
    "gpt-5.4":               (2.50, 15.00),
    "claude-opus-4-8":       (5.00, 25.00),
}

client = OpenAI(base_url="https://your-gateway/v1", api_key="...")

def grade(prediction: str, truth: str) -> bool:
    # Swap in your own grader: exact match, JSON field match,
    # numeric tolerance, or an LLM judge. Keep it deterministic.
    return prediction.strip().lower() == truth.strip().lower()

def run_eval(model: str, examples: list[dict]) -> dict:
    pin, pout = PRICES[model]
    in_tok = out_tok = correct = 0
    for ex in examples:
        r = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": ex["prompt"]}],
            temperature=0,
        )
        pred = r.choices[0].message.content
        in_tok  += r.usage.prompt_tokens
        out_tok += r.usage.completion_tokens
        correct += grade(pred, ex["answer"])
    cost = in_tok/1e6*pin + out_tok/1e6*pout
    n = len(examples)
    return {
        "model": model, "n": n, "accuracy": correct/n,
        "cost_run": round(cost, 4),
        "cost_per_1k_correct": round(cost / max(correct, 1) * 1000, 4),
    }

examples = [json.loads(l) for l in open("eval.jsonl")]   # {"prompt","answer"}
for m in PRICES:
    print(run_eval(m, examples))

Two rules keep the numbers honest. Set temperature=0 so a re-run grades the same way. Read tokens from r.usage, never estimate them, because reasoning models and tool calls add tokens your character count will miss. We covered that reconstruction problem in the structured-output token overhead post.

What a run costs

Assume 500 examples, 800 input and 250 output tokens each, a common shape for a classification or short-answer task. Cost per run across six models, from cheapest to frontier. Prices verified 2026-07-20.

Model                                                                      Input $/1M  Output $/1M  Cost per run (500 ex)
-------------------------------------------------------------------------  ----------  -----------  ---------------------
[deepseek-v4-flash](https://vynaris.com/models#deepseek-v4-flash)          $0.14       $0.28        $0.091
[Gemini 3.1 Flash-Lite](https://vynaris.com/models#gemini-3-1-flash-lite)  $0.25       $1.50        $0.287
[gpt-5.4-mini](https://vynaris.com/models#gpt-5-4-mini)                    $0.75       $4.50        $0.863
[Claude Haiku 4.5](https://vynaris.com/models#claude-haiku-4-5)            $1.00       $5.00        $1.025
[gpt-5.4](https://vynaris.com/models#gpt-5-4)                              $2.50       $15.00       $2.875
[Claude Opus 4.8](https://vynaris.com/models#claude-opus-4-8)              $5.00       $25.00       $5.125

The whole sweep costs $10.27. Running your own benchmark on all six frontier-and-workhorse models, on your own data, is cheaper than the coffee you drink reading their marketing. There is no budget excuse to guess. If you want to plug your own example count and token shape into these prices, the calculator does the per-run arithmetic directly.

The trap: cost per run is the wrong metric

Here is where the harness earns its keep. Cost per run ranks deepseek-v4-flash cheapest and Claude Opus 4.8 most expensive, a 56x spread. But you do not buy runs, you buy correct answers, and a wrong answer is not free. Someone or something has to catch it and fix it.

The illustrative accuracies below are placeholders to show the formula. They are not measured scores, and the whole point of the harness is that you replace them with your own graded results. What matters is the shape of the conclusion, not these specific fractions.

Suppose your graded run produced these accuracies, and every wrong answer costs a person $2.00 to catch and correct downstream. Total cost per run becomes cost_per_run + wrong_answers * $2.00.

Model                  Accuracy*  Cost per run  Wrong of 500  + human fix @ $2  Total
---------------------  ---------  ------------  ------------  ----------------  -------
Claude Opus 4.8        0.95       $5.13         25            $50.00            $55.13
gpt-5.4                0.92       $2.88         40            $80.00            $82.87
gpt-5.4-mini           0.86       $0.86         70            $140.00           $140.86
Claude Haiku 4.5       0.84       $1.03         80            $160.00           $161.03
deepseek-v4-flash      0.80       $0.09         100           $200.00           $200.09
Gemini 3.1 Flash-Lite  0.78       $0.29         110           $220.00           $220.29

*Illustrative, not measured. Replace with your graded numbers.

The ranking inverts completely. deepseek-v4-flash, cheapest per run at $0.09, becomes the second most expensive option at $200 once its 100 wrong answers each cost a human $2 to fix. Claude Opus 4.8, the priciest per run at $5.13, becomes the cheapest overall at $55, because it is wrong on 25 examples instead of 100. The 15-percentage-point accuracy gap is worth $145, and the model price difference of $5 is noise against it.

This is the entire case for right-sizing on measured accuracy rather than the sticker price we tabulated for 14 models. When wrong answers are cheap to catch, a cheap model that re-runs on a frontier model via model routing wins: at a $0.01 re-run cost per error, the totals stay within $1.12 to $5.38 and the cheap model keeps its lead. When wrong answers are expensive, because a human fixes them or because a bad answer reaches a customer, accuracy dominates and the frontier model wins. The harness tells you which regime you are in. A leaderboard cannot.

Grading: the part that decides everything

The cost numbers are trivial arithmetic. The accuracy number is where the work is, and where most eval efforts go wrong.

When you do not need this

The honest tradeoff: if your task is genuinely covered by a public benchmark that matches your distribution, and a wrong answer costs almost nothing, skip the harness and take the cheapest model that clears a smoke test. Building an eval set has a real cost in labeling time, and for a low-stakes internal tool it can be over-engineering. The harness earns its keep when the accuracy gap between models is large enough to matter, when wrong answers are expensive, or when you are about to commit to a model across a high-volume workload where a few points of accuracy compound into real money. Below that bar, a two-example gut check is fine, and you should not pretend otherwise.

FAQ

How much does it cost to benchmark LLMs on my own data? About $10.27 to run 500 examples through six models at 800 input and 250 output tokens each, verified 2026-07-20. deepseek-v4-flash costs $0.09 per run and Claude Opus 4.8 costs $5.13; the full sweep is under the price of a coffee.

What metric should decide which model to use? Cost per correct answer, not cost per API call. It is cost_per_run / (N * accuracy), and once you add the cost of handling wrong answers it can rank the models in the opposite order from raw token price.

Why not just use a public leaderboard? A leaderboard measures a public test set, not your workload. Your prompt, distribution, and grading differ, and the score is tuned to impress. Measuring on your own examples costs about $10 and gives a number that maps to your bill.

How do I grade open-ended outputs? Use a fixed rubric or a stronger model as judge with a deterministic prompt, and route the judge's calls through the same harness so its cost is counted. Spot-check its grades against human labels on a sample before trusting it.

Does a cheaper model always win on cost? No. When wrong answers are cheap to catch, a cheap model with a frontier re-run stays cheapest. When wrong answers cost a human to fix, a more accurate model can be cheaper overall despite a higher per-token price, because it is wrong less often.

Sources

Prices change. We re-verify every figure in this post monthly and stamp updates. Numbers here are current as of 2026-07-20.

The lesson from the harness: the benchmark that decides your model is cheap to run and specific to your data, and cost per correct answer beats cost per token. Vynaris is an OpenAI-compatible gateway that exposes every model behind one base URL and reports per-request token usage, so the harness above runs against all of them without six SDKs. One base URL swap. Get an API key at vynaris.com.