Blog · 2026-08-09 · Vynaris Team
Coding agents miss their own budgets: pricing a 30x token spread and 0.39 estimate ceiling
Same-task coding agents can burn 30x more tokens; self-estimates top out at 0.39 correlation. On the paper GPT-5 shape, GPT-5.6 Terra is $1.80 at p50 and $54.10 at 30x — hard ceilings beat preflight guesses.
Same-task coding-agent runs can burn up to 30x more tokens than another attempt, and self-estimates top out at 0.39 correlation while systematically under-predicting. On the paper's GPT-5 token shape, today's GPT-5.6 Terra list price is $1.80 at p50 and $54.10 at a 30x stretch, so hard ceilings beat asking the agent for a preflight number.
Prices verified 2026-08-09.
TL;DR
- Bai et al. study eight frontier models on SWE-bench Verified with OpenHands. Same-task runs differ by up to 30x in total tokens. Average max/min on the same problem is about 2x.
- Self-prediction correlations peak at 0.39. In the authors' public CSV, GPT-5 under-predicts input on 100% of 500 instances, by 13.5x on the mean.
- Repricing the GPT-5 p50/p90/p99 shape at 2026-08-09 Terra rates gives $1.80 / $4.26 / $6.91 per completed task with all input billed uncached. A 30x stretch of that p50 shape is $54.10.
- Claude Sonnet 4.5 and Kimi-K2 used 3.17M and 2.25M more tokens than GPT-5 on average. At Terra uncached rates that is $6.53 and $4.61 of extra spend per task, not a quality rebate.
Verdict table
Control idea What the public evidence supports Cost verdict
-------------------------------------------- ----------------------------------------------------------- ------------------------------------
Ask the agent for a preflight token estimate Correlation up to 0.39; models systematically underestimate Too weak for a hard budget
Budget from human difficulty labels Kendall τ_b = 0.32; large overlap across bands Weak predictor of spend
Spend more tokens to raise accuracy Accuracy peaks at intermediate cost, then saturates Extra tokens are not a success lever
Meter p50 / p90 / p99 and kill the tail Empirical distributions are published and repriceable Use this as the operating control
Cap calls and tokens before the run Directly bounds the 2x typical and 30x extreme spreads Beats point estimatesThe useful product change is not a smarter self-estimate prompt. It is percentile metering plus a kill switch.
What we computed and why
Bai, Huang, Wang, Sun, Mihalcea, Brynjolfsson, Pentland, and Pei publish How Do AI Agents Spend Your Money? (arXiv:2604.22750). They run OpenHands on SWE-bench Verified across eight models, four independent runs per problem, and then ask the same agent to estimate tokens before solving.
This article is not a paper recap. We take their published averaged token CSV, compute p50/p90/p99 shapes, and reprice those shapes on live 2026-08-09 stickers. That turns stochastic token findings into cost per task controls.
It is also distinct from our pre-build workflow forecast. That post starts from editable call-graph assumptions. This one starts from measured run distributions and asks what budget machinery still works when the agent cannot price itself.
Assumptions table
Assumption Value Why it is editable
--------------------------- ------------------------------------------------------------------------ ---------------------------------------------------------------------------------------
Workload SWE-bench Verified, OpenHands, 500 instances Public paper setting
Token source Authors' public averaged CSV (`all_models_averaged_predictions_new.csv`) Primary receipt for means and percentiles
Shape used for live pricing GPT-5 empirical p50 / p90 / p99 input and output Cheapest mean model in the study; conservative for “efficient agent” baselines
Live rates OpenAI / Anthropic / DeepSeek / Gemini first-party pages, 2026-08-09 Historical paper models are not today's SKUs
Scenario A 100% input at list input rate Upper-bound invoice if caching fails
Scenario B 85% input as cache read, 15% fresh Disclosed proxy for “cache reads dominate volume”; not a measured hit rate from the CSV
Same-task multipliers 1x, 2x, 30x on the GPT-5 p50 shape Paper average ~2x; abstract maximum 30x
Dollar outputs USD per completed task Not per 1M tokens alonePaper model names stay as published: GPT-5, GPT-5.2, Claude Sonnet 3.7 / 4 / 4.5, Kimi-K2, Qwen3-Coder-480B-A35B-Instruct, Gemini-3-Pro. Live dollar rows use today's SKUs on the measured GPT-5 shape.
Empirical token shapes
From the public CSV of 500 instance means:
Paper model Mean input Mean output p50 total p90 total p99 total Mean input / predicted input
----------------- ---------- ----------- --------- --------- ---------- ----------------------------
GPT-5 978,305 22,775 806,475 1,938,348 3,068,015 13.5x
GPT-5.2 1,304,002 24,104 948,876 2,576,683 5,680,857 13.3x
Claude Sonnet 4.5 4,142,511 25,337 3,804,293 6,465,025 10,386,576 64.0x
Kimi-K2 3,235,263 16,211 2,409,361 6,559,038 11,672,512 8.7x
Gemini-3-Pro 2,179,129 14,982 1,639,912 4,436,765 9,407,963 26.9xInput tokens dominate every row. That matches the paper: agentic coding is input-heavy even with prompt caching, because history and tool output keep re-entering context.
Across problems, GPT-5 mean totals already span 20.8x from cheapest to dearest instance. The paper's sharper operational warning is run-to-run: the same problem can still move by about 2x on average and up to 30x in the extreme.
Dollars on today's stickers
We hold the GPT-5 p50 / p90 / p99 token shape fixed and apply 2026-08-09 list rates. Scenario A bills every input token at the uncached input rate.
Live model $/1M in $/1M out p50 $/task p90 $/task p99 $/task
------------------------------------------------------------------- ------- -------- ---------- ---------- ----------
[GPT-5.6 Luna](https://vynaris.com/models#gpt-5-6-luna) 0.20 1.20 0.18 0.43 0.69
GPT-5.6 Terra 2.00 12.00 1.80 4.26 6.91
[GPT-5.6 Sol](https://vynaris.com/models#gpt-5-6-sol) 5.00 30.00 4.51 10.65 17.28
[Claude Haiku 4.5](https://vynaris.com/models#claude-haiku-4-5) 1.00 5.00 0.88 2.09 3.38
[Claude Sonnet 5](https://vynaris.com/models#claude-sonnet-5) intro 2.00 10.00 1.76 4.18 6.75
[Claude Opus 5](https://vynaris.com/models#claude-opus-5) 5.00 25.00 4.41 10.45 16.88
DeepSeek v4-flash 0.14 0.28 0.12 0.28 0.44
Gemini 3.1 Pro (<=200k tier) 2.00 12.00 1.80 4.26 6.91Scenario B (85% cache reads) cuts the Terra p50 row from $1.80 to $0.60. That is a disclosed sensitivity, not a promise. If your prefix is unstable, use Scenario A for the budget.
Plug your own p50/p90/p99 counts into the LLM cost calculator at the moment you set the monthly agent budget. Keep input, output, cache-hit fraction, and kill thresholds visible. A single mean estimate is not a plan.
Same-task spread priced in dollars
Hold the GPT-5 p50 shape and scale it by the paper's average and extreme same-task multipliers.
Spread Input tokens Output tokens Terra uncached Terra 85% cache Luna uncached Sonnet 5 uncached
------------------- ------------ ------------- -------------- --------------- ------------- -----------------
1x (p50) 786,472 19,187 $1.80 $0.60 $0.18 $1.76
2x (paper average) 1,572,944 38,374 $3.61 $1.20 $0.36 $3.53
30x (paper maximum) 23,594,168 575,602 $54.10 $18.00 $5.41 $52.94
A team that budgets only the p50 mean will be wrong on ordinary 2x days and catastrophically wrong on a 30x day. The control that survives is a hard token or dollar ceiling near your chosen percentile, not a one-shot self-estimate.
Efficiency gaps are behavioral, not task difficulty
On the same 500 problems, Claude Sonnet 4.5 averaged 3.17M more total tokens than GPT-5. Kimi-K2 averaged 2.25M more. The paper shows the ranking holds on the shared-success subset (n=230) and the shared-failure subset (n=100). The expensive models are not simply “working harder on harder tasks.”
Priced as extra Terra-uncached spend on each model's own in/out mix, those gaps are $6.53 and $4.61 per completed task. High-cost runs also show more repeated file views and edits. That is wasted context growth, not a reliable path to higher accuracy.
This is why model routing still matters after sticker comparison. Route for measured token efficiency and stop rules, not for the model's self-reported confidence about cost. Our coding-agent cost-per-task model prices a fixed step graph; here the lesson is that the graph length itself is stochastic across runs.
Why self-estimates fail as budget controls
The paper's prediction task lets the agent inspect the repo and tools, then output input, output, and total estimates before solving. Correlations with real usage are weak to moderate. Output is easier than input. Sonnet 4.5 peaks at 0.39 on output tokens. Gemini-3-Pro trails on both.
Direction is worse than correlation. Models systematically underestimate. In the public CSV, GPT-5's mean predicted input is 72,272 tokens against 978,305 actual, a 13.5x miss, and every one of the 500 instances under-predicts input.
Human difficulty labels are not a rescue hatch. Expert time bands only weakly track spend (Kendall τ_b = 0.32). 6.7% of “under 15 minute” tasks used more tokens than the average “over 1 hour” task.
So neither the agent nor the human label is decision-grade for a hard dollar cap. Use them as coarse warnings at most.
What to put in production instead
- Log input, output, cache-read, and cache-write tokens per run. Store p50/p90/p99 by workflow and model.
- Set a kill switch at a chosen percentile, plus a max-call ceiling. Default candidates from this study: warn at p90, stop at p99, or stop at 2x your measured p50 if you lack history.
- Prefer early stop on repeated file-view / modify loops. The paper links high-cost failures to redundant exploration.
- Do not raise spend hoping for accuracy. Intermediate cost already captures most of the available success.
- Reprice the retained percentile shape whenever stickers move. The token distribution is the durable asset; the dollar row is a date-stamped overlay.
Per-request cost and token ceilings are boring. They are also the only controls that survive a 30x draw.
Honest tradeoff: when not to over-constrain
Do not put a tight p50 ceiling on exploratory research runs where a human is watching and can abort. You will truncate useful search and create false “model is dumb” conclusions.
Do put hard ceilings on unattended batch agents, overnight repo sweeps, and any workflow where a 30x tail can silently burn the monthly budget. If the task is worth unbounded exploration, price that as a human-supervised mode, not as default production.
Also do not treat Scenario B's 85% cache fraction as universal. Cache helps only while prefixes stay stable. Our scaffolding overhead report shows how fixed instruction and tool schema taxes compound before the agent reads the user request.
FAQ
Is the 30x figure the average same-task spread?
No. The paper reports up to 30x between runs on the same task. The average max/min across instances is about 2x. Budget for both.
Can we trust an agent's preflight token estimate?
Not for a hard cap. The best reported correlation is 0.39, and underestimation is systematic.
Why reprice on GPT-5.6 Terra if the paper used GPT-5?
The paper models and rates are historical. The durable receipt is the token shape. We keep that shape and apply live stickers so the dollar rows are actionable on 2026-08-09.
Does spending more tokens raise SWE-bench accuracy?
Not reliably. Same-problem accuracy rises from the cheapest run to an intermediate band, then saturates. The highest-cost quartile is not the accuracy winner.
What should replace point estimates?
p50/p90/p99 metering, call ceilings, and automatic stop on redundant tool loops. Use self-estimates only as a coarse relative warning.
Sources
- Bai et al., arXiv:2604.22750, captured 2026-08-09: 30x same-task spread, 0.39 self-estimate ceiling, accuracy saturation, 1.5M+ token efficiency gaps, difficulty misalignment, and prediction underestimation.
- Public averaged predictions CSV, captured 2026-08-09: 500-instance mean input/output tokens and self-predictions used for percentiles and underprediction ratios.
- OpenAI API pricing, captured 2026-08-09: GPT-5.6 Luna / Terra / Sol list and cache-read rates.
- Anthropic pricing docs, re-verified 2026-08-14: Claude Haiku 4.5, Sonnet 5 standard, and Opus 5 rates.
- Jiaxin Pei's announcement, captured 2026-08-09: public demand signal only.
All arithmetic is in the accompanying Python script. No Vynaris traffic, customers, or routing internals appear in these numbers.
Further reading: Autolith's three public agent traces reprice 59.6K to 601.1K-token runs at current Terra and Sol API rates while separating subscription execution from API cost.
Further reading: OpenRouter’s Hermes session-cost medians show why turn bands beat token stickers.