VynarisEarly betaGet your API key

Blog · 2026-09-14 · Vynaris Team

16,893 agent experiments need an $866 Terra token budget before search and sandboxes

At current rates, a modeled 16,893-run Terra experiment costs $865.77 in model tokens before search and sandbox compute.

Armature ran 16,893 coding-agent sessions and published 5,292 of them. A reproducible scenario using GPT-5.6 Terra as the actor costs $865.77 in model tokens. Search and a five-minute E2B CPU floor lift it to $1,159.03. That is our model, not Armature's invoice. Prices verified 2026-09-14.

TL;DR

Verdict table

Illustrative actor                                               Price per 1M input/output tokens  Actor/run  Total/run with evaluation  16,893 runs  5,292 published sessions
---------------------------------------------------------------  --------------------------------  ---------  -------------------------  -----------  ------------------------
[Gemini 3.7 Flash](https://vynaris.com/models#gemini-3-7-flash)  $0.75 / $3.75                     $0.01350   $0.02475                   $418.10      $130.98
[Claude Sonnet 5](https://vynaris.com/models#claude-sonnet-5)    $2 / $10                          $0.03600   $0.04725                   $798.19      $250.05
GPT-5.6 Terra                                                    $2 / $12                          $0.04000   $0.05125                   $865.77      $271.22
[GPT-5.6 Sol](https://vynaris.com/models#gpt-5-6-sol)            $4 / $20                          $0.07200   $0.08325                   $1,406.34    $440.56
[Claude Opus 5](https://vynaris.com/models#claude-opus-5)        $5 / $25                          $0.09000   $0.10125                   $1,710.42    $535.82

The table is a planning tool. Armature did not publish its token counts, actor model IDs, search-query count, runtime, or invoice. The 5,292 sessions are a publication subset, not a pass-rate denominator.

What Armature disclosed

Armature tested which developer tools Claude Code, Codex, and Cursor select and install. Its public study covers 75 repositories, 1,163 prompt variations, four user profiles, three coding agents, and 18 tool sectors. Each run used an ephemeral sandbox.

The experiment added two more model roles. One Gemini 3.7 Flash instance acted as a simulated human. It approved the top recommendation or asked the coding agent to choose. Another instance judged whether the session was valid and extracted the winner from the conversation and code diff.

Armature says it ran 16,893 sessions and initially published 5,292 valid sessions. It explicitly says the remaining runs were not necessarily discarded. Dividing 5,292 by 16,893 gives 31.33%, but calling that a success rate would be false precision.

This distinction matters. A publication filter can reflect incomplete analysis, scope choices, invalid repositories, or task failures. The source does not provide counts for each reason.

Editable assumptions

Parameter                                Scenario value                          Status
---------------------------------------  --------------------------------------  -------------------------------------------------------
Total sessions                           16,893                                  Armature disclosure
Published sessions                       5,292                                   Armature disclosure; not a pass rate
Actor usage per run                      8,000 input / 2,000 output tokens       Illustrative
Supervisor usage per run                 2,000 input / 500 output tokens         Illustrative
Judge usage per run                      8,000 input / 500 output tokens         Illustrative
Supervisor and judge model               gemini-3.7-flash                        Armature names Gemini 3.7 Flash; stable ID verified
Search-using sessions                    64%                                     Rounded equal-weight average of about 67%, 94%, and 30%
Search queries per search-using session  1                                       Deliberately conservative floor
Sandbox runtime                          300 seconds per run                     Illustrative
Sandbox compute                          E2B default 2 vCPU at $0.000028/second  Public rate; CPU only

The actor usage is a fixed task shape, not a claim about the three coding products. It treats each full agent loop as one token bucket. That bucket includes the cumulative input token cost and output token cost from all model calls in a run.

The $865.77 token bill, line by line

At $2 per 1M input tokens and $12 per 1M output tokens, the Terra actor costs:

8,000 × $2 / 1,000,000 + 2,000 × $12 / 1,000,000 = $0.0400 per run

The supervisor uses Gemini 3.7 Flash at $0.75/$3.75 per 1M tokens:

2,000 × $0.75 / 1,000,000 + 500 × $3.75 / 1,000,000 = $0.003375

The judge reads more context:

8,000 × $0.75 / 1,000,000 + 500 × $3.75 / 1,000,000 = $0.007875

Together, the evaluator calls cost $0.01125 per run. The modeled per-request cost is therefore $0.0400 + $0.01125 = $0.05125. Across the disclosed run count:

16,893 × $0.05125 = $865.76625, rounded to $865.77.

The actor portion is $675.72. The supervisor and judge add $190.05. Terra therefore drives 675.72 / 865.77 = 78.0% of the token bill.

Use the cost calculator when your trace has a different token shape. The companion Python model also emits a CSV with every row.

Search is a separate meter

Armature reports that Cursor used the web in roughly two-thirds of sessions, Codex in 94%, and Claude Code in about 30%. The study does not disclose the run mix across those agents or the number of queries per session.

An equal-weight average, rounded to 64%, gives 16,893 × 0.64 = 10,811.52 search-using sessions. At Google's $14 per 1,000 search queries, a one-query floor costs:

10,811.52 × $14 / 1,000 = $151.36128, or $151.36 gross.

Google lists 5,000 free requests per month across Gemini 3.x models. If the full allowance were unused, the paid remainder would be (10,811.52 - 5,000) × $0.014 = $81.36. A submitted request may create more than one billable search query, so neither number is a ceiling.

This is a sensitivity test, not a reconstruction. Codex and Cursor do not necessarily buy Google grounding for these searches. Their internal search costs are not public in the study.

Sandbox compute can rival the search line

Armature rotated among E2B, Blaxel, and Daytona. It gives no provider split or runtime. We can still build one transparent floor from E2B's live rate card.

E2B lists its default 2-vCPU sandbox at $0.000028 per second. Five minutes costs 300 × $0.000028 = $0.0084 per run. Across 16,893 runs, CPU alone costs $141.90.

That floor excludes RAM, storage, paid-plan fees, and the other two providers. Adding gross search and this CPU floor to the Terra token model yields $865.77 + $151.36 + $141.90 = $1,159.03. It still is not a complete experiment budget.

What this means for routing

The actor model is the first budget decision. Cost per task ranges from $0.02475 with Gemini 3.7 Flash to $0.10125 with Claude Opus 5 after holding both evaluator roles fixed. That is a 4.09x spread.

The order changes at the middle. Claude Sonnet 5 costs $798.19, which is $67.57 less than Terra under this token shape. Terra and Sonnet have the same input price, but Sonnet output costs $10 instead of $12 per 1M tokens.

When the actor is Gemini 3.7 Flash, the fixed evaluation layer is 45.5% of the token bill. When the actor is Opus, it is 11.1%. Optimizing the judge first makes sense only after the actor is already cheap.

The missing denominator is quality. Armature's public article does not split valid sessions by actor model or publish accepted-task rates for these illustrative API models. A model routing decision needs spend divided by accepted outcomes, not spend alone. Our evaluation harness guide shows how to collect that denominator before a large run.

Our LLM judge sampling analysis reaches the same operational conclusion at a different scale: the evaluator bill is easy to shrink, but a bad judge silently corrupts the result.

The token trace matters more than the headline run count. Holding the $0.01125 evaluation layer fixed gives this Terra sensitivity:

Actor input/output tokens per run  Terra token budget for 16,893 runs
---------------------------------  ----------------------------------
8,000 / 2,000                      $865.77
20,000 / 5,000                     $1,879.35
50,000 / 10,000                    $3,906.51
100,000 / 20,000                   $7,622.97

The 100k/20k scenario costs 8.8x the base scenario even though the session count never changes. Before choosing a route, export usage by role. An average across the whole harness hides which actor or judge created the long tail.

Honest tradeoff

Do not route the actor to the cheapest model from this table without an evaluation. The $418.10 Gemini scenario saves $447.66 versus Terra, but the public study does not show whether that substitution preserves tool choice, implementation quality, or completion rate.

Batch inference can reduce model prices when a provider supports it. It may not fit this design. The simulated human sits inside a multi-turn loop, so delaying one role can leave an expensive sandbox idle.

Prompt caching may help repeated judge instructions. We exclude it because Armature did not disclose cache hits or stable-prefix lengths. A lower unverified number would make the spreadsheet prettier and the budget worse.

Caveats

Sources