VynarisEarly betaGet your API key

Blog · 2026-09-04 · Vynaris Team

Coding agent cost-per-task leaderboard: $0.39 Hermes vs $1.47 Claude Code across six agents

Composio ran Kimi K3 through six coding-agent harnesses on 26 tasks. Hermes cost $0.39/task, Claude Code $1.47, a 3.77x spread. At Uber's $1,500 cap, Hermes delivers 3,846 tasks vs 1,020.

Composio ran Kimi K3 through six coding-agent harnesses on 26 tasks. Hermes Agent cost $0.39 per task, Claude Code $1.47, a 3.77x spread. At Uber's $1,500 monthly cap, that is 3,846 tasks on Hermes versus 1,020 on Claude Code. Prices verified 2026-09-04.

TL;DR

What we computed, and why

Most cost-per-task comparisons hold the agent fixed and swap the model. Composio did the opposite: they held the model constant (Kimi K3 via OpenRouter) and swapped the software around it. The result is a leaderboard where every difference comes from the agent harness, not the model's sticker price.

This matters because teams pick models for cost when the harness is the larger lever. Our earlier coding-agent cost model priced one fixed trace across six models and found a 41x spread driven by per-token pricing. Composio's test holds pricing constant and shows that orchestration alone produces a 3.77x spread on the same bill.

The test used Kimi K3 at its list rate of $3.00/MTok input (cache miss), $0.30/MTok input (cache hit), and $15.00/MTok output, verified live from kimi.ai on 2026-09-04. All six harnesses ran the same 26 agentic tasks through the same hosted Composio MCP tools with maximum reasoning enabled.

The assumptions (edit these)

Assumption    Value                           Note
------------  ------------------------------  -----------------------------------------------------
Model         Kimi K3 (`moonshotai/kimi-k3`)  held constant across all harnesses
Provider      OpenRouter                      list pricing, no volume discount
Input price   $3.00/MTok                      cache miss; $0.30/MTok on cache hit
Output price  $15.00/MTok
Tasks         26                              identical instructions, tools, and connected app data
Reasoning     Maximum                         same level for every harness
Denominator   Cost per task                   not per token, not per call

The denominator is the load-bearing choice. Per-token pricing hides the harness tax because it cannot see how many times a harness re-sends context, how many tool calls it makes, or how its context window management inflates input tokens. Cost per task captures all of that in one number.

Results: cost per task, six harnesses

Harness       Avg $/task  Median $/task  Cost/1,000 tasks  vs cheapest (avg)
------------  ----------  -------------  ----------------  -----------------
Hermes Agent  $0.39       $0.29          $390              1.00x
Pi Agent      $0.40       $0.29          $400              1.03x
Codex         $0.47       $0.39          $470              1.21x
OpenCode      $0.51       $0.35          $510              1.31x
Kimi Code     $0.54       $0.38          $540              1.38x
Claude Code   $1.47       $0.72          $1,470            3.77x
Cost per task across six coding-agent harnesses running Kimi K3 on 26 identical tasks
Average and median cost per task across six harnesses, all running Kimi K3. Hermes $0.39, Pi Agent $0.40, Codex $0.47, OpenCode $0.51, Kimi Code $0.54, Claude Code $1.47. Source: Composio, captured 2026-09-04.

The average and median tell the same story. Hermes and Pi Agent are tied at the bottom on both measures. Claude Code sits alone at the top, 2.48x above the median leader and 3.77x above the average leader. The middle four harnesses cluster between $0.47 and $0.54 per task, a narrow band that suggests most harnesses land in the same neighborhood. Claude Code is the outlier.

If you are sizing a real budget, this is the moment to plug your own task count in: run the per-task math in the calculator with your daily task volume and the cost-per-task figures above.

Where the tokens go

Composio's earlier three-harness test (28 tasks) published the token counts behind the costs. The median task consumed 61,000 tokens on Kimi Code, 67,000 on Hermes, and 340,000 on Claude Code. That is a 5.6x token gap on the same task, same model, same tools.

Input tokens dominate agentic workloads. Composio estimates they make up roughly 95% of the token bill. At Kimi K3's $3.00/MTok input rate, 61,000 tokens cost $0.18 and 340,000 tokens cost $1.02 before any output tokens are counted. The published costs ($0.22 and $2.00) are higher, which means Claude Code's harness also generates more output tokens per task, not just more input.

Sebastian Raschka independently observed the same pattern with Qwen 3.6: Claude Code used 2 to 3 times more tokens than other harnesses at similar success rates. His analysis traced the gap to input tokens, not output. Claude Code's harness accumulates more context across multi-turn interactions, feeding previous messages, tool results, and file contents back into the model on each step.

The Uber frame: what $1,500/month buys

Uber capped AI coding tools at $1,500 per engineer per month after average costs hit $150 to $250 and heavy users reached $2,000. That cap is the buyer threshold. The question is not whether coding agents generate value (Uber reported 70% of committed code came from AI-assisted engineers) but how much work fits under a fixed budget.

Harness       Tasks/month at $1,500 cap (avg)  Tasks/month at $1,500 cap (median)
------------  -------------------------------  ----------------------------------
Hermes Agent  3,846                            5,172
Pi Agent      3,750                            5,172
Codex         3,191                            3,846
OpenCode      2,941                            4,286
Kimi Code     2,778                            3,947
Claude Code   1,020                            2,083

An engineer spending Uber's average $150 to $250 per month on Claude Code at $1.47/task completes 102 to 170 tasks. The same spend on Hermes at $0.39/task completes 385 to 641 tasks. At the $1,500 cap, Hermes delivers 3.77x more completed tasks than Claude Code for the same budget.

This is the per-request cost argument applied at the team level. The harness choice multiplies or divides your effective capacity before you touch the model.

The 8-harness corroboration

Composio later expanded the test to eight harnesses on 25 tasks. The cost-per-success spread widened. Claude Code's total estimated cost was $35.37 and Hermes Agent's was $9.28, a 3.81x ratio. Cost per successful task was $1.96 on Claude Code versus $0.46 on Hermes, a 4.26x spread that accounts for pass-rate differences.

Oh My Pi, a harness not in the six-agent test, posted the highest pass rate (22/25, 88%) at $0.52 per success. Claude Code passed 18/25 (72%), the same as Pi Agent and Grok Build. The harness with the highest cost did not produce the highest quality.

What it means for routing

The routing implication is narrow but direct. If your team runs Kimi K3 and your harness is Claude Code, switching to Hermes or Pi Agent cuts your per-task cost by 73% without changing the model. That is a larger saving than most model routing decisions produce, and it requires no gateway, no fallback logic, and no quality arbitration. You swap one binary for another.

For teams on Claude Code with Anthropic models rather than Kimi K3, the harness tax still applies. Our Hermes session-cost analysis showed that at 50+ turns, a Hermes session on Kimi K3 costs $4.56 versus $27.04 on Claude Fable 5. The harness gap compounds with the model gap. The per-call metering guide shows how to measure this on your own traces.

When this does not apply

Three honest limits on this data.

First, task complexity is not controlled. Composio's 26 tasks are business-application workflows (Gmail, Sheets, GitHub, Slack), not software-engineering tasks. A harness optimized for code editing may behave differently on a codebase refactor than on a calendar update. Claude Code is built for code, and its context-heavy strategy may pay off on harder tasks that this test does not measure.

Second, Moonshot warns that Kimi K3's performance can become unstable when a harness fails to return its thinking history correctly. This is a model-harness pairing test, not a general verdict on Claude Code. A harness optimized for one model family can impose overhead when used with another.

Third, Composio sells agent infrastructure. The test reinforces their product category. The token counts and costs are public and reproducible from Kimi K3's list pricing, but the task set and scoring are vendor-controlled.

FAQ

Should I switch from Claude Code to Hermes?

If you run Kimi K3 on business-application tasks similar to Composio's test set, the cost saving is 73% per task. If you run Anthropic models on software-engineering tasks, the gap may be smaller or larger. Measure your own per-task cost with the metering recipe before switching.

Why is Claude Code so much more expensive?

Its harness re-sends more context on each step. The median task used 340,000 tokens on Claude Code versus 61,000 on Kimi Code. Most of that is input tokens billed at $3.00/MTok. The harness accumulates previous messages, tool results, and file contents and feeds them back into the model across turns.

Is cost-per-task better than cost-per-token?

For agent workloads, yes. Per-token pricing cannot see how many times a harness re-sends context or how many tool calls it makes. Cost per task captures the full request shape in one number. The cost-per-token rate is still the input, but the task is the unit buyers budget against.

Does Claude Code produce better results?

In the 8-harness test, Claude Code passed 18 of 25 tasks (72%). Hermes Agent passed 20 (80%) and Oh My Pi passed 22 (88%). The most expensive harness was not the most accurate.

Sources

All arithmetic is reproducible in artifacts/coding-agent-cost-per-task-leaderboard-math.py.