VynarisEarly betaGet your API key

Blog · 2026-08-15 · Vynaris Team

Hard LLM spend caps: close the 32-worker race before dispatch

Atomic reserve/settle closes the LLM budget race across workers. A stale price table or low estimate can still break the cap. Here is the safe pattern.

A post-hoc budget check can let 32 workers dispatch $1.28 of calls against the same final $0.04. Atomic reserve/settle admits one $0.04 call when the estimate is exact. It still cannot save a stale or optimistic price table. Prices verified 2026-08-15.

TL;DR

Verdict table

Pattern                                    Concurrency-safe?  Estimate-safe?                 Crash-safe?        Use it?
-----------------------------------------  -----------------  -----------------------------  -----------------  -------------------------------------
Read remaining, dispatch, charge later     No                 No                             No                 Never for a hard cap
Atomic reserve, dispatch, settle actual    Yes                Only with a conservative hold  With TTL recovery  Default pattern
Atomic reserve with rolling p75            Yes                No, p75 has a tail             With TTL recovery  Good soft cap and throughput tradeoff
Provider-enforced max plus atomic reserve  Yes                Strongest available bound      With TTL recovery  Use when overshoot must be impossible

The core distinction is simple. A token budget can govern work before an API call. An invoice only exists after it. Your ledger needs a provisional liability between those moments.

The race: every worker sees the same money

Assume a $10.00 session has $0.04 left for research. The next call is estimated at $0.04. Thirty-two workers wake together.

The broken flow looks harmless:

if remaining_budget(session_id) >= estimated_cost(job):
    result = provider.call(job)
    charge(session_id, actual_cost(result))

Each worker reads $0.04 before any worker charges it. All 32 pass. If each call costs the estimate, the process dispatches 32 × $0.04 = $1.28 and exceeds the final $0.04 by (32 - 1) × $0.04 = $1.24.

Concurrent workers  Post-hoc dispatched cost  Overage above the last $0.04  Atomic reserve, exact estimate
------------------  ------------------------  ----------------------------  ------------------------------
1                   $0.04                     $0.00                         $0.04
4                   $0.16                     $0.12                         $0.04
8                   $0.32                     $0.28                         $0.04
16                  $0.64                     $0.60                         $0.04
32                  $1.28                     $1.24                         $0.04
Log-scale bars show post-hoc dispatched cost growing from $0.04 with one worker to $1.28 with 32 workers, while atomic reserve stays at $0.04.
Dispatched cost before settlement when every worker reads the same final $0.04. The post-hoc path grows with worker count; atomic reserve admits one exact-estimate call. Values are derived assumptions, not production data.

The worker count multiplies exposure, not the price itself. This is why a monthly alert cannot enforce a per-run limit. Alerts describe money already spent. Admission control decides whether the next liability may exist.

Reserve and check in one transaction

The safe sequence has four states: reserve, dispatch, settle or release.

BEGIN
  session = SELECT ... FOR UPDATE
  available = budget - spent - held - escrow
  IF estimate > available: reject
  INSERT reservation(status='held', amount=estimate, expires_at=...)
  UPDATE session SET held = held + estimate
COMMIT

dispatch provider call

BEGIN
  INSERT append_only_cost_rows(actual_usage, actual_dollars)
  UPDATE reservation SET status='settled'
  UPDATE session SET spent = spent + actual, held = held - estimate
COMMIT

The availability check and hold must commit together. A mutex inside one process is insufficient once jobs run across processes or machines. Use a row lock, serializable transaction, conditional update or another atomic compare-and-write supported by your database.

The settlement transaction matters too. The append-only cost row, spent increment and hold release should succeed or fail together. Otherwise a crash can leave the aggregate lower than the cost ledger or leave money held after it was charged.

Treat insufficient budget as normal control flow. It means backpressure worked. Do not retry it as a transient API error and create a more expensive loop.

A reservation estimate is not a hard bound

Atomicity closes the concurrency race. It does not predict how many output tokens the model will emit, how many retries a provider SDK will make, or whether a tool adds a separate fee.

Suppose six calls each reserve $0.04 from a $0.24 balance. If every call settles at $0.06, the ledger records $0.06 × 6 = $0.36. That is $0.12 above the available balance even though reservation admission was perfectly serialized. The estimate failed; the transaction did not.

Use one of three reservation policies:

  1. Conservative ceiling. Price max_input_tokens + max_output_tokens + worst_allowed_tool_fees + allowed_retries. This underuses budget when calls finish early, but it is the closest application-level hard bound.
  2. Quantile estimate. Reserve a rolling p75 or p95 by model and task class. This admits more work. It is a soft cap because the unreserved tail still exists.
  3. Two-stage bound. Reserve the likely cost, then require another atomic hold before any retry, escalation or paid tool. A cheap first attempt cannot silently authorize an expensive second attempt.

For an actual hard dollar cap, combine the conservative hold with provider controls: set max_tokens, disable unbounded automatic retries, cap paid tool calls and cancel streaming when the provider supports enforceable limits. You cannot reverse an API call after the provider has done the work.

The public Mole repository implements a rolling p75 after five observations, seeded conservatively before that. Its README reports 0% budget overshoot across its test corpus. That is publisher-reported evidence for this implementation, not proof that p75 reservations guarantee zero overshoot on another workload.

Settle failures because failed calls still cost money

A provider timeout can arrive after token generation. A validation failure can reject an answer after the model has billed it. A retry can create a second bill before the first response's accounting reaches your worker.

Settle every usage record you receive, including partial usage from failed work. Then reserve again for the retry. The public Mole ledger implementation follows this rule: failures still settle cost, and each retry takes its own reservation.

Make settlement idempotent. Give every provider attempt and reservation a stable ID. A worker that times out after the database commits can replay settlement without double-charging. This is the budget version of request deduplication.

Release dead-worker holds with a TTL

A worker can die after reserving and before dispatch. Without recovery, the hold lives forever and the budget appears exhausted even though no call happened.

Store expires_at on each reservation. Sweep expired holds and release them atomically. Mole's public default is 15 minutes. That number is an implementation choice from its source, not a universal timeout. Your TTL should exceed the normal provider-call duration plus queue jitter, and workers should renew the hold for legitimately long calls.

Do not release a hold merely because the client stopped waiting. First determine whether the provider accepted the request. If that answer is unknown, keep the liability until reconciliation. Premature release can admit replacement work while the original call is still billable.

Keep output escrow separate

Research agents have a particularly ugly failure mode: they spend the entire budget collecting evidence and leave nothing to generate the answer.

Reserve an output escrow when the session starts. If the total budget is $10.00 and the configured escrow is 15%, research begins with $10.00 × (1 - 0.15) = $8.50; $1.50 stays unavailable until the output phase. Mole uses 15% as a starting default and says it should be calibrated from real ledgers.

Escrow is not free money. Release it only when research stops, then reserve the final generation call from that balance. If output needs less, return the remainder. Our team spend-ceiling guide covers the slower monthly pace problem; this pattern handles concurrent liabilities inside one run.

Version prices like code

A ledger can reconcile perfectly and still disagree with the provider invoice because its rates are stale. That failure is silent: every internal sum matches the wrong table.

The public Mole pricing table gives two useful publish-day examples on our 8,000-input/2,000-output task:

Model and rate state                                                                  Table bill /1,000 tasks  Live/effective bill                        Reservation effect
------------------------------------------------------------------------------------  -----------------------  -----------------------------------------  -------------------------------------------
[Claude Sonnet 5](https://vynaris.com/models#claude-sonnet-5), stale $3/$15           $54.00                   $36.00 at live $2/$10                      50% over-reserve; safe but strands capacity
[DeepSeek-V4-Flash](https://vynaris.com/models#deepseek-v4-flash), stale $0.14/$0.28  $1.68                    $3.08 off-peak after 2026-08-16 16:00 UTC  Actual is 1.83× the hold
DeepSeek-V4-Flash, same stale row                                                     $1.68                    $6.16 peak after cutover                   Actual is 3.67× the hold

Before fixing a hold size, price your own input/output shape in the calculator. The 8,000/2,000 shape here is an editable example, not a universal call profile.

Anthropic's live pricing page says Sonnet 5's $2/$10 rate is now standard and the planned $3/$15 increase will not occur. DeepSeek's live pricing page publishes the 2026-08-16 cutover and new off-peak/peak rows. Prices verified 2026-08-15.

On DeepSeek, stale off-peak reservation is 45.5% below the actual bill: 1 - $1.68 / $3.08 = 45.5%. At peak it is 72.7% low: 1 - $1.68 / $6.16 = 72.7%. A default overshoot alert that fires only above 2× would miss the 1.83× off-peak error.

Store prices with provider, model_id, effective_from, effective_to, input/output/cache/tool rates, source URL and captured timestamp. Resolve the rate at reservation time and persist the version on the hold. Reprice open holds when a scheduled cutover occurs before dispatch.

Our per-call dollar metering guide maps provider usage fields to billable categories. The DeepSeek price-reset analysis supplies the full cutover math used here.

Honest tradeoff

Conservative reservations reduce utilization. If every call reserves its worst case, a $10 budget may admit far fewer calls than it eventually could have paid for. Quantile holds admit more work but stop being a hard guarantee.

Choose based on damage. For a demo, a small bounded overage may be cheaper than idle budget. For a tenant credit limit, prepaid account or compliance rule, reserve the conservative maximum and accept lower utilization. Calling both policies “hard caps” is how finance discovers the distinction for you.

Write that policy beside the cap so an incident responder knows whether to preserve utilization or stop spend immediately.

Vynaris option

Vynaris can keep model choice and per-request cost receipts outside application code. Use it when one gateway should apply model policy across workers. The reserve/settle ledger still belongs in your application because only your application knows the tenant budget and business outcome.

Implementation checklist

FAQ

Why not use a Redis counter?

You can, if one atomic operation checks availability and creates the hold, and settlement remains idempotent. The data store matters less than the state transition. A GET followed by a separate DECR has the same race as the SQL anti-pattern.

Is rolling p75 enough for a hard cap?

No. It is an admission estimate. One quarter of the observed distribution is above p75 by definition, and price or workload drift can move the whole distribution. Use a conservative upper bound when the cap must not move.

What happens when actual cost exceeds the reservation?

Record the actual charge because the provider already earned it, flag the estimate miss, and stop admitting work if the remaining budget is gone. Never clip the ledger to the reservation; that makes the dashboard look compliant while the invoice is not.

How should retries be budgeted?

Each retry gets a new atomic reservation. Do not let the first call's hold authorize the second call. Cap retries explicitly and include any provider-side automatic retry behavior in the worst-case bound.

When is this machinery unnecessary?

If one worker makes one bounded call and a small overage has no consequence, a simple post-call meter plus a max_tokens limit may be enough. Add the ledger when concurrency, tenant isolation or prepaid balances make overshoot material.

Sources