Blog · 2026-08-09 · Vynaris Team

LMCache's 14x throughput claim: when it means 92.9% lower GPU cost

LMCache's 14x throughput implies 92.9% lower GPU cost per request only after a saturated fleet is right-sized. Fixed GPUs keep the same invoice.

LMCache's 14x throughput result can mean 92.9% lower GPU cost per request, but only when demand saturates the baseline fleet and operators remove the freed capacity. At the paper's low QPS example, keeping the same GPUs cuts latency, not the invoice. The paper publishes no dollar bill for DRAM, storage, transfer, or operations.

Prices verified 2026-08-09.

TL;DR

• The LMCache paper reports 2.3x to 14x higher query throughput at the same time to first token than its strongest baseline across five models in its CPU-offload experiment.
• Pure capacity math converts 2.3x to 14x throughput into 56.5% to 92.9% lower GPU cost per completed request: 1 - 1 / throughput factor.
• In a disclosed scenario with eight H100s at GMI Cloud's $2 per GPU-hour starting price and one baseline request per second, GPU-only cost falls from $4.444 to $1.932 or $0.317 per 1,000 requests.
• The paper did not measure that dollar row. It used an eight-H100 server, allocated the minimum GPUs needed per model, and allowed up to 500GB of CPU DRAM.
• A fixed eight-GPU fleet still bills $11,680 over a 730-hour month in our price scenario. Higher throughput saves money only after consolidation, scale-down, or demand growth.

Verdict table

Claim                      What the public evidence supports                                                        Cost verdict
-------------------------  ---------------------------------------------------------------------------------------  --------------------------------------------------
“14x throughput”           Yes, in the paper's CPU-offload experiment at equal TTFT                                 Conditional, workload-specific result
“93% cheaper”              92.9% lower GPU cost per request if capacity scales down by 14x                          Valid GPU-only derivation, not an observed invoice
“Faster at low load”       1.9x to 8.1x lower TTFT at QPS=1                                                         Better latency; same fixed fleet bill
“Cheaper infrastructure”   GPU saving may be offset by CPU DRAM, storage, transfer, and operations                  Benchmark the all-in stack
“Works for every context”  No; at 32Gbps, the paper's B200 sensitivity beats prefill only beyond 256K input tokens  Cache loading has a crossover point

The useful answer is narrower than the viral one. LMCache can turn repeated long prefixes into GPU capacity. It cannot turn a latency multiplier into invoice savings without a scaling event.

What the paper actually measured

LMCache moves KV cache outside GPU memory and reuses it across requests and inference engines. Its hierarchy can use CPU memory, local disk, remote disk, or Redis. The implementation integrates with vLLM and SGLang and exposes lookup, movement, pinning, and compression controls through its open-source repository.

The main CPU-offload experiment uses multi-round document question answering. Each query contains 10K input tokens by default. The paper uses 20K input tokens for Llama-3.1-8B-Instruct and caps output at 100 tokens. Forty users start the session, then new users arrive at a configured QPS. LMCache may use up to 500GB of CPU memory.

The testbed is an eight-H100 server from GMI Cloud. The researchers allocate the minimum H100 count required to start each model. That distinction matters: “an eight-H100 server” does not prove every row burned eight GPUs.

The setup lists meta-llama/Llama-3.1-8B-Instruct, Llama-3.1-70B-Instruct, Qwen/Qwen2.5-Coder-32B-Instruct, Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8, Qwen/Qwen2.5-72B-Instruct, and a paper-reported Sao10K-L3-8B model. We preserve the source's identifiers instead of renaming model families.

At QPS=1, LMCache reports 1.9x to 8.1x lower time to first token. At equal TTFT, it processes 2.3x to 14x more queries than the strongest baseline across five evaluated models. Inter-token latency is 7% to 92% lower at QPS=1. Those are performance results, not cloud charges.

The paper's abstract says up to 15x throughput across its broader set of workloads. We audit 14x because that is the stated upper bound for the specific CPU-offload comparison behind the “93% cheaper” interpretation.

Turning throughput into GPU cost

The conversion is simple when the assumptions are explicit:

new GPU cost per request = baseline GPU cost per request / throughput factor
GPU cost reduction       = 1 - 1 / throughput factor

At 2.3x, 1 - 1/2.3 = 56.5%. At 14x, 1 - 1/14 = 92.9%. This is the same capacity identity used in any saturated service. It does not require a special LLM billing rule.

To make the dollars concrete, we use GMI Cloud's current starting rate of $2 per H100 GPU-hour. The paper names GMI Cloud as its hardware provider, but does not say that its experiment paid this exact rate. We therefore label the row as a scenario.

Throughput factor  Scenario throughput   GPU-only $/1K requests  Saving vs baseline
-----------------  --------------------  ----------------------  ------------------
1x baseline        3,600 requests/hour   $4.444                  0%
2.3x               8,280 requests/hour   $1.932                  56.5%
14x                50,400 requests/hour  $0.317                  92.9%

The scenario assumes eight GPUs, $2 x 8 = $16/hour, and one baseline request per second. Baseline cost is $16 / 3,600 x 1,000 = $4.444 per 1,000 requests. Divide by each throughput factor for the other rows.

Replace the one-request-per-second baseline with your measured rate in the LLM cost calculator. Keep the H100 count, hourly rate, token shape, latency target, cache-hit distribution, and batch policy visible. A result without those fields is not a capacity plan.

Why a faster server can send the same invoice

A reserved GPU bills by time, not completed requests. In our scenario, eight H100s running for 730 hours cost 8 x $2 x 730 = $11,680. That invoice remains $11,680 whether the fleet processes one request per second or fourteen.

The saving appears in one of three ways:

1. A multi-node fleet consolidates onto fewer replicas while holding the same latency target.
2. Autoscaling removes idle GPU-hours after demand falls.
3. Demand grows into the new headroom without buying more GPUs.

The first two lower spend. The third avoids future spend. None happens automatically after a benchmark.

Hardware granularity also bites. A service using one minimum-size model replica cannot buy one-fourteenth of a server. If the model needs eight H100s to fit, a 14x throughput gain may create headroom but no immediate GPU reduction. The economic curve becomes stepwise, not smooth.

Our self-hosting break-even audit reaches the same conclusion from the opposite direction: GPU utilization and minimum fleet size decide cost per task, not the hourly sticker alone.

The missing non-GPU denominator

LMCache trades recomputation for a cache hierarchy. The total cost equation is:

total hourly cost = GPU-hours + CPU DRAM + local storage + remote storage + transfers + operations

Cost line                Public paper input                                      What an operator must add
-----------------------  ------------------------------------------------------  ----------------------------------------------------
GPUs                     H100 testbed; minimum count varies by model             Actual GPU count and contracted $/GPU-hour
CPU DRAM                 Up to 500GB in the CPU-offload experiment               $/GB-hour and reserved host cost
Local or remote storage  Supported backends are described                        Capacity, read/write, and request charges
Network transfers        Ethernet, RDMA, NVLink, and remote paths are evaluated  Egress, cross-zone, fabric, and switch costs
Operations               Open-source software                                    Deployment, upgrades, monitoring, and incident labor

The paper publishes no absolute dollar bill for these lines. GMI Cloud's page lists H100 from $2 per GPU-hour and says its bare-metal offer has no egress fees. That does not price another operator's DRAM, storage, or network topology.

This is why “14x throughput equals 93% lower infrastructure cost” overclaims the paper. “Up to 92.9% lower GPU cost per request after right-sizing” is defensible.

Where the result stops transferring

The workload has long, repeated prefixes and short outputs. That shape rewards prompt caching. A stream of unrelated prompts has little reusable state and will not reproduce the result.

Network speed changes the decision. In the paper's B200 sensitivity study, KV loading over a 32Gbps network beats fresh prefill only when input exceeds 256K tokens. At 64Gbps or 128Gbps, loading wins across the tested context lengths. Cache placement needs a measured crossover, not a global “always load” rule.

Context truncation can halve prefix-cache hit ratio, according to the paper's deployment lessons. An application that trims or rewrites its prefix may destroy the reuse assumed by the benchmark. So can frequent document updates, low repeat-query probability, or an unstable tokenizer.

Commercial baselines are black boxes in the paper. Their internal cache policy is unknown. The 14x result therefore does not establish a universal lead over every tuned vLLM deployment, every storage backend, or a future version of the same systems.

A deployment test that produces a decision

Replay public or synthetic traces that preserve your input-length, output-length, arrival-rate, and prefix-reuse distributions. Do not upload private customer prompts into an external benchmark harness.

Measure p50 and p95 TTFT, inter-token latency, completed requests per second, GPU utilization, CPU DRAM occupancy, cache-hit tokens, bytes transferred, and errors. Run baseline and LMCache at the same latency target. Saturate both systems before comparing capacity.

Next, reduce replicas until the optimized system approaches the target. That step converts performance into a bill. Price every remaining node and storage line from the same deployment region. Use cost per task, not tokens per second, as the final operating metric.

Cache-aware model routing can improve reuse, but it can also create hot replicas. The paper's controller exposes cache locations so a router can balance expected hits against load. Our prompt-caching production guide covers the billing and prefix-stability checks on managed APIs; self-hosted stacks need the same discipline at the GPU layer.

Honest tradeoff: when not to add LMCache

Do not add a distributed cache layer to low-QPS, short-context, low-reuse inference. Basic vLLM may already meet the latency target, and an extra controller, storage tier, and transfer path add failure modes without releasing a GPU.

Add LMCache when repeated long contexts drive prefill work, GPU memory cannot retain the useful working set, and the fleet is large enough to consolidate. It can also be justified for latency alone, but call that a latency investment. Do not relabel it as savings.

SGLang users need another caveat. The paper says LMCache's CPU offloading is comparable to SGLang's native CPU offloading on Qwen3-32B. LMCache's advantage there is distributed hierarchical storage, not a blanket local-throughput win.

FAQ

Does 14x throughput mean 14x lower cost?

It means 92.9% lower GPU cost per request if the same work can run on one-fourteenth the GPU capacity. Fixed fleets do not save until capacity is removed or avoided.

Did the LMCache paper report 93% lower cloud bills?

No. It reports throughput and latency. The 92.9% figure is a transparent derivation from 14x throughput, limited to GPU capacity.

What workload produced the 14x result?

The CPU-offload experiment uses multi-round document question answering with long inputs, short outputs, rising QPS, and up to 500GB of CPU DRAM.

What should be included in an all-in comparison?

GPU-hours, CPU DRAM, local and remote storage, network transfers, operations, errors, and the minimum replica count needed to meet the latency target.

When can KV loading be slower than recomputing prefill?

On the paper's 32Gbps B200 sensitivity, loading wins only beyond 256K input tokens. The crossover depends on network bandwidth, model, context length, and storage tier.

Sources

• LMCache paper, version 2, captured 2026-08-09: architecture, model IDs, workload shapes, hardware setup, 2.3x to 14x CPU-offload throughput, latency results, sensitivity study, and deployment lessons.
• LMCache repository, captured 2026-08-09: Apache-2.0 implementation, supported engines, connectors, storage backends, and deployment documentation.
• GMI Cloud H100 pricing, captured 2026-08-09: H100 PCIe from $2.00 per GPU-hour and H100 SXM from $2.40 per GPU-hour.
• Akshay Pachaar's LMCache explainer, captured 2026-08-09: public demand signal only, not a benchmark or pricing receipt.

All cost arithmetic is reproducible in the accompanying Python script. The private-company distributions mentioned by the paper are not Vynaris data and are not used as dollar inputs here.