Blog · 2026-11-05 · By Dhrumil Barot, Founder

What Is an LLM Gateway? Definition, Costs and Examples

An LLM gateway is a proxy between your app and model providers that adds one API, keys, logging, fallbacks and caching. Fees on $1,000/month range from $0 to $55, checked 2026-11-05.

TL;DR: An LLM gateway is a proxy that sits between your application and model providers such as OpenAI, Anthropic and Google, giving you one API plus keys, budgets, logs, fallbacks and caching. On $1,000 a month of inference, the gateway fee ranges from $0 (self-hosted LiteLLM, Vercel AI Gateway) to $55 (OpenRouter's 5.5% credit fee), prices checked 2026-11-05. It pays for itself when its caching and fallbacks save more than that fee, or when more than one team shares the same provider keys.

What is an LLM gateway, in one paragraph? It is a single endpoint, usually OpenAI-compatible, that receives every model call your apps make, applies your rules (who may call which model, how much they may spend, what to log, what to retry), forwards the call to the right provider, and returns the answer in one format. LiteLLM describes its gateway as "one OpenAI-compatible endpoint for 100+ LLM providers" with virtual keys, spend tracking, fallbacks and guardrails (LiteLLM docs). Cloudflare, Vercel, OpenRouter, Portkey, Helicone and Vynaris all sell a version of the same layer. The differences that matter are the fee shape and which controls you get.

Six jobs a gateway takes over from your services

An LLM gateway does six jobs that otherwise end up copied into every service that calls a model:

The point is centralisation. Without a gateway, the retry logic, the key rotation and the cost logging live in five codebases and drift apart. With one, they live in one place.

How is an LLM gateway different from an LLM router?

An LLM router decides which model should answer a given request; an LLM gateway is the infrastructure layer that carries the request, whichever model is chosen. A gateway can be completely static: every call to gpt-6-luna goes to OpenAI, with logging and a fallback. A router adds a decision: this prompt is simple, send it to the cheap model; that one needs reasoning, send it to the frontier model.

The line is blurry because most products ship both. OpenRouter's routing variants are a good example: :floor sorts providers by price, :nitro by throughput, :exacto by tool-calling quality (OpenRouter FAQ). That is provider routing for one model, not model routing. If you want the economics of model routing specifically, we worked them in confidence-gated cascade routing cost math.

Does an LLM gateway add cost?

Yes, almost always, but rarely as a markup on tokens. Every major gateway now says it passes provider prices through; the fee sits somewhere else. Here is the fee on the same $1,000 of monthly inference, from each vendor's page on 2026-11-05:

Sources: OpenRouter pricing, Cloudflare AI Gateway pricing, Vercel AI Gateway pricing, Vynaris pricing, all checked 2026-11-05.

The Vynaris line, worked: 3% of $500 is $15, plus 1% of the remaining $500 is $5, total $20.

One change since our July audit: OpenRouter's BYOK allowance is now measured in dollars, $25,000 a month of list-price inference with no BYOK fee on pay-as-you-go, instead of a free request count. For most small teams that makes BYOK on OpenRouter free in practice.

Watch the add-ons too. Vercel bills Custom Reporting at $0.075 per 1,000 tag writes and $5 per 1,000 report queries, and team-wide Zero Data Retention at $0.10 per 1,000 requests. Those are small, but they are not zero.

Does a gateway add latency?

A hosted gateway adds one network hop and some processing per call; a self-hosted one adds a hop inside your own network. Vendors publish their own overhead claims, and they depend on region and payload, so measure it: send 100 identical requests direct and through the gateway, and compare median and p95 time to first token. For chat UIs the added hop is usually small next to model generation time; for tight agent loops with dozens of calls per task, it compounds.

Worked example: fee versus caching savings on $1,000

A gateway saves money in three ways: response caching, prompt caching passed through correctly, and budgets that stop runaway spend. Here is the math on one assumed workload. Every input is an assumption you can change.

Prompt caching on Anthropic models. Cache reads cost 0.1x the base input price and 5-minute cache writes cost 1.25x (Anthropic prompt caching docs). The $400 prefix becomes 400 x (0.9 x 0.1 + 0.1 x 1.25) = 400 x 0.215 = $86. Saving: $314 a month. This is a provider feature, not a gateway feature; the gateway's job is to pass cache_control through untouched and report cache hits, which is worth checking before you adopt one.

Exact-match response caching. Cloudflare's gateway cache hashes the full request with SHA-256 and serves an identical request from cache, marked cf-aig-cache-status: HIT (Cloudflare caching docs). At a 5% duplicate rate, $50 of the $1,000 never reaches the provider. Any change in the body, even one parameter, is a new cache entry, so this only pays on truly repeated calls: FAQ bots, classification of recurring strings, eval reruns.

The break-even rule falls out of the second line: a percentage gateway funded only by exact-match caching pays for itself when your duplicate rate is above its fee rate. At 5.5%, you need more than 5.5% of requests to be byte-identical. Most interactive apps never get there, which is why the real case for a gateway is usually control and reliability, not caching.

When do you not need an LLM gateway?

You do not need an LLM gateway when one application, owned by one team, calls one provider. LiteLLM's own docs say the same: use the gateway when more than one application or person sends requests, and the plain SDK when one Python app does (LiteLLM docs). Skip it, or postpone it, when:

• You call a single provider and its own dashboard already gives you usage per key.
• Your spend is under about $100 a month: a $49 flat plan is a 49% surcharge at that size.
• Every call is latency-critical and you have not measured the extra hop.
• Your provider's prompt caching already captures most of the repeated input, and your duplicate rate is near zero.

Add one when the second team asks for a key, when an outage costs you real revenue (see what model fallback chains save during outages), or when finance asks for cost per feature and nobody can answer.

Common pitfalls

• Counting the "no markup" claim as the price. The fee lives in credit purchases, flat plans or add-ons. Compute it on your own monthly spend.
• BYOK fallbacks billed elsewhere. On Vercel, a request that fails with your key retries on Vercel's credentials and is charged to your credits, and budgets exclude BYOK spend (Vercel docs). Set provider-side limits as well.
• Breaking prompt caching. A gateway that rewrites or reorders messages can turn cache reads at 0.1x into full-price input. Check reported cache-read tokens after switching.
• Self-hosting as "free". LiteLLM's $0 is a software line; you still run Postgres, patch releases and carry the pager. We priced that tradeoff in LiteLLM vs OpenRouter vs managed gateways.

Where Vynaris fits

If the problem after reading this is "we want one OpenAI-compatible endpoint with per-request cost in the response headers and a fee that drops as we grow", that is what Vynaris is: provider list price plus 3% on the first $500 a month and 1% after. Above roughly $3,900 a month of steady spend with no routing needs, Portkey's flat $49 plan costs less on fee alone, as our fee audit shows.

FAQ

What is an LLM gateway in simple terms?

An LLM gateway is one front door for all your AI model calls. Your apps talk to the gateway, and the gateway talks to OpenAI, Anthropic, Google or any other provider, adding keys, spend limits, logs, retries and caching on the way.

Is an AI gateway the same as an LLM gateway?

Mostly, yes. Cloudflare and Vercel call their products "AI Gateway" and they proxy LLM calls with the same features. "AI gateway" is sometimes used more widely, to include image, speech and MCP tool traffic.

Is an LLM gateway the same as an API gateway?

No. A general API gateway (Kong, AWS API Gateway) routes HTTP traffic without understanding tokens. An LLM gateway knows model names, token counts and prices, so it can enforce token budgets, compute cost per request and fall back between model providers.

Does an LLM gateway make AI calls cheaper?

Only indirectly. Providers' token prices pass through unchanged on OpenRouter, Cloudflare and Vercel. Savings come from cached responses, correctly passed-through prompt caching, budgets that stop runaway jobs, and routing cheaper models where a router is included.

Can I build my own LLM gateway?

Yes. A thin proxy with one base URL, a retry on 429 and 5xx errors and a cost log is a weekend of work, and LiteLLM's open-source gateway is a ready starting point. The ongoing cost is upkeep: new models, changed prices and provider API changes every month.

Which LLM gateway is cheapest?

On fee alone at $1,000 a month: self-hosted LiteLLM, Vercel AI Gateway and Cloudflare with your own keys are $0 in gateway fees, OpenRouter BYOK is $0 under its $25,000 allowance, and OpenRouter credits cost $55. Compare total cost, including hosting and engineering time, not just the fee.