Glossary · Category

Routing & cost

73 plain-English definitions.

A/B testing (models): Comparing two models or prompt versions on live traffic to measure which performs better on real user outcomes.
AI gateway: Another common name for an LLM gateway: middleware that centralizes access, observability, and governance for calls to multiple models.
blended cost per token: A single average cost-per-token figure that combines a request's input and output token pricing into one number.
budget alert: A notification triggered when LLM spend approaches or exceeds a predefined threshold.
BYOK (bring your own key): A deployment model where a customer supplies their own provider API keys to a gateway or tool instead of routing spend through the vendor's account.
canary routing: Gradually shifting a small percentage of production traffic to a new model or version to validate it before a full rollout.
cascading inference: The technique of routing easy queries to small models and only invoking large models on the subset of queries that need them.
chargeback (AI spend): Allocating shared LLM infrastructure costs back to the specific teams or products that generated the usage.
committed use discount: A cloud/provider pricing arrangement where committing to a minimum spend or usage level unlocks lower per-token rates.
cost anomaly detection: Automatically flagging unusual spikes in LLM spend that may indicate a bug, abuse, or inefficient routing.
cost ceiling: A hard cap configured to stop or throttle requests once spend for a project, user, or time window reaches a set limit.
cost observability: Tooling that gives visibility into where and why LLM spend is being incurred across an application or organization.
cost per query: The average dollar amount spent to answer one user query, factoring in tokens, model choice, and any retries.
cost receipt: An itemized breakdown showing exactly how much a given LLM request cost, often broken down by input tokens, output tokens, and model used.
cost-aware routing: A routing strategy that factors in per-token or per-request pricing when deciding which model should handle a query.
egress-free routing: Architecting inference traffic to avoid cross-region or cross-cloud data transfer fees.
fallback model: A backup model that a routing system calls automatically when the primary model fails, times out, or is rate-limited.
FinOps for AI: The practice of applying cloud financial management discipline (budgeting, forecasting, chargeback) to LLM and GPU spend.
geographic routing (inference): Directing requests to the nearest or most compliant regional inference endpoint to reduce latency or satisfy data residency rules.
health-check-based routing: Routing traffic away from model endpoints that are failing health checks toward healthy alternatives automatically.
inference cost optimization: The general practice of reducing the dollar cost of running LLM inference through routing, caching, quantization, and right-sizing.
inference margin: The difference between what a company charges for AI-powered features and what it actually costs to run the underlying inference.
input/output token pricing: The common LLM billing model where prompt tokens and generated tokens are priced differently, usually with output costing more.
latency-aware routing: A routing strategy that picks a model or endpoint based on real-time response speed rather than cost or quality alone.
least-latency routing: A routing policy that always sends the next request to whichever available endpoint currently has the lowest observed response time.
LLM gateway: A proxy layer sitting between an application and one or more LLM providers that handles routing, caching, auth, logging, and cost control.
LLM router: A system or model that decides, per request, which underlying LLM should handle it to optimize for cost, quality, or speed.
LLM unit economics: The cost-per-request or cost-per-output-token math that determines whether an AI feature is profitable at scale.
load balancing (LLM): Distributing inference requests across multiple model replicas or providers to avoid overloading any single endpoint.
model arbitrage: Automatically choosing the cheapest capable model across providers for each request to minimize spend without sacrificing quality.
model cascade: A pipeline that tries a cheap, fast model first and only escalates to a more expensive model if the first response fails a confidence or quality check.
model routing: Automatically directing each incoming request to the most appropriate model based on factors like task complexity, cost, or latency requirements.
model swap without code change: The ability to switch which underlying model handles requests purely via configuration, thanks to a unified API layer.
model tiering: Organizing available models into cost/quality tiers (e.g. small, medium, frontier) and routing requests to the appropriate tier.
multi-model fallback: A resilience strategy that automatically retries a failed request against an alternative model or provider.
multi-provider abstraction: A unified API layer that lets developers call multiple LLM providers (OpenAI, Anthropic, etc.) through one consistent interface.
outcome-based pricing: A pricing model that charges based on successful results delivered (e.g. resolved tickets) rather than raw token or compute usage.
overage billing: Charging extra fees once usage exceeds a plan's included quota, common in metered AI API pricing.
Pareto frontier (model selection): The curve of models where no alternative offers strictly better quality and lower cost simultaneously, used to guide model choice.
per-request cost: The actual dollar cost incurred by a single LLM API call, based on input/output tokens and the model's pricing.
per-tenant cost tracking: Attributing LLM usage and spend to individual customers or business units in a multi-tenant application.
prepaid credits (AI API): An account balance purchased in advance and drawn down as API usage occurs, common among inference providers.
price-performance ratio: A comparison metric weighing a model's cost against its quality or speed, used to decide which model is the best value for a task.
prompt caching discount: A pricing benefit some providers offer where cached prompt tokens are billed at a fraction of the normal input rate.
prompt compression: Automatically shortening a prompt (e.g. via summarization or token pruning) before sending it to a model to cut cost and latency.
provider failover: Automatically switching an inference request to a different LLM provider when the primary provider is down or degraded.
provisioned throughput: A pricing tier where a customer pays for a guaranteed rate of tokens-per-minute capacity rather than metered per-token billing.
quality-aware routing: A routing strategy that predicts which model is likely to produce the best output for a given query before sending it.
quality-cost frontier: The set of models that represent the best achievable quality at each cost level, with no other model beating them on both dimensions.
rate limiting (LLM API): Restricting the number of requests or tokens a client can send to an API within a given time window.
request deduplication: Detecting and collapsing identical or near-identical concurrent requests into a single model call to save cost.
reserved capacity (inference): Pre-purchasing guaranteed GPU or throughput capacity from a provider, often at a lower rate than on-demand pricing.
right-sizing (models): Matching the smallest, cheapest model capable of handling a given task instead of defaulting to the largest available model.
semantic router: A routing layer that uses embeddings or classifiers to understand a request's intent or difficulty and route it to the best-fit model.
session affinity (LLM): Keeping all requests in a conversation routed to the same backend instance so its prompt/KV cache stays warm.
shadow deployment: Running a new model or routing rule in parallel with production traffic, without serving its outputs to users, to validate it before cutover.
showback: Reporting AI usage costs to teams for visibility without actually billing them internally, as a lighter version of chargeback.
spend forecasting (AI): Predicting future LLM costs based on historical usage trends and planned feature growth.
spillover routing: Sending overflow traffic to a secondary provider or model once the primary one hits its rate limit or capacity.
spot instance inference: Running LLM inference on discounted, preemptible cloud GPU capacity to cut costs, accepting the risk of interruption.
sticky routing: Consistently routing requests from the same user or session to the same model instance to preserve cache locality.
token accounting: Tracking and attributing token usage across requests, users, or projects for cost visibility and billing.
token arbitrage: Exploiting price differences for equivalent capability across providers by routing requests to whichever is cheapest at the moment.
token bucket rate limiting: A rate-limiting algorithm that allows bursts of requests up to a capped "bucket" that refills at a steady rate.
token budget: A cap on the number of tokens an application or agent is allowed to consume for a task or time period.
token cost: The price charged per unit (usually per 1,000 or per million) of input or output tokens by an LLM provider.
token-level billing: Charging customers based on the exact number of tokens consumed rather than flat per-request or subscription pricing.
traffic splitting: Dividing incoming requests between multiple models or model versions according to a defined percentage split.
unified API: A single API surface that normalizes requests and responses across multiple underlying LLM providers so developers write code once.
usage quota: A cap on the number of requests or tokens a user, team, or API key can consume within a given period.
usage-based pricing: A billing model where customers pay according to actual consumption (tokens, requests) rather than a flat subscription fee.
weighted round robin routing: A load-balancing method that distributes requests across model endpoints proportionally to assigned weights.
zero data retention (ZDR): A provider policy or contractual commitment that request and response data is not stored or used for training after processing.