Glossary · Category

Inference API & Deployment Concepts

9 plain-English definitions.

Fallback routing

A gateway/router behavior that automatically retries a request against an alternate model or provider when the primary choice fails or is unavailable.

Idempotency key

A client-supplied identifier that lets an API safely retry a request without risk of duplicate side effects (e.g., duplicate billed generations).

Load shedding

Deliberately rejecting or degrading some requests under extreme load to protect overall system availability.

Multi-tenancy

Serving multiple distinct customers or workloads from shared underlying infrastructure while maintaining isolation and fair resource allocation.

Provider abstraction layer

A software layer (common in gateways) that normalizes different model providers' APIs into one consistent interface for the application.

Rate limiting

Restricting the number of requests or tokens a client can send in a given time window, used to protect infrastructure and enforce fair usage/billing tiers.

Streaming response

An API mode that returns generated tokens incrementally as they're produced rather than waiting for the full completion, reducing perceived latency.

Structured output mode

An API feature guaranteeing the model's response conforms to a specified JSON schema via constrained decoding.

Token usage accounting

The practice of tracking and billing based on input and output token counts consumed per request, the standard LLM API pricing unit.