Glossary · Category

Additional Serving Reliability & Observability Terms

8 plain-English definitions.

Circuit breaker (inference routing)

A resilience pattern that automatically stops routing traffic to a failing model/provider endpoint for a cooldown period after detecting repeated errors.

Health check (model endpoint)

A periodic lightweight request used to verify that a model-serving endpoint is responsive and healthy before routing production traffic to it.

Jitter (latency)

Variability in response latency across otherwise identical requests, which can degrade user experience even when average latency is acceptable.

Observability (LLM ops)

The practice of instrumenting an LLM application with logging, tracing, and metrics to understand latency, cost, quality, and failure modes in production.

P50/P95/P99 latency

Percentile latency metrics (median, 95th, 99th percentile) used to characterize typical versus worst-case response times for an inference service.

Tail latency

The latency experienced by the slowest fraction of requests (e.g., P99), often the key metric for user-perceived reliability even when average latency looks fine.

Token-level logging

Recording generation at the individual token level (including logprobs or timing) for detailed debugging, evaluation, or safety auditing purposes.

Trace (LLM request)

A structured record of a single request's full lifecycle — including prompts, tool calls, retries, and model responses — used for debugging and evaluation.