Glossary · Category

Fine-Grained Serving & Inference Optimization Terms

13 plain-English definitions.

Batch size (inference): The number of concurrent requests processed together in one forward pass during serving, distinct from training batch size, tuned for throughput/latency tradeoffs.
Canary deployment (model): Gradually routing a small percentage of production traffic to a new model version to validate quality and stability before a full rollout.
Continuous integration for model evaluation: Automated pipelines that re-run benchmark and regression evaluations whenever a model, prompt, or serving configuration changes, to catch quality regressions before deployment.
Draft-target model pairing: The practice of selecting a smaller model from the same family or distillation lineage as a target model specifically because their output distributions align well for speculative decoding.
Eviction policy (KV cache): The rule (e.g., LRU) governing which cached KV-cache blocks or prompt prefixes are discarded first when memory is full.
Max batch tokens: A serving configuration limit capping the total token count (across all requests) allowed in a single inference batch, used to bound GPU memory usage.
Prompt prefix sharing: A serving optimization that lets multiple concurrent requests sharing an identical prompt prefix (e.g., a common system prompt) reuse the same computed KV-cache.
Regression testing (LLM outputs): Systematically re-running a fixed evaluation set against a new model or prompt version to detect unintended quality degradations versus a prior baseline.
Request coalescing: Combining multiple compatible incoming requests into a single batched inference pass to improve GPU utilization.
Session affinity / sticky routing: Routing repeated requests from the same conversation or user to the same backend instance to maximize prompt-cache hit rate.
Shadow traffic / shadow deployment: Sending a copy of live production requests to a new model version for evaluation without returning its responses to users, used to validate before cutover.
Speculative batch verification: Running the target model's verification pass over multiple requests' draft tokens simultaneously in speculative decoding to preserve batching efficiency.
Warm cache hit rate: The fraction of requests that benefit from a previously cached prompt prefix or KV-cache state, directly affecting effective latency and cost.