Glossary · Category

Serving, Batching & Systems

10 plain-English definitions.

Autoscaling

Dynamically adjusting the number of active inference replicas/GPUs based on real-time request load to balance latency and cost.

EAGLE

A speculative decoding method that predicts future tokens using the target model's own feature (hidden-state) sequence rather than token sequence, improving draft accuracy over Medusa.

Expert parallelism (MoE serving)

Distributing different MoE experts across GPUs so that only the activated experts' devices participate in each token's forward pass during inference.

Medusa

A speculative decoding method that adds multiple extra prediction heads directly onto the target model to propose several future tokens simultaneously, avoiding a separate draft model.

Model sharding

Splitting a single model's weights across multiple devices (via tensor or pipeline parallelism) because it exceeds one device's memory capacity.

Multi-LoRA serving

A serving pattern that hosts one base model in memory while dynamically swapping in many small LoRA adapters per request, letting one GPU serve many fine-tuned variants cheaply.

Offloading

Moving model weights, KV cache, or activations to CPU RAM or disk when GPU memory is insufficient, trading latency for the ability to run larger models.

Prefill

The initial forward pass over the entire input prompt that computes and caches key/value states before autoregressive token generation begins.

Request queuing / admission control

Serving-layer logic that decides when to accept, delay, or reject incoming inference requests to keep latency and GPU memory within bounds.

Throughput-latency tradeoff

The inherent tension in LLM serving where increasing batch size raises total tokens-per-second throughput but also raises per-request latency.