VynarisEarly beta Kimi K3Get your API key

Glossary · Category

Inference & serving

124 plain-English definitions.

acceptance rate
In speculative decoding, the fraction of draft-model-proposed tokens that the target model confirms as correct.
admission control
Serving-layer logic that rejects or queues new requests once a system is at capacity, to protect latency for already-accepted requests.
arithmetic intensity
The ratio of compute operations to memory accesses in a workload, which determines whether it's compute-bound or memory-bound.
asynchronous inference
Submitting a request and polling for or being notified of the result later, instead of holding a live connection open.
autoregressive decoding
Generating text one token at a time, where each new token depends on all previously generated ones.
autoscaling (inference)
Automatically adding or removing model-serving replicas based on real-time request load.
backpressure
A signal propagated upstream when a downstream inference service is overloaded, telling callers to slow down or retry later.
batch API
A provider endpoint (e.g. OpenAI, Anthropic) for submitting large volumes of non-urgent requests that are processed asynchronously at a lower cost.
batch inference
Processing many prompts together as a bulk job rather than as individual real-time requests, often at a discounted price.
block size (paged KV cache)
The fixed-size unit into which the KV cache is divided when using paged memory management like vLLM's.
blue-green deployment (model)
Running two identical serving environments and switching traffic from the old model version to the new one instantly, enabling instant rollback.
chunked prefill
Splitting a long prompt's prefill computation into smaller chunks interleaved with decode steps from other requests, smoothing out latency spikes.
circuit breaker (LLM API)
A resilience pattern that stops sending requests to a failing model or provider for a cooldown period instead of retrying endlessly.
cold start
The latency penalty incurred when a model has to be loaded into GPU memory before it can serve its first request.
compute bound
A workload (like prefill on long prompts) whose speed is limited by GPU FLOPs rather than memory movement.
context caching
A provider-level feature (e.g. Gemini, Claude) that lets you cache large reusable context so repeated calls don't re-pay for the same tokens.
context rot
The observed degradation in a model's ability to accurately use information as the amount of context it must attend to grows.
context window
The maximum number of tokens (input plus output) a model can process or attend to in a single request.
context window overflow
The error or truncation that occurs when a conversation's accumulated tokens exceed a model's maximum context length.
continuous batching
A serving technique that dynamically adds and removes requests from a running GPU batch instead of waiting for a fixed batch to finish, boosting throughput.
CUDA graphs
A GPU feature that captures a sequence of kernel launches as a single replayable graph, cutting per-step CPU overhead during inference.
data parallelism
Running identical copies of a model on multiple devices, each processing a different batch of data simultaneously.
decode phase
The token-by-token generation stage after prefill, typically bottlenecked by memory bandwidth rather than compute.
disaggregated prefill and decode
Running the prefill and decode phases on separate GPU pools so each can be scaled and optimized independently.
draft model
The small, cheap model used in speculative decoding to propose candidate tokens for the larger model to verify.
dynamic batching
Grouping incoming requests on the fly based on arrival time and load rather than a fixed schedule.
EAGLE decoding
A speculative decoding technique that predicts future tokens using the target model's own feature representations for higher acceptance rates.
expert parallelism
Distributing the different experts of a mixture-of-experts model across multiple GPUs or nodes.
exponential backoff
A retry strategy that increases the wait time between successive retries after failures, commonly used against rate-limited LLM APIs.
first-come-first-served scheduling
A simple request scheduling policy that serves requests strictly in arrival order, without prioritization.
FlashAttention
A GPU-optimized attention algorithm that avoids materializing the full attention matrix in memory, making long-context inference and training much faster.
FlashAttention-2
An improved version of FlashAttention with better GPU parallelism and work partitioning for even faster attention computation.
FlashAttention-3
A further-optimized FlashAttention release tuned for Hopper GPUs, exploiting asynchrony and low-precision compute.
FlashDecoding
A FlashAttention variant optimized specifically for the memory-bound decode phase by parallelizing over the KV cache sequence dimension.
GGUF
A binary file format for storing quantized LLM weights, widely used by llama.cpp and Ollama for local inference.
goodput
The throughput of requests that are actually served within acceptable latency SLOs, as opposed to raw throughput that ignores latency violations.
GPU memory fragmentation
Wasted GPU memory caused by inefficient allocation patterns, a key problem PagedAttention was designed to solve for the KV cache.
GPU pooling
Sharing a pool of GPUs across multiple models or tenants to improve utilization instead of dedicating hardware per model.
GPU utilization
The percentage of a GPU's compute or memory bandwidth actively being used, a key efficiency metric for inference cost.
grouped-query attention (GQA)
A middle ground between multi-head and multi-query attention where heads are grouped to share key/value projections, balancing quality and KV cache savings.
health check endpoint
An API route inference servers expose so load balancers can detect and route around unhealthy instances.
idempotency key (LLM API)
A client-supplied unique identifier that lets a provider safely deduplicate retried requests without double-processing them.
in-flight batching
NVIDIA's term for continuous batching in TensorRT-LLM, where requests join and leave a batch mid-generation.
inference endpoint
The API URL a client calls to send a model a request and receive a generated response.
inter-token latency
The time gap between consecutive generated tokens, which determines how "smooth" streamed output feels.
JIT compilation (inference)
Just-in-time compiling model computation graphs into optimized GPU kernels the first time they run, trading startup latency for steady-state speed.
kernel fusion
Combining multiple GPU operations into a single kernel launch to reduce memory reads/writes and overhead.
KV cache
The stored key and value tensors from previous tokens that let a transformer avoid recomputing attention over the whole sequence at each step.
KV cache compression
Techniques that shrink the memory footprint of the KV cache, such as quantizing it or sharing it across layers.
KV cache eviction
Strategies for discarding older or less useful cached key-value pairs when GPU memory runs out.
KV cache offloading
Moving KV cache data from GPU memory to CPU RAM or disk to free up space for more concurrent requests.
llama.cpp
A lightweight C/C++ inference engine for running LLaMA-family and other open-weight models efficiently on CPUs and consumer GPUs.
llamafile
A single self-contained executable that bundles a model and an inference engine so it runs on multiple OSes without installation.
LLM serving engine
Software (like vLLM, SGLang, TGI) purpose-built to run trained models efficiently at scale for production inference traffic.
LMCache
An open-source KV cache layer that persists and shares prefix caches across requests and even across serving engine instances.
long-context inference
Serving techniques and optimizations specifically aimed at handling very large context windows (100K+ tokens) efficiently.
lookahead decoding
A parallel decoding technique that generates and verifies multiple token guesses simultaneously using a Jacobi-iteration-style approach, without a separate draft model.
lost in the middle
The phenomenon where LLMs recall information from the beginning and end of a long context more reliably than from the middle.
max tokens parameter
The API setting that caps how many tokens a model is allowed to generate in a single response.
Medusa decoding
A speculative decoding method that adds extra prediction heads to a model so it can propose multiple future tokens in a single forward pass.
memory bandwidth bound
A workload (like LLM decode) whose speed is limited by how fast data can move through memory rather than by raw compute power.
MLX
Apple's array framework optimized for machine learning on Apple Silicon, used for running and fine-tuning models locally on Macs.
model loading time
The time it takes to load a model's weights from disk into GPU memory before inference can begin.
model parallelism
The general practice of splitting a single model's computation across multiple devices because it doesn't fit on one.
model server
A general term for the software process that loads a model and exposes it via an API for inference requests.
model swapping
Unloading one model from GPU memory and loading another to serve a different request, trading latency for hardware flexibility.
Mooncake
A disaggregated KV-cache-centric serving architecture (from Moonshot AI) that separates prefill and decode clusters connected by a fast KV transfer layer.
multi-head latent attention (MLA)
DeepSeek's attention variant that compresses keys and values into a shared low-rank latent vector to shrink KV cache memory.
multi-model serving
Running several different models on shared infrastructure, dynamically loading and unloading them based on demand.
multi-query attention (MQA)
An attention variant where all query heads share a single key/value head, drastically shrinking KV cache size.
multi-turn conversation
An exchange spanning several back-and-forth messages between user and model, requiring the full history to be resent or cached each turn.
needle in a haystack test
A benchmark that hides a specific fact ("needle") inside a long document ("haystack") to test whether a model can retrieve it from anywhere in its context.
NIXL
An NVIDIA-backed low-latency data transfer library used to move KV cache data between disaggregated prefill and decode nodes.
Ollama
A developer tool for downloading, running, and serving open-weight LLMs locally with a simple CLI and API.
PagedAttention
A memory management algorithm (from vLLM) that stores the KV cache in non-contiguous fixed-size blocks like OS virtual memory pages, cutting GPU memory waste.
PagedAttention v2
An updated version of vLLM's paged KV cache algorithm with further throughput improvements.
parallel tool calls
A model capability where multiple independent tool/function calls are issued in a single turn instead of one at a time.
partial JSON parsing
Incrementally parsing a structured output stream before it's fully complete, so a client can react to fields as they arrive.
pipeline parallelism
Splitting a model's layers across multiple GPUs so different devices process different stages of the forward pass in a pipeline.
preemption (inference)
Pausing a lower-priority in-progress generation to free GPU resources for a higher-priority request.
prefill phase
The initial pass where a model processes the entire input prompt at once to build its KV cache before generating any output.
prefill-decode disaggregation
Same as disaggregated serving: splitting compute-bound prefill from memory-bound decode onto different hardware.
prefix caching
Caching the KV cache of a common prompt prefix (like a system prompt) so subsequent requests only need to compute the new suffix.
prompt caching
Reusing the computed KV cache for a shared prompt prefix across multiple requests to skip redundant computation and cut cost and latency.
queueing delay
The time a request waits in line before a GPU worker becomes available to process it.
RadixAttention
SGLang's technique for automatically sharing KV cache across requests with matching prefixes via a radix tree, enabling efficient prompt caching.
Ray Serve
A scalable model-serving library built on the Ray distributed computing framework, commonly used to deploy LLM inference pipelines.
request scheduling
The logic an inference server uses to decide the order and grouping in which incoming requests are processed.
request timeout tuning
Configuring how long a client waits for a model response before giving up, balanced against the variability of LLM latency.
ring attention
A distributed attention technique that shards a long sequence across devices arranged in a ring, passing KV blocks around to compute attention without holding the full sequence on one device.
rolling context summarization
Periodically compressing older parts of a long conversation into a summary to stay within the context window while preserving key facts.
rolling deployment (model)
Gradually replacing old model server instances with new ones to avoid downtime during an update.
scale to zero
An autoscaling behavior where all compute for a model is shut down when there's no traffic, eliminating idle cost.
self-speculative decoding
A speculative decoding variant where a subset of a single model's own layers acts as the draft model, avoiding the need for a separate smaller model.
semantic caching
Caching LLM responses keyed by the meaning of a query (via embedding similarity) rather than an exact string match, so paraphrased requests can still hit cache.
sequence length bucketing
Grouping requests with similar input/output lengths together to make batching more efficient.
sequence parallelism
Splitting a single long input sequence across multiple GPUs to process it in parallel, useful for very long context windows.
server-sent events (SSE)
A web standard for streaming one-way text updates from server to client over HTTP, commonly used to stream LLM tokens.
serverless inference
A deployment model where GPU compute is provisioned on demand per request and billed only for active usage, with no idle infrastructure to manage.
service level objective (SLO)
A target performance threshold (e.g. TTFT under 500ms) that an inference service commits to meeting.
SGLang
An open-source LLM serving framework known for RadixAttention-based prefix caching and a structured generation language for complex prompting.
sliding window attention
An attention mechanism that limits each token to attending only to a fixed-size local window of nearby tokens, reducing compute for long sequences.
speculative decoding
Using a small, fast draft model to guess several tokens ahead, which the larger target model then verifies in a single parallel pass to speed up generation.
stateless API (LLM)
An API design where each request is self-contained and the server retains no memory of prior turns between calls.
static batching
Processing a fixed group of requests together and waiting for the entire batch to complete before starting the next, less efficient than continuous batching.
stop sequences
User-defined strings that, when generated, tell the model to immediately halt output.
streaming chunk
A single incremental piece of a model's response delivered to the client as part of a streamed generation.
streaming inference
Sending generated tokens to the client incrementally as they're produced rather than waiting for the full response.
swap space (inference)
CPU memory reserved by an inference server to temporarily hold KV cache data evicted from GPU memory under pressure.
target model
The larger, more accurate model in speculative decoding that verifies or rejects tokens proposed by the draft model.
tensor parallelism
Splitting a single model's weight matrices across multiple GPUs so each computes a slice of every layer in parallel.
TensorRT-LLM
NVIDIA's library for compiling and optimizing LLM inference on NVIDIA GPUs, including in-flight batching and custom kernels.
Text Generation Inference (TGI)
Hugging Face's production-ready LLM serving toolkit supporting continuous batching, tensor parallelism, and quantization.
throughput vs. latency tradeoff
The fundamental serving tension between maximizing requests processed per second and minimizing response time for any single request.
time per output token (TPOT)
The average latency to generate each subsequent token after the first, a key measure of decode speed.
time to first token (TTFT)
The latency between sending a request and receiving the first generated token, dominated by the prefill phase.
tokens per second (TPS)
The throughput rate at which a model generates output tokens, used to compare inference speed across models and hardware.
tree attention
A speculative decoding technique that verifies multiple candidate token sequences arranged in a tree structure within a single forward pass.
Triton Inference Server
NVIDIA's general-purpose model-serving platform supporting multiple frameworks and backends, including LLMs.
vLLM
An open-source, high-throughput LLM inference and serving engine built around PagedAttention and continuous batching.
warm pool
A set of pre-loaded, ready-to-serve model instances kept running to avoid cold-start latency.
warm-up requests
Dummy or synthetic requests sent to a newly started model server to trigger JIT compilation and cache population before real traffic arrives.
weight streaming
Loading model weights from storage into GPU memory incrementally during inference rather than all at once upfront.