Glossary · Category
Distributed Training & Hardware Systems
11 plain-English definitions.
- Accelerator interconnect topology
- The physical/network arrangement (e.g., ring, mesh, fat-tree) connecting accelerators in a cluster, which shapes achievable parallelism efficiency at scale.
- All-gather
- A collective communication operation where every device ends up with the concatenation of a value contributed by every other device, used in sharded parameter/optimizer training.
- All-reduce
- A collective communication operation that aggregates (e.g., sums) values (typically gradients) across all devices and distributes the result back to each, core to synchronous distributed training.
- Checkpoint sharding
- Splitting a large model's saved weights across multiple files/shards so it can be loaded and written in parallel across devices.
- Compute cluster scheduler
- Software (e.g., Slurm, Kubernetes-based schedulers) that allocates GPU/accelerator resources to training and inference jobs across a shared cluster.
- CUDA graph
- A captured, replayable sequence of GPU operations that eliminates per-kernel launch overhead, used to speed up repetitive inference workloads.
- Elastic training
- A distributed training setup that can dynamically add or remove worker nodes during a training run without restarting from scratch.
- FLOPs (floating point operations)
- A measure of computational work; total training or inference FLOPs are a standard way to compare the raw compute cost of running or training different models.
- GPU memory bandwidth
- The rate at which data can be read from or written to a GPU's on-device memory (HBM), often the binding constraint during the memory-bound LLM decode phase.
- TPU (Tensor Processing Unit)
- Google's custom AI accelerator chip, an alternative to GPUs optimized for large matrix multiplications used in training and serving transformer models.
- Triton (compiler)
- An open-source language and compiler for writing efficient custom GPU kernels in Python-like syntax, widely used to implement fused attention and quantization kernels.