Glossary · Category
Quantization
18 plain-English definitions.
- AWQ (Activation-aware Weight Quantization)
- A post-training quantization method that identifies and preserves the small fraction of "salient" weights most affected by activation magnitude, protecting them from precision loss.
- BF16 (bfloat16)
- A 16-bit floating point format with the same exponent range as FP32 but fewer mantissa bits, widely used for stable mixed-precision training.
- Calibration dataset
- A small representative sample of data used during post-training quantization to determine optimal scaling factors and minimize accuracy loss.
- FP16
- 16-bit floating point with a smaller dynamic range than BF16, historically common for training and inference on GPUs supporting it.
- FP32
- 32-bit IEEE floating point, the traditional full-precision format for training, now rarely used for inference due to memory and speed cost.
- FP8
- An 8-bit floating point format (with variants like E4M3/E5M2) enabling faster, lower-memory inference and training on newer GPUs while retaining more dynamic range than integer formats.
- GGML
- The original tensor library and file format for efficient CPU inference of quantized LLMs, precursor to GGUF.
- GPTQ
- A post-training quantization method that uses approximate second-order (Hessian) information to minimize the error introduced when rounding weights to low bit-width, layer by layer.
- INT4
- 4-bit integer quantization, cutting memory roughly 4x versus FP16, typically applied to weights only with careful calibration to limit quality loss.
- INT8
- 8-bit integer quantization of weights/activations, roughly halving memory versus FP16 with a small quality tradeoff, common in production inference.
- KV-cache quantization
- Reducing the numerical precision of stored key/value attention cache entries (e.g., to FP8 or INT8) to cut the memory cost of long-context and high-concurrency serving.
- Mixed-precision inference
- Running different parts of a model (or different tensors) at different numerical precisions to balance speed and accuracy.
- Per-channel quantization
- A quantization granularity that assigns a separate scale factor to each channel (e.g., output neuron) rather than one scale for an entire tensor, improving accuracy.
- Quantization
- The process of representing model weights and/or activations with lower numerical precision to reduce memory footprint and increase inference speed.
- Quantization-aware training (QAT)
- A training approach that simulates quantization effects during training itself, so the model learns weights robust to low-precision inference.
- SmoothQuant
- A quantization technique that migrates the quantization difficulty from activations to weights via a per-channel scaling factor, easing INT8 activation quantization.
- W4A16
- A quantization scheme that stores weights in 4-bit precision while keeping activations in 16-bit, popular because it preserves activation dynamic range while shrinking weight memory.
- W8A8
- A quantization scheme quantizing both weights and activations to 8-bit, enabling faster integer matrix multiplication on supporting hardware.