Glossary · Category
Additional Quantization & Compression Detail
11 plain-English definitions.
- bitsandbytes
- A popular open-source library implementing 8-bit and 4-bit quantization (including QLoRA's NF4 format) for efficient LLM training and inference in PyTorch.
- Blockwise quantization
- Applying separate quantization scale factors to small blocks of weights (rather than a whole tensor), improving accuracy at a small metadata overhead cost.
- Double quantization
- A memory-saving technique (used in QLoRA) that quantizes the quantization constants themselves, squeezing out additional memory savings.
- EXL2
- A mixed-precision quantization format (used by ExLlamaV2) that allows different bit-widths per layer to optimize the size/quality tradeoff.
- ExLlama / ExLlamaV2
- Optimized inference runtimes specialized for running GPTQ/EXL2-quantized LLaMA-family models efficiently on consumer GPUs.
- GGUF quantization levels (Q4_K_M, Q5_K_M, Q8_0)
- Named presets in the GGUF format specifying bit-width and blockwise quantization strategy, trading file size and speed against output quality.
- HQQ (Half-Quadratic Quantization)
- A fast calibration-free post-training quantization method that solves a half-quadratic optimization problem to set quantization parameters without needing a calibration dataset.
- NF4 (4-bit NormalFloat)
- A quantization data type designed for normally distributed weights, used by QLoRA to minimize quantization error versus standard integer 4-bit formats.
- Outlier feature
- An activation value that is far larger in magnitude than typical, disproportionately affecting quantization error and motivating techniques like SmoothQuant and AWQ.
- Post-training quantization (PTQ)
- Quantizing a model's weights (and optionally activations) after training is complete, without further gradient-based fine-tuning, as opposed to quantization-aware training.
- Symmetric vs asymmetric quantization
- Two schemes for mapping floating-point ranges to integers: symmetric uses a zero-centered range, asymmetric adds a zero-point offset to better fit skewed distributions.