Glossary · Category

Model Architecture Families & Training Concepts

21 plain-English definitions.

AdamW optimizer: A widely used gradient-descent optimizer combining adaptive per-parameter learning rates (Adam) with decoupled weight decay, the default choice for training transformers.
Autoregressive model: A model that generates output sequentially, with each new token conditioned on all previously generated tokens.
Batch size (training): The number of training examples processed together before a model's weights are updated; larger batches can improve hardware efficiency and gradient estimate stability.
Cosine learning rate schedule: A training schedule that decreases the learning rate following a cosine curve over the course of training, a common choice for LLM pretraining.
Data deduplication: Removing near-duplicate documents from a pretraining corpus, shown to improve model quality and reduce memorization for a fixed compute budget.
Distillation-trained model: A model whose training specifically used outputs or reasoning traces from a larger teacher model as supervision, common for producing efficient "mini" model variants.
Emergent capability: An ability that appears abruptly at a certain model scale rather than improving smoothly, observed on some benchmarks as models grow larger.
Epoch: One complete pass through the entire training dataset.
Hybrid attention-SSM architecture: A model design that interleaves standard attention layers with state-space-model layers to balance quality and long-sequence efficiency.
Learning rate warmup: A training schedule technique that gradually increases the learning rate from near-zero at the start of training to avoid early instability.
Mamba: A state-space-model architecture with input-dependent (selective) parameters, achieving transformer-competitive quality with linear-time sequence processing.
Masked language modeling (MLM): A pretraining objective (used by BERT-style models) that hides random tokens in the input and trains the model to predict them from bidirectional context.
Next-token prediction: The self-supervised training objective where a model learns to predict the next token in a sequence given all previous tokens, the core pretraining task for autoregressive LLMs.
Overfitting: When a model learns patterns specific to its training data that don't generalize to new data, showing a gap between training and validation performance.
Parameter count: The total number of learnable weights in a model, a rough (imperfect) proxy for model capacity and inference cost.
Retrieval-augmented pretraining: Training a model with access to a retrieval mechanism over an external corpus during pretraining itself, rather than only at inference time.
State space model (SSM): A sequence-modeling architecture (e.g., Mamba) based on linear state-space equations rather than attention, offering linear-time inference and constant memory per generated token.
Synthetic data: Training data generated by a model (rather than collected from humans) increasingly used to augment or replace human-authored pretraining and fine-tuning data.
Tokenizer-free / byte-level model: An architecture that operates directly on raw bytes rather than a learned subword vocabulary, avoiding tokenization artifacts at the cost of longer sequences.
Tokens seen / training tokens: The total count of tokens a model was trained on, a key scale metric alongside parameter count for describing pretraining compute.
Total parameters: The full parameter count of a model including all experts in a mixture-of-experts architecture, whether or not they're active for a given token.