Glossary · Category

Training & tuning

97 plain-English definitions.

active parameters: In a mixture-of-experts model, the subset of total parameters actually used to process a given token, as opposed to the full parameter count.
adapters (fine-tuning): Small trainable modules inserted into a frozen pretrained model's layers to adapt its behavior without full retraining.
agentic RL: Reinforcement learning applied to train models specifically for multi-step, tool-using agentic tasks rather than single-turn responses.
alignment (LLM): The broad effort to make a model's behavior match human intentions and values, typically via RLHF, DPO, or constitutional methods.
Axolotl: A popular open-source tool for configuring and running LLM fine-tuning jobs, especially LoRA/QLoRA.
backdoor attack (LLM): A hidden trigger deliberately trained into a model that causes it to behave maliciously only when a specific input pattern is present.
base model: A pretrained model before any instruction tuning or RLHF alignment has been applied.
best-of-N sampling: Generating multiple candidate responses and selecting the best one (via a scoring model or heuristic) to improve output quality.
budget forcing: A technique to control how much test-time reasoning compute a model uses, by capping or extending its thinking tokens.
byte pair encoding (BPE): A common tokenization algorithm that iteratively merges the most frequent pairs of characters or subwords into single tokens.
catastrophic forgetting: When fine-tuning on new data causes a model to lose previously learned capabilities.
chain-of-thought distillation: Training a smaller model to reproduce a larger model's step-by-step reasoning traces, not just its final answers.
chat template: The formatting convention (special tokens marking roles like system/user/assistant) a model expects its input structured in.
Chinchilla optimal: The compute-optimal ratio of model size to training data tokens identified by DeepMind's Chinchilla scaling law study.
cold-start data: An initial set of curated examples used to bootstrap a model's behavior before reinforcement learning takes over, especially in reasoning model training.
constitutional AI: An alignment technique (from Anthropic) that trains a model to critique and revise its own outputs according to a written set of principles, reducing reliance on human labels.
context length extension: Techniques (like RoPE scaling or fine-tuning on longer sequences) used to extend a model's usable context window beyond its original training length.
continued pretraining: Extending a model's pretraining on additional domain-specific data before task-specific fine-tuning.
curriculum learning: Ordering training examples from easy to hard to improve how efficiently a model learns.
data distillation: Generating a curated synthetic training dataset from a strong model's outputs to train a smaller model.
data poisoning: An attack that corrupts a model's training data to induce harmful, biased, or backdoored behavior after training.
dataset deduplication: Removing near-duplicate examples from a training corpus, which improves training efficiency and reduces memorization.
DeepSpeed: Microsoft's open-source library for efficient large-scale distributed model training, known for its ZeRO memory optimization.
dense model: A model architecture where every parameter is used for every token, as opposed to a sparse mixture-of-experts model.
direct preference optimization (DPO): A simpler alternative to RLHF that fine-tunes a model directly on preference pairs without training a separate reward model.
distributed training: Training a single model across many GPUs or machines simultaneously, using parallelism strategies to split the workload.
domain adaptation: Fine-tuning or adapting a general-purpose model to perform well on a specific domain's vocabulary and tasks.
expert routing (MoE): The learned gating mechanism in a mixture-of-experts model that decides which experts process each token.
extended thinking: A model mode where it's allowed to generate a longer internal reasoning trace before producing its final answer.
fine-tuning: Further training a pretrained model on a smaller, task-specific dataset to specialize its behavior.
FSDP (fully sharded data parallel): A PyTorch distributed training strategy that shards model parameters, gradients, and optimizer states across GPUs.
full fine-tuning: Updating all of a model's parameters during fine-tuning, as opposed to parameter-efficient methods like LoRA.
gradient checkpointing: A memory-saving training technique that recomputes intermediate activations during the backward pass instead of storing them all.
group relative policy optimization (GRPO): A reinforcement learning algorithm (used by DeepSeek) that estimates advantage by comparing a group of sampled outputs, avoiding the need for a separate value model.
Hugging Face Transformers: The widely used open-source Python library providing pretrained model architectures and tools for training, fine-tuning, and inference.
inference-time scaling: Improving model performance by allocating more compute during generation (longer reasoning chains, more samples) rather than training a bigger model.
instruct model: A model that has undergone instruction tuning to follow natural-language commands, as opposed to a raw base model.
instruction tuning: Fine-tuning a base model on examples of instructions and desired responses to make it follow directions reliably.
knowledge distillation: Training a smaller "student" model to mimic the outputs or behavior of a larger "teacher" model, transferring capability at lower cost.
KTO (Kahneman-Tversky optimization): A preference-tuning method that optimizes directly on unpaired binary "good/bad" labels instead of requiring paired preference comparisons like DPO.
load balancing loss (MoE): An auxiliary training objective that encourages a mixture-of-experts router to distribute tokens evenly across experts.
LoRA (low-rank adaptation): A parameter-efficient fine-tuning method that trains small low-rank matrices injected into a frozen model instead of updating all its weights.
machine unlearning: Techniques for removing the influence of specific training data from an already-trained model without retraining from scratch.
majority voting (self-consistency): Sampling multiple reasoning paths for the same question and picking the most common final answer.
mixed precision training: Training with a mix of lower-precision (e.g. FP16/BF16) and higher-precision (FP32) numbers to speed up training while preserving stability.
mixture of experts (MoE): A model architecture with many specialized "expert" sub-networks where only a subset is activated per token, increasing capacity without proportionally increasing compute.
model card: A standardized document describing a model's training data, intended use, limitations, and evaluation results.
model checkpoint: A saved snapshot of a model's weights at a particular point during training, which can be resumed from or evaluated.
model distillation: Another term for knowledge distillation: compressing a large model's capabilities into a smaller, cheaper model.
model merging: Combining the weights of multiple fine-tuned models into a single model, often to blend capabilities without retraining.
model souping: Averaging the weights of multiple independently fine-tuned models to produce a single, often more robust, model.
model watermarking: Embedding a statistically detectable but imperceptible pattern into a model's generated output to later verify it came from that model.
model weights: The learned numerical parameters of a neural network that encode everything it "knows" after training.
NTK-aware scaling: A technique for extending a model's effective context length by adjusting its rotary position embedding frequencies.
o1-style reasoning: Reasoning behavior modeled after OpenAI's o1, where a model generates a hidden chain of thought before producing a final visible answer.
on-policy vs off-policy RL: The distinction between training on data generated by the current model policy versus data from a different (often older) policy.
open source model: A model whose weights (and sometimes training code/data) are released publicly, though licensing terms vary widely.
open weights: Model parameters that are publicly downloadable and usable, as opposed to a model only accessible through a hosted API.
ORPO (odds ratio preference optimization): A fine-tuning method that combines supervised fine-tuning and preference alignment into a single training objective, skipping a separate reward model.
outcome reward model (ORM): A reward model that scores only the final answer of a reasoning chain, without evaluating intermediate steps.
parameter-efficient fine-tuning (PEFT): A family of fine-tuning techniques (like LoRA, adapters, prefix tuning) that update only a small fraction of a model's parameters.
post-training: The set of steps (SFT, RLHF, DPO, etc.) applied after pretraining to align a base model's behavior for real-world use.
prefix tuning: A parameter-efficient fine-tuning method that learns a small set of continuous prompt vectors prepended to the input, leaving model weights frozen.
pretraining: The initial, compute-intensive phase of training a model on massive amounts of raw text to learn general language patterns.
process reward model (PRM): A reward model that scores the correctness of each intermediate reasoning step, not just the final answer.
prompt tuning: Learning soft, continuous prompt embeddings via gradient descent instead of hand-writing discrete text prompts.
proximal policy optimization (PPO): The reinforcement learning algorithm most commonly used in the RLHF pipeline to optimize the LLM's policy against a reward model.
QLoRA: A memory-efficient variant of LoRA that fine-tunes a quantized (typically 4-bit) base model, drastically cutting the GPU memory needed.
quality filtering (pretraining data): Automatically scoring and discarding low-quality documents from a pretraining dataset before training begins.
reasoning model: An LLM specifically trained (often via RL) to produce extended step-by-step reasoning before answering, improving performance on complex tasks.
reasoning tokens: The intermediate "thinking" tokens a reasoning model generates before its final answer, often billed separately from output tokens.
red teaming (LLM): Systematically probing a model for harmful, unsafe, or exploitable behaviors before deployment.
reinforcement learning from AI feedback (RLAIF): Like RLHF, but using an AI model's judgments instead of human raters to generate the preference signal.
reinforcement learning from human feedback (RLHF): A training method that uses human preference ratings to train a reward model, then optimizes the LLM's policy against that reward model via reinforcement learning.
rejection sampling (fine-tuning): Generating many candidate outputs, filtering to keep only the best ones by some criterion, and using those to fine-tune the model further.
reward hacking: When a model finds unintended ways to score highly on its reward signal without actually satisfying the intended goal.
reward model: A model trained to predict how much a human (or AI) would prefer one output over another, used to guide RL-based fine-tuning.
RLVR (reinforcement learning with verifiable rewards): An RL fine-tuning approach that uses automatically checkable rewards (like unit tests passing or math answers matching) instead of a learned reward model.
RoPE (rotary position embedding): A method for encoding token position information into attention computations using rotation matrices, widely used in modern LLMs.
safetensors: A safe, fast file format for storing model weights, designed to avoid the security risks of Python's pickle format.
scaling laws: Empirical relationships describing how model performance improves predictably with increases in parameters, data, and compute.
self-play (LLM training): A training approach where a model generates and learns from interactions with copies of itself, without external data.
SimPO: A simplified preference optimization method that removes the need for a reference model used in DPO, reducing training memory and complexity.
sparse MoE: A mixture-of-experts design where a router activates only a small number of experts per token, keeping inference cost sparse relative to total parameters.
sparse reward: A reinforcement learning setup where feedback is only given at the end of a long task, making credit assignment difficult.
supervised fine-tuning (SFT): Fine-tuning a model on labeled input-output pairs to teach it a specific task or response style.
synthetic data generation: Using an LLM to generate training examples (instructions, reasoning traces, dialogues) instead of relying solely on human-collected data.
task vectors: The weight differences between a fine-tuned model and its base model, which can be added, subtracted, or merged like vectors to combine capabilities.
test-time compute: Spending additional computation at inference time (like extended reasoning or multiple samples) to improve answer quality, rather than only scaling training.
tokenizer: The component that converts raw text into the discrete tokens (subword units) an LLM actually processes.
tool-integrated reasoning: Training a model to interleave tool calls (like code execution) directly into its reasoning process rather than reasoning purely in text.
training data attribution: Techniques for tracing which specific training examples most influenced a particular model output.
Unsloth: An open-source library that speeds up and reduces memory usage for LoRA/QLoRA fine-tuning of open-weight models.
verifiable rewards: Reward signals derived from objectively checkable outcomes (code compiles, tests pass, math is correct) rather than subjective human or model judgment.
vocabulary size: The number of distinct tokens a model's tokenizer can represent, which affects both model size and encoding efficiency.
weight license: The specific legal terms (e.g. Apache 2.0, Llama Community License) governing how a model's released weights can be used, modified, or redistributed.
ZeRO (zero redundancy optimizer): A distributed training technique (from DeepSpeed) that shards optimizer states, gradients, and parameters across GPUs to train larger models with less memory per device.