Glossary · Category

Fine-Tuning & Alignment Methods

7 plain-English definitions.

Adapter layers

Small trainable modules inserted between a frozen pretrained model's layers, an early parameter-efficient fine-tuning approach predating LoRA.

DPO (Direct Preference Optimization)

An alignment method that directly optimizes a model on pairwise preference data using a closed-form loss, avoiding the need to train a separate reward model or run RL.

GRPO (Group Relative Policy Optimization)

An RL algorithm (used in DeepSeek-R1) that estimates advantage by comparing a group of sampled outputs to each other's average reward, removing the need for a separate value-function critic model.

PPO (Proximal Policy Optimization)

The reinforcement learning algorithm most commonly used in RLHF to update the policy (LLM) against a learned reward model while constraining how far it drifts per update.

RLAIF (Reinforcement Learning from AI Feedback)

A variant of RLHF where an AI model (rather than humans) generates the preference labels used to train the reward model, reducing labeling cost.

RLHF (Reinforcement Learning from Human Feedback)

An alignment technique that trains a reward model from human preference comparisons, then optimizes the LLM against that reward model using reinforcement learning.

Test-time compute / inference-time scaling

Improving model output quality by spending more computation at inference time (e.g., longer reasoning chains, sampling multiple candidates) rather than only via larger training runs.