Glossary · Category

Classic Optimizers & Training Infrastructure Detail

8 plain-English definitions.

Adam optimizer

An adaptive optimizer combining momentum with per-parameter learning-rate scaling based on gradient variance, the direct predecessor to AdamW.

Gradient accumulation

A technique that sums gradients over several small forward/backward passes before applying a single parameter update, simulating a larger batch size on limited memory.

Gradient clipping

A training stabilization technique that caps the magnitude of gradients before applying an update, preventing destabilizing large steps.

Loss spike

A sudden, large increase in training loss during a large-scale training run, often requiring a rollback to a prior checkpoint and hyperparameter adjustment.

Momentum (optimization)

An optimization technique that accumulates a running average of past gradients to smooth updates and accelerate convergence, incorporated into optimizers like SGD-with-momentum and Adam.

Muon optimizer

A newer optimizer designed specifically for training transformer weight matrices that orthogonalizes update directions, showing faster convergence than AdamW in some large-scale runs.

RMSProp

An adaptive-learning-rate optimizer that divides each parameter's gradient by a running average of recent squared gradients, a precursor to Adam.

Stochastic gradient descent (SGD)

The foundational optimization algorithm that updates parameters using gradients estimated from small random batches of data rather than the full dataset.