Glossary · Category

Activation Functions & Normalization Variants

7 plain-English definitions.

Batch normalization

A normalization technique that rescales activations across the batch dimension during training, common in CNNs but largely replaced by layer/RMS normalization in transformers.

GELU (Gaussian Error Linear Unit)

A smooth activation function that weights inputs by their value under a Gaussian CDF, commonly used in transformer feed-forward layers (e.g., BERT, GPT-2).

Group normalization

A normalization technique that divides channels into groups and normalizes within each group, an alternative to batch normalization that doesn't depend on batch size.

QK-normalization

A stabilization technique that applies normalization to query and key vectors before computing attention scores, used in some large-scale models to prevent attention logit blowup.

ReLU (Rectified Linear Unit)

An activation function that outputs the input directly if positive and zero otherwise, computationally cheap and historically dominant in deep learning before transformer-era alternatives.

SiLU / Swish

A smooth, self-gated activation function (x times sigmoid(x)) used as the nonlinearity inside SwiGLU feed-forward blocks in modern LLMs.

Softplus

A smooth approximation of ReLU defined as the log of one plus the exponential of the input, occasionally used where a differentiable-everywhere activation is needed.