Glossary · Category
Attention Mechanisms & Transformer Architecture
22 plain-English definitions.
- Attention sink
- The empirical phenomenon where initial tokens in a sequence absorb disproportionate attention mass, exploited by some long-context and streaming methods to preserve stability.
- Causal attention mask
- A mask applied during attention so each token can only attend to itself and earlier tokens, enforcing autoregressive generation.
- Cross-attention
- An attention mechanism where queries come from one sequence and keys/values from another, used to let a decoder attend to encoder outputs or other modalities.
- Decoder-only architecture
- A transformer design that generates tokens autoregressively using only a stack of decoder (causal-attention) blocks; the standard architecture for modern chat LLMs.
- Encoder-decoder architecture
- A two-part transformer design (e.g., T5) where an encoder processes the full input and a decoder generates output conditioned on it, common in translation and summarization.
- Encoder-only architecture
- A transformer design (e.g., BERT) that processes the whole input bidirectionally without generation, used for classification and embeddings.
- Feed-forward network (FFN)
- The position-wise two-layer (or gated) MLP block in a transformer layer that transforms each token's representation independently after attention.
- Gated linear unit (GLU)
- A feed-forward variant that multiplies one linear projection by a gated (activated) second projection, improving expressiveness over a plain MLP.
- GeGLU
- A gated linear unit variant using the GELU activation function in place of SwiGLU's Swish.
- Layer normalization
- A normalization technique that rescales activations across the feature dimension for each token, stabilizing training.
- Linear attention
- An attention approximation that reformulates the softmax kernel to achieve linear rather than quadratic time/memory complexity in sequence length.
- Load balancing loss
- An auxiliary training loss added to MoE models to discourage the router from sending too many tokens to a small subset of experts.
- Multi-head attention
- An attention layer that runs several attention "heads" in parallel on different learned projections of the input so the model can attend to different relationship types at once.
- Pre-norm vs post-norm
- Two placements of normalization layers relative to the residual connection; pre-norm (before the sublayer) is now standard for training stability at scale.
- Residual connection
- A skip connection that adds a layer's input directly to its output, easing gradient flow in deep networks.
- RMSNorm
- A simplified layer normalization that scales by the root-mean-square of activations without subtracting the mean, cheaper and widely used in modern LLMs.
- Scaled dot-product attention
- The core attention formula that divides the dot product of query and key vectors by the square root of the key dimension before softmax, to keep gradients stable.
- Self-attention
- A mechanism where each token in a sequence computes a weighted combination of all other tokens in the same sequence to build context-aware representations.
- Sparse attention
- Any attention variant that skips computing full pairwise attention scores, instead attending to a subset of tokens, to scale to longer contexts cheaply.
- Sparse MoE routing
- The gating mechanism that decides which top-k experts process each token in a mixture-of-experts layer.
- SwiGLU
- A gated linear unit variant using the Swish/SiLU activation function, widely used in modern transformer feed-forward blocks (e.g., LLaMA).