Glossary · Category

Positional Encoding & Context Length

9 plain-English definitions.

Absolute positional encoding

Fixed or learned embeddings added to token embeddings that directly encode each token's position index.

ALiBi (Attention with Linear Biases)

A positional method that adds a fixed linear penalty to attention scores based on the distance between tokens, extrapolating to longer sequences without retraining.

Context length extrapolation

The ability (or techniques to grant the ability) of a model to handle sequences longer than those seen during training.

Effective context length

The context length at which a model actually retains strong retrieval/reasoning performance, often shorter than its advertised maximum context window.

Long-context fine-tuning

Additional training on long documents, often with adjusted positional encodings, to extend a model's usable context window.

Position interpolation

A context-extension technique that linearly rescales position indices to fit a longer sequence within the range the model saw during pretraining.

Positional encoding

Any scheme that injects information about token order into a transformer, since self-attention itself is order-agnostic.

Sinusoidal positional encoding

The original transformer positional scheme using fixed sine/cosine functions of different frequencies to represent position.

YaRN

"Yet another RoPE extensioN" method that interpolates rotary frequencies with a temperature correction, enabling efficient context-length extension with minimal fine-tuning.