Glossary · Category

Additional Tokenizer & Language Coverage Terms

5 plain-English definitions.

Context length in characters vs tokens

The distinction between a model's token-based context limit and the variable number of characters/words that limit represents, which shifts depending on language and content type.

Fertility (tokenizer)

The average number of tokens a tokenizer produces per word for a given language, used to compare how efficiently different tokenizers encode different languages.

Merge rules (BPE)

The ordered list of subword-pair merge operations learned during BPE training that defines exactly how new text is tokenized.

Multilingual tokenizer coverage

How evenly a tokenizer's vocabulary represents different languages and scripts; poor coverage inflates token counts (and thus cost/latency) for underrepresented languages.

Subword regularization

A tokenization training technique that introduces randomness into how words are split into subwords during training, improving model robustness to tokenization variation.