Glossary · Category

Tokenization

9 plain-English definitions.

Byte-level BPE

A BPE variant that operates on raw UTF-8 bytes rather than characters, guaranteeing any input text can be tokenized without unknown-token failures.

Detokenization

The process of converting a sequence of model output tokens back into human-readable text.

Out-of-vocabulary (OOV) token

A word or symbol not present in a tokenizer's vocabulary; subword tokenizers avoid true OOV by falling back to smaller pieces or bytes.

SentencePiece

A language-agnostic tokenizer library that trains subword vocabularies (BPE or unigram) directly on raw text without pre-tokenizing on whitespace.

Special tokens

Reserved tokens (e.g., beginning-of-sequence, end-of-sequence, padding, chat role markers) that carry structural rather than linguistic meaning.

Tiktoken

OpenAI's fast BPE tokenizer library used by GPT models, implementing byte-level BPE with model-specific vocabularies (e.g., cl100k_base, o200k_base).

Token-to-word ratio

The average number of tokens needed to represent one word, which varies by language and tokenizer and directly affects real-world cost per request.

Unigram language model tokenizer

A subword tokenization algorithm that starts from a large vocabulary and iteratively prunes tokens to maximize the likelihood of the training corpus.

WordPiece

A subword tokenization algorithm similar to BPE but choosing merges that maximize training-data likelihood rather than raw frequency; used in BERT.