Glossary · Category
Sampling, Decoding & Generation Parameters
18 plain-English definitions.
- Contrastive search
- A decoding method that balances token probability with a penalty for similarity to previous tokens' representations, aiming to reduce degenerate repetition.
- Frequency penalty
- A decoding adjustment that penalizes tokens proportionally to how often they've already appeared in the output.
- Greedy decoding
- A decoding strategy that always picks the single highest-probability next token, producing deterministic but sometimes repetitive output.
- Guided JSON / structured output
- A feature where the model's decoding is constrained to conform to a target JSON schema, common in tool-calling and function-calling APIs.
- Logit bias
- A per-token adjustment applied directly to model logits before sampling, used to encourage or suppress specific tokens (e.g., banning a word).
- Logprobs
- The log-probabilities a model assigns to candidate tokens at each generation step, often exposed via API for confidence estimation or classification.
- Max tokens / max output tokens
- The upper limit on how many tokens a model is allowed to generate in a single response.
- Min-p sampling
- A sampling method that keeps only tokens whose probability is at least a fraction of the top token's probability, adapting the cutoff to the model's confidence.
- Mirostat
- An adaptive sampling algorithm that dynamically tunes a perplexity target during generation to maintain consistent output "surprise" and avoid repetition or incoherence.
- Presence penalty
- A decoding adjustment that applies a flat penalty to any token that has appeared at least once already, encouraging topic diversity.
- Repetition penalty
- A decoding adjustment that lowers the probability of tokens already generated, reducing repetitive loops in output.
- Stochastic sampling
- Any decoding approach that draws the next token randomly according to the (possibly modified) probability distribution, as opposed to deterministic greedy decoding.
- Stop sequence
- A user-specified string that halts generation immediately once produced, used to control output boundaries.
- Tail-free sampling
- A sampling method that truncates low-probability tokens based on the curvature of the second derivative of the sorted probability distribution.
- Temperature
- A sampling parameter that scales the logits before softmax; lower values make output more deterministic/peaked, higher values increase randomness.
- Top-k sampling
- A sampling method that restricts token choices to the k highest-probability tokens before sampling.
- Top-p (nucleus sampling)
- A sampling method that restricts token choices to the smallest set whose cumulative probability exceeds p, dynamically adjusting the candidate pool size.
- Typical sampling
- A sampling method that selects tokens close to the expected entropy of the distribution rather than just the highest-probability ones, aiming for more natural text.