Glossary · Category

Data Engineering for Model Training

10 plain-English definitions.

Common Crawl

A large, freely available web-crawl dataset used as a major source of raw text for pretraining most large language models.

Data contamination detection

Techniques (e.g., n-gram overlap checks) used to detect and remove benchmark test data that has leaked into a pretraining or fine-tuning corpus.

Data mixture / mixture weights

The proportions in which different data sources (web text, code, books, papers, etc.) are sampled during pretraining, tuned to influence downstream model capabilities.

Data provenance

Documentation and tracking of where training data originated, used for compliance, licensing, and reproducibility purposes.

Instruction dataset

A dataset of (instruction, response) pairs used for supervised fine-tuning to teach a base model to follow commands.

Multilingual corpus

A pretraining data mixture spanning many languages, used to build models with broad cross-lingual capability rather than English-only performance.

Preference dataset

A dataset of paired responses with human or AI-labeled preference judgments (which response is better), used to train reward models or run DPO.

Pretraining corpus

The large-scale collection of text (and possibly other modalities) a model is trained on during pretraining, spanning web crawls, books, code, and curated sources.

Quality filtering

Automated or heuristic filtering of raw web-scraped text to remove low-quality, spam, or boilerplate content before it enters a pretraining corpus.

Red-team dataset

A curated set of adversarial prompts designed to probe for unsafe, harmful, or policy-violating model outputs, used both for evaluation and safety fine-tuning.