Glossary · Category
Multimodal & Vision-Language Terminology
14 plain-English definitions.
- Automatic speech recognition (ASR)
- A model or system that transcribes spoken audio into written text.
- Autoregressive image generation
- Generating images token-by-token (treating image patches or codes as a sequence) rather than through diffusion denoising.
- CLIP (Contrastive Language-Image Pretraining)
- A model trained to align image and text embeddings in a shared space via contrastive learning, widely used as a vision encoder or for zero-shot image classification.
- Cross-modal alignment
- Training techniques that map different modalities (text, image, audio) into a shared representation space so a model can reason across them jointly.
- Diffusion model
- A generative model class that learns to reverse a gradual noising process, iteratively denoising random noise into structured output (commonly images, increasingly used for other modalities).
- Image tokenization / patchification
- Splitting an image into fixed-size patches that are each embedded and treated as a token in a vision transformer's input sequence.
- Interleaved multimodal input
- A prompt format that mixes text, images, and other modalities in arbitrary order within the same context window.
- Latent diffusion
- A diffusion model variant that performs the denoising process in a compressed latent space rather than raw pixel space, greatly reducing compute cost.
- Multimodal fusion
- The point and method (early, late, or intermediate) at which information from different modalities is combined inside a model's architecture.
- Native multimodality
- An architecture trained from the start on interleaved multiple modalities in one shared model, as opposed to bolting a vision encoder onto a pretrained text-only LLM.
- Optical character recognition (OCR)
- Extracting machine-readable text from images of printed or handwritten text; increasingly performed end-to-end by vision-language models.
- Text-to-speech (TTS)
- A model or system that converts written text into synthesized spoken audio.
- Vision encoder
- The sub-network (often a Vision Transformer) that converts raw image pixels into a sequence of embeddings a multimodal model can process alongside text tokens.
- Vision Transformer (ViT)
- A transformer architecture applied to images by splitting them into fixed-size patches treated as a token sequence.