Glossary · Category

Hardware

47 plain-English definitions.

A100: NVIDIA's Ampere-architecture data center GPU, the dominant training/inference chip before Hopper, still widely used.
B200: NVIDIA's Blackwell-architecture data center GPU, successor to the Hopper generation, built for large-scale AI training and inference.
bare-metal GPU: Renting a physical GPU server directly without a virtualization layer, for maximum performance and control.
capex vs opex (AI infra): The distinction between buying GPU hardware outright (capital expense) versus renting cloud compute (operating expense) for AI workloads.
Cerebras: An AI hardware company known for its wafer-scale engine chips designed for extremely fast LLM training and inference.
compute cluster scheduling: Software (like Kubernetes, Slurm, or Ray) that allocates GPU jobs to available cluster nodes efficiently.
consumer GPU inference: Running LLM inference on gaming/consumer-grade GPUs (like an RTX 4090) rather than data-center hardware, common for local/hobbyist use.
custom silicon (AI accelerator): Purpose-built chips designed specifically to accelerate AI workloads, as opposed to general-purpose GPUs.
CXL (Compute Express Link): An emerging interconnect standard that lets GPUs and CPUs share memory pools more flexibly, relevant to future inference memory scaling.
edge inference: Running model inference directly on local or edge devices (phones, laptops, IoT) instead of a remote data center.
FLOPs: Floating-point operations; the standard unit for measuring the raw compute capacity used or required by a model.
FP4 precision: An ultra-low 4-bit floating point format emerging on newest GPUs (like Blackwell) for even more aggressive inference compression.
FP8 precision: An 8-bit floating point number format increasingly used for both training and inference to cut memory and compute cost with minimal quality loss.
GB200: NVIDIA's Blackwell superchip that combines two B200 GPUs with a Grace CPU on a single module for large-scale AI clusters.
GPU: Graphics processing unit; the parallel-compute hardware that trains and runs virtually all modern deep learning models.
GPU broker: A marketplace or intermediary that matches buyers of AI compute with available GPU supply across multiple data centers.
GPU class: A tier or category of GPU (e.g. consumer, data-center, high-memory) chosen based on the performance and cost needs of a workload.
GPU cluster: A collection of interconnected GPU servers used together to train or serve very large models beyond a single machine's capacity.
GPU memory capacity: The total amount of VRAM a GPU has, which directly limits how large a model (or how many concurrent requests) it can serve.
GPU sharding: Splitting model weights or computation across multiple GPUs when a single GPU's memory is insufficient.
GPU shortage: The persistent supply-demand imbalance for data center AI chips that has driven up prices and lead times since 2023.
GPU-hour: A billing unit representing one GPU running for one hour, commonly used to price cloud AI compute.
H100: NVIDIA's Hopper-architecture data center GPU, the workhorse for most large-scale LLM training and inference through 2024-2025.
H200: An upgraded version of NVIDIA's H100 GPU with significantly more and faster HBM memory, aimed at large-model inference.
HBM (high bandwidth memory): A type of stacked, high-throughput memory used on data center GPUs, critical for feeding data fast enough to keep up with compute.
Inferentia: Amazon's custom chip designed specifically for cost-efficient LLM inference on AWS.
InfiniBand: A high-throughput, low-latency networking technology commonly used to interconnect GPU nodes in large AI training clusters.
Kubernetes for GPUs: Using Kubernetes to orchestrate and schedule containerized workloads that require GPU resources, common for inference deployments.
liquid cooling (data center): A cooling method using circulated liquid instead of air to manage the heat output of dense GPU clusters.
LPU: Groq's Language Processing Unit, a custom chip architecture designed for extremely fast, low-latency LLM token generation.
MFU (model FLOPs utilization): The percentage of a GPU's theoretical peak compute that's actually being used productively during training or inference.
MIG (multi-instance GPU): An NVIDIA feature that partitions a single physical GPU into multiple isolated smaller GPU instances, each servable independently.
multi-GPU inference: Splitting a single model's inference workload across more than one GPU because the model doesn't fit on one device.
neocloud: A newer category of cloud provider that specializes specifically in renting out GPU compute for AI workloads, as opposed to general-purpose hyperscalers.
NVLink: NVIDIA's high-speed interconnect that lets multiple GPUs communicate directly with much higher bandwidth than standard PCIe.
NVSwitch: NVIDIA's switch chip that enables all-to-all high-bandwidth GPU communication within a server node.
on-device inference: Running an LLM's computation entirely on the end-user's own hardware, avoiding network calls to a server.
power density (data center): The amount of electrical power consumed and heat generated per rack in AI data centers, a growing infrastructure constraint.
power usage effectiveness (PUE): A data center efficiency metric comparing total facility power consumption to the power actually delivered to computing equipment.
SambaNova: An AI hardware company offering custom dataflow-architecture chips for enterprise AI training and inference.
tensor core: Specialized GPU hardware units designed to accelerate the matrix multiply operations that dominate neural network compute.
total cost of ownership (GPU): The full cost of operating GPU infrastructure over its lifetime, including purchase or rental price, power, cooling, and maintenance.
TPU: Google's custom Tensor Processing Unit, an ASIC designed specifically for accelerating neural network training and inference.
Trainium: Amazon's custom AI training chip designed as a cost-efficient alternative to NVIDIA GPUs on AWS.
vGPU: A virtualized slice of a physical GPU allocated to a single VM or container, enabling GPU resource sharing in virtualized environments.
VRAM: The dedicated memory on a GPU that holds model weights, activations, and the KV cache during inference or training.
wafer-scale chip: A chip design (pioneered by Cerebras) that uses an entire silicon wafer as a single giant processor instead of cutting it into individual dies.