Glossary · Category

Speculative Decoding & Model Compression (Distillation, Pruning)

6 plain-English definitions.

Layer skipping / early exit

An inference technique that allows a model to stop processing at an intermediate layer for "easy" tokens, saving compute.

Model pruning

Removing weights, neurons, attention heads, or entire layers from a trained model to reduce its size and inference cost, ideally with minimal quality loss.

Structured pruning

Pruning that removes entire structural units (channels, heads, layers) rather than individual weights, yielding hardware-friendly speedups.

Task arithmetic / task vectors

A model-merging technique that treats the weight difference between a fine-tuned and base model as a vector that can be added, subtracted, or scaled to combine or remove capabilities.

Unstructured pruning

Pruning that zeroes out individual weights anywhere in the network based on magnitude or importance, requiring sparse-matrix support to realize speedups.

Weight sharing

An architectural technique where multiple layers reuse the same parameters, reducing model size at the cost of representational flexibility.