Inference
KV cache
A KV cache stores attention keys and values from earlier tokens during autoregressive decoding.
How it works
Without the cache, every new token would recompute projections for the full prefix. The cache saves compute but grows with sequence length, layer count, and key-value width.
A concrete example
Illustrative: 32 layers, 32 KV heads, head width 128, 2 bytes, and 2,048 tokens require about 1 GiB.
Do not confuse it with Prefix cache
a KV cache belongs to an active sequence; prefix caching reuses shared prefixes across requests.