Inference
Decode phase
Decode is the autoregressive phase that generates one new token per active sequence per iteration.
How it works
Each iteration reads model weights and the existing KV cache. It is often memory-bandwidth constrained at small batch sizes.
A concrete example
Illustrative: 100 decode iterations produce 100 new tokens for one active sequence.
Do not confuse it with Prefill
decode handles new tokens sequentially; prefill handles the prompt in parallel.