Neural Mastery

Inference

Time to first token

Time to first token measures latency from request arrival until the first generated token is available.

queueprefilltoken 1token 2request80ms220ms|40msTTFT is the highlighted wait: 300 ms here.
Time to first token measures latency from request arrival until the first generated token is available. Values shown in the worked visual are illustrative unless the labels state otherwise.

How it works

It includes queueing, prompt processing, and initial sampling. Long prompts can increase it even when later token generation is fast.

A concrete example

Illustrative: 35 ms queueing plus 120 ms prefill plus 5 ms sampling gives 160 ms TTFT.

Do not confuse it with Inter-token latency

TTFT covers startup; inter-token latency covers gaps during generation.

Study the full lesson