Inference
Tokens per second
Tokens per second measures generation throughput over time.
How it works
The value must state whether it is per request, per device, or aggregate. Batching can improve aggregate throughput while worsening individual latency.
A concrete example
Illustrative: 800 output tokens completed in 10 seconds equals 80 aggregate tokens per second.
Do not confuse it with Requests per second
token throughput counts generated units; request throughput counts completed requests.