Inference
Continuous batching
Continuous batching admits and removes requests as individual decoding slots become available.
How it works
Static batches wait for every sequence to finish, wasting capacity on completed rows. Continuous scheduling replaces finished work without draining the whole batch.
A concrete example
Illustrative: when 3 of 16 sequences finish, the scheduler admits 3 waiting requests at the next scheduling boundary.
Do not confuse it with Dynamic batching
dynamic batching groups arrivals before execution; continuous batching reschedules during decoding.