Inference
Speculative decoding
Speculative decoding uses a cheaper model to propose tokens that a target model verifies in parallel.
How it works
Accepted draft tokens advance generation several positions per target-model pass. The method preserves the target distribution when acceptance and correction are implemented correctly.
A concrete example
Illustrative: a draft proposes 5 tokens and the target accepts the first 4 before correcting token 5.
Do not confuse it with Distillation
speculative decoding uses two models at runtime; distillation trains a smaller model beforehand.