Neural Mastery

LLMs

Next-token prediction

Next-token prediction estimates a probability distribution for the token after a prefix.

satranateflew"The cat ..." gives a probability for each next token.
Next-token prediction estimates a probability distribution for the token after a prefix. Values shown in the worked visual are illustrative unless the labels state otherwise.

How it works

Autoregressive language models repeat this operation to generate a sequence. Training uses the known next token at every position as the target.

A concrete example

Illustrative: after “2 + 2 =”, probabilities may assign 0.96 to token “4”.

Do not confuse it with Masked language modeling

next-token prediction is causal; masked modeling reconstructs hidden tokens using both sides.

Study the full lesson