LLMs
Token
A token is one unit in the sequence processed by a language model.
How it works
A tokenizer converts text into integer token identifiers before model computation. Tokens may be words, word pieces, punctuation, or bytes.
A concrete example
Illustrative: “unbelievable” might split into 3 tokens, depending on the tokenizer.
Do not confuse it with Word
a token is a model unit and does not necessarily match a linguistic word.