LLMs
Tokenization
Tokenization converts text into a sequence of vocabulary identifiers.
How it works
The tokenizer defines the model’s input alphabet and how text length translates into compute. Its segmentation choices affect multilingual coverage and rare strings.
A concrete example
Illustrative: a 12-character identifier can become 1 token or 8 tokens under different vocabularies.
Do not confuse it with Embedding
tokenization chooses IDs; embedding lookup converts those IDs into vectors.