Multimodal & Generative Models

Attention doesn't care whether a "token" represents a word or an image patch — which is exactly why the same architecture generalizes across modalities.

Vision-Language Models (VLMs)

Combine a vision encoder (often a Vision Transformer, see CNNs and Attention & Transformers) with an LLM: image patches are encoded, projected into the same embedding space as text tokens, and fed into the language model alongside a text prompt — letting the model reason jointly over both. This is how models can answer questions about an image, describe it, or follow instructions that reference visual content.

Image

→

Vision Encoder

→

Projector

→

LLM

Maps those image features into the LLM's embedding space -- the piece that makes image features look like token embeddings to the LLM.

Vision-Language-Action Models (VLAs)

Extend a VLM with the ability to output actions (e.g. robot motor commands) instead of just text — used in robotics, where a model needs to perceive a scene, understand an instruction, and output a concrete physical action, all from a shared multimodal understanding. The interesting engineering question isn't "can a VLM see and read" (VLMs already do that) — it's how you get a model that only ever learned to output tokens to output a continuous motor command instead.

Two answers dominate in practice, and they trade off in the same discretize-vs-regress way that shows up all over ML:

Discretize the action space into tokens. RT-2 (Google DeepMind) fine-tunes a large VLM — PaLI-X (55B) or PaLM-E (12B) — that was already pretrained on web-scale image-text data, and represents each robot action as a short string of digit tokens (e.g. "1 128 91 241 5 101 127 217") appended to its existing vocabulary. Because actions are just tokens, RT-2 needs no new output head or architecture at all — it reuses next-token prediction unchanged, which is exactly what lets it inherit the base VLM's web-scale semantic knowledge (recognizing objects, following instructions phrased in novel ways) directly into robot control. OpenVLA follows the same discretized-token recipe on a fully open stack: a 7B Llama 2 backbone with a dual DINOv2 + SigLIP vision encoder, trained on 970k real-world robot demonstrations, and LoRA-fine-tunable to a new robot. It reportedly outperforms the much larger RT-2-X (55B) by 16.5 points of absolute task success across 29 tasks — with 7x fewer parameters — evidence that the training data and recipe matter as much as raw scale for this task.
Regress continuous actions directly. Discretized tokens are a reasonable fit for a single low-frequency pick-or-place decision, but dexterous manipulation (folding laundry, assembling a box) needs smooth, high-frequency motor commands — up to 50 Hz — where snapping every action to a token bin gets jerky. π₀ (Physical Intelligence) instead builds on a 3B PaliGemma VLM and adds flow matching — the same continuous, diffusion-style generation idea covered below for images, but generating an action trajectory instead of an image — trained on 10,000+ hours of real robot data across tasks like folding laundry and bagging groceries. Its successor, π₀.₅, hedges between both worlds: a hierarchical setup where coarse action tokens are predicted first (discretized, via a "FAST" tokenizer), then refined into continuous motor commands via flow matching.

Either way, the core VLA idea is the same as the VLM diagram above: encode vision and language into one shared representation — the only difference is what comes out the other end.

Diffusion Models

Instead of generating autoregressively (one token/pixel at a time), diffusion models learn to reverse a gradual noising process: starting from pure noise, the model iteratively denoises step by step until a coherent image (or audio, or video) emerges. Trained by teaching the model to predict the noise that was added at each step of a forward "noising" process — a form of the same gradient-descent-on-a-loss-function idea from Calculus & Optimization, just with a different objective. This is the dominant approach behind modern image generation models. See Generative Models for the named variants (DDPM, DDIM, Latent/Stable Diffusion, ControlNet) and how diffusion compares to its main alternative, GANs.

The real forward-process math behind that noising, applied here too:

t = 1 / 1001

x_t: structure at low t, indistinguishable from noise as t → 100

alpha_bar_t (fraction of original signal retained) vs. t

Real linear beta schedule (beta: 1e-4 -> 0.02 over 100 steps), real cumulative alpha_bar_t = 1.000 at t = 1, real forward formula x_t = sqrt(alpha_bar_t)*x0 + sqrt(1-alpha_bar_t)*epsilon applied to a fixed toy signal and a fixed noise draw. By t ~= 100, alpha_bar_t is near 0 -- almost no original signal left, matching how DDPM's forward process ends at pure noise regardless of the input image.

Diffusion vs. Autoregressive Generation

Autoregressive (standard LLMs): generates sequentially, one token conditioned on all previous ones — naturally suited to text, where order and causality matter.
Diffusion: generates the whole output "at once," refined iteratively — naturally suited to images, where there's no inherent left-to-right ordering.

Diffusion Language Models (DLMs): an active research direction applying diffusion-style generation to text instead of autoregression — potentially enabling faster parallel generation (denoising a whole sequence at once rather than token-by-token), though autoregressive models remain dominant in production as of today. The structural difference made concrete, step by step:

generation mode

step = 4 / 164

each step commits exactly one new token, left to right

Autoregressive: token i only resolves once i < step (real index comparison) -- strictly left to right, one token per step, 4/16 resolved.

LLMs & GenAI section complete. Next: Agents — where models stop just answering and start taking action.