Attention Step-Through
Type any sentence and watch the real computation from Attention & Transformers run on it live — an actual attention heatmap and per-token attention breakdown, not a diagram of one.
Interactive
Attention Step-Through
Attention head
Every value below is real, computed live from “the” — color encodes sign and magnitude (green = positive, red = negative, opacity = size), the same value-as-color convention used throughout this walkthrough, now live instead of a fixed worked example:
Embedding
-0.62
0.34
0.55
-0.11
Query
-0.51
-0.14
0.18
-0.03
Key
-0.23
0.50
-0.26
-0.31
Value
-0.08
-0.59
-0.17
0.10
Attention from “the” to every token:
the
17%
cat
18%
sat
17%
on
17%
the
17%
mat
15%
Real Q·Kᵀ/√d_k → softmax → weighted-sum-of-V math, computed on deterministic demo embeddings (see the component note for exactly what's simplified and why).
What to Try
- Click different rows (query tokens) and watch the bar chart on the right update — this is the attention weight distribution Self-Attention: Query, Key, Value describes: how much a given token "looks at" every other token when building its output representation.
- Switch heads on the same sentence — since each head has independently-seeded Q/K/V weights, you'll see visibly different attention patterns for the exact same input, a concrete (if untrained) illustration of why Multi-Head Attention gives a model several "representation subspaces" instead of one.
- Try a sentence with a repeated word ("the cat sat on the mat and the dog watched the cat") and see how attention distributes across the multiple occurrences.
What's Simplified Here, and Why
Two honest simplifications, both to keep this component small and fast rather than bundling a full tokenizer/model: tokenization is whitespace/punctuation-level, not real subword BPE (see Foundation Model Internals — Tokenization for the real thing), and the embeddings/weights are deterministic demo values, not a trained model's. The computation — the actual matrix multiplies, the scaling, the softmax, the weighted sum — is exactly what a real Transformer layer performs; only the inputs to that computation are simplified.
Back to Visual Lab Overview.