LLMs
Self-attention
Self-attention mixes information among positions in the same sequence.
How it works
Each position scores other positions using query-key similarity and combines their value vectors. The learned weights make the context-dependent mixture explicit.
A concrete example
Illustrative: one token assigns weights [0.05, 0.15, 0.70, 0.10] across four positions.
Do not confuse it with Cross-attention
self-attention reads the same sequence; cross-attention reads another sequence or modality.