LLMs

Multi-head attention

Multi-head attention runs several learned attention projections in parallel.

Multi-head attention runs several learned attention projections in parallel. Values shown in the worked visual are illustrative unless the labels state otherwise.

How it works

Each head can represent a different relation in its own projected subspace. The head outputs are concatenated and projected back to the model width.

A concrete example

Illustrative: width 768 split across 12 heads gives 64 dimensions per head.

Do not confuse it with Multi-query attention

multi-head attention has separate key-value heads; MQA shares one key-value head.

Study the full lesson