Inference
Tensor parallelism
Tensor parallelism splits individual tensor operations across devices.
How it works
Each device computes a shard of a layer and communicates partial results. It reduces per-device weight memory but introduces collective communication.
A concrete example
Illustrative: an 8,192-column matrix is split into four 2,048-column shards across 4 GPUs.
Do not confuse it with Pipeline parallelism
tensor parallelism splits layers internally; pipeline parallelism assigns different layers to stages.