Training

LoRA

LoRA learns low-rank update matrices while keeping the original model weights frozen.

LoRA learns low-rank update matrices while keeping the original model weights frozen. Values shown in the worked visual are illustrative unless the labels state otherwise.

How it works

A weight update is represented as the product of two small matrices. This reduces trainable parameters and optimizer state while preserving the base model.

A concrete example

Illustrative: a 4,096×4,096 update at rank 16 uses 131,072 trainable values instead of 16,777,216.

Do not confuse it with Adapter layer

LoRA modifies weight updates through low-rank factors; adapters insert new network modules.

Study the full lesson