Fundamentals
Gradient descent
Gradient descent updates parameters in the direction that reduces loss.
How it works
The gradient gives the local slope of the objective with respect to each parameter. The learning rate scales how far the optimizer moves.
A concrete example
Illustrative: with gradient 0.8 and learning rate 0.1, a weight moves from 2.0 to 1.92.
Do not confuse it with Backpropagation
backpropagation computes gradients; gradient descent uses them to update parameters.