Optimizers, In Full Depth

Calculus & Optimization introduced gradient descent and briefly named the adaptive optimizers. This page derives each one and shows, visually, why the field moved from plain SGD to today's defaults.

SGD, Momentum, and Adam racing toward the minimum of an elongated ("ravine") loss surface

Watch the three paths above on this deliberately elongated bowl (steep in one direction, shallow in the other — a stand-in for real loss surfaces, which are rarely nicely circular). Plain SGD (red) crawls along the shallow direction and stalls far from the minimum within the step budget. Momentum (gold) overshoots and oscillates before settling — it builds up speed but has to fight that speed to turn. Adam (green) adapts its step size per-dimension and reaches the minimum's neighborhood fastest, though it overshoots once too. This single picture motivates every optimizer below.

Now try it yourself — same bowl, your starting point, your learning rate:

Gradient Descent Variants (by batch size)

• Batch Gradient Descent: compute the gradient using the entire dataset, then take one step. Accurate direction, but one step per full pass over the data — painfully slow for large datasets, and requires holding the whole dataset's gradients in memory at once.
• Stochastic Gradient Descent (SGD): one example at a time. Extremely noisy gradient estimates, but that noise actually helps escape shallow local minima and saddle points — and it enables starting to learn before even seeing the full dataset once.
• Mini-Batch Gradient Descent: the practical universal default — a small batch (32-512 examples) per step. Balances gradient accuracy against update frequency, and maps naturally onto GPU parallelism (see Algorithms & Data Structures). When people say "SGD" in a deep learning context, they almost always mean this.

The base update rule, regardless of batch size: $\theta \leftarrow \theta - \alpha \nabla L(\theta)$.

Momentum-Based Methods

Momentum: accumulate a running (exponentially-weighted) average of past gradients, and step in that direction instead of the raw current gradient:

$v \leftarrow \beta v + (1-\beta)\nabla L(\theta), \qquad \theta \leftarrow \theta - \alpha v$

Like a ball rolling downhill: it builds speed in a consistent direction and dampens oscillation across a ravine's steep walls (see the chart above). $\beta$ (typically 0.9) controls how much history persists.

Nesterov Momentum: a lookahead correction — compute the gradient not at the current position, but at the position momentum is about to carry you to, then adjust. This "look before you leap" adjustment gives noticeably better convergence than vanilla momentum on the same budget, at almost no extra cost.

Adaptive Learning Rate Methods

Momentum uses the same effective learning rate for every parameter. Adaptive methods scale the learning rate per-parameter, based on that parameter's own gradient history.

AdaGrad: divide each parameter's learning rate by the square root of the sum of all its squared past gradients: $\theta_i \leftarrow \theta_i - \frac{\alpha}{\sqrt{G_i + \epsilon}}\nabla_i L$, where $G_i$ accumulates $\nabla_i L^2$ over every step so far. Parameters that get frequent large gradients (common features) end up with a smaller effective learning rate; rare features keep a larger one — good for sparse data. Fatal flaw: $G_i$ only ever grows, so the effective learning rate eventually decays to ≈0 and training stalls, even if far from converged.

RMSProp: fixes AdaGrad's decay problem directly — replace the sum of squared gradients with an exponentially-decaying moving average, so old gradients get forgotten instead of accumulating forever: $G_i \leftarrow \gamma G_i + (1-\gamma)\nabla_i L^2$. The learning rate can now go back up if recent gradients shrink, instead of monotonically dying.

AdaDelta: a further refinement of RMSProp that removes the need to manually set a global learning rate at all — it uses the ratio of recent parameter update magnitudes to recent gradient magnitudes to derive an implicit step size automatically.

Adam and Its Descendants

Adam (Adaptive Moment Estimation): combines momentum (a moving average of the gradient itself — the "first moment") with RMSProp-style adaptive scaling (a moving average of the squared gradient — the "second moment"):

$m \leftarrow \beta_1 m + (1-\beta_1)\nabla L, \qquad v \leftarrow \beta_2 v + (1-\beta_2)(\nabla L)^2$ $\hat{m} = \frac{m}{1-\beta_1^t}, \quad \hat{v} = \frac{v}{1-\beta_2^t}, \qquad \theta \leftarrow \theta - \alpha \frac{\hat{m}}{\sqrt{\hat{v}}+\epsilon}$

The $\hat{m}, \hat{v}$ step is bias correction — early in training, $m$ and $v$ start at zero and are biased toward zero for the first several steps; dividing by $(1-\beta^t)$ corrects for this so early updates aren't artificially shrunk. Adam is the default optimizer for the overwhelming majority of deep learning today, including Transformers (see Attention & Transformers).

AdamW: a small but important fix to Adam — standard Adam applies L2 weight decay by adding it into the gradient before the adaptive scaling, which means the effective decay strength gets distorted by each parameter's own $\hat{v}$. AdamW instead applies weight decay directly to the weights, decoupled from the adaptive step — this is now the actual default in most modern training code (including every major LLM), not plain Adam.

AdaMax: a variant of Adam using the $L^\infty$ norm (max of past gradient magnitudes) instead of the $L^2$ norm for the second moment — occasionally more stable, rarely the default.

Nadam: Adam with Nesterov's lookahead correction folded in, combining both ideas.

Lion (EvoLved Sign Momentum): a much simpler, more memory-efficient recent optimizer that only tracks momentum (no second moment) and uses the sign of the momentum for the update rather than its magnitude — found via automated search rather than hand-derivation, and competitive with AdamW on some large-scale training runs while using less memory (no $v$ to store).

LAMB (Layer-wise Adaptive Moments): wraps Adam's update with an additional per-layer scaling based on the ratio of weight norm to update norm — specifically designed to keep training stable when using very large batch sizes (thousands+), where plain Adam can become unstable.

Choosing an Optimizer

• Default, almost always: AdamW.
• Memory-constrained / very large models: Lion (fewer optimizer states to store) or 8-bit Adam variants.
• Very large batch, distributed training: LAMB.
• Classical ML / convex problems (see Linear Regression): plain SGD or even the closed-form solution, since there's no messy non-convex surface to navigate.
• Sparse features (e.g. large embedding tables): AdaGrad's original motivation still applies, though Adam-family optimizers now dominate in practice regardless.

Minimal Implementation

import numpy as np

def adam_step(theta, grad, m, v, t, lr=0.001, b1=0.9, b2=0.999, eps=1e-8):
    m = b1 * m + (1 - b1) * grad
    v = b2 * v + (1 - b2) * grad ** 2
    m_hat = m / (1 - b1 ** t)
    v_hat = v / (1 - b2 ** t)
    theta = theta - lr * m_hat / (np.sqrt(v_hat) + eps)
    return theta, m, v

Next: Weight Initialization, Regularization & LR Scheduling — the remaining training-loop decisions that determine whether an optimizer like this actually converges well.