Training Deep Networks

Making a network deep (many layers) makes it more expressive, but also much harder to train — this page covers the techniques that make deep training actually work.

What "training" looks like in practice: loss dropping fastest early on, noisier and slower as it approaches a floor — every technique below exists to make this curve drop faster, more smoothly, or further

Loss Functions

Cross-entropy: the standard loss for classification — measures how far the predicted probability distribution is from the true one (see Probability & Statistics). This is also the loss behind next-token prediction in every LLM.
MSE: standard for regression.
Contrastive / triplet loss: used when the goal is learning good embeddings rather than direct predictions — pulls similar examples together and pushes dissimilar ones apart in embedding space. Core to how modern embedding models (used for RAG retrieval) are trained.

Normalization

Batch Normalization: normalizes each layer's activations across the batch dimension during training, stabilizing and speeding up training significantly. Introduces a dependency on batch statistics, which complicates things at inference time (running averages are used instead).
Layer Normalization: normalizes across the feature dimension for each individual example, independent of batch size. This independence is why Transformers use LayerNorm instead of BatchNorm — sequence models often deal with variable-length inputs and small/variable batch sizes.
RMSNorm: a simplified LayerNorm variant that skips re-centering (mean subtraction) and only rescales by the root-mean-square — cheaper to compute, used in most modern LLMs (LLaMA, etc.) with no meaningful quality loss.

Same tensor, different axis normalized over — click a norm type to see exactly which cells get grouped together:

BatchNorm

LayerNorm

RMSNorm

GroupNorm

← batchfeature/channel →

Normalizes across a single example's OWN features instead of across the batch -- batch-size-independent, the default in Transformers.

Regularization for Deep Nets

Dropout: randomly zero out a fraction of neurons each training step, forcing the network to not over-rely on any single neuron — effectively training an ensemble of sub-networks that share weights.
Standard L1/L2 weight regularization still applies (see Model Evaluation & Metrics), though it's used more sparingly in very large models where data volume itself is often the main regularizer.

Vanishing & Exploding Gradients

In a deep network, gradients are a product of many layers' local derivatives (chain rule). If those derivatives are consistently < 1, the gradient shrinks toward zero as it propagates back through many layers ("vanishing") — early layers barely learn. If consistently > 1, gradients blow up ("exploding"), causing unstable, diverging training.

Fixes: careful initialization (He/Xavier), normalization layers, gradient clipping, and — most importantly — residual connections.

Drag $w_h$ below and watch the same product-of-Jacobians mechanism either vanish or explode as it's carried back through more steps:

w_h = 0.90 (per-step factor tanh'(z)·w_h ≈ 0.810)0.9

factor < 1 — gradient vanishes exponentially with distance

\frac{\partial h_T}{\partial h_t} = \prod_{k=t+1}^{T} \mathrm{diag}(\tanh'(z_k))\, W_h

Drag w_h -- below the threshold, |dh_T/dh_t| decays exponentially with distance (vanishing); above it, the same product blows up (exploding). Both come from the identical mechanism: (T-t) repeated multiplications by the same W_h.

Residual (Skip) Connections

Instead of a layer computing $y = f(x)$ , a residual block computes $y = x + f(x)$ — the input is added directly to the output. This gives gradients a direct path backward that bypasses $f$ entirely, largely solving the vanishing gradient problem and enabling networks with hundreds of layers (ResNet) or dozens of Transformer blocks (every modern LLM) to train successfully. This is one of the single most important architectural ideas in deep learning.

\text{LayerNorm}(x + \text{Sublayer}(x))

Sublayer(x) is self-attention in one Add & Norm, the feed-forward network in the other.

Post-norm, as in the original paper: LayerNorm(x + Sublayer(x)). The skip connection (bottom path) means depth never forces gradients through a transformation they can't get around.

Learning Rate Schedules & Gradient Clipping

Warmup + decay: start small, ramp up, then decay — covered in Calculus & Optimization; essential for training stability in Transformers specifically.
Gradient clipping: rescale the gradient if its norm exceeds a threshold, preventing a single bad batch from destabilizing training — near-universal in large-scale training runs.
Mixed precision training: use lower-precision (fp16/bf16) numbers for most computation to speed up training and reduce memory, while keeping certain operations (like the loss) in higher precision to avoid numerical instability.

warmup steps100

⬤ warmup + cosine decay⬤ warmup + linear decay⬤ step decay

Real formulas evaluated at every step: warmup ramps linearly from 0 to the peak LR over the first 100 steps, then each schedule decays differently. Watch the warmup slider -- too-short warmup means the very first, noisiest gradients get a large step; too-long wastes training time at a suboptimal LR.

Next: Convolutional Neural Networks — the architecture that made deep learning work for images.