Deep Learning Overview
Everywhere Machine Learning fits models with a small, hand-chosen set of features and a fixed functional form — a line, a tree, a kernel — deep learning takes the opposite bet: give a network enough layers and enough data, and let it learn its own features, end to end, directly from raw pixels, raw audio, or raw text. That single idea — stack simple, differentiable operations, and let gradient descent find the weights — is what this entire section is about, from the smallest building block up to the architecture behind every modern large language model.
Why "deep" specifically
A single layer of weighted sums and a nonlinearity (a perceptron, see Neural Network Fundamentals) can only separate data that's linearly separable. Stack layers, and each one learns a progressively more abstract transformation of the last — edges into shapes into object parts into objects, for a vision network; characters into words into phrases into meaning, for a language model. Depth is what buys that hierarchy. The cost is that training a deep stack is a genuinely harder optimization problem than training a shallow one (vanishing/exploding gradients chief among the reasons — derived in full in Sequence Models and Training Deep Networks), which is exactly why so much of this section is about the mechanics of training, not just the architectures themselves.
How this section is organized
Four clusters, roughly in the order the field actually solved them:
1. Foundations — how a network computes and learns at all. Neural Network Fundamentals derives the forward pass and backpropagation from scratch; Activation Functions, Loss Functions, Optimizers, and Initialization, Regularization & Scheduling cover every choice you actually have to make to get a network training at all, and Training Deep Networks covers what changes once "a network" becomes "a deep network" — normalization, residual connections, and the vanishing-gradient problem they exist to fix.
2. Architectures for structured data. Raw fully-connected layers ignore the structure of an input — a pixel's neighbors, a word's position in a sentence — and that structure is exactly what the next architectures are built to exploit. CNNs exploit spatial locality (a pixel is related to its neighbors) via convolution and pooling; Vision Architectures covers what's built on top of that foundation (ViTs, detection, segmentation). Sequence Models exploits temporal order via recurrence (RNN → LSTM/GRU → Seq2Seq+Attention) and derives exactly why recurrence itself becomes the bottleneck at scale — the problem Attention & Transformers exists to solve, removing recurrence entirely in favor of direct, parallel attention between every pair of positions. This is the architecture behind GPT, Claude, LLaMA, and effectively every frontier model today, so it's worth treating as the section's centerpiece, not just one topic among many.
3. Learning without labels. Autoencoders & VAEs and Generative Models: GANs & Diffusion cover architectures trained without hand-labeled targets — reconstruction error, adversarial competition, or iterative denoising standing in for supervision. GNNs, RL, Metric Learning, SSL & Multimodal Nets rounds out the architectures that don't fit neatly into "images" or "sequences": graphs, reinforcement learning, contrastive/self-supervised pretraining, and CLIP-style joint vision-language models.
4. Reference. NN Layers Reference is a single lookup page for every layer type introduced across the architectures above — useful once you've read the derivations and just need to check a shape or a formula.
How to actually use this section
Read Foundations in order — everything after it assumes you can derive a forward pass and backward pass by hand. After that, Sequence Models → Attention & Transformers is the single most valuable path if your goal is understanding modern LLMs, since it's written as one continuous argument (RNNs' limitations motivate LSTM/GRU, which motivate Seq2Seq+Attention, which motivates removing recurrence entirely). CNNs/Vision, and the unsupervised-learning cluster, are largely independent of that path and of each other — jump to whichever matches what you're building.
Prerequisite: the derivations here lean on calculus (chain rule, gradients) and linear algebra (matrix/vector operations) — see Mathematics for AI if any step feels unfamiliar. If you haven't seen the classical ML toolkit (linear/logistic regression, bias-variance, regularization) yet, Machine Learning covers the concepts deep learning specializes and extends.
See the roadmap for the full ordered path through this section.