SGD Classifier & Regressor, In Full Depth

Not a new algorithm — a framing. SGDClassifier/SGDRegressor (as named in libraries like scikit-learn) are the same linear model family covered throughout this section, unified by training them all with the same generic engine: mini-batch stochastic gradient descent.

The Unifying Idea

Every linear model on this site — Linear Regression, Ridge, Lasso, Logistic Regression, even linear SVM — has the same underlying shape: a linear function $\mathbf{w}^T\mathbf{x}+b$ , paired with a loss function, optionally paired with a regularization penalty. What actually differs between them is just which loss and which penalty get plugged in:

Model	Loss	Penalty
Linear Regression	Squared error	None
Ridge	Squared error	L2
Lasso	Squared error	L1
Logistic Regression	Log loss	L2 (default)
Linear SVM	Hinge loss	L2

An "SGD Classifier/Regressor" is exactly this table, implemented as one generic training loop — pick a loss, pick a penalty, and SGD optimizes whichever combination you chose. The exact same loop, the exact same data, three different (loss, penalty) pairs plugged in:

Loss + penalty combination

Switch configurations and watch the real boundary shift -- same engine, same data, genuinely different fitted line because the loss function's gradient is genuinely different.

Same generic SGD loop (`for each point: score = w·x+b, grad = loss'(y, score), w -= lr*(grad*x + penalty), b -= lr*grad`), same real 120-point dataset -- only the loss/penalty plugged in differs. Real fitted boundary for Linear SVM (hinge + L2): w=(1.03, 0.84), b=0.12.

This is why the gradient descent derivation from Linear Regression and the gradient derivation from Logistic Regression look structurally identical — they're literally the same training procedure with a different loss plugged in.

Why This Framing Matters in Practice

Scales to huge datasets: closed-form solutions (like the normal equation) don't scale past a modest number of features, and even batch gradient descent requires holding meaningful chunks of data in memory. SGD updates on one example or mini-batch at a time (see Optimizers — Gradient Descent Variants), so it scales to datasets far larger than memory, streaming through the data incrementally.
Online learning: because SGD naturally processes data incrementally, an SGD-framed model can keep learning from new data arriving after deployment, without retraining from scratch — a genuinely different capability from the batch/closed-form methods elsewhere in this section.
One implementation, many models: a library only needs to implement the SGD loop once, then expose loss/penalty as configuration — which is exactly why scikit-learn (and similar libraries) implement SGDClassifier/SGDRegressor as configurable wrappers rather than reimplementing near-identical training loops per model.

Practical Considerations

Feature scaling matters — like KNN, SGD-trained models converge much better when features are standardized first, since a badly-scaled feature can dominate the gradient and destabilize the learning rate choice (see Calculus & Optimization — Learning Rate Schedules).
Learning rate schedule matters more here than for the closed-form versions of these same models — see LR Scheduling for the standard shapes (constant, invscaling, adaptive) libraries expose for SGD-trained linear models.
When to reach for it over the closed-form/batch version: dataset too large for memory, need online/incremental learning, or you specifically want to swap losses/penalties without switching model classes entirely.

This completes the classical linear/margin/generative classifier family. Next: Deep Learning — where the same "linear function + loss + gradient descent" recipe gets stacked into many layers.