Supervised Learning
The workhorse of classical ML: learn a mapping from inputs to known outputs, then generalize to new inputs.
Linear & Logistic Regression
Linear regression fits , minimizing squared error. Assumes a linear relationship between features and target, and that errors are roughly normally distributed. Its simplicity is a feature — it's interpretable (each weight tells you the effect of that feature) and has a convex loss surface (see Calculus & Optimization), so training is guaranteed to find the global optimum. See Linear Regression, In Full Depth for the full gradient derivation, the closed-form normal equation, and a from-scratch implementation.
Regularized variants add a penalty on the weights to fight overfitting when features are correlated or numerous: Ridge Regression (L2 penalty, shrinks weights smoothly), Lasso Regression (L1 penalty, shrinks some weights to exactly zero — automatic feature selection), and Elastic Net (a tunable blend of both, generally the safest default of the three).
Logistic regression applies a sigmoid to the linear output to produce a probability, and is trained with cross-entropy loss instead of squared error. Despite the name, it's a classification algorithm — one of the most-used baseline classifiers because it's fast, interpretable, and surprisingly hard to beat on many tabular problems. See Logistic Regression, In Full Depth for the full derivation, including why its gradient has the exact same shape as linear regression's despite a completely different loss function.
Tree-Based Methods
Decision trees split the feature space recursively (e.g. "is age > 30?") to separate classes/values, producing axis-aligned rectangular decision regions. Easy to interpret, but prone to overfitting if grown too deep. See Decision Trees, In Full Depth for the Gini/entropy split criteria, the full recursive algorithm, and why the boundary looks like a staircase.
Random Forests and Extra Trees: train many decision trees on bootstrapped samples of the data and random subsets of features, then average their predictions (bagging). Reduces variance dramatically compared to a single tree. See Random Forest & Extra Trees for the bias-variance math behind why averaging works, out-of-bag error, and feature importance.
AdaBoost and Gradient Boosting: build trees sequentially, where each new tree corrects the errors of the ensemble so far — trading Random Forest's variance-reduction for bias-reduction. See Boosting for the full derivation, including why gradient boosting is literally gradient descent in function space.
XGBoost, LightGBM, CatBoost: the production-grade gradient boosting implementations that dominate tabular ML and Kaggle-style competitions. Almost always the strongest off-the-shelf method for structured/tabular data. See XGBoost, LightGBM & CatBoost for what actually differs between them (regularization, leaf-wise growth, categorical handling) and when to reach for which.
Bagging vs. Boosting: bagging trains models independently in parallel to reduce variance; boosting trains models sequentially, each focused on the previous ensemble's mistakes, to reduce bias. Stacking goes further — training a "meta-model" to combine the outputs of several different base models.
Support Vector Machines
Finds the hyperplane that separates classes with the maximum margin — the widest possible gap between classes. Uses the "kernel trick" to handle non-linear boundaries by implicitly mapping data into a higher-dimensional space without ever computing that mapping directly. Historically dominant before deep learning for problems with limited data and a clear margin between classes. See Support Vector Machines (SVM & SVR), In Full Depth for the margin-maximization derivation, the soft margin, and exactly how the kernel trick avoids computing the high-dimensional mapping.
k-Nearest Neighbors
Predicts a new point's label by looking at the closest points in the training set (by some distance metric — often Euclidean or cosine, see Linear Algebra) and taking a majority vote (classification) or average (regression). No real training phase — all the cost is at prediction time, which is exactly why approximate nearest-neighbor search (covered in Databases) matters at scale. See K-Nearest Neighbors, In Full Depth for the curse of dimensionality and why KNN's boundary looks so different from every other classifier's.
Naive Bayes, LDA & QDA
Naive Bayes applies Bayes' theorem (see Probability & Statistics) with the "naive" assumption that features are conditionally independent given the class. That assumption is almost never literally true, yet Naive Bayes remains a strong, fast baseline for text classification and spam filtering. LDA and QDA are close relatives — generative classifiers that model each class as a Gaussian, differing only in whether classes share one covariance (LDA, linear boundary) or each get their own (QDA, curved boundary). See Naive Bayes, LDA & QDA, In Full Depth for the full derivation and a direct visual comparison of the two boundary shapes.
SGD Classifier & Regressor
Not a separate algorithm — a generic training engine that unifies linear/logistic regression, ridge/lasso, and linear SVM into one configurable loss+penalty framework, trained via mini-batch SGD instead of a closed form. See SGD Classifier & Regressor, In Full Depth for the full loss/penalty comparison table and why this framing matters for datasets too large to fit in memory.
Time Series Forecasting
A structurally different problem from everything above: predicting the future of a single sequence, where order carries the information and examples are never independent. See Time Series Forecasting: ARIMA, SARIMA, Prophet & TFT, In Full Depth for trend/seasonality decomposition, the classical statistical models, and how attention-based deep learning (TFT) extends the same problem.
Survival Analysis
Predicting time until an event, where some subjects haven't experienced the event yet by the time you have to analyze the data (censoring) — churn timing, equipment failure, clinical outcomes. See Survival Analysis: Kaplan-Meier & Cox Proportional Hazards, In Full Depth for why this needs its own machinery rather than plain regression, and how Cox regression gets interpretable hazard ratios without ever specifying the baseline hazard's shape.
Recommender Systems & Learning-to-Rank
Predicting preference and ordering, not a single output value: matrix factorization, embeddings, the retrieval→rank→re-rank production pipeline behind Netflix/Amazon/YouTube-style recommendations, and the pairwise/listwise objectives (RankNet, LambdaMART) that directly optimize ranking quality metrics like NDCG instead of proxying through classification. See Recommender Systems and Learning-to-Rank for the full depth — the same retrieval-then-rank pattern that also shows up in search and RAG.
Next: Unsupervised Learning — finding structure without labels.