Elastic Net, In Full Depth

Ridge handles correlated features gracefully but never drops any. Lasso drops features automatically but picks unstably among correlated ones. Elastic Net combines both penalties to get most of the benefit of each.

What Is Elastic Net?

Same hypothesis as Linear Regression, with both the L2 (Ridge) and L1 (Lasso) penalties added together:

$J_{\text{elastic}}(\mathbf{w}, b) = \frac{1}{n}\sum_{i=1}^n\left(\hat{y}^{(i)} - y^{(i)}\right)^2 + \lambda\left(\alpha\sum_{j=1}^d|w_j| + (1-\alpha)\sum_{j=1}^d w_j^2\right)$

Two hyperparameters now: $\lambda$ controls overall regularization strength (same role as in Ridge/Lasso), and $\alpha \in [0, 1]$ controls the mix between the two penalty types. $\alpha = 1$ recovers pure Lasso; $\alpha = 0$ recovers pure Ridge; anything in between blends them.

Why Mix the Two Penalties?

The correlated-features problem, concretely: suppose two features are near-duplicates of each other (highly correlated). Lasso's L1 penalty, left alone, tends to arbitrarily pick one and zero out the other — small changes in the data (a different train/test split, a different random seed) can flip which one survives, making the model's selected feature set unstable and hard to trust.

The L2 term fixes this specifically: Ridge's penalty doesn't have Lasso's "pick one, zero the rest" behavior — it spreads weight across correlated features roughly evenly. Adding even a little L2 penalty (small but nonzero $(1-\alpha)$) alongside the L1 term stabilizes which features get selected, while the L1 term still does the actual work of zeroing out genuinely irrelevant features.

Gradient

Simply the sum of the Ridge and Lasso gradient terms, weighted by $\alpha$:

$\frac{\partial J_{\text{elastic}}}{\partial w_j} = \underbrace{\frac{2}{n}\sum_i\left(\hat{y}^{(i)} - y^{(i)}\right)x_j^{(i)}}_{\text{OLS gradient}} + \lambda\alpha \cdot \text{sign}(w_j) + \lambda(1-\alpha)\cdot 2w_j$

In practice, Elastic Net is solved with the same coordinate-descent + soft-thresholding approach as Lasso (see Lasso — Coordinate Descent), just with a rescaled threshold that accounts for the added L2 term. Try it on the exact same synthetic correlated dataset as the Ridge and Lasso studios — drag α between the two extremes and watch the regularization path itself change shape, not just the numbers:

Choosing λ and α

Both are hyperparameters selected via cross-validation (see ML Workflow Fundamentals) — typically a 2D grid search over candidate $(\lambda, \alpha)$ pairs, since the best mix depends entirely on how correlated the actual features are and how many are truly irrelevant.

Ridge vs. Lasso vs. Elastic Net

Minimal Implementation

import numpy as np

def soft_threshold(rho, lam):
    return np.sign(rho) * max(abs(rho) - lam, 0)

def fit_elastic_net(X, y, lam=1.0, alpha=0.5, epochs=200):
    n, d = X.shape
    w = np.zeros(d)
    for _ in range(epochs):
        for j in range(d):
            residual = y - X @ w + w[j] * X[:, j]
            rho = X[:, j] @ residual
            z = (X[:, j] ** 2).sum() + n * lam * (1 - alpha)   # L2 term adds to denominator
            w[j] = soft_threshold(rho, lam * alpha * n / 2) / z if z > 0 else 0
    return w

The linear model family is complete: Linear Regression → Ridge → Lasso → Elastic Net, each one relaxing an assumption of the last. Next: Logistic Regression — the same linear machinery, adapted for classification instead of regression.