Neural Mastery
← Practice
Linear Regression via the Normal Equationmedium

Linear Regression via the Normal Equation

medium Classical ML from Scratch 15–20 min

NumPy allowed · plain Python earns 10 bonus points

Gradient descent step 1/4

Loss moves downhill

The update rule changes the parameters on each step.
Open lesson →
01parameters
02prediction
03loss
04gradient
θ = 4.000loss = 16.000∂L/∂θ = 8.000

θ ← θ − 0.14 · ∂L/∂θ

Implement the closed-form (normal equation) solution to ordinary least squares -- the exact solution gradient descent only approximates iteratively, and the textbook baseline every regularized variant (Ridge, Lasso) is a variation of.

Task

Implement linear_regression_normal_equation(X, y): prepend a column of ones to X (for the intercept), then return w = (X_aug^T X_aug)^-1 X_aug^T y as a plain Python list, ordered [intercept, w1, w2, ...].

Function Signature

linear_regression_normal_equation(X: list[list[float]], y: list[float]) -> list[float]

Examples

Example 1: Exact 1D Fit (y = 2x + 1)
Input: X = [[1], [2], [3], [4]] y = [3, 5, 7, 9]
Output: [1, 2]
Explanation: Noise-free data recovers the exact generating weights
Example 2: Constant Target
Input: X = [[0], [1], [2]] y = [1, 1, 1]
Output: [1, 0]
Explanation: A flat target fits intercept = mean(y), slope = 0

Constraints

  • NumPy is allowed and recommended (np.linalg.inv or np.linalg.solve) -- the test matrices are small, but a pure-Python Gauss-Jordan inversion also works and earns the bonus.
  • The returned list's first element is the intercept (bias term); the rest are the per-feature weights in the same order as X's columns.
  • Assume X^T X is invertible (X has more rows than columns and no perfectly collinear features).
Python3Saved ✓
import numpy as np

def linear_regression_normal_equation(X: list[list[float]], y: list[float]) -> list[float]:
"""X: list of feature rows (no bias column). y: list of targets.
Return the fitted weight vector [intercept, w1, w2, ...] solving
ordinary least squares via the closed-form normal equation
w = (X^T X)^-1 X^T y, after prepending a column of ones to X."""
# Your implementation here
pass

Case 1: Exact 1D Fit (y = 2x + 1)
Input: X = [[1], [2], [3], [4]] y = [3, 5, 7, 9]
Expected: [1, 2]
Case 2: Constant Target
Input: X = [[0], [1], [2]] y = [1, 1, 1]
Expected: [1, 0]