Numerical Gradient via Finite Differences

Difficulty: Medium · Concept: Calculus & Optimization — Derivatives and Gradients

Every analytic gradient on this site — linear regression's $\partial J/\partial w$, backpropagation's chain rule — has an entirely different way to compute (approximately) the same number, with no calculus required: just perturb each input slightly and measure how the output changes. The central difference approximation:

$f'(x) \approx \frac{f(x+h) - f(x-h)}{2h}$

for a small $h$ (commonly $10^{-5}$). This is slower and only approximate — but it's exactly how you'd sanity-check that a hand-derived analytic gradient (like the one in One Gradient Descent Step) is actually correct, a real technique called gradient checking.

Your task: implement numerical_gradient(f, x, h=1e-5), where f takes a list of numbers and returns a single number, and x is the point (a list) to evaluate the gradient at. Return a list — one partial derivative per dimension of x.

Next: Cosine Similarity From Scratch