Linear Algebra for AI

Every number an AI model works with — a pixel, a word, a weight — is stored and manipulated as a vector or a matrix. Linear algebra is the language that makes that manipulation precise.

Intuition: Numbers, Directions, and Functions

Three ideas carry this entire page. A vector isn't just a list of numbers — it's a direction and magnitude in space, and comparing two vectors (dot product, cosine similarity) is really asking "how aligned are these two directions." A matrix isn't just a grid — it's a function that takes a vector in and produces a vector out, and everything from "what is rank" to "what is an eigenvector" is really a question about what that function does to space. And a quadratic form ($\mathbf{x}^T A \mathbf{x}$) is what curvature looks like in this language — the concept every optimization method on the next page is secretly built from.

Vectors

A vector is an ordered list of numbers, e.g. $\mathbf{v} = [2, -1, 3]$. In AI, a vector usually represents something's position in a space of features: a word's embedding, an image's pixel values flattened out, a user's preference profile.

Dot product: $\mathbf{a} \cdot \mathbf{b} = \sum_i a_i b_i$. Measures how much two vectors point in the same direction. Norm (length): $\|\mathbf{v}\|_2 = \sqrt{\sum_i v_i^2}$. Cosine similarity: $\cos(\theta) = \dfrac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\|\|\mathbf{b}\|}$. Drag both vectors below and watch all three recompute live:

If the dot product is large and positive, the vectors are aligned; if it's zero, they're orthogonal (unrelated); if negative, they point in opposite directions. Cosine similarity is the similarity metric for embeddings — it measures direction, ignoring magnitude, which is exactly what you want when comparing two text or image embeddings for semantic similarity (this is the metric vector databases use for nearest-neighbor search).

Try it yourself: implement the dot product from scratch, against real test cases.

Matrices

A matrix is a grid of numbers — think of it as a function that transforms vectors. A neural network layer is nothing more than a matrix multiplication followed by a nonlinearity: $\mathbf{y} = W\mathbf{x} + \mathbf{b}$.

Matrix multiplication: combining two matrices $A$ (shape $m \times n$) and $B$ (shape $n \times p$) produces $C$ (shape $m \times p$), where each entry $C_{ij}$ is the dot product of row $i$ of $A$ and column $j$ of $B$. Hover any output cell to see the real dot product behind it:

This is the single most executed operation on every GPU running AI workloads today. Transpose ($A^T$) flips rows and columns — shows up constantly in backpropagation (gradients flow backward through the transposed weight matrix) and in attention (Q · Kᵀ). Inverse ($A^{-1}$) is the matrix that "undoes" $A$, so $A A^{-1} = I$ — only exists for square, full-rank matrices, rarely computed directly in deep learning (too expensive/unstable) but essential to understand conceptually.

A matrix acting as a function is easiest to see geometrically — drag the four entries below and watch the whole grid warp in real time, the two bold arrows always landing exactly at the matrix's two columns:

Try it yourself: implement matrix multiplication from scratch, against real test cases.

Rank, Independence, and Span

• Linear independence: a set of vectors where none can be written as a combination of the others.
• Span: all the vectors you can reach by combining a given set of vectors.
• Rank: the number of linearly independent rows/columns in a matrix — effectively, how much "real" information the matrix carries.

Drag the second vector below toward (or away from) the first and watch span and rank change in real time:

Why this matters: LoRA (Low-Rank Adaptation), one of the most widely used fine-tuning techniques for LLMs, works precisely because weight updates during fine-tuning tend to have low rank — so you can approximate them with two small matrices instead of one huge one.

Eigenvalues and Eigenvectors

For a square matrix $A$, an eigenvector $\mathbf{v}$ is a vector whose direction $A$ doesn't change — it only gets scaled: $A\mathbf{v} = \lambda \mathbf{v}$, where $\lambda$ is the eigenvalue. Step through real power iteration — repeatedly apply $A$ and renormalize — and watch an arbitrary starting vector converge onto the real eigenvector direction:

Intuition: eigenvectors are the "natural axes" of a transformation — the directions where the matrix acts like simple scaling instead of full rotation/shearing. This is the basis of PCA (Principal Component Analysis): the eigenvectors of a dataset's covariance matrix are the directions of maximum variance in the data. Power iteration, exactly as shown above, is also how real large-scale eigensolvers find a dominant eigenvector without ever forming a full eigendecomposition — the same core idea behind PageRank.

Singular Value Decomposition (SVD)

Any matrix $A$ (even non-square) can be decomposed as $A = U \Sigma V^T$, where $U$ and $V$ are orthogonal matrices and $\Sigma$ is diagonal with non-negative values (the singular values), sorted largest to smallest. A real, hand-verifiable decomposition — genuinely orthonormal $U$/$V$, three rank-1 terms with decreasing singular values — truncated live:

If you understand SVD, you understand why "compressing" a matrix by keeping only the top-$k$ singular values loses the least information possible for that compression level (the Eckart-Young theorem, made concrete above) — this is exactly the intuition behind low-rank approximations used in efficient LLM fine-tuning (LoRA), recommender systems (matrix factorization: decomposing a user-item ratings matrix), and model compression.

Quadratic Forms and Positive (Semi-)Definite Matrices

A quadratic form is the scalar $\mathbf{x}^T A \mathbf{x}$ — a matrix $A$ "sandwiched" between a vector and its transpose. It shows up everywhere curvature matters: it's the general shape of a loss surface near a minimum (see Jacobians & Hessians), the Mahalanobis distance in statistics, and the energy term in many kernel methods. Real contour shading plus the real gradient at five sample points, for three qualitatively different matrices:

A symmetric matrix $A$ is positive semi-definite (PSD) if $\mathbf{x}^T A \mathbf{x} \geq 0$ for every vector $\mathbf{x}$, and positive definite (PD) if that inequality is strict for every nonzero $\mathbf{x}$ — ruling out flat directions, exactly the difference visible between the first two presets above. Covariance matrices are always PSD (the variance of any linear combination of variables can't be negative). This property guarantees that optimization problems built on them (many kernel methods, second-order optimization) behave predictably — no "negative curvature" surprises. A quick test: a symmetric matrix is PD if and only if all its eigenvalues are strictly positive — this is exactly the condition Newton's method checks on the Hessian to confirm it's actually at a minimum, not a saddle point (the third preset above).

Matrix Calculus: Gradients w.r.t. Vectors and Matrices

Backpropagation is, mechanically, repeated application of a handful of matrix-calculus identities:

• $\nabla_{\mathbf{x}} (A\mathbf{x}) = A^T$ — the gradient of a linear map w.r.t. its input is just the matrix's transpose. This is why backprop through a linear layer multiplies by the transposed weight matrix: the forward pass computes $A\mathbf{x}$, the backward pass needs $\nabla_{\mathbf{x}}$, which the identity above hands you directly.
• $\nabla_{\mathbf{x}} (\mathbf{x}^T A \mathbf{x}) = (A + A^T)\mathbf{x}$, which simplifies to $2A\mathbf{x}$ when $A$ is symmetric — the real gradient computed and plotted in the quadratic-form diagram above. This single identity is the derivation of the OLS normal equations in Linear Regression: differentiate the squared-error loss (a quadratic form in the weights), set it to zero, solve.
• $\nabla_W (\mathbf{w}^T \mathbf{x}) = \mathbf{x}$ and, for a full weight matrix, $\nabla_W (W\mathbf{x} + \mathbf{b})$ w.r.t. each entry of $W$ is an outer product $\mathbf{\delta}\, \mathbf{x}^T$ — the exact computation every deep learning framework's autograd performs at every Linear/Dense layer during the backward pass.

You rarely derive these by hand in practice (autograd does it), but recognizing them is what makes a backprop derivation on a whiteboard readable instead of a wall of chain-rule symbols.

Code: Power Iteration, For Real

The exact algorithm the eigenvector diagram above steps through:

import numpy as np

def power_iteration(A: np.ndarray, num_steps: int = 20) -> tuple[np.ndarray, float]:
    v = np.random.randn(A.shape[0])
    v = v / np.linalg.norm(v)
    for _ in range(num_steps):
        v = A @ v
        v = v / np.linalg.norm(v)          # renormalize every step
    eigenvalue = v @ A @ v                  # Rayleigh quotient at convergence
    return v, eigenvalue                    # v converges to the dominant eigenvector

Where this shows up in the rest of the curriculum

Next: Calculus & Optimization — how models actually learn by following gradients through this linear-algebra machinery.