Deep Learning & LLM Interpretability

Classical Interpretability treats a model as a black box (or, at most, a tree structure) and explains its outputs. This page goes a level deeper: tools built specifically for a differentiable, gradient-accessible neural network, from a single gradient-based heatmap up through the current frontier of mechanistic interpretability research.

Intuition: A Rigor Ladder, Not a Toolbox of Equals

Every tool below answers "what's going on inside this network," but they don't all answer it with the same strength of evidence. A saliency map is cheap and fast, but purely correlational — it tells you what correlates with the output, not what causes it. A verified circuit is expensive to find, but it's an actual causal, mechanistic account. Keep this ladder in mind as you go — it's also how the page ends:

Saliency Maps and Gradient-Based Attribution

The most direct way to ask "which input pixels/tokens mattered" for a neural network specifically: compute the gradient of the output with respect to the input (see Jacobians) — large gradient magnitude at a given input position means small changes there would meaningfully change the output. For images, this produces a saliency map, a heatmap over the input highlighting which pixels most influenced the prediction. Simple and fast, but with a specific, well-documented failure mode: gradient saturation.

Gradient Saturation, Demonstrated

If a model's score is already near 0 or 1 for a given input (deep into a sigmoid's flat region), the local gradient there is tiny — almost zero — even for input pixels that were genuinely decisive in getting the score that high in the first place. Watch it happen with a real toy scorer, $F(x) = \text{sigmoid}(k \cdot w{\cdot}x)$, evaluated at an input where $F(x) \approx 1$:

Integrated Gradients fixes this by not evaluating the gradient only at the saturated endpoint — it accumulates the gradient along the entire straight-line path from a baseline (typically all-zeros) to the actual input, most of which isn't saturated:

$\text{IG}_i(x) = (x_i - x_i^{\text{baseline}}) \int_0^1 \frac{\partial F(x^{\text{baseline}} + \alpha(x - x^{\text{baseline}}))}{\partial x_i}\, d\alpha$

The diagram above computes that integral for real, via a 40-step Riemann sum — which is exactly how it's done in practice, since the integral rarely has a closed form for a real network. Grad-CAM is a different fix for a different failure mode: rather than raw input-pixel gradients (noisy at pixel granularity), it uses gradients flowing into a CNN's late convolutional layers, producing coarser but often more semantically meaningful heatmaps.

Activation Visualization

A complementary strategy to saliency: instead of asking "which pixels of this input mattered," ask "what input would make this neuron fire maximally" — directly inspecting what a neuron or channel represents, rather than working backward from one specific example. In practice this means running real gradient ascent on the input itself, climbing the neuron's actual activation function step by step:

This is literally how classic feature-visualization results were produced (the widely-seen "what does this neuron want to see" synthesized images): not by inspecting weights directly, but by optimizing an input against a real, differentiable activation function, exactly like the toy version above, just with a full trained network's activation surface instead of two Gaussian bumps. Classic results from this line of work showed early CNN layers learn recognizable edge and color detectors — directly consistent with the classical Sobel-filter convolution intuition, the network learns something similar to what was previously hand-designed — while deeper layers learn increasingly abstract, complex feature detectors (textures, then object parts, then whole objects), direct empirical evidence for the "hierarchy of increasingly abstract features" intuition CNNs describes conceptually.

Attention Analysis

Specific to Transformers (see Attention & Transformers): visualizing attention weights directly — since they're already a normalized (softmax) distribution, they're directly viewable as a heatmap with no gradients or extra machinery needed. Two of the most-cited qualitative patterns found this way are shown below — a head that always looks one token back, and an induction head that pattern-matches against earlier context to predict a repeat:

A well-known caution applies: high attention weight on a token doesn't necessarily mean that token was causally important to the output — attention is one signal among several the model's computation combines, and treating attention weights as a complete, sufficient explanation for a model's behavior has been repeatedly shown to be an overclaim in the interpretability literature. Use it as a hypothesis-generating tool, not a final answer — which is exactly why the induction pattern above only became a confirmed circuit (below), not just a suggestive picture, once it was verified causally.

Probing Classifiers

Train a small, separate classifier to predict some property of interest (part-of-speech, syntactic structure, a factual attribute) from a model's internal activations at a specific layer — if the probe can accurately predict the property, that's evidence the model's representation at that layer linearly encodes that information, whether or not the model's own output ever makes it explicit. Three real logistic-regression probes below, fit by real gradient descent to synthetic activations that only differ in how separable the underlying property is — a direct stand-in for "how cleanly a real layer encodes it":

Probing revealed, for instance, that BERT-style encoders build up increasingly abstract linguistic structure across their layers — roughly: earlier layers encode more surface-level features like part-of-speech, later layers encode more abstract syntactic/semantic structure — evidence for what a model represents internally, distinct from evidence about how it uses that representation to produce output (which circuits, below, address more directly). A caveat mirroring attention's: a probe finding a property decodable doesn't prove the model actually reads it out and uses it downstream — a powerful enough probe can sometimes decode information the model itself never acts on.

Sparse Autoencoders and Superposition

A more recent technique specifically for addressing superposition — the empirical observation that a neural network's individual neurons often don't correspond to single, clean, human-interpretable concepts; instead, many neurons each participate in representing several different concepts simultaneously, and any one concept is spread across many neurons. This isn't a training accident to be fixed — it's close to mathematically forced whenever a model has more genuinely useful features to represent than it has neurons to represent them with:

A sparse autoencoder is trained on a model's internal activations to reconstruct them through a much wider, sparsely-activated hidden layer (see Autoencoders) — the sparsity constraint pressures it to discover a larger set of more individually-interpretable "features" than the original neuron basis provides, effectively un-mixing the superposed concepts into cleaner, separately-interpretable units, the same geometric move the diagram above shows: give the same features more room and their interference collapses. This has become one of the most actively used tools in current mechanistic interpretability research specifically because it addresses superposition head-on rather than working around it.

Circuits

A circuit is a specific, identified subgraph of a network's components (attention heads, MLP neurons, or sparse-autoencoder features) that together implement a specific, identifiable piece of the model's computation. The induction pattern from the attention-analysis diagram above is the canonical example — traced end-to-end into an actual verified circuit:

Finding and verifying a circuit means being able to point to the actual mechanism a model uses to perform a piece of its behavior, not just correlational evidence (like a probe or a saliency map) that some information is present somewhere. That's the qualitative jump circuits make over everything earlier on this page: from "this correlates with the behavior" to "this is how the behavior is computed, and here's the causal chain proving it."

Mechanistic Interpretability

The overarching research program the last several tools serve: reverse-engineering a neural network's learned computation into human-understandable algorithms — treating a trained model less like a black-box statistical function and more like compiled code to be decompiled, with sparse autoencoders providing cleaner interpretable units and circuit analysis providing the causal story of how those units combine to produce behavior. It's the most rigorous, and most labor-intensive, tier of the ladder at the top of this page — genuinely explaining how a specific capability works mechanistically, rather than what correlates with it — and directly the tool AI Safety — Scalable Oversight points to for inspecting a model's actual computation rather than trusting its self-reported reasoning.

Interpretability section complete. Next: Reinforcement Learning — a separate track with its own theoretical foundation underlying RLHF, agents, and game-playing systems.