Dense Layer Backward Passmedium

Dense Layer Backward Pass

medium Neural Networks by Hand 20 min

NumPy allowed · plain Python earns 10 bonus points

Derive and compute the analytical backward gradients for a fully-connected affine layer.

Related Skills

Task

Implement dense_backward(dout, x, w): given upstream gradient dout (N, D_out), input cache x (N, D_in), and weight matrix w (D_in, D_out), compute dx = dout @ w.T, dw = x.T @ dout, and db = sum(dout, axis=0). Return a dictionary {'dx': dx, 'dw': dw, 'db': db} with all numbers rounded to 4 decimal places.

Function Signature

dense_backward(dout: list[list[float]], x: list[list[float]], w: list[list[float]]) -> dict

Examples

Example 1: Batch of 2, Din 2, Dout 2
Input: dout = [[1, 0.5], [-0.5, 2]]
x = [[2, 1], [0, -1]]
w = [[0.5, -1], [1, 0]]
Output: {'dx': [[0, 1], [-2.25, -0.5]], 'dw': [[2, 1], [1.5, -1.5]], 'db': [0.5, 2.5]}

Example 2: Single sample projection
Input: dout = [[1, -1]]
x = [[3, 4]]
w = [[1, 0.5], [2, -0.5]]
Output: {'dx': [[0.5, 2.5]], 'dw': [[3, -3], [4, -4]], 'db': [1, -1]}

Example 3: Zero upstream gradient
Input: dout = [[0, 0]]
x = [[1, 2]]
w = [[1, 1], [1, 1]]
Output: {'dx': [[0, 0]], 'dw': [[0, 0], [0, 0]], 'db': [0, 0]}

Constraints

1 <= N <= 100
1 <= D_in, D_out <= 64
Consistent input dimensions

Python3Saved ✓

def dense_backward(dout: list[list[float]], x: list[list[float]], w: list[list[float]]) -> dict:

"""Compute gradients of loss with respect to inputs, weights, and biases of a dense layer.

Args:

dout: Upstream gradient matrix of shape (N, D_out).

x: Forward input matrix of shape (N, D_in).

w: Forward weight matrix of shape (D_in, D_out).

Returns:

Dictionary with 'dx': (N, D_in), 'dw': (D_in, D_out), 'db': (D_out,), all rounded to 4 decimals.

"""

pass

Case 1: Batch of 2, Din 2, Dout 2

Input: dout = [[1, 0.5], [-0.5, 2]]
x = [[2, 1], [0, -1]]
w = [[0.5, -1], [1, 0]]

Expected: {'dx': [[0, 1], [-2.25, -0.5]], 'dw': [[2, 1], [1.5, -1.5]], 'db': [0.5, 2.5]}

Case 2: Single sample projection

Input: dout = [[1, -1]]
x = [[3, 4]]
w = [[1, 0.5], [2, -0.5]]

Expected: {'dx': [[0.5, 2.5]], 'dw': [[3, -3], [4, -4]], 'db': [1, -1]}

Case 3: Zero upstream gradient

Input: dout = [[0, 0]]
x = [[1, 2]]
w = [[1, 1], [1, 1]]

Expected: {'dx': [[0, 0]], 'dw': [[0, 0], [0, 0]], 'db': [0, 0]}