Neural Mastery
← Practice
ReLU Backward Gradient Passeasy

ReLU Backward Gradient Pass

easy Neural Networks by Hand 15 min

NumPy allowed · plain Python earns 10 bonus points

Gradient descent step 1/4

Loss moves downhill

The update rule changes the parameters on each step.
Open lesson →
01parameters
02prediction
03loss
04gradient
θ = 4.000loss = 16.000∂L/∂θ = 8.000

θ ← θ − 0.14 · ∂L/∂θ

Backpropagate upstream gradients through a Rectified Linear Unit (ReLU) activation function.

Task

Implement relu_backward(dout, x) which computes the element-wise gradient dx given upstream gradient dout and forward input cache x. For each element, if x[i][j] > 0, the derivative is 1 so dx[i][j] = dout[i][j]; if x[i][j] <= 0, the derivative is 0 so dx[i][j] = 0.0. Round results to 4 decimal places.

Function Signature

relu_backward(dout: list[list[float]], x: list[list[float]]) -> list[list[float]]

Examples

Example 1: Positive and negative inputs
Input: dout = [[1, 2], [0.5, -1]] x = [[3, -2], [0, 1.5]]
Output: [[1, 0], [0, -1]]
Example 2: All negative inputs
Input: dout = [[2, 3]] x = [[-1, -0.5]]
Output: [[0, 0]]
Example 3: All positive inputs
Input: dout = [[1.5, 0.8]] x = [[2, 0.1]]
Output: [[1.5, 0.8]]

Constraints

  • dout and x have identical 2D dimensions
  • 1 <= N, D <= 100
  • Floats are finite
Python3Saved ✓
def relu_backward(dout: list[list[float]], x: list[list[float]]) -> list[list[float]]:
"""Compute the gradient of the loss with respect to input x through a ReLU activation.

Args:
dout: Upstream gradient matrix of shape (N, D).
x: Forward input matrix of shape (N, D).

Returns:
Gradient matrix dx where dx[i][j] = dout[i][j] if x[i][j] > 0 else 0.0, rounded to 4 decimals.
"""
pass

Case 1: Positive and negative inputs
Input: dout = [[1, 2], [0.5, -1]] x = [[3, -2], [0, 1.5]]
Expected: [[1, 0], [0, -1]]
Case 2: All negative inputs
Input: dout = [[2, 3]] x = [[-1, -0.5]]
Expected: [[0, 0]]
Case 3: All positive inputs
Input: dout = [[1.5, 0.8]] x = [[2, 0.1]]
Expected: [[1.5, 0.8]]