LSTM Cell Forward Passhard

LSTM Cell Forward Pass

hard Neural Networks by Hand 25 min

NumPy allowed · plain Python earns 10 bonus points

Regulate sequence memory and overcome vanishing gradients using gated LSTM memory cells.

Related Skills

Task

Implement lstm_cell_forward(...): compute combined gate pre-activations raw = x @ W_x + h_prev @ W_h + b of length 4 * H. Split into input gate i = sigmoid(raw[0:H]), forget gate f = sigmoid(raw[H:2H]), candidate cell g = tanh(raw[2H:3H]), and output gate o = sigmoid(raw[3H:4H]). Update cell state c_next = f * c_prev + i * g, and hidden state h_next = o * tanh(c_next). Return {'h_next': h_next, 'c_next': c_next} with elements rounded to 4 decimal places.

Function Signature

lstm_cell_forward(x: list[list[float]], h_prev: list[list[float]], c_prev: list[list[float]], W_x: list[list[float]], W_h: list[list[float]], b: list[float]) -> dict

Examples

Example 1: Single sample 1D H=1
Input: x = [[1]]
h_prev = [[0]]
c_prev = [[0]]
W_x = [[0.5, 0.5, 0.5, 0.5]]
W_h = [[0, 0, 0, 0]]
b = [0, 0, 0, 0]
Output: {'h_next': [[0.1743]], 'c_next': [[0.2876]]}

Example 2: Retention through forget gate
Input: x = [[0]]
h_prev = [[0.5]]
c_prev = [[1]]
W_x = [[0, 0, 0, 0]]
W_h = [[0, 0, 0, 0]]
b = [-10, 10, 0, 10]
Output: {'h_next': [[0.7615]], 'c_next': [[1]]}

Example 3: H=2 dimensional state
Input: x = [[1, 0]]
h_prev = [[0.1, -0.1]]
c_prev = [[0.5, -0.5]]
W_x = [[0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1],
       [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2]]
W_h = [[0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05],
       [-0.05, -0.05, -0.05, -0.05, -0.05, -0.05, -0.05, -0.05]]
b = [0, 0, 0, 0, 0, 0, 0, 0]
Output: {'h_next': [[0.164, -0.1071]], 'c_next': [[0.3215, -0.2059]]}

Constraints

1 <= N <= 32
1 <= D_in, H <= 32
W_x shape is (D_in, 4*H)
W_h shape is (H, 4*H)
len(b) == 4*H

Python3Saved ✓

def lstm_cell_forward(x: list[list[float]], h_prev: list[list[float]], c_prev: list[list[float]], W_x: list[list[float]], W_h: list[list[float]], b: list[float]) -> dict:

"""Execute single timestep forward pass of a Long Short-Term Memory (LSTM) cell.

Args:

x: Input batch of shape (N, D_in).

h_prev: Previous hidden state of shape (N, H).

c_prev: Previous cell state of shape (N, H).

W_x: Input weight matrix of shape (D_in, 4 * H).

W_h: Hidden weight matrix of shape (H, 4 * H).

b: Bias vector of length 4 * H (order: i, f, g, o).

Returns:

Dictionary with 'h_next' and 'c_next', both of shape (N, H) rounded to 4 decimals.

"""

pass

Case 1: Single sample 1D H=1

Input: x = [[1]]
h_prev = [[0]]
c_prev = [[0]]
W_x = [[0.5, 0.5, 0.5, 0.5]]
W_h = [[0, 0, 0, 0]]
b = [0, 0, 0, 0]

Expected: {'h_next': [[0.1743]], 'c_next': [[0.2876]]}

Case 2: Retention through forget gate

Input: x = [[0]]
h_prev = [[0.5]]
c_prev = [[1]]
W_x = [[0, 0, 0, 0]]
W_h = [[0, 0, 0, 0]]
b = [-10, 10, 0, 10]

Expected: {'h_next': [[0.7615]], 'c_next': [[1]]}

Case 3: H=2 dimensional state

Input: x = [[1, 0]]
h_prev = [[0.1, -0.1]]
c_prev = [[0.5, -0.5]]
W_x = [[0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1],
       [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2]]
W_h = [[0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05],
       [-0.05, -0.05, -0.05, -0.05, -0.05, -0.05, -0.05, -0.05]]
b = [0, 0, 0, 0, 0, 0, 0, 0]

Expected: {'h_next': [[0.164, -0.1071]], 'c_next': [[0.3215, -0.2059]]}