Neural Mastery
← Practice
Adam Optimizer Step with Bias Correctionmedium

Adam Optimizer Step with Bias Correction

medium Neural Networks by Hand 20 min

NumPy allowed · plain Python earns 10 bonus points

Gradient descent step 1/4

Loss moves downhill

The update rule changes the parameters on each step.
Open lesson →
01parameters
02prediction
03loss
04gradient
θ = 4.000loss = 16.000∂L/∂θ = 8.000

θ ← θ − 0.14 · ∂L/∂θ

Implement Kingma and Ba's adaptive moment estimation (Adam) optimizer step with bias correction.

Related Skills

Task

Implement adam_optimizer_step(...): at timestep t >= 1, compute for each parameter: m_t = beta1 * m + (1 - beta1) * g, v_t = beta2 * v + (1 - beta2) * g^2. Correct bias: m_hat = m_t / (1 - beta1^t), v_hat = v_t / (1 - beta2^t). Update parameter: param = param - lr * m_hat / (sqrt(v_hat) + eps). Return {'param': ..., 'm': ..., 'v': ...} rounded to 6 decimal places.

Function Signature

adam_optimizer_step(param: list[float], grad: list[float], m: list[float], v: list[float], t: int, lr: float = 0.001, beta1: float = 0.9, beta2: float = 0.999, eps: float = 1e-8) -> dict

Examples

Example 1: First Adam step t=1
Input: param = [1, 2] grad = [0.1, -0.2] m = [0, 0] v = [0, 0] t = 1 lr = 0.001 beta1 = 0.9 beta2 = 0.999 eps = 1e-8
Output: {'param': [0.999, 2.001], 'm': [0.01, -0.02], 'v': [0.00001, 0.00004]}
Example 2: Second Adam step t=2 with memory
Input: param = [0.999, 2.001] grad = [0.1, -0.2] m = [0.01, -0.02] v = [0.0001, 0.0004] t = 2 lr = 0.001 beta1 = 0.9 beta2 = 0.999 eps = 1e-8
Output: {'param': [0.998574, 2.001426], 'm': [0.019, -0.038], 'v': [0.00011, 0.00044]}
Example 3: Zero gradient step
Input: param = [0.5] grad = [0] m = [0.1] v = [0.05] t = 5 lr = 0.01 beta1 = 0.9 beta2 = 0.999 eps = 1e-8
Output: {'param': [0.499305], 'm': [0.09], 'v': [0.04995]}

Constraints

  • t >= 1
  • len(param) == len(grad) == len(m) == len(v) >= 1
  • 0 < beta1, beta2 < 1.0
  • lr > 0.0
Python3Saved ✓
def adam_optimizer_step(param: list[float], grad: list[float], m: list[float], v: list[float], t: int, lr: float = 0.001, beta1: float = 0.9, beta2: float = 0.999, eps: float = 1e-8) -> dict:
"""Execute one update step of the Adam optimizer with bias correction.

Args:
param: Current parameter values.
grad: Gradient vector.
m: First moment vector.
v: Second raw moment vector.
t: Current timestep index (t >= 1).
lr: Learning rate.
beta1: First moment decay factor.
beta2: Second moment decay factor.
eps: Small epsilon denominator term.

Returns:
Dictionary with 'param', 'm', 'v' lists, each rounded to 6 decimal places.
"""
pass

Case 1: First Adam step t=1
Input: param = [1, 2] grad = [0.1, -0.2] m = [0, 0] v = [0, 0] t = 1 lr = 0.001 beta1 = 0.9 beta2 = 0.999 eps = 1e-8
Expected: {'param': [0.999, 2.001], 'm': [0.01, -0.02], 'v': [0.00001, 0.00004]}
Case 2: Second Adam step t=2 with memory
Input: param = [0.999, 2.001] grad = [0.1, -0.2] m = [0.01, -0.02] v = [0.0001, 0.0004] t = 2 lr = 0.001 beta1 = 0.9 beta2 = 0.999 eps = 1e-8
Expected: {'param': [0.998574, 2.001426], 'm': [0.019, -0.038], 'v': [0.00011, 0.00044]}
Case 3: Zero gradient step
Input: param = [0.5] grad = [0] m = [0.1] v = [0.05] t = 5 lr = 0.01 beta1 = 0.9 beta2 = 0.999 eps = 1e-8
Expected: {'param': [0.499305], 'm': [0.09], 'v': [0.04995]}