Neural Mastery
← Practice
AdamW Optimizer Step From Scratchmedium

AdamW Optimizer Step From Scratch

mediumDeep Learning⏱ 15–20 min
⚡ Libraries allowed · Pure Python earns +10 bonus XP
🎯 Mission
Implement AdamW adaptive learning rate optimization with decoupled weight decay.

Task

Implement `adamw_step` computing first/second moments, bias corrections, and decoupled weight decay updates.

Function Signature

adamw_step(param: list, grad: list, m: list, v: list, t: int) -> tuple

Examples

Example 1: Step 1 Update
Input: {"param":[1],"grad":[0.1],"m":[0],"v":[0],"t":1,"lr":0.1,"weight_decay":0.01}
Output: [[0.899,0.01,0.0001]]

Constraints

  • Must apply bias correction m_hat = m / (1 - beta1^t) and v_hat = v / (1 - beta2^t).
  • Weight decay must be decoupled: param = param - lr * (m_hat / (sqrt(v_hat) + eps) + weight_decay * param).
Python3Saved ✓
def adamw_step(param, grad, m, v, t, lr=0.001, beta1=0.9, beta2=0.999, eps=1e-8, weight_decay=0.01):
"""
Execute one AdamW update step.
Return tuple (next_param, next_m, next_v) as lists.
"""
# Your implementation here
pass

Case 1: Step 1 Update
Input: {"param":[1],"grad":[0.1],"m":[0],"v":[0],"t":1,"lr":0.1,"weight_decay":0.01}
Expected: [[0.899,0.01,0.0001]]