Neural Mastery
← Practice
Global Gradient Norm Clippingmedium

Global Gradient Norm Clipping

medium Neural Networks by Hand 20 min

NumPy allowed · plain Python earns 10 bonus points

Gradient descent step 1/4

Loss moves downhill

The update rule changes the parameters on each step.
Open lesson →
01parameters
02prediction
03loss
04gradient
θ = 4.000loss = 16.000∂L/∂θ = 8.000

θ ← θ − 0.14 · ∂L/∂θ

Mitigate exploding gradients in deep neural networks and recurrent models by scaling down gradients when their L2 norm exceeds a threshold.

Task

Implement grad_clip_by_norm(grads, max_norm): compute the global L2 norm norm = sqrt(sum(g_i^2)). If norm > max_norm, scale each gradient element by max_norm / norm; otherwise leave gradients unchanged. Return {'clipped_grads': [...], 'global_norm': norm} with all numbers rounded to 6 decimal places.

Function Signature

grad_clip_by_norm(grads: list[float], max_norm: float) -> dict

Examples

Example 1: Gradient norm exceeds threshold
Input: grads = [3, 4] max_norm = 2.5
Output: {'clipped_grads': [1.5, 2], 'global_norm': 5}
Example 2: Gradient norm within threshold
Input: grads = [1, -1] max_norm = 5
Output: {'clipped_grads': [1, -1], 'global_norm': 1.414214}
Example 3: Zero gradients
Input: grads = [0, 0] max_norm = 1
Output: {'clipped_grads': [0, 0], 'global_norm': 0}

Constraints

  • len(grads) >= 1
  • max_norm > 0.0
Python3Saved ✓
def grad_clip_by_norm(grads: list[float], max_norm: float) -> dict:
"""Clip gradients by global Euclidean L2 norm to avoid exploding gradients.

Args:
grads: 1D list of parameter gradient values.
max_norm: Maximum permitted L2 norm.

Returns:
Dictionary with 'clipped_grads' and 'global_norm', rounded to 6 decimal places.
"""
pass

Case 1: Gradient norm exceeds threshold
Input: grads = [3, 4] max_norm = 2.5
Expected: {'clipped_grads': [1.5, 2], 'global_norm': 5}
Case 2: Gradient norm within threshold
Input: grads = [1, -1] max_norm = 5
Expected: {'clipped_grads': [1, -1], 'global_norm': 1.414214}
Case 3: Zero gradients
Input: grads = [0, 0] max_norm = 1
Expected: {'clipped_grads': [0, 0], 'global_norm': 0}