Neural Mastery
← Practice
Kaiming / He Initialization Varianceeasy

Kaiming / He Initialization Variance

easy Neural Networks by Hand 15 min

NumPy allowed · plain Python earns 10 bonus points

Calculate the exact standard deviation for Kaiming (He) normal initialization to preserve activation variance across deep layers.

Task

Implement he_initialization_std(fan_in, mode='relu'): compute std = gain / sqrt(fan_in). The gain depends on mode: for 'relu', gain = sqrt(2.0); for 'leaky_relu', gain = sqrt(2.0 / (1.0 + 0.01^2)); for any other string (e.g. 'linear'), gain = 1.0. Return std rounded to 6 decimal places.

Function Signature

he_initialization_std(fan_in: int, mode: str = 'relu') -> float

Examples

Example 1: ReLU with fan_in 512
Input: fan_in = 512 mode = 'relu'
Output: 0.0625
Example 2: Linear with fan_in 256
Input: fan_in = 256 mode = 'linear'
Output: 0.0625
Example 3: Small layer fan_in 32
Input: fan_in = 32 mode = 'relu'
Output: 0.25

Constraints

  • fan_in >= 1
  • mode in ['relu', 'leaky_relu', 'linear']
Python3Saved ✓
def he_initialization_std(fan_in: int, mode: str = "relu") -> float:
"""Compute theoretical standard deviation for He (Kaiming) weight initialization.

Args:
fan_in: Number of input units in the weight tensor.
mode: Non-linearity type ('relu', 'leaky_relu', or 'linear').

Returns:
Standard deviation float rounded to 6 decimal places.
"""
pass

Case 1: ReLU with fan_in 512
Input: fan_in = 512 mode = 'relu'
Expected: 0.0625
Case 2: Linear with fan_in 256
Input: fan_in = 256 mode = 'linear'
Expected: 0.0625
Case 3: Small layer fan_in 32
Input: fan_in = 32 mode = 'relu'
Expected: 0.25