Neural Mastery
← Practice
Practice: RMS Normalization From Scratcheasy

Practice: RMS Normalization From Scratch

easy Inside an LLM 10–15 min

NumPy allowed · plain Python earns 10 bonus points

New to this topic? Deep Learning Overview
Study this concept first →

Implement Root Mean Square Normalization (RMSNorm), the computationally efficient LayerNorm alternative used in modern LLMs.

Task

Implement rms_norm(x, gamma, eps=1e-6). For each row x_i, compute rms = sqrt(mean(x_i^2) + eps), and return (x / rms) * gamma.

Function Signature

rms_norm(x: list[list[float]], gamma: list[float], eps: float = 1e-6) -> list[list[float]]

Examples

Example 1: Identity scaling gamma = 1s
Input: x = [[1, 1], [2, 2]] gamma = [1, 1]
Output: [[1, 1], [1, 1]]
Example 2: Non-uniform features and scaling
Input: x = [[3, 4]] gamma = [0.5, 2]
Output: [[0.424264, 2.262742]]

Constraints

  • x has shape (N, D)
  • gamma has shape (D,)
  • eps > 0 (defaults to 1e-6)
  • Returns list of lists or numpy array
Python3Saved ✓
import numpy as np

def rms_norm(x: list[list[float]], gamma: list[float], eps: float = 1e-6) -> list[list[float]]:
"""Compute RMS Normalization across the feature dimension.
x: 2D array or list of lists (batch_size, d)
gamma: 1D scale parameters (d,)
Return normalized output as list of lists.
"""
# Your code here
pass

Case 1: Identity scaling gamma = 1s
Input: x = [[1, 1], [2, 2]] gamma = [1, 1]
Expected: [[1, 1], [1, 1]]
Case 2: Non-uniform features and scaling
Input: x = [[3, 4]] gamma = [0.5, 2]
Expected: [[0.424264, 2.262742]]