Perplexity from Token Log-Probabilitieshard

Perplexity from Token Log-Probabilities

hard Inside an LLM 15–20 min

NumPy allowed · plain Python earns 10 bonus points

Tokens attend to tokens

Q · Kᵀ / √d → softmax → weighted values, computed in the page.

Open lesson →

01tokens

02Q K V

03scores

04attention

modelslearnrelationships

models32%

learn38%

relationships30%

Q · Kᵀ / √d scores normalized with softmax

Implement perplexity computation — the standard metric for evaluating language model quality, measuring how well the model predicts held-out text.

Task

Implement compute_perplexity(log_probs) that computes exp(-mean(log_probs)) from per-token natural log-probabilities.

Function Signature

compute_perplexity(log_probs: list[float]) -> float

Examples

Example 1: Uniform probs over 10-way (PPL=10)
Input: log_probs = [-2.302585092994046, -2.302585092994046, -2.302585092994046, -2.302585092994046, -2.302585092994046]
Output: 10.000000000000002

Example 2: Perfect predictions (PPL=1)
Input: log_probs = [0, 0, 0, 0, 0]
Output: 1

Constraints

Use natural logarithm (math.log base e).
Perplexity = exp(-sum(log_probs) / len(log_probs)).
Raise ValueError on empty input.

Python3Saved ✓

def compute_perplexity(log_probs: list[float]) -> float:

"""Compute perplexity from per-token log-probabilities (natural log).

Perplexity = exp(-mean(log_probs))

A lower perplexity means the model is less "surprised" by the text.

Args:

log_probs: List of per-token log-probabilities (natural log, non-positive).

Returns:

float: Perplexity of the sequence.

Raises:

ValueError: If log_probs is empty.

"""

import math

# Your implementation here

pass

Case 1: Uniform probs over 10-way (PPL=10)

Input: log_probs = [-2.302585092994046, -2.302585092994046, -2.302585092994046, -2.302585092994046, -2.302585092994046]

Expected: 10.000000000000002

Case 2: Perfect predictions (PPL=1)

Input: log_probs = [0, 0, 0, 0, 0]

Expected: 1