Neural Mastery
← Practice
Top-P (Nucleus) Filteringeasy

Top-P (Nucleus) Filtering

easy Inside an LLM 15–20 min

NumPy allowed · plain Python earns 10 bonus points

LLM step 1/4

Logits become a next-token distribution

Softmax turns a model score vector into probabilities.
Open lesson →
01prompt
02tokens
03logits
04probabilities
attention→next token
mechanism65%
layer20%
pattern10%
model5%

Softmax(logits) → probability distribution

Dynamically adapt sampling vocabulary width using Holtzman et al. Nucleus (Top-P) filtering.

Task

Implement top_p_filter(logits, p, filter_value=-float('inf')). Compute softmax probabilities. Sort token indices descending by probability (breaking ties by lower index). Accumulate probabilities until the cumulative sum is $ge p$. Retain this nucleus of tokens and mask all other tokens with filter_value. If $p le 0.0$ or $p > 1.0$, raise ValueError. At least one token (the top token) must always be retained.

Function Signature

top_p_filter(logits: list[float], p: float, filter_value: float = -float('inf')) -> list[float]

Examples

Example 1: Nucleus p=0.8 captures top mass
Input: logits = [3, 2, 0.5, -1] p = 0.8
Output: [3, 2, float('-inf'), float('-inf')]
Example 2: p=1.0 keeps all tokens
Input: logits = [2, 1, 0.5] p = 1
Output: [2, 1, 0.5]
Example 3: Invalid p <= 0 error
Input: logits = [1, 2] p = 0
Output: Raises ValueError

Constraints

  • p <= 0.0 or p > 1.0 must raise ValueError.
  • Always retain at least the single highest probability token.
  • Ties in probability must be broken by lowest token index.
  • A plain-Python solution earns 10 bonus points.
Python3Saved ✓
def top_p_filter(logits: list[float], p: float, filter_value: float = -float('inf')) -> list[float]:
"""Perform nucleus (top-p) filtering over vocabulary logits.
Args:
logits: Raw token logits.
p: Cumulative probability threshold in (0, 1].
filter_value: Value assigned to masked tokens.
Returns:
Filtered logits list with tail tokens masked.
"""
# Your implementation here
pass

Case 1: Nucleus p=0.8 captures top mass
Input: logits = [3, 2, 0.5, -1] p = 0.8
Expected: [3, 2, float('-inf'), float('-inf')]
Case 2: p=1.0 keeps all tokens
Input: logits = [2, 1, 0.5] p = 1
Expected: [2, 1, 0.5]
Case 3: Invalid p <= 0 error
Input: logits = [1, 2] p = 0
Expected: Raises ValueError