Neural Mastery
← Practice
Shannon Entropy, Cross-Entropy, and KL Divergencemedium

Shannon Entropy, Cross-Entropy, and KL Divergence

medium Math for ML 15-20 min

NumPy allowed · plain Python earns 10 bonus points

Linear algebra step 1/4

A vector transforms

Matrix multiplication is the first computation behind every model.
Open lesson →
01vector
02matrix
03transform
04result
[2, −1] × [[1, 2], [−.5, 1]] = [0, −2]

Computed matrix × vector transformation

Evaluate fundamental information-theoretic metrics on discrete probability distributions.

Related Skills

Task

Implement shannon_and_cross_entropy(p, q): compute base-2 Shannon entropy H(p) = -sum(p_i * log2(p_i)), cross-entropy H(p, q) = -sum(p_i * log2(q_i)), and Kullback-Leibler divergence D_KL(p || q) = H(p, q) - H(p). Skip terms where p_i < 1e-12. Return {"entropy_p": float, "cross_entropy": float, "kl_div": float} rounded to 4 decimals.

Function Signature

shannon_and_cross_entropy(p: list[float], q: list[float]) -> dict[str, float]

Examples

Example 1: Identical Distributions
Input: p = [0.5, 0.5] q = [0.5, 0.5]
Output: {'entropy_p': 1, 'cross_entropy': 1, 'kl_div': 0}
Explanation: Identical distributions have KL divergence 0.
Example 2: Shifted Binary Distributions
Input: p = [0.9, 0.1] q = [0.5, 0.5]
Output: {'entropy_p': 0.469, 'cross_entropy': 1, 'kl_div': 0.531}
Explanation: Biased coin vs fair coin.

Constraints

  • p and q are valid probability vectors summing to 1.0.
  • q_i > 0 whenever p_i > 0.
Python3Saved ✓
def shannon_and_cross_entropy(p: list[float], q: list[float]) -> dict[str, float]:
"""Compute Shannon entropy H(p), cross-entropy H(p, q), and KL divergence D_KL(p || q) in base 2 bits.
Return {'entropy_p': float, 'cross_entropy': float, 'kl_div': float} rounded to 4 decimals."""
# Your implementation here
pass

Case 1: Identical Distributions
Input: p = [0.5, 0.5] q = [0.5, 0.5]
Expected: {'entropy_p': 1, 'cross_entropy': 1, 'kl_div': 0}
Case 2: Shifted Binary Distributions
Input: p = [0.9, 0.1] q = [0.5, 0.5]
Expected: {'entropy_p': 0.469, 'cross_entropy': 1, 'kl_div': 0.531}