Mixture-of-Experts Top-K Routinghard

Mixture-of-Experts Top-K Routing

hard Inside an LLM 15 min

NumPy allowed · plain Python earns 10 bonus points

Tokens attend to tokens

Q · Kᵀ / √d → softmax → weighted values, computed in the page.

Open lesson →

01tokens

02Q K V

03scores

04attention

modelslearnrelationships

models32%

learn38%

relationships30%

Q · Kᵀ / √d scores normalized with softmax

Implement Mixture-of-Experts top-k routing, the real gating mechanism deciding which experts process a given token and how much weight each contributes.

Task

Implement moe_topk_routing(router_logits, top_k). Apply softmax to router_logits to get expert probabilities. Select the top_k experts by probability. RENORMALIZE just those top_k probabilities to sum to 1 (so the token's output is a proper weighted combination of only the selected experts). Return a list of {"expert_idx": int, "weight": float}, sorted by expert_idx ascending.

Function Signature

moe_topk_routing(router_logits: list[float], top_k: int) -> list[dict]

Examples

Example 1: top-2 of 4 experts
Input: router_logits = [1, 3, 0.5, 2]
top_k = 2
Output: [{'expert_idx': 1, 'weight': 0.7310585786300049}, {'expert_idx': 3, 'weight': 0.26894142136999516}]

Example 2: top-1 routing
Input: router_logits = [1, 5, 0.5]
top_k = 1
Output: [{'expert_idx': 1, 'weight': 1}]

Example 3: top-k equals all experts
Input: router_logits = [1, 2]
top_k = 2
Output: [{'expert_idx': 0, 'weight': 0.2689414213699951}, {'expert_idx': 1, 'weight': 0.7310585786300049}]

Constraints

top_k <= len(router_logits)

Python3Saved ✓

import math

def moe_topk_routing(router_logits, top_k):

# Your implementation here

pass

Case 1: top-2 of 4 experts

Input: router_logits = [1, 3, 0.5, 2]
top_k = 2

Expected: [{'expert_idx': 1, 'weight': 0.7310585786300049}, {'expert_idx': 3, 'weight': 0.26894142136999516}]

Case 2: top-1 routing

Input: router_logits = [1, 5, 0.5]
top_k = 1

Expected: [{'expert_idx': 1, 'weight': 1}]

Case 3: top-k equals all experts

Input: router_logits = [1, 2]
top_k = 2

Expected: [{'expert_idx': 0, 'weight': 0.2689414213699951}, {'expert_idx': 1, 'weight': 0.7310585786300049}]