Will It Hold 50 Requests a Second?medium

Will It Hold 50 Requests a Second?

medium Specify, Verify, Debug 25 min

Standard library only · runs in your browser

Black Friday is coming and the owner expects 50 support requests a second at peak. The agent can deploy any of four setups. They all work. Pick the cheapest one that answers within 4 seconds for 95% of customers, and be able to show the arithmetic.

Task

For each option, estimate:

latency of one request = input tokens ÷ prefill speed + output tokens ÷ decode speed;
p95 latency = latency × p95_factor. The option is out if this exceeds slo_p95_seconds;
requests in flight = requests per second × latency (Little's law);
replicas = ceil(in flight ÷ (max concurrent × utilization)), at least 1. Running replicas at 100% leaves no headroom for bursts;
monthly cost = replicas × cost per hour × 730, rounded to 2 decimals.

Write plan_capacity(...). Return the cheapest option that meets the SLO:

{"option": "small-a100", "replicas": 3, "p95_seconds": 2.77, "monthly_cost": 4818.0}

with p95_seconds rounded to 2 decimals. Ties on cost go to the alphabetically first name. Return None if nothing meets the SLO.

Function Signature

plan_capacity(load, options, slo_p95_seconds, utilization=0.7, p95_factor=1.5) -> dict | None

Examples

Example 1: 50 requests a second, p95 under 4 seconds
Input: load = {'requests_per_s': 50, 'input_tokens': 1800, 'output_tokens': 250}
options = [{'name': 'small-l4', 'prefill_tokens_per_s': 12000, 'decode_tokens_per_s': 90, 'max_concurrent': 32, 'cost_per_hour': 0.8}, {'name': 'small-a100', 'prefill_tokens_per_s': 30000, 'decode_tokens_per_s': 140, 'max_concurrent': 64, 'cost_per_hour': 2.2}, {'name': 'large-h100', 'prefill_tokens_per_s': 40000, 'decode_tokens_per_s': 110, 'max_concurrent': 128, 'cost_per_hour': 4.5}, {'name': 'small-h100', 'prefill_tokens_per_s': 60000, 'decode_tokens_per_s': 200, 'max_concurrent': 80, 'cost_per_hour': 4}]
slo_p95_seconds = 4
Output: {'option': 'small-a100', 'replicas': 3, 'p95_seconds': 2.77, 'monthly_cost': 4818}
Explanation: Without headroom, small-h100 looks cheapest. Using mean latency instead of p95, small-l4 does.

Example 2: An SLO nothing can meet
Input: load = {'requests_per_s': 50, 'input_tokens': 1800, 'output_tokens': 250}
options = [{'name': 'small-l4', 'prefill_tokens_per_s': 12000, 'decode_tokens_per_s': 90, 'max_concurrent': 32, 'cost_per_hour': 0.8}, {'name': 'small-a100', 'prefill_tokens_per_s': 30000, 'decode_tokens_per_s': 140, 'max_concurrent': 64, 'cost_per_hour': 2.2}, {'name': 'large-h100', 'prefill_tokens_per_s': 40000, 'decode_tokens_per_s': 110, 'max_concurrent': 128, 'cost_per_hour': 4.5}, {'name': 'small-h100', 'prefill_tokens_per_s': 60000, 'decode_tokens_per_s': 200, 'max_concurrent': 80, 'cost_per_hour': 4}]
slo_p95_seconds = 1
Output: None

Example 3: A quiet day: 2 requests a second
Input: load = {'requests_per_s': 2, 'input_tokens': 1800, 'output_tokens': 250}
options = [{'name': 'small-l4', 'prefill_tokens_per_s': 12000, 'decode_tokens_per_s': 90, 'max_concurrent': 32, 'cost_per_hour': 0.8}, {'name': 'small-a100', 'prefill_tokens_per_s': 30000, 'decode_tokens_per_s': 140, 'max_concurrent': 64, 'cost_per_hour': 2.2}, {'name': 'large-h100', 'prefill_tokens_per_s': 40000, 'decode_tokens_per_s': 110, 'max_concurrent': 128, 'cost_per_hour': 4.5}, {'name': 'small-h100', 'prefill_tokens_per_s': 60000, 'decode_tokens_per_s': 200, 'max_concurrent': 80, 'cost_per_hour': 4}]
slo_p95_seconds = 5
Output: {'option': 'small-l4', 'replicas': 1, 'p95_seconds': 4.39, 'monthly_cost': 584}

Constraints

Use the formulas exactly as written. The tests compare numbers.
`math` is imported.
Two shortcuts give a cheaper but wrong answer on the main test. That's what it's for.

Python3Saved ✓

import json

import math

import re

from datetime import date, datetime, timedelta

def plan_capacity(load, options, slo_p95_seconds, utilization=0.7, p95_factor=1.5):

"""The cheapest option that meets the p95 latency SLO, with replicas and monthly cost."""

# Your code here

pass

Case 1: 50 requests a second, p95 under 4 seconds

Input: load = {'requests_per_s': 50, 'input_tokens': 1800, 'output_tokens': 250}
options = [{'name': 'small-l4', 'prefill_tokens_per_s': 12000, 'decode_tokens_per_s': 90, 'max_concurrent': 32, 'cost_per_hour': 0.8}, {'name': 'small-a100', 'prefill_tokens_per_s': 30000, 'decode_tokens_per_s': 140, 'max_concurrent': 64, 'cost_per_hour': 2.2}, {'name': 'large-h100', 'prefill_tokens_per_s': 40000, 'decode_tokens_per_s': 110, 'max_concurrent': 128, 'cost_per_hour': 4.5}, {'name': 'small-h100', 'prefill_tokens_per_s': 60000, 'decode_tokens_per_s': 200, 'max_concurrent': 80, 'cost_per_hour': 4}]
slo_p95_seconds = 4

Expected: {'option': 'small-a100', 'replicas': 3, 'p95_seconds': 2.77, 'monthly_cost': 4818}

Case 2: An SLO nothing can meet

Input: load = {'requests_per_s': 50, 'input_tokens': 1800, 'output_tokens': 250}
options = [{'name': 'small-l4', 'prefill_tokens_per_s': 12000, 'decode_tokens_per_s': 90, 'max_concurrent': 32, 'cost_per_hour': 0.8}, {'name': 'small-a100', 'prefill_tokens_per_s': 30000, 'decode_tokens_per_s': 140, 'max_concurrent': 64, 'cost_per_hour': 2.2}, {'name': 'large-h100', 'prefill_tokens_per_s': 40000, 'decode_tokens_per_s': 110, 'max_concurrent': 128, 'cost_per_hour': 4.5}, {'name': 'small-h100', 'prefill_tokens_per_s': 60000, 'decode_tokens_per_s': 200, 'max_concurrent': 80, 'cost_per_hour': 4}]
slo_p95_seconds = 1

Expected: None

Case 3: A quiet day: 2 requests a second

Input: load = {'requests_per_s': 2, 'input_tokens': 1800, 'output_tokens': 250}
options = [{'name': 'small-l4', 'prefill_tokens_per_s': 12000, 'decode_tokens_per_s': 90, 'max_concurrent': 32, 'cost_per_hour': 0.8}, {'name': 'small-a100', 'prefill_tokens_per_s': 30000, 'decode_tokens_per_s': 140, 'max_concurrent': 64, 'cost_per_hour': 2.2}, {'name': 'large-h100', 'prefill_tokens_per_s': 40000, 'decode_tokens_per_s': 110, 'max_concurrent': 128, 'cost_per_hour': 4.5}, {'name': 'small-h100', 'prefill_tokens_per_s': 60000, 'decode_tokens_per_s': 200, 'max_concurrent': 80, 'cost_per_hour': 4}]
slo_p95_seconds = 5

Expected: {'option': 'small-l4', 'replicas': 1, 'p95_seconds': 4.39, 'monthly_cost': 584}