Neural Mastery
← Practice
Ship It or Roll It Backhard

Ship It or Roll It Back

hard Specify, Verify, Debug 25 min

Standard library only · runs in your browser

The agent's new prompt goes to 10% of traffic. Twenty minutes in, the canary has 30 errors and production has 100. Someone says the canary looks fine, since it has fewer errors. Someone else wants to roll back. Settle it with a rule you'd use every time, not a feeling.

Task

Each side is {"requests": int, "errors": int, "p95_ms": number}.

Write canary_decision(control, canary, min_requests=500, enough_requests=2000, z_threshold=2.33, latency_ratio_limit=1.2):

  1. Canary has fewer than min_requests → {"decision": "keep_waiting", "z": None}.
  2. Compute the one-sided two-proportion z-score of the canary's error rate against the control's, using the pooled rate:
    • p = (errors_control + errors_canary) ÷ (requests_control + requests_canary)
    • z = (rate_canary − rate_control) ÷ √(p(1−p)(1/n_control + 1/n_canary))
    • z = 0 when the denominator is 0. Round z to 2 decimals.
  3. z > z_threshold → rollback.
  4. Canary p95 > latency_ratio_limit × control p95 → rollback.
  5. Canary has at least enough_requests → promote. Otherwise → keep_waiting.

Return {"decision": ..., "z": z}.

Function Signature

canary_decision(control: dict, canary: dict, ...) -> dict

Examples

Example 1: 30 errors in 2,500 against 100 in 20,000
Input: control = {'requests': 20000, 'errors': 100, 'p95_ms': 800} canary = {'requests': 2500, 'errors': 30, 'p95_ms': 820}
Output: {'decision': 'rollback', 'z': 4.35}
Explanation: Fewer errors, but a 1.2% error rate against 0.5%.
Example 2: 14 errors in 2,500
Input: control = {'requests': 20000, 'errors': 100, 'p95_ms': 800} canary = {'requests': 2500, 'errors': 14, 'p95_ms': 830}
Output: {'decision': 'promote', 'z': 0.4}
Example 3: Five minutes in: 300 requests
Input: control = {'requests': 20000, 'errors': 100, 'p95_ms': 800} canary = {'requests': 300, 'errors': 9, 'p95_ms': 790}
Output: {'decision': 'keep_waiting', 'z': None}

Constraints

  • Compare rates, never raw counts: the canary gets a tenth of the traffic.
  • `math` is imported.
  • z_threshold 2.33 is roughly a 1% false-alarm rate for a one-sided test.
Python3Saved ✓
import json
import math
import re
from datetime import date, datetime, timedelta


def canary_decision(control, canary, min_requests=500, enough_requests=2000, z_threshold=2.33,
latency_ratio_limit=1.2):
"""promote, rollback or keep_waiting, with the z-score of the error-rate difference."""
# Eyeballing it: fewer errors than production means it's fine.
if canary["errors"] > control["errors"]:
return {"decision": "rollback", "z": None}
return {"decision": "promote", "z": None}

Case 1: 30 errors in 2,500 against 100 in 20,000
Input: control = {'requests': 20000, 'errors': 100, 'p95_ms': 800} canary = {'requests': 2500, 'errors': 30, 'p95_ms': 820}
Expected: {'decision': 'rollback', 'z': 4.35}
Case 2: 14 errors in 2,500
Input: control = {'requests': 20000, 'errors': 100, 'p95_ms': 800} canary = {'requests': 2500, 'errors': 14, 'p95_ms': 830}
Expected: {'decision': 'promote', 'z': 0.4}
Case 3: Five minutes in: 300 requests
Input: control = {'requests': 20000, 'errors': 100, 'p95_ms': 800} canary = {'requests': 300, 'errors': 9, 'p95_ms': 790}
Expected: {'decision': 'keep_waiting', 'z': None}