Capstone: The Incident Reviewhard

Capstone: The Incident Review

hard Specify, Verify, Debug 30 min

Standard library only · runs in your browser

It's the morning after the checkout incident. The review starts at ten, and someone has already written "cause: the 14:00 checkout deploy" on the whiteboard, because that's the most recent change. Bring the facts, computed from the evidence rather than from the timeline everyone remembers.

Related Skills

Observability Deployment strategies

Task

You have the deploy history and the incident's logs. first_failure from stage 3 is provided.

Write incident_review(deploys, lines), returning:

{"root_service": "payments-db", "first_failure_at": "2026-10-20T14:03:01Z",
 "trigger_deploy": "d-43", "rollback_to": "d-41", "affected_requests": 6}

root_service and first_failure_at come from the first failure.
trigger_deploy: the most recent deploy of the root service at or before the first failure. Deploys of other services don't count. The list isn't sorted.
rollback_to: the root service's deploy before the trigger, or None if there isn't one.
affected_requests: distinct request ids on ERROR lines that have a valid timestamp.
No errors at all → None.

Function Signature

incident_review(deploys: list[dict], lines: list[str]) -> dict | None

Examples

Example 1: Yesterday's checkout incident
Input: deploys = [{'id': 'd-41', 'service': 'payments-db', 'at': '2026-10-19T09:00:00Z'}, {'id': 'd-43', 'service': 'payments-db', 'at': '2026-10-20T13:58:00Z'}, {'id': 'd-42', 'service': 'checkout', 'at': '2026-10-20T14:00:00Z'}, {'id': 'd-44', 'service': 'payments-db', 'at': '2026-10-20T14:30:00Z'}]
lines = ['2026-10-20T14:03:12Z level=ERROR svc=checkout req=r-1043 msg=payment_failed status=500', '2026-10-20T14:03:12Z level=ERROR svc=checkout req=r-1044 msg=payment_failed status=500', '2026-10-20T14:02:58Z level=WARN svc=payments-db req=- msg=slow_query ms=2400', '2026-10-20T14:03:05Z level=ERROR svc=payments req=r-1041 msg=db_timeout after_ms=3000', '2026-10-20T14:03:01Z level=ERROR svc=payments-db req=r-1040 msg=connection_pool_exhausted', '2026-10-20T14:03:13Z level=ERROR svc=checkout req=r-1045 msg=payment_failed status=500', '2026-10-20T14:03:06Z level=INFO svc=payments req=r-1046 msg=retrying attempt=2', 'payments-db: pool size 20, in use 20', '2026-10-20T14:03:14Z level=ERROR svc=checkout req=r-1046 msg=payment_failed status=500']
Output: {'root_service': 'payments-db', 'first_failure_at': '2026-10-20T14:03:01Z', 'trigger_deploy': 'd-43', 'rollback_to': 'd-41', 'affected_requests': 6}
Explanation: The 14:00 checkout deploy is the most recent change, and it's not the cause.

Example 2: No errors
Input: deploys = [{'id': 'd-41', 'service': 'payments-db', 'at': '2026-10-19T09:00:00Z'}, {'id': 'd-43', 'service': 'payments-db', 'at': '2026-10-20T13:58:00Z'}, {'id': 'd-42', 'service': 'checkout', 'at': '2026-10-20T14:00:00Z'}, {'id': 'd-44', 'service': 'payments-db', 'at': '2026-10-20T14:30:00Z'}]
lines = ['2026-10-20T09:00:00Z level=INFO svc=api req=r-1 msg=ok']
Output: None

Example 3: The service's first deploy caused it
Input: deploys = [{'id': 'd-1', 'service': 'search', 'at': '2026-10-20T08:00:00Z'}]
lines = ['2026-10-20T08:05:00Z level=ERROR svc=search req=r-9 msg=index_missing']
Output: {'root_service': 'search', 'first_failure_at': '2026-10-20T08:05:00Z', 'trigger_deploy': 'd-1', 'rollback_to': None, 'affected_requests': 1}

Constraints

Deploy times are ISO strings ending in Z. Compare them as times.
A deploy after the first failure (the hotfix) is never the trigger.
Standard library only: `datetime` is imported, and `first_failure` is below the line.

Python3Saved ✓

import json

import math

import re

from datetime import date, datetime, timedelta

def incident_review(deploys, lines):

"""The facts for the postmortem: root service, first failure, trigger deploy,

rollback target and how many requests were affected."""

# Your code here

pass

# =============================================================================

# PROVIDED BELOW -- part of Helpdesk. Read it, don't edit it.

# =============================================================================

def first_failure(lines):

errors = []

for position, line in enumerate(lines):

fields = {}

parts = line.split(" ")

for part in parts[1:]:

if "=" in part:

key, _, value = part.partition("=")

fields[key] = value

try:

at = datetime.fromisoformat(parts[0].replace("Z", "+00:00"))

except ValueError:

continue

if fields.get("level") != "ERROR" or "svc" not in fields:

continue

errors.append((at, position, fields["svc"], fields.get("req"), parts[0]))

if not errors:

return None

at, _, service, request_id, raw_time = min(errors)

return {"service": service, "request_id": request_id, "time": raw_time}

Case 1: Yesterday's checkout incident

Input: deploys = [{'id': 'd-41', 'service': 'payments-db', 'at': '2026-10-19T09:00:00Z'}, {'id': 'd-43', 'service': 'payments-db', 'at': '2026-10-20T13:58:00Z'}, {'id': 'd-42', 'service': 'checkout', 'at': '2026-10-20T14:00:00Z'}, {'id': 'd-44', 'service': 'payments-db', 'at': '2026-10-20T14:30:00Z'}]
lines = ['2026-10-20T14:03:12Z level=ERROR svc=checkout req=r-1043 msg=payment_failed status=500', '2026-10-20T14:03:12Z level=ERROR svc=checkout req=r-1044 msg=payment_failed status=500', '2026-10-20T14:02:58Z level=WARN svc=payments-db req=- msg=slow_query ms=2400', '2026-10-20T14:03:05Z level=ERROR svc=payments req=r-1041 msg=db_timeout after_ms=3000', '2026-10-20T14:03:01Z level=ERROR svc=payments-db req=r-1040 msg=connection_pool_exhausted', '2026-10-20T14:03:13Z level=ERROR svc=checkout req=r-1045 msg=payment_failed status=500', '2026-10-20T14:03:06Z level=INFO svc=payments req=r-1046 msg=retrying attempt=2', 'payments-db: pool size 20, in use 20', '2026-10-20T14:03:14Z level=ERROR svc=checkout req=r-1046 msg=payment_failed status=500']

Expected: {'root_service': 'payments-db', 'first_failure_at': '2026-10-20T14:03:01Z', 'trigger_deploy': 'd-43', 'rollback_to': 'd-41', 'affected_requests': 6}

Case 2: No errors

Input: deploys = [{'id': 'd-41', 'service': 'payments-db', 'at': '2026-10-19T09:00:00Z'}, {'id': 'd-43', 'service': 'payments-db', 'at': '2026-10-20T13:58:00Z'}, {'id': 'd-42', 'service': 'checkout', 'at': '2026-10-20T14:00:00Z'}, {'id': 'd-44', 'service': 'payments-db', 'at': '2026-10-20T14:30:00Z'}]
lines = ['2026-10-20T09:00:00Z level=INFO svc=api req=r-1 msg=ok']

Expected: None

Case 3: The service's first deploy caused it

Input: deploys = [{'id': 'd-1', 'service': 'search', 'at': '2026-10-20T08:00:00Z'}]
lines = ['2026-10-20T08:05:00Z level=ERROR svc=search req=r-9 msg=index_missing']

Expected: {'root_service': 'search', 'first_failure_at': '2026-10-20T08:05:00Z', 'trigger_deploy': 'd-1', 'rollback_to': None, 'affected_requests': 1}