Neural Mastery
← Practice
Provider Fallback Cascademedium

Provider Fallback Cascade

medium Agents in Production 20–25 min

NumPy allowed · plain Python earns 10 bonus points

LLM step 1/4

Logits become a next-token distribution

Softmax turns a model score vector into probabilities.
Open lesson →
01prompt
02tokens
03logits
04probabilities
attention→next token
mechanism65%
layer20%
pattern10%
model5%

Softmax(logits) → probability distribution

Maintain high availability across LLM APIs by automatically failing over across primary and secondary model providers.

Task

Implement execute_fallback_cascade(providers, request). Iterate through providers in order. For each provider, accumulate latency_ms. If fail_rate_limit or fail_server_error is True, record the failure attempt and proceed to the next provider. On the first provider without failure flags, record successful attempt and return {'successful_provider': name, 'response': response_text, 'attempts': attempts, 'total_latency_ms': total_latency}. If all fail, raise RuntimeError('All providers in fallback cascade failed'). If providers is empty, raise ValueError.

Function Signature

execute_fallback_cascade(providers: list[dict], request: dict) -> dict

Examples

Example 1: Primary succeeds immediately
Input: providers = [{'name': 'primary-openai', 'fail_rate_limit': False, 'fail_server_error': False, 'latency_ms': 120, 'response_text': 'Success primary'}] request = {'prompt': 'Hello'}
Output: {'successful_provider': 'primary-openai', 'response': 'Success primary', 'attempts': [{'provider': 'primary-openai', 'status': 'success', 'latency_ms': 120}], 'total_latency_ms': 120}
Example 2: Primary rate limited secondary succeeds
Input: providers = [{'name': 'primary-openai', 'fail_rate_limit': True, 'fail_server_error': False, 'latency_ms': 50, 'response_text': ''}, {'name': 'secondary-anthropic', 'fail_rate_limit': False, 'fail_server_error': False, 'latency_ms': 180, 'response_text': 'Claude response'}] request = {'prompt': 'Hello'}
Output: {'successful_provider': 'secondary-anthropic', 'response': 'Claude response', 'attempts': [{'provider': 'primary-openai', 'status': 'failed', 'error': 'rate_limit_exceeded', 'latency_ms': 50}, {'provider': 'secondary-anthropic', 'status': 'success', 'latency_ms': 180}], 'total_latency_ms': 230}
Example 3: All providers fail raises RuntimeError
Input: providers = [{'name': 'p1', 'fail_rate_limit': True, 'latency_ms': 30}, {'name': 'p2', 'fail_server_error': True, 'latency_ms': 40}] request = {'prompt': 'Hello'}
Output: Raises RuntimeError
Example 4: Empty provider list raises ValueError
Input: providers = [] request = {'prompt': 'Hello'}
Output: Raises ValueError

Constraints

  • Total latency must be cumulative sum of all tried providers.
  • Preserve chronological attempt log.
  • Raise RuntimeError when all providers fail.
Python3Saved ✓
def execute_fallback_cascade(providers: list[dict], request: dict) -> dict:
"""Simulate prioritized LLM provider fallback cascade with error logging and latency accounting.
Args:
providers: Ordered list of provider configurations.
request: API request dictionary.
Returns:
Dict with 'successful_provider', 'response', 'attempts', and 'total_latency_ms'.
"""
# Your implementation here
pass

Case 1: Primary succeeds immediately
Input: providers = [{'name': 'primary-openai', 'fail_rate_limit': False, 'fail_server_error': False, 'latency_ms': 120, 'response_text': 'Success primary'}] request = {'prompt': 'Hello'}
Expected: {'successful_provider': 'primary-openai', 'response': 'Success primary', 'attempts': [{'provider': 'primary-openai', 'status': 'success', 'latency_ms': 120}], 'total_latency_ms': 120}
Case 2: Primary rate limited secondary succeeds
Input: providers = [{'name': 'primary-openai', 'fail_rate_limit': True, 'fail_server_error': False, 'latency_ms': 50, 'response_text': ''}, {'name': 'secondary-anthropic', 'fail_rate_limit': False, 'fail_server_error': False, 'latency_ms': 180, 'response_text': 'Claude response'}] request = {'prompt': 'Hello'}
Expected: {'successful_provider': 'secondary-anthropic', 'response': 'Claude response', 'attempts': [{'provider': 'primary-openai', 'status': 'failed', 'error': 'rate_limit_exceeded', 'latency_ms': 50}, {'provider': 'secondary-anthropic', 'status': 'success', 'latency_ms': 180}], 'total_latency_ms': 230}
Case 3: All providers fail raises RuntimeError
Input: providers = [{'name': 'p1', 'fail_rate_limit': True, 'latency_ms': 30}, {'name': 'p2', 'fail_server_error': True, 'latency_ms': 40}] request = {'prompt': 'Hello'}
Expected: Raises RuntimeError
Case 4: Empty provider list raises ValueError
Input: providers = [] request = {'prompt': 'Hello'}
Expected: Raises ValueError