Neural Mastery
← Practice
Capstone: A Production Agent Loophard

Capstone: A Production Agent Loop

hard Build an Agent 45–60 min

Standard library only · runs in your browser

Put it all together: one loop that handles single and parallel tool calls, recovers from errors, detects loops, trims its context, enforces a real cost budget, and leaves a trace you could debug at 3am. This is the core of a real agent runtime, written by you.

Task

Implement parse_reply(text) and run_agent_v2(task, script, config).

parse_reply extends stage 1. Return None if the text has no "tool substring. Extract the JSON the same way as stage 1. If it has a tool_calls list, each item is a call; otherwise the object itself is one call. Return a list of {"id", "tool", "arguments"} (id may be None). Raise ValueError on bad JSON or bad shape.

run_agent_v2. config keys, all optional: max_steps (8), max_consecutive_errors (3), max_tokens (None), context_limit (None). Each step:

  1. context = trim_messages(messages, context_limit) if a limit is set, else messages. Its token cost is the sum of count_tokens over context. If max_tokens is set and the tokens already spent plus this cost would exceed it, return "token_budget" with steps = step - 1. Otherwise add the cost to the tokens spent, which accumulate across calls, and call llm(context).
  2. Append the reply as an assistant message, then:
    • Parse error: feedback exactly as in stage 5, count an error, and add {"step", "action": "parse_error"} to trace.
    • Final answer: trace {"step", "action": "final_answer"}, then return.
    • Calls: the signature is json.dumps([[tool, arguments], ...], sort_keys=True). If it's the same as the last two signatures, trace "loop_detected" and return. Otherwise run execute_tool_calls(calls), append every result message, add each tool name to tools_called, count the failed results as errors, and trace {"step", "action": "tools", "tools": [...], "failed": n}.
  3. Consecutive errors go up after a parse error, or after a tool step where every call failed. A tool step with at least one success resets them to 0. Stop with "too_many_errors" when they reach the limit.

Return {"answer", "steps", "stop_reason", "tools_called", "errors", "trace"}.

Function Signature

run_agent_v2(task: str, script: list[str], config: dict) -> dict

Examples

Example 1: Single call then answer
Input: task = 'Weather in Lima?' script = ['{"tool": "get_weather", "arguments": {"city": "Lima"}}', 'Lima: {last_tool_result}'] config = {}
Output: {'answer': 'Lima: 16C, overcast', 'steps': 2, 'stop_reason': 'final_answer', 'tools_called': ['get_weather'], 'errors': 0, 'trace': [{'step': 1, 'action': 'tools', 'tools': ['get_weather'], 'failed': 0}, {'step': 2, 'action': 'final_answer'}]}
Example 2: Parallel calls in one reply
Input: task = 'Paris and Tokyo?' script = ['{"tool_calls": [{"id": "a", "tool": "get_weather", "arguments": {"city": "Paris"}}, {"id": "b", "tool": "get_weather", "arguments": {"city": "Tokyo"}}]}', 'Last result: {last_tool_result}'] config = {}
Output: {'answer': 'Last result: 24C, clear', 'steps': 2, 'stop_reason': 'final_answer', 'tools_called': ['get_weather', 'get_weather'], 'errors': 0, 'trace': [{'step': 1, 'action': 'tools', 'tools': ['get_weather', 'get_weather'], 'failed': 0}, {'step': 2, 'action': 'final_answer'}]}
Example 3: Context limit trims what the model sees
Input: task = 'Weather tour' script = ['{"tool": "get_weather", "arguments": {"city": "Paris"}}', '{"tool": "get_weather", "arguments": {"city": "Tokyo"}}', '{"tool": "get_weather", "arguments": {"city": "Lima"}}', 'I saw {context_size} messages; latest: {last_tool_result}'] config = {'context_limit': 60}
Output: {'answer': 'I saw 4 messages; latest: 16C, overcast', 'steps': 4, 'stop_reason': 'final_answer', 'tools_called': ['get_weather', 'get_weather', 'get_weather'], 'errors': 0, 'trace': [{'step': 1, 'action': 'tools', 'tools': ['get_weather'], 'failed': 0}, {'step': 2, 'action': 'tools', 'tools': ['get_weather'], 'failed': 0}, {'step': 3, 'action': 'tools', 'tools': ['get_weather'], 'failed': 0}, {'step': 4, 'action': 'final_answer'}]}
Example 4: Cumulative token budget
Input: task = 'Weather tour' script = ['{"tool": "get_weather", "arguments": {"city": "Paris"}}', '{"tool": "get_weather", "arguments": {"city": "Tokyo"}}', '{"tool": "get_weather", "arguments": {"city": "Lima"}}', 'done'] config = {'max_tokens': 150}
Output: {'answer': None, 'steps': 2, 'stop_reason': 'token_budget', 'tools_called': ['get_weather', 'get_weather'], 'errors': 0, 'trace': [{'step': 1, 'action': 'tools', 'tools': ['get_weather'], 'failed': 0}, {'step': 2, 'action': 'tools', 'tools': ['get_weather'], 'failed': 0}]}

Constraints

  • Provided: `trim_messages` (stage 7) and `execute_tool_calls` (stage 8), plus `dispatch` and `validate_arguments`.
  • Token spend is cumulative: you pay for the whole context on every call.
  • The scripted model fills `{context_size}` with the number of messages you sent it, so trimming is visible to the tests.
Python3Saved ✓
import json
import math
import re


def parse_reply(text):
"""None for a final answer, else a list of {"id", "tool", "arguments"} calls.
Supports a single call or {"tool_calls": [...]}. Raise ValueError if malformed."""
# Your code here
pass


def run_agent_v2(task, script, config):
"""The full loop: parallel calls, recovery, loop detection, trimming, budget, trace."""
llm = make_llm(script)
messages = [{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": task}]
# Your code here
pass


# =============================================================================
# PROVIDED BELOW -- the sandbox and your earlier stages. Read it, don't edit it.
# =============================================================================
# ---------------------------------------------------------------------------
# The sandbox world your agent lives in (provided -- you don't edit this).
# Four deterministic tools, each with a JSON-Schema-style argument spec,
# exactly the shape you'd hand a real model's tool-calling API.
# ---------------------------------------------------------------------------
WEATHER = {"paris": "18C, light rain", "tokyo": "24C, clear", "lima": "16C, overcast"}
DOCS = {
"refund-policy": "Refunds are accepted within 30 days with a receipt.",
"shipping": "Standard shipping takes 3-5 business days.",
}


Case 1: Single call then answer
Input: task = 'Weather in Lima?' script = ['{"tool": "get_weather", "arguments": {"city": "Lima"}}', 'Lima: {last_tool_result}'] config = {}
Expected: {'answer': 'Lima: 16C, overcast', 'steps': 2, 'stop_reason': 'final_answer', 'tools_called': ['get_weather'], 'errors': 0, 'trace': [{'step': 1, 'action': 'tools', 'tools': ['get_weather'], 'failed': 0}, {'step': 2, 'action': 'final_answer'}]}
Case 2: Parallel calls in one reply
Input: task = 'Paris and Tokyo?' script = ['{"tool_calls": [{"id": "a", "tool": "get_weather", "arguments": {"city": "Paris"}}, {"id": "b", "tool": "get_weather", "arguments": {"city": "Tokyo"}}]}', 'Last result: {last_tool_result}'] config = {}
Expected: {'answer': 'Last result: 24C, clear', 'steps': 2, 'stop_reason': 'final_answer', 'tools_called': ['get_weather', 'get_weather'], 'errors': 0, 'trace': [{'step': 1, 'action': 'tools', 'tools': ['get_weather', 'get_weather'], 'failed': 0}, {'step': 2, 'action': 'final_answer'}]}
Case 3: Context limit trims what the model sees
Input: task = 'Weather tour' script = ['{"tool": "get_weather", "arguments": {"city": "Paris"}}', '{"tool": "get_weather", "arguments": {"city": "Tokyo"}}', '{"tool": "get_weather", "arguments": {"city": "Lima"}}', 'I saw {context_size} messages; latest: {last_tool_result}'] config = {'context_limit': 60}
Expected: {'answer': 'I saw 4 messages; latest: 16C, overcast', 'steps': 4, 'stop_reason': 'final_answer', 'tools_called': ['get_weather', 'get_weather', 'get_weather'], 'errors': 0, 'trace': [{'step': 1, 'action': 'tools', 'tools': ['get_weather'], 'failed': 0}, {'step': 2, 'action': 'tools', 'tools': ['get_weather'], 'failed': 0}, {'step': 3, 'action': 'tools', 'tools': ['get_weather'], 'failed': 0}, {'step': 4, 'action': 'final_answer'}]}
Case 4: Cumulative token budget
Input: task = 'Weather tour' script = ['{"tool": "get_weather", "arguments": {"city": "Paris"}}', '{"tool": "get_weather", "arguments": {"city": "Tokyo"}}', '{"tool": "get_weather", "arguments": {"city": "Lima"}}', 'done'] config = {'max_tokens': 150}
Expected: {'answer': None, 'steps': 2, 'stop_reason': 'token_budget', 'tools_called': ['get_weather', 'get_weather'], 'errors': 0, 'trace': [{'step': 1, 'action': 'tools', 'tools': ['get_weather'], 'failed': 0}, {'step': 2, 'action': 'tools', 'tools': ['get_weather'], 'failed': 0}]}