Neural Mastery
← Practice
Evaluate Your Agenthard

Evaluate Your Agent

hard Build an Agent 20–25 min

Standard library only · runs in your browser

'It worked when I tried it' isn't an evaluation. Build the harness that runs your agent over a suite of cases and reports the numbers teams actually track: task success, tool-call accuracy, steps, and why each failure failed.

Task

Implement evaluate_agent(cases). Each case has name, task, script, expected_answer_contains, expected_tools (a list), and an optional max_steps (default 8).

For each case, call the provided run_agent(case["task"], case["script"], max_steps=...) (your stage-6 agent). Then classify it, checking in this order:

  1. answer is None: failure reason "no_answer:" + stop_reason
  2. tools_called != expected_tools: "wrong_tools"
  3. expected_answer_contains isn't in the answer (case-insensitive): "wrong_answer"
  4. Otherwise the case passed.

Return:

{"pass_rate": passed / n,                       rounded to 3 places
 "tool_accuracy": exact-tool-match cases / n,   rounded to 3 places
 "avg_steps": mean steps,                       rounded to 2 places
 "failures": {case name: reason}}

Raise ValueError for an empty list of cases.

Function Signature

evaluate_agent(cases: list[dict]) -> dict

Examples

Example 1: A passing suite
Input: cases = [{'name': 'weather-paris', 'task': 'Weather in Paris?', 'script': ['{"tool": "get_weather", "arguments": {"city": "Paris"}}', 'Paris: {last_tool_result}'], 'expected_answer_contains': 'light rain', 'expected_tools': ['get_weather']}, {'name': 'sum', 'task': 'What is 1 + 2?', 'script': ['{"tool": "add", "arguments": {"a": 1, "b": 2}}', 'It\'s {last_tool_result}.'], 'expected_answer_contains': '3', 'expected_tools': ['add']}]
Output: {'pass_rate': 1, 'tool_accuracy': 1, 'avg_steps': 2, 'failures': {}}
Example 2: Wrong answer vs wrong tools
Input: cases = [{'name': 'weather-paris', 'task': 'Weather in Paris?', 'script': ['{"tool": "get_weather", "arguments": {"city": "Paris"}}', 'Paris: {last_tool_result}'], 'expected_answer_contains': 'light rain', 'expected_tools': ['get_weather']}, {'name': 'hallucinated', 'task': 'Weather in Tokyo?', 'script': ['{"tool": "get_weather", "arguments": {"city": "Tokyo"}}', 'Tokyo is snowing.'], 'expected_answer_contains': 'clear', 'expected_tools': ['get_weather']}, {'name': 'no-lookup', 'task': 'Shipping time?', 'script': ['Shipping takes 3-5 business days.'], 'expected_answer_contains': '3-5', 'expected_tools': ['lookup_doc']}]
Output: {'pass_rate': 0.333, 'tool_accuracy': 0.667, 'avg_steps': 1.67, 'failures': {'hallucinated': 'wrong_answer', 'no-lookup': 'wrong_tools'}}
Example 3: A looping agent produces no answer
Input: cases = [{'name': 'sum', 'task': 'What is 1 + 2?', 'script': ['{"tool": "add", "arguments": {"a": 1, "b": 2}}', 'It\'s {last_tool_result}.'], 'expected_answer_contains': '3', 'expected_tools': ['add']}, {'name': 'stuck', 'task': 'Weather in Lima?', 'script': ['{"tool": "get_weather", "arguments": {"city": "Lima"}}', '{"tool": "get_weather", "arguments": {"city": "Lima"}}', '{"tool": "get_weather", "arguments": {"city": "Lima"}}', 'never'], 'expected_answer_contains': 'overcast', 'expected_tools': ['get_weather']}]
Output: {'pass_rate': 0.5, 'tool_accuracy': 0.5, 'avg_steps': 2.5, 'failures': {'stuck': 'no_answer:loop_detected'}}
Example 4: Empty suite is an error
Input: cases = []
Output: Raises ValueError

Constraints

  • `tool_accuracy` counts every case whose tools match, including cases that failed for another reason.
  • Use `round()` exactly as specified. Floats are compared with a 1e-5 tolerance.
  • Don't re-implement the agent. Call the provided `run_agent`.
Python3Saved ✓
import json
import math
import re


def evaluate_agent(cases):
"""Run run_agent over every case; return pass_rate, tool_accuracy, avg_steps, failures."""
# Your code here
pass


# =============================================================================
# PROVIDED BELOW -- the sandbox and your earlier stages. Read it, don't edit it.
# =============================================================================
# ---------------------------------------------------------------------------
# The sandbox world your agent lives in (provided -- you don't edit this).
# Four deterministic tools, each with a JSON-Schema-style argument spec,
# exactly the shape you'd hand a real model's tool-calling API.
# ---------------------------------------------------------------------------
WEATHER = {"paris": "18C, light rain", "tokyo": "24C, clear", "lima": "16C, overcast"}
DOCS = {
"refund-policy": "Refunds are accepted within 30 days with a receipt.",
"shipping": "Standard shipping takes 3-5 business days.",
}


def add(a, b):
return a + b


def divide(a, b):
return a / b


def get_weather(city):
key = city.strip().lower()
Case 1: A passing suite
Input: cases = [{'name': 'weather-paris', 'task': 'Weather in Paris?', 'script': ['{"tool": "get_weather", "arguments": {"city": "Paris"}}', 'Paris: {last_tool_result}'], 'expected_answer_contains': 'light rain', 'expected_tools': ['get_weather']}, {'name': 'sum', 'task': 'What is 1 + 2?', 'script': ['{"tool": "add", "arguments": {"a": 1, "b": 2}}', 'It\'s {last_tool_result}.'], 'expected_answer_contains': '3', 'expected_tools': ['add']}]
Expected: {'pass_rate': 1, 'tool_accuracy': 1, 'avg_steps': 2, 'failures': {}}
Case 2: Wrong answer vs wrong tools
Input: cases = [{'name': 'weather-paris', 'task': 'Weather in Paris?', 'script': ['{"tool": "get_weather", "arguments": {"city": "Paris"}}', 'Paris: {last_tool_result}'], 'expected_answer_contains': 'light rain', 'expected_tools': ['get_weather']}, {'name': 'hallucinated', 'task': 'Weather in Tokyo?', 'script': ['{"tool": "get_weather", "arguments": {"city": "Tokyo"}}', 'Tokyo is snowing.'], 'expected_answer_contains': 'clear', 'expected_tools': ['get_weather']}, {'name': 'no-lookup', 'task': 'Shipping time?', 'script': ['Shipping takes 3-5 business days.'], 'expected_answer_contains': '3-5', 'expected_tools': ['lookup_doc']}]
Expected: {'pass_rate': 0.333, 'tool_accuracy': 0.667, 'avg_steps': 1.67, 'failures': {'hallucinated': 'wrong_answer', 'no-lookup': 'wrong_tools'}}
Case 3: A looping agent produces no answer
Input: cases = [{'name': 'sum', 'task': 'What is 1 + 2?', 'script': ['{"tool": "add", "arguments": {"a": 1, "b": 2}}', 'It\'s {last_tool_result}.'], 'expected_answer_contains': '3', 'expected_tools': ['add']}, {'name': 'stuck', 'task': 'Weather in Lima?', 'script': ['{"tool": "get_weather", "arguments": {"city": "Lima"}}', '{"tool": "get_weather", "arguments": {"city": "Lima"}}', '{"tool": "get_weather", "arguments": {"city": "Lima"}}', 'never'], 'expected_answer_contains': 'overcast', 'expected_tools': ['get_weather']}]
Expected: {'pass_rate': 0.5, 'tool_accuracy': 0.5, 'avg_steps': 2.5, 'failures': {'stuck': 'no_answer:loop_detected'}}
Case 4: Empty suite is an error
Input: cases = []
Expected: Raises ValueError