Measure Retrieval Before Tuning Itmedium

Measure Retrieval Before Tuning It

medium Specify, Verify, Debug 25 min

Standard library only · runs in your browser

"The bot gets order questions wrong." Someone proposes a bigger embedding model. Someone else wants to rewrite the prompt. Before anyone changes anything, measure where retrieval fails, per kind of question, so the fix goes where the misses are.

Task

You have the ranked documents retrieval returned for seven test questions, and labels saying which documents are relevant.

Write retrieval_report(results, labels, k), returning:

{"overall": {"recall_at_k": 0.75, "mrr": 0.667, "queries": 6},
 "by_category": {"order_status": {"recall_at_k": 0.25, "mrr": 0.5, "queries": 2}, ...},
 "worst_category": "order_status",
 "unlabeled": 1}

Use only the first k distinct documents of each result list. The same id appearing twice counts once.
recall@k for one query = relevant documents in the top k ÷ all relevant documents. Average over queries.
MRR: average of 1 ÷ (rank of the first relevant document within the top k), or 0 if none is there.
Queries with no labels are skipped and counted in unlabeled.
Round every number to 3 decimals. worst_category has the lowest recall@k, ties broken alphabetically. If nothing is labelled, overall and worst_category are None.

Function Signature

retrieval_report(results: list[dict], labels: dict, k: int) -> dict

Examples

Example 1: Helpdesk's seven test questions, k=3
Input: results = [{'query_id': 'q1', 'category': 'policy', 'retrieved': ['refunds', 'shipping', 'warranty']}, {'query_id': 'q2', 'category': 'policy', 'retrieved': ['warranty', 'refunds', 'refunds', 'returns']}, {'query_id': 'q3', 'category': 'order_status', 'retrieved': ['faq', 'shipping', 'contact']}, {'query_id': 'q4', 'category': 'order_status', 'retrieved': ['tracking', 'faq', 'contact']}, {'query_id': 'q5', 'category': 'product', 'retrieved': ['sizes', 'materials', 'care']}, {'query_id': 'q6', 'category': 'product', 'retrieved': ['care', 'materials', 'faq']}, {'query_id': 'q7', 'category': 'policy', 'retrieved': ['privacy', 'terms', 'refunds']}]
labels = {'q1': ['refunds'], 'q2': ['refunds', 'returns'], 'q3': ['tracking'], 'q4': ['tracking', 'shipping'], 'q5': ['sizes'], 'q6': ['materials']}
k = 3
Output: {'overall': {'recall_at_k': 0.75, 'mrr': 0.667, 'queries': 6}, 'by_category': {'order_status': {'recall_at_k': 0.25, 'mrr': 0.5, 'queries': 2}, 'policy': {'recall_at_k': 1, 'mrr': 0.75, 'queries': 2}, 'product': {'recall_at_k': 1, 'mrr': 0.75, 'queries': 2}}, 'worst_category': 'order_status', 'unlabeled': 1}

Example 2: One query, relevant document at rank 2
Input: results = [{'query_id': 'a', 'category': 'policy', 'retrieved': ['x', 'y', 'z']}]
labels = {'a': ['y']}
k = 3
Output: {'overall': {'recall_at_k': 1, 'mrr': 0.5, 'queries': 1}, 'by_category': {'policy': {'recall_at_k': 1, 'mrr': 0.5, 'queries': 1}}, 'worst_category': 'policy', 'unlabeled': 0}

Example 3: No labels yet
Input: results = [{'query_id': 'q1', 'category': 'policy', 'retrieved': ['refunds', 'shipping', 'warranty']}, {'query_id': 'q2', 'category': 'policy', 'retrieved': ['warranty', 'refunds', 'refunds', 'returns']}]
labels = {}
k = 3
Output: {'overall': None, 'by_category': {}, 'worst_category': None, 'unlabeled': 2}

Constraints

`by_category` holds the same three keys as `overall`, one entry per category with labelled queries.
Hidden tests use k=1 and k=5 and duplicate ids.
Standard library only.

Python3Saved ✓

import json

import math

import re

from datetime import date, datetime, timedelta

def retrieval_report(results, labels, k):

"""recall@k and MRR, overall and per category, plus the worst category."""

# Your code here

pass

Case 1: Helpdesk's seven test questions, k=3

Input: results = [{'query_id': 'q1', 'category': 'policy', 'retrieved': ['refunds', 'shipping', 'warranty']}, {'query_id': 'q2', 'category': 'policy', 'retrieved': ['warranty', 'refunds', 'refunds', 'returns']}, {'query_id': 'q3', 'category': 'order_status', 'retrieved': ['faq', 'shipping', 'contact']}, {'query_id': 'q4', 'category': 'order_status', 'retrieved': ['tracking', 'faq', 'contact']}, {'query_id': 'q5', 'category': 'product', 'retrieved': ['sizes', 'materials', 'care']}, {'query_id': 'q6', 'category': 'product', 'retrieved': ['care', 'materials', 'faq']}, {'query_id': 'q7', 'category': 'policy', 'retrieved': ['privacy', 'terms', 'refunds']}]
labels = {'q1': ['refunds'], 'q2': ['refunds', 'returns'], 'q3': ['tracking'], 'q4': ['tracking', 'shipping'], 'q5': ['sizes'], 'q6': ['materials']}
k = 3

Expected: {'overall': {'recall_at_k': 0.75, 'mrr': 0.667, 'queries': 6}, 'by_category': {'order_status': {'recall_at_k': 0.25, 'mrr': 0.5, 'queries': 2}, 'policy': {'recall_at_k': 1, 'mrr': 0.75, 'queries': 2}, 'product': {'recall_at_k': 1, 'mrr': 0.75, 'queries': 2}}, 'worst_category': 'order_status', 'unlabeled': 1}

Case 2: One query, relevant document at rank 2

Input: results = [{'query_id': 'a', 'category': 'policy', 'retrieved': ['x', 'y', 'z']}]
labels = {'a': ['y']}
k = 3

Expected: {'overall': {'recall_at_k': 1, 'mrr': 0.5, 'queries': 1}, 'by_category': {'policy': {'recall_at_k': 1, 'mrr': 0.5, 'queries': 1}}, 'worst_category': 'policy', 'unlabeled': 0}

Case 3: No labels yet

Input: results = [{'query_id': 'q1', 'category': 'policy', 'retrieved': ['refunds', 'shipping', 'warranty']}, {'query_id': 'q2', 'category': 'policy', 'retrieved': ['warranty', 'refunds', 'refunds', 'returns']}]
labels = {}
k = 3

Expected: {'overall': None, 'by_category': {}, 'worst_category': None, 'unlabeled': 2}