Write a Check That Agrees With Peoplemedium

Write a Check That Agrees With People

medium Specify, Verify, Debug 25–30 min

Standard library only · runs in your browser

Support reviewed 200 conversations by hand and marked each answer grounded or not. The most common failure was answers that cite a document and then state something it doesn't say. Nobody can review 5,000 conversations a day, so turn what the reviewers did into a check.

Task

A trace is the question, the documents retrieval returned, and the bot's answer, which cites documents like [refunds]. The reviewers' rule, as they explained it:

Refusals are fine. An answer that says "I couldn't find", "I could not find", "I don't know" or "I do not know" (any case) is grounded.
It must cite, and only what it was given. No [id] citation at all → not grounded. Citing an id that wasn't retrieved → not grounded.
Numbers must come from what it cites. Every dollar amount ($50, $4.99) and every duration (30 days, 5 business days, 12 hours) in the answer must appear word for word in the text of a *cited* document.

Write is_grounded(trace) to match the reviewers' labels.

Function Signature

is_grounded(trace: dict) -> bool

Examples

Example 1: Cites refunds, says 30 days
Input: trace = {'question': 'Can I get a refund?', 'retrieved': [{'id': 'refunds', 'text': 'Refunds are accepted within 30 days of delivery. Orders under $5 are not refundable.'}], 'answer': 'You can get a refund within 30 days of delivery [refunds].'}
Output: True

Example 2: Cites refunds, says 60 days
Input: trace = {'question': 'Can I get a refund?', 'retrieved': [{'id': 'refunds', 'text': 'Refunds are accepted within 30 days of delivery. Orders under $5 are not refundable.'}], 'answer': 'You have 60 days from delivery to ask for a refund [refunds].'}
Output: False
Explanation: The citation is real. The number isn't in it.

Example 3: Cites a document it was never given
Input: trace = {'question': 'Can I get a refund?', 'retrieved': [{'id': 'refunds', 'text': 'Refunds are accepted within 30 days of delivery. Orders under $5 are not refundable.'}], 'answer': 'Electronics have a 12 month warranty [warranty].'}
Output: False

Example 4: Right answer, no citation
Input: trace = {'question': 'Can I get a refund?', 'retrieved': [{'id': 'refunds', 'text': 'Refunds are accepted within 30 days of delivery. Orders under $5 are not refundable.'}], 'answer': 'Refunds are accepted within 30 days of delivery.'}
Output: False

Example 5: An honest refusal
Input: trace = {'question': 'Do you sell gift cards?', 'retrieved': [{'id': 'refunds', 'text': 'Refunds are accepted within 30 days of delivery. Orders under $5 are not refundable.'}], 'answer': 'I couldn\'t find anything about gift cards in our policies.'}
Output: True

Example 6: A dollar amount from the cited document
Input: trace = {'question': 'Is shipping free?', 'retrieved': [{'id': 'shipping', 'text': 'Standard shipping takes 5 business days. Free shipping on orders over $50.'}], 'answer': 'Shipping is free on orders over $50 [shipping].'}
Output: True

Constraints

Citation ids are lowercase letters, digits and hyphens inside square brackets.
A duration is a number followed by day(s), hour(s) or business day(s).
The starter is the check the agent wrote: "does it cite something?" Hidden tests are more reviewed traces.

Python3Saved ✓

import json

import math

import re

from datetime import date, datetime, timedelta

def is_grounded(trace):

"""True when the answer is supported by the documents it cites."""

# The agent's version: an answer with a citation counts as grounded.

return "[" in trace["answer"]

Case 1: Cites refunds, says 30 days

Input: trace = {'question': 'Can I get a refund?', 'retrieved': [{'id': 'refunds', 'text': 'Refunds are accepted within 30 days of delivery. Orders under $5 are not refundable.'}], 'answer': 'You can get a refund within 30 days of delivery [refunds].'}

Expected: True

Case 2: Cites refunds, says 60 days

Input: trace = {'question': 'Can I get a refund?', 'retrieved': [{'id': 'refunds', 'text': 'Refunds are accepted within 30 days of delivery. Orders under $5 are not refundable.'}], 'answer': 'You have 60 days from delivery to ask for a refund [refunds].'}

Expected: False

Case 3: Cites a document it was never given

Input: trace = {'question': 'Can I get a refund?', 'retrieved': [{'id': 'refunds', 'text': 'Refunds are accepted within 30 days of delivery. Orders under $5 are not refundable.'}], 'answer': 'Electronics have a 12 month warranty [warranty].'}

Expected: False

Case 4: Right answer, no citation

Input: trace = {'question': 'Can I get a refund?', 'retrieved': [{'id': 'refunds', 'text': 'Refunds are accepted within 30 days of delivery. Orders under $5 are not refundable.'}], 'answer': 'Refunds are accepted within 30 days of delivery.'}

Expected: False

Case 5: An honest refusal

Input: trace = {'question': 'Do you sell gift cards?', 'retrieved': [{'id': 'refunds', 'text': 'Refunds are accepted within 30 days of delivery. Orders under $5 are not refundable.'}], 'answer': 'I couldn\'t find anything about gift cards in our policies.'}

Expected: True

Case 6: A dollar amount from the cited document

Input: trace = {'question': 'Is shipping free?', 'retrieved': [{'id': 'shipping', 'text': 'Standard shipping takes 5 business days. Free shipping on orders over $50.'}], 'answer': 'Shipping is free on orders over $50 [shipping].'}

Expected: True