Neural Mastery
← Practice
Stale Answers After a Policy Changemedium

Stale Answers After a Policy Change

medium Specify, Verify, Debug 20 min

Standard library only · runs in your browser

On Monday the owner shortens the refund window to 14 days and updates the policy document. On Wednesday the bot is still telling customers 30. Retrieval picked up the new document within seconds. The answer never changed.

Task

Helpdesk answers from documents, and the agent added a cache so the same question doesn't call the model twice. The cache key is the question text, so an answer stays cached after the document behind it changes.

Fix cache_key(question, retrieved):

  • retrieved is the list of documents retrieval returned for this question, each {"id": ..., "version": ...}. A document's version goes up by one every time it's edited.
  • Freshness: when any retrieved document changes, the key must change.
  • Hit rate: editing a document that wasn't retrieved for this question must *not* change its key.
  • Same question, same key: ignore case, extra spaces, and trailing ?, . or !.

run_session(docs, events) (provided) replays a day of questions and document edits, and returns every answer plus the number of cache hits.

Function Signature

cache_key(question: str, retrieved: list[dict]) -> str   (run_session is provided)

Examples

Example 1: The Monday policy change
Input: docs = [{'id': 'refunds', 'version': 1, 'topics': ['refund', 'refunds', 'return', 'returns'], 'text': 'Refunds are accepted within 30 days of delivery.'}, {'id': 'shipping', 'version': 1, 'topics': ['shipping', 'delivery', 'ship'], 'text': 'Standard shipping takes 5 business days.'}] events = [{'ask': 'What is the refund policy?'}, {'update': 'refunds', 'text': 'Refunds are accepted within 14 days of delivery.'}, {'ask': 'What is the refund policy?'}]
Output: {'answers': ['Refunds are accepted within 30 days of delivery.', 'Refunds are accepted within 14 days of delivery.'], 'cache_hits': 0}
Explanation: The second answer must say 14 days.
Example 2: The same question asked twice
Input: docs = [{'id': 'refunds', 'version': 1, 'topics': ['refund', 'refunds', 'return', 'returns'], 'text': 'Refunds are accepted within 30 days of delivery.'}, {'id': 'shipping', 'version': 1, 'topics': ['shipping', 'delivery', 'ship'], 'text': 'Standard shipping takes 5 business days.'}] events = [{'ask': 'What is the refund policy?'}, {'ask': 'What is the refund policy?'}]
Output: {'answers': ['Refunds are accepted within 30 days of delivery.', 'Refunds are accepted within 30 days of delivery.'], 'cache_hits': 1}
Example 3: An edit to a document this question doesn't use
Input: docs = [{'id': 'refunds', 'version': 1, 'topics': ['refund', 'refunds', 'return', 'returns'], 'text': 'Refunds are accepted within 30 days of delivery.'}, {'id': 'shipping', 'version': 1, 'topics': ['shipping', 'delivery', 'ship'], 'text': 'Standard shipping takes 5 business days.'}] events = [{'ask': 'What is the refund policy?'}, {'update': 'shipping', 'text': 'Standard shipping takes 3 business days.'}, {'ask': 'What is the refund policy?'}]
Output: {'answers': ['Refunds are accepted within 30 days of delivery.', 'Refunds are accepted within 30 days of delivery.'], 'cache_hits': 1}
Explanation: The refund answer is still valid, so this should be a cache hit.

Constraints

  • The key must not depend on the order of `retrieved`.
  • Hidden tests count cache hits, so a key that changes on every edit anywhere fails too.
  • Standard library only.
Python3Saved ✓
import json
import math
import re
from datetime import date, datetime, timedelta


def cache_key(question, retrieved):
"""The cache key for an answer built from these retrieved documents."""
# The agent's version: one answer per question, forever.
return question


# =============================================================================
# PROVIDED BELOW -- part of Helpdesk. Read it, don't edit it.
# =============================================================================
def retrieve(docs, question):
"""Keyword retrieval: a document matches when it shares a topic word with the question."""
words = set(question.lower().replace("?", " ").split())
hits = [d for d in docs.values() if words & set(d["topics"])]
return [{"id": d["id"], "version": d["version"]} for d in sorted(hits, key=lambda d: d["id"])]


def answer_from(docs, retrieved):
if not retrieved:
return "I couldn't find that in our policies."
return " ".join(docs[r["id"]]["text"] for r in retrieved)


def run_session(docs, events):
"""Replay a day of traffic. Each event is {"ask": question} or
{"update": doc_id, "text": new_text}. Returns every answer given and
how many questions were served from the cache."""
docs = {d["id"]: dict(d) for d in docs}
cache, answers, hits = {}, [], 0
for event in events:
if "update" in event:
Case 1: The Monday policy change
Input: docs = [{'id': 'refunds', 'version': 1, 'topics': ['refund', 'refunds', 'return', 'returns'], 'text': 'Refunds are accepted within 30 days of delivery.'}, {'id': 'shipping', 'version': 1, 'topics': ['shipping', 'delivery', 'ship'], 'text': 'Standard shipping takes 5 business days.'}] events = [{'ask': 'What is the refund policy?'}, {'update': 'refunds', 'text': 'Refunds are accepted within 14 days of delivery.'}, {'ask': 'What is the refund policy?'}]
Expected: {'answers': ['Refunds are accepted within 30 days of delivery.', 'Refunds are accepted within 14 days of delivery.'], 'cache_hits': 0}
Case 2: The same question asked twice
Input: docs = [{'id': 'refunds', 'version': 1, 'topics': ['refund', 'refunds', 'return', 'returns'], 'text': 'Refunds are accepted within 30 days of delivery.'}, {'id': 'shipping', 'version': 1, 'topics': ['shipping', 'delivery', 'ship'], 'text': 'Standard shipping takes 5 business days.'}] events = [{'ask': 'What is the refund policy?'}, {'ask': 'What is the refund policy?'}]
Expected: {'answers': ['Refunds are accepted within 30 days of delivery.', 'Refunds are accepted within 30 days of delivery.'], 'cache_hits': 1}
Case 3: An edit to a document this question doesn't use
Input: docs = [{'id': 'refunds', 'version': 1, 'topics': ['refund', 'refunds', 'return', 'returns'], 'text': 'Refunds are accepted within 30 days of delivery.'}, {'id': 'shipping', 'version': 1, 'topics': ['shipping', 'delivery', 'ship'], 'text': 'Standard shipping takes 5 business days.'}] events = [{'ask': 'What is the refund policy?'}, {'update': 'shipping', 'text': 'Standard shipping takes 3 business days.'}, {'ask': 'What is the refund policy?'}]
Expected: {'answers': ['Refunds are accepted within 30 days of delivery.', 'Refunds are accepted within 30 days of delivery.'], 'cache_hits': 1}