A Judge You Can Trusthard

A Judge You Can Trust

hard Specify, Verify, Debug 25–30 min

Standard library only · runs in your browser

The team wants to compare two prompts by asking an LLM which answer is better. The judge agrees with humans 60% of the time. Before anyone trusts that number, find out how much of it is the judge reading the answers and how much is the judge picking whichever answer came first.

Task

judge(question, first, second) (provided) is the LLM judge. It returns "first" or "second". It's scripted so the tests are exact, and it has the two biases real judges have.

Each pair has two answers a and b, and the answer a person preferred ("human": "a" or "b").

Write evaluate_judge(pairs), returning three numbers rounded to 3 decimals:

raw_agreement: ask the judge once, a first and b second, and count how often its pick matches the human.
swap_checked_agreement: ask twice, a-then-b and b-then-a. If both runs pick the same answer, that's the verdict. If they disagree, the verdict is a tie, and a tie never matches the human.
position_flip_rate: how often the two runs disagree.

Function Signature

evaluate_judge(pairs: list[dict]) -> dict   (judge is provided)

Examples

Example 1: Five pairs the team labelled
Input: pairs = [{'question': 'Can I return last week\'s order?', 'a': 'Yes. Refunds are accepted within 30 days of delivery, so an order delivered last week still qualifies. Start the return from your order page.', 'b': 'Probably, check the site.', 'human': 'a'}, {'question': 'How long do I have?', 'a': 'Refunds are accepted within 30 days of delivery.', 'b': 'You have 30 days after delivery to request one.', 'human': 'b'}, {'question': 'How long do I have?', 'a': 'You have 30 days after delivery to request one.', 'b': 'Refunds are accepted within 30 days of delivery.', 'human': 'a'}, {'question': 'What\'s the refund policy?', 'a': 'Refunds are accepted within 30 days of delivery.', 'b': 'Great question! Our generous refund policy lets you return almost anything at any time, and we always do our best to make customers happy, so just reach out whenever you like and we\'ll sort it.', 'human': 'a'}, {'question': 'Can I return last week\'s order?', 'a': 'Probably, check the site.', 'b': 'Yes. Refunds are accepted within 30 days of delivery, so an order delivered last week still qualifies. Start the return from your order page.', 'human': 'b'}]
Output: {'raw_agreement': 0.6, 'swap_checked_agreement': 0.4, 'position_flip_rate': 0.4}
Explanation: Swap-checking lowers the number. The raw 60% included two answers chosen by position.

Example 2: One pair with an obvious winner
Input: pairs = [{'question': 'Can I return last week\'s order?', 'a': 'Yes. Refunds are accepted within 30 days of delivery, so an order delivered last week still qualifies. Start the return from your order page.', 'b': 'Probably, check the site.', 'human': 'a'}]
Output: {'raw_agreement': 1, 'swap_checked_agreement': 1, 'position_flip_rate': 0}

Example 3: Two near-identical answers
Input: pairs = [{'question': 'How long do I have?', 'a': 'Refunds are accepted within 30 days of delivery.', 'b': 'You have 30 days after delivery to request one.', 'human': 'b'}]
Output: {'raw_agreement': 0, 'swap_checked_agreement': 0, 'position_flip_rate': 1}

Constraints

Map positions back to answers carefully: in the swapped run, "first" means b.
Each pair is judged exactly twice for the swap check.
Hidden tests use other pairs and hand-computed expectations.

Python3Saved ✓

import json

import math

import re

from datetime import date, datetime, timedelta

def evaluate_judge(pairs):

"""Agreement with humans, raw and swap-checked, plus how often position flips the verdict."""

# Your code here

return {"raw_agreement": 0.0, "swap_checked_agreement": 0.0, "position_flip_rate": 0.0}

# =============================================================================

# PROVIDED BELOW -- part of Helpdesk. Read it, don't edit it.

# =============================================================================

def judge(question, first, second):

"""The LLM judge, as it behaves in production (scripted so tests are exact):

it prefers the clearly longer answer, and when the two are close in length

it picks whichever one it saw first."""

if len(second) > 1.2 * len(first):

return "second"

if len(first) > 1.2 * len(second):

return "first"

Case 1: Five pairs the team labelled

Input: pairs = [{'question': 'Can I return last week\'s order?', 'a': 'Yes. Refunds are accepted within 30 days of delivery, so an order delivered last week still qualifies. Start the return from your order page.', 'b': 'Probably, check the site.', 'human': 'a'}, {'question': 'How long do I have?', 'a': 'Refunds are accepted within 30 days of delivery.', 'b': 'You have 30 days after delivery to request one.', 'human': 'b'}, {'question': 'How long do I have?', 'a': 'You have 30 days after delivery to request one.', 'b': 'Refunds are accepted within 30 days of delivery.', 'human': 'a'}, {'question': 'What\'s the refund policy?', 'a': 'Refunds are accepted within 30 days of delivery.', 'b': 'Great question! Our generous refund policy lets you return almost anything at any time, and we always do our best to make customers happy, so just reach out whenever you like and we\'ll sort it.', 'human': 'a'}, {'question': 'Can I return last week\'s order?', 'a': 'Probably, check the site.', 'b': 'Yes. Refunds are accepted within 30 days of delivery, so an order delivered last week still qualifies. Start the return from your order page.', 'human': 'b'}]

Expected: {'raw_agreement': 0.6, 'swap_checked_agreement': 0.4, 'position_flip_rate': 0.4}

Case 2: One pair with an obvious winner

Input: pairs = [{'question': 'Can I return last week\'s order?', 'a': 'Yes. Refunds are accepted within 30 days of delivery, so an order delivered last week still qualifies. Start the return from your order page.', 'b': 'Probably, check the site.', 'human': 'a'}]

Expected: {'raw_agreement': 1, 'swap_checked_agreement': 1, 'position_flip_rate': 0}

Case 3: Two near-identical answers

Input: pairs = [{'question': 'How long do I have?', 'a': 'Refunds are accepted within 30 days of delivery.', 'b': 'You have 30 days after delivery to request one.', 'human': 'b'}]

Expected: {'raw_agreement': 0, 'swap_checked_agreement': 0, 'position_flip_rate': 1}