Inter-Rater Agreement (Cohen's Kappa)hard

Inter-Rater Agreement (Cohen's Kappa)

hard Agents in Production 15 min

NumPy allowed · plain Python earns 10 bonus points

Implement Cohen's Kappa, the real, standard statistic for measuring agreement between two evaluators (human or LLM-judge) beyond what chance alone would predict.

Task

Implement cohens_kappa(rater_a, rater_b). Observed agreement p_o = fraction where rater_a[i] == rater_b[i]. Expected agreement p_e = sum(freq_a[c] * freq_b[c] for c in categories), where freq_a[c]/freq_b[c] are each rater's fraction of ratings equal to category c. Kappa = (p_o - p_e) / (1 - p_e). Return 1.0 if p_e == 1.

Function Signature

cohens_kappa(rater_a: list[str], rater_b: list[str]) -> float

Examples

Example 1: perfect agreement kappa=1
Input: rater_a = ['good', 'bad', 'good']
rater_b = ['good', 'bad', 'good']
Output: 1

Example 2: no agreement beyond chance
Input: rater_a = ['good', 'good', 'bad', 'bad']
rater_b = ['bad', 'bad', 'good', 'good']
Output: -1

Example 3: high but not perfect agreement
Input: rater_a = ['good', 'bad', 'good', 'bad']
rater_b = ['good', 'bad', 'good', 'good']
Output: 0.5

Constraints

len(rater_a) == len(rater_b) > 0

Python3Saved ✓

def cohens_kappa(rater_a, rater_b):

# Your implementation here

pass

Case 1: perfect agreement kappa=1

Input: rater_a = ['good', 'bad', 'good']
rater_b = ['good', 'bad', 'good']

Expected: 1

Case 2: no agreement beyond chance

Input: rater_a = ['good', 'good', 'bad', 'bad']
rater_b = ['bad', 'bad', 'good', 'good']

Expected: -1

Case 3: high but not perfect agreement

Input: rater_a = ['good', 'bad', 'good', 'bad']
rater_b = ['good', 'bad', 'good', 'good']

Expected: 0.5