Faithfulness Evaluation Scoremedium

Faithfulness Evaluation Score

medium RAG & Retrieval 15 min

NumPy allowed · plain Python earns 10 bonus points

A query retrieves context

Embedding similarity ranks the documents before generation.

Open lesson →

01query

02embed

03retrieve

04context

What is RAG?→ embed →top context

retrieval.md44%

rag-systems.md33%

agents.md24%

Cosine-like similarity scores ranked before context assembly

Implement the RAG faithfulness metric: what fraction of a generated answer's claims are actually grounded in retrieved sources.

Task

Implement faithfulness_score(answer_sentences, grounded_flags). Both lists are the same length (one flag per sentence, True meaning that sentence is grounded). Return sum(grounded_flags) / len(answer_sentences). Return 1.0 for an empty answer (vacuously faithful).

Function Signature

faithfulness_score(answer_sentences: list[str], grounded_flags: list[bool]) -> float

Examples

Example 1: fully grounded
Input: answer_sentences = ['a', 'b']
grounded_flags = [True, True]
Output: 1

Example 2: partially grounded
Input: answer_sentences = ['a', 'b', 'c']
grounded_flags = [True, False, True]
Output: 0.6666666666666666

Example 3: fully ungrounded
Input: answer_sentences = ['a']
grounded_flags = [False]
Output: 0

Constraints

len(answer_sentences) == len(grounded_flags)

Python3Saved ✓

def faithfulness_score(answer_sentences, grounded_flags):

# Your implementation here

pass

Case 1: fully grounded

Input: answer_sentences = ['a', 'b']
grounded_flags = [True, True]

Expected: 1

Case 2: partially grounded

Input: answer_sentences = ['a', 'b', 'c']
grounded_flags = [True, False, True]

Expected: 0.6666666666666666

Case 3: fully ungrounded

Input: answer_sentences = ['a']
grounded_flags = [False]

Expected: 0