Design Challenge: A RAG System Over 100M Enterprise Documents

Difficulty: Hard · Concept: The 9-Step Framework

Genuinely different from every other practice problem on this site: there's no single function to implement and no test suite that can check your answer. You're designing a real system end to end, in free text, then self-assessing against a real rubric — the same honest, no-auto-grading approach every "should we add LLM grading" question on this site runs into (there's no backend here beyond Firebase, and pretending otherwise wouldn't be honest).

The problem: you've been asked to design a system that lets people ask questions and get answers grounded in an organization's own documents. The real numbers: 100 million documents, sub-second latency expected, and answers that have to be right, not just plausible-sounding, because a wrong answer in an enterprise setting has real consequences.

Design it end to end — what would you actually build, in what order, and why? Think through chunking, retrieval, ranking, and everything downstream of getting an answer in front of a real user, including how you'd know it's actually working once it's live.

Next: Design Challenge: Real-Time Fraud Detection — a completely different shape of problem, same framework.