Build From Scratch — Overview

Every page on this site explains a concept. This section is different: it's a list of things to actually build, from scratch, because building something is a stronger, more durable form of understanding than reading about it — the gap between "I understand how attention works" and "I have implemented attention" is where most illusory understanding hides. Andrej Karpathy's educational repositories (micrograd, nanoGPT, and the "Neural Networks: Zero to Hero" series) are the best-known example of exactly this philosophy, and worth working through directly alongside this list.

What's in this section

• The Build List — fifteen from-scratch builds, roughly in order of dependency (each one uses ideas the previous ones established), each scoped with what "done" looks like and cross-linked to the conceptual page it's implementing.

How to Use This List

Don't build all fifteen before moving on to anything else — that's not how this site is meant to be used. Instead, build the one or two most relevant to whatever you're currently learning, right after reading the conceptual explanation elsewhere on the site — read Attention & Transformers, then build attention from scratch, while the concept is fresh, rather than saving all the building for later. Each entry below is scoped to be buildable in a focused session or two, not a multi-week project — the goal is the specific insight that comes from implementation, not a production-quality artifact.

See The Paper Timeline for how these building blocks connect historically, and Projects for larger, more complete systems to build once the from-scratch fundamentals feel solid.