Inside an LLM

Attention, masks, positional encodings, sampling, the KV cache and the tricks that make inference fast.

Intermediate · 22 problems · 0 passed · Background reading: Attention & Transformers

1. 01Scaled Dot-Product Attention Scoreeasy
2. 02Construct Causal Attention Maskeasy
3. 03Temperature Scaled Softmaxeasy
4. 04Greedy Decoding Stepeasy
5. 05Top-K Logit Filteringeasy
6. 06Top-P (Nucleus) Filteringeasy
7. 07Cross-Entropy Loss from Logitseasy
8. 08Multi-Head Attention: Split Headseasy
9. 09RMS Normalization From Scratcheasy
10. 10Sinusoidal Positional Encodingmedium
11. 11Rotary Positional Embedding (RoPE) From Scratchhard
12. 12Perplexity from Token Log-Probabilitieshard
13. 13Calculate KV-Cache Memory Footprinteasy
14. 14KV Cache Size With Grouped Query Attentionmedium
15. 15KV Cache State Managermedium
16. 16Beam Search Step with Pruningmedium
17. 17Merge a LoRA Adapter Into the Base Weightmedium
18. 18Greedy Longest-Match WordPiece Tokenizationhard
19. 19Speculative Decoding Rejection Samplermedium
20. 20PagedAttention Block Allocatormedium
21. 21Continuous Batching Iteration Schedulermedium
22. 22Mixture-of-Experts Top-K Routinghard

Next track: RAG & Retrieval