Inference Engineering — Roadmap

Training an AI model creates potential; Inference Engineering transforms that potential into production reality.

In modern AI systems, inference is not a passive function call (model.forward()). It is a high-stakes systems discipline bounded by two opposing hardware extremes:

1. The Hyperscale Cloud Frontier: Serving 70B+ parameter models across clusters of 80GB GPUs with sub-second Time-To-First-Token (TTFT), high throughput, continuous batching, and virtual memory paging.
2. The Ultra-Low-Power Edge Frontier (TinyML): Executing computer vision, keyword spotting, and sensor anomaly detection on 80MHz microcontrollers with less than 256 KB of SRAM, zero operating system, and milliwatt battery power budgets.

We engineer our inference stacks across this entire spectrum: from custom Triton kernels and PagedAttention down to static zero-copy microcontroller tensor arenas.

1. The Full-Stack Inference Spectrum

Explore the Roofline Model across modern inference hardware:

The Curriculum Track

┌─────────────────────────────────────────────────────────────────────────────┐
│                 INFERENCE ENGINEERING & EDGE AI CURRICULUM                  │
└─────────────────────────────────────────────────────────────────────────────┘
                                       │
         ┌─────────────────────────────┼─────────────────────────────┐
         ▼                             ▼                             ▼
┌─────────────────┐           ┌─────────────────┐           ┌─────────────────┐
│ 1. Foundations  │           │ 2. Roofline     │           │ 3. Continuous   │
│ & Systems Stack │           │ & KV Cache      │           │    Batching     │
│ [Overview]      │           │ [PagedAttention]│           │ [Speculative]   │
└─────────────────┘           └─────────────────┘           └─────────────────┘
         │                             │                             │
         └─────────────────────────────┼─────────────────────────────┘
                                       ▼
         ┌───────────────────────────────────────────────────────────┐
         ▼                                                           ▼
┌─────────────────────────────────┐         ┌─────────────────────────────────┐
│ 4. Quantization & Compression   │         │ 5. Edge AI & TinyML             │
│ FP8, INT8, AWQ, Sparsity        │         │ MCU Arenas & TFLite Micro       │
└─────────────────────────────────┘         └─────────────────────────────────┘
                                       │
         ┌─────────────────────────────┴─────────────────────────────┐
         ▼                                                           ▼
┌─────────────────────────────────────┐     ┌─────────────────────────────────────┐
│ 6. Hardware Kernels & Acceleration  │     │ 7. Disaggregation & Prefix Caching  │
│ FlashAttention, Triton, TensorRT    │     │ PD Separation, Radix Trees, MoE EP  │
└─────────────────────────────────────┘     └─────────────────────────────────────┘

1. Inference Engineering Overview

• The end-to-end inference lifecycle: Prefill vs Decode phases.
• Latency metrics that matter: Time To First Token (TTFT), Time Per Output Token (TPOT), and P99 latency.
• The cost equation of model serving: VRAM footprint, FLOPS utilization (MFU), and dollar-per-million-tokens.
• Architectural deep dive: Cloud cluster serving vs on-device edge execution.

2. The Roofline Model & KV Cache Mechanics

• The Roofline Model: Arithmetic Intensity, memory-bound vs compute-bound regimes.
• The KV Cache memory explosion: deriving the mathematical footprint ($2 \times 2 \times n_{\text{layers}} \times n_{\text{heads}} \times d_{\text{head}} \times L$).
• PagedAttention (vLLM): Virtual memory paging for KV cache blocks, eliminating internal and external fragmentation.
• Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) memory bandwidth reduction.

3. Continuous Batching & Speculative Decoding

• Why static request batching fails: the bubble latency trap.
• Iteration-level scheduling and continuous batching (Orca / vLLM).
• Speculative Decoding (Leviathan et al., Chen et al.): Draft-Verify acceleration engines.
• Tree Attention, Medusa heads, and EAGLE: multi-token verification in $O(1)$ forward passes.

4. Quantization & Model Compression

• Numerical representations: FP32, FP16, BF16, and FP8 (E4M3 vs E5M2).
• Post-Training Quantization (PTQ) vs Quantization-Aware Training (QAT).
• Symmetric vs Asymmetric INT8 quantization math: scale factors and zero-points ($q = \text{round}(x/S) + Z$).
• Advanced 4-bit Quantization: AWQ (Activation-aware Weight Quantization) and GPTQ.
• Structured vs unstructured weight pruning and 2:4 sparse Tensor Core acceleration.

5. Edge AI & TinyML Architecture

• The microcontroller constraint envelope: 256 KB SRAM, 1 MB Flash ROM, milliwatt power budgets.
• Eliminating dynamic allocation: Pre-planned static Tensor Arenas (TensorFlow Lite Micro).
• MicroNet and MCUNet: system-algorithm co-design for microcontrollers.
• Edge runtimes: ONNX Runtime Mobile, ExecuTorch (PyTorch Edge), Apple CoreML & Neural Engine (ANE).
• In-browser inference: WebGPU and WebAssembly (WASM).

6. Hardware Acceleration & Custom Kernels

• GPU Memory Hierarchy: High-Bandwidth Memory (HBM) vs On-Chip SRAM cache.
• FlashAttention-2 & 3: Tiling, online softmax, and eliminating intermediate attention matrices.
• Custom kernel authoring with OpenAI Triton and CUDA C++.
• Compilation and graph capture: TensorRT-LLM, TorchDynamo, and CUDA Graphs.

7. Disaggregated Serving, Prefix Caching & Distributed Inference

• Prefill-Decode (PD) Disaggregation: Decoupling compute-bound prefill from memory-bound decode over RDMA/NVLink fabrics to eliminate head-of-line blocking and ITL jitter.
• Prefix Caching & Radix Trees: SGLang RadixAttention and vLLM Automatic Prefix Caching (APC) for multi-turn chats, RAG prompts, and agent workflows.
• Distributed Model Parallelism: Megatron-LM Tensor Parallelism (TP), Pipeline Parallelism (PP), and Expert Parallelism (EP) for frontier MoE architectures (DeepSeek-V3).
• Production Serving Metrics: TTFT, ITL, TPOT, P99 tail latency SLOs, and concurrency-based Kubernetes autoscaling.