Inference Engineering: Systems, Scale & Edge Deployment

In modern AI engineering, training a foundation model represents less than 15% of the total lifecycle cost. The remaining 85% is consumed by inference serving—the operational compute required to execute millions of forward passes across production applications, robotics, smartphones, and edge devices.

Inference is not a simple function execution. It is a systems-level battle against hardware physical boundaries:

1. The Prefill vs Decode Phase Dilemma: Processing the user prompt is compute-bound; generating subsequent tokens autoregressively is strictly memory-bandwidth bound.
2. The KV Cache Memory Explosion: Storing past Key and Value activations for 4k-128k context windows rapidly exhausts GPU VRAM, causing memory fragmentation and limiting concurrent batch sizes.
3. The Edge Constraint Envelope: Deploying vision and sensory models to microcontrollers (TinyML) where SRAM is capped at 256 KB, requiring zero dynamic memory allocation and INT8 quantization.

We engineer our inference systems across this complete continuum: from multi-GPU vLLM clusters down to bare-metal ARM Cortex-M microcontrollers.

1. The Roofline Model: Hardware Operational Regimes

Every piece of compute hardware—from an 80GB NVIDIA H100 GPU down to a $2 ESP32 microcontroller—operates under the mathematical boundaries of the Roofline Model:

The Two Operational Regimes

The peak attainable throughput of a model on a given hardware accelerator is bounded by:

$\text{Attainable Performance (TFLOPs/s)} = \min\left(\text{Peak Compute}, \text{Operational Intensity} \times \text{Memory Bandwidth}\right)$

Where Operational Intensity ($I$) is defined as:

$I = \frac{\text{Total Floating Point Operations (FLOPs)}}{\text{Total Bytes Transferred from Memory (Bytes)}}$

The Knee of the Roofline occurs at:

$I_{\text{knee}} = \frac{\text{Peak Compute Capacity (FLOPs/s)}}{\text{Peak Memory Bandwidth (Bytes/s)}}$

• Compute-Bound Regime ($I > I_{\text{knee}}$): The arithmetic intensity is high enough that the execution units (ALUs / Tensor Cores) are saturated. Performance is limited by compute FLOPs.
• Memory-Bandwidth-Bound Regime ($I < I_{\text{knee}}$): The execution units sit idle waiting for weights and activations to arrive across the memory bus. Performance is limited by memory bandwidth.

2. KV Cache Optimization & PagedAttention

In modern transformer models, recomputing past keys and values at every token generation step scales quadratically: $O(L^2)$. To achieve linear generation time, models store past Key and Value projection vectors in a KV Cache.

However, the memory footprint of the KV Cache grows rapidly with sequence length:

$\text{KV Cache Size per Request (Bytes)} = 2 \times 2 \times n_{\text{layers}} \times n_{\text{heads}} \times d_{\text{head}} \times L \times \text{Precision Bytes}$

PagedAttention vs Contiguous Allocation

In naive inference servers, memory for each request was allocated contiguously for the request's max_seq_len. This caused 60% to 80% of VRAM to be wasted due to over-reservation and memory fragmentation.

Explore how PagedAttention (vLLM) resolves memory fragmentation using virtual memory paging:

3. High-Throughput Serving: Continuous Batching & Speculative Decoding

Traditional batching (Static Batching) waits for all sequences in a batch to finish before accepting new requests. Because different sequences generate different lengths, shorter sequences sit idle (creating "inference bubbles") while waiting for the longest sequence.

Continuous Batching (iteration-level scheduling) evaluates scheduling after every single token generation step: finished sequences are evicted immediately and new requests are inserted into the running iteration.

Accelerating Generation: Speculative Decoding

Autoregressive token generation requires sequential execution. Speculative Decoding breaks this sequential dependency by pairing a small, fast Draft Model with a large Target Model:

The Speculative Decoding Guarantees

1. Parallel Verification: The target model evaluates all $K$ draft tokens in a single parallel forward pass ($O(1)$ verification).
2. Exact Distributional Fidelity: Using modified rejection sampling, tokens are accepted with probability: $\alpha_i = \min\left(1, \frac{P_{\text{target}}(x_i)}{P_{\text{draft}}(x_i)}\right)$ If a token is rejected, the target model resamples a replacement token from the normalized residual distribution: $P_{\text{resample}}(x) = \text{relu}\left(P_{\text{target}}(x) - P_{\text{draft}}(x)\right)$ This guarantees that the final output distribution is mathematically identical to sampling directly from the large target model!

4. Quantization: Compressing Precision for Datacenter & Edge

Quantization maps continuous 32-bit or 16-bit floating point numbers into discrete low-bit integer or floating representations (FP8, INT8, INT4):

Uniform Affine INT8 Quantization

The fundamental quantization mapping from real float $x \in [\alpha, \beta]$ to an 8-bit integer $q \in [-128, 127]$ is:

$q = \text{clamp}\left(\left\lfloor \frac{x}{S} \right\rceil + Z, -128, 127\right)$

Where the Scale factor ($S$) and Zero-point ($Z$) are defined as:

$S = \frac{\beta - \alpha}{q_{\max} - q_{\min}} = \frac{\beta - \alpha}{255}$

$Z = \text{round}\left(\frac{-\alpha}{S}\right) + q_{\min}$

De-quantization reconstructs the approximate floating-point value:

$\hat{x} = S \times (q - Z)$

In Symmetric Quantization (used for zero-centered weights), $Z = 0$ and $S = \frac{\max(|x|)}{127}$, eliminating integer offset additions during matrix multiplications.

5. Inference on the Edge: TinyML & Microcontrollers

Moving inference to embedded microcontrollers (ARM Cortex-M0/M4/M7, ESP32, RISC-V) introduces an entirely different set of physical constraints:

• Zero Operating System: Bare-metal C++ execution with no virtual memory or swap space.
• Microsecond Memory Budget: Typically 64 KB to 256 KB of SRAM and 512 KB to 1 MB of Flash ROM.
• Zero Dynamic Allocation: malloc and free lead to heap fragmentation and embedded Hard Fault crashes.

The Static Tensor Arena

Explore how TensorFlow Lite Micro structures memory execution inside embedded microcontrollers:

The Three Invariants of Microcontroller Inference

1. eXecute In Place (XIP): Quantized INT8 model weights reside permanently in non-volatile Flash memory (ROM) and are read directly, consuming zero bytes of precious SRAM.
2. The Static Tensor Arena: A single static byte buffer (uint8_t tensor_arena[ARENA_SIZE]) is allocated at startup.
3. Activation Lifetime Reuse: Intermediate tensor activations share memory offsets. Once Layer 1 finishes, Layer 3 reuses that identical physical SRAM address.

6. Live Interactive Inference Calculation in Python

Test this real, in-browser Python implementation to calculate the exact VRAM footprint, KV cache size, and memory bandwidth throughput for any model architecture:

Explore the deep-dive chapters in this track:

• The Roofline Model & KV Cache Mechanics
• Continuous Batching & Speculative Decoding
• Quantization & Model Compression
• Edge AI & TinyML Architecture
• Hardware Acceleration & Custom Kernels
• Disaggregated Serving, Prefix Caching & Distributed Inference