Edge AI & TinyML Architecture

While datacenter inference operates in a world of high-bandwidth memory (HBM3), 80-gigabyte VRAM pools, and kilowatt power supplies, the physical frontier of AI lives at the opposite extreme: Edge AI and TinyML.

TinyML is the technical discipline of running machine learning inference on ultra-low-power microcontrollers (MCUs)—such as ARM Cortex-M0+, Cortex-M4, Cortex-M7, ESP32, and RISC-V:

• Compute Power: 40 MHz to 200 MHz single-core clocks.
• Memory Envelope: 64 KB to 256 KB of SRAM (working memory) and 512 KB to 2 MB of Flash ROM.
• Power Budget: Milliwatts or microwatts (operating on coin-cell batteries or solar energy for years without human intervention).
• Zero Operating System: Bare-metal C++ execution without virtual memory, paging, or swap files.

This chapter details the systems architecture of microcontroller inference engines, zero-copy Flash execution, and memory planning algorithms.

1. The Microcontroller Memory Architecture

Explore the division between non-volatile Flash memory (ROM) and volatile SRAM:

The Embedded Memory Split

In an embedded system, memory is divided into two physically distinct semiconductor technologies:

1. Flash Memory (ROM, Non-Volatile):
   • Size: Typically 512 KB to 2 MB.
   • Speed: Moderate read speed (~30-50ns), zero standby power.
   • Purpose: Stores compiled firmware instructions, quantized INT8 weights, bias vectors, and graph topology.
   • XIP (eXecute In Place): The microcontroller reads weights directly from Flash via memory-mapped buses (SPI / AHB). Weights never enter SRAM!
2. SRAM (Static RAM, Volatile):
   • Size: 64 KB to 256 KB total.
   • Speed: High-speed single-cycle access (~5ns), high static leakage current.
   • Purpose: Reserved strictly for mutable state: input sensor buffers (microphones, accelerometers, camera pixels) and intermediate layer activations.

2. The Static Tensor Arena: Zero Dynamic Allocation

To eliminate heap fragmentation, TensorFlow Lite for Microcontrollers (TFLite Micro) allocates a single static byte buffer at initialization:

// Static allocation in global BSS segment -- zero heap malloc()!
constexpr int kTensorArenaSize = 60 * 1024; // 60 KB budget
alignas(16) uint8_t tensor_arena[kTensorArenaSize];

// Initialize TFLite Micro runtime
tflite::MicroInterpreter interpreter(
    model,
    resolver,
    tensor_arena,
    kTensorArenaSize,
    error_reporter
);

Offline Activation Lifetime Planning

Because the layer execution order of a neural network is deterministic, an offline memory planner computes the exact lifetime intervals for every intermediate activation tensor:

Layer Execution Timeline:
Layer 0 (Input):   [======== Lifetime ========]
Layer 1 (Conv):              [======== Lifetime ========]  (Reuses Offset A)
Layer 2 (Pool):                         [======== Lifetime ========]  (Reuses Offset B)
Layer 3 (Dense):                                   [==== Lifetime ====] (Reuses Offset A!)

Tensors whose lifetimes do not overlap are assigned the exact same byte offset inside tensor_arena. This buffer reuse reduces peak SRAM consumption by up to 75%, allowing models with 180 KB of total activations to execute inside a 48 KB arena!

3. System-Algorithm Co-Design: MCUNet & Depthwise Convolutions

Standard convolutional layers are computationally prohibitive on microcontrollers.

For a standard convolution with input channels $C_{\text{in}}$, output channels $C_{\text{out}}$, and kernel size $K \times K$:

$\text{FLOPs}_{\text{Conv}} = H \times W \times C_{\text{in}} \times C_{\text{out}} \times K^2$

In Depthwise-Separable Convolutions (MobileNet / MCUNet):

1. Depthwise Step: Applies a single $K \times K$ spatial filter per input channel independently.
2. Pointwise Step: Applies a $1 \times 1$ linear projection across channels.

$\text{FLOPs}_{\text{Sep}} = H \times W \times C_{\text{in}} \times K^2 + H \times W \times C_{\text{in}} \times C_{\text{out}}$

$\text{Compute Reduction} = \frac{H W C_{\text{in}} K^2 + H W C_{\text{in}} C_{\text{out}}}{H W C_{\text{in}} C_{\text{out}} K^2} = \frac{1}{C_{\text{out}}} + \frac{1}{K^2} \approx \frac{1}{9} \text{ of original compute!}$

4. Python Simulation: Microcontroller Memory Arena Planner

Run this live in-browser Python implementation to simulate how an offline activation memory planner assigns buffer offsets to minimize peak SRAM: