Disaggregated Serving, Prefix Caching & Distributed Inference

When scaling LLMs to thousands of concurrent users, traditional monolithic serving engines hit a fundamental wall: the structural conflict between Prefill and Decode.

Prefill is compute-bound, processing thousands of input tokens in parallel with high operational intensity. Decode is memory-bandwidth bound, generating one token at a time with near-zero operational intensity. When both run on the same GPU, long prefills cause severe head-of-line blocking, inducing massive Inter-Token Latency (ITL) jitter for active users.

In this chapter, we engineer the modern distributed serving architectures that solve this bottleneck: Prefill-Decode (PD) Disaggregation, Radix Tree Prefix Caching, and Topology-Aware Model Parallelism (Tensor, Pipeline, and Expert Parallelism for frontier MoE architectures).

1. The Prefill-Decode Interference Problem

In traditional monolithic serving (e.g., standard vLLM or Hugging Face TGI), incoming requests enter a shared continuous batch:

Monolithic Co-located Batch (Severe ITL Jitter):
Iter 1: [Req A: decode tok 12] [Req B: decode tok 45] [Req C: PREFILL 4,096 tokens!] 
         └───────────────────────┬──────────────────────────────────────────┘
                                 │
                   GPU Tensor Cores 100% saturated with GEMM;
                   Req A and Req B experience an ITL latency spike of 450ms!

The Two Contrasting Regimes

When a 4,096-token prefill shares a forward pass with ongoing decode requests, the forward pass time jumps from 15ms to 400ms. To the user reading streamed text, generation suddenly freezes—violating real-time voice and streaming Service Level Objectives (SLOs).

2. Prefill-Decode (PD) Disaggregated Serving Architecture

Disaggregated serving (pioneered by systems like Splitwise, DistServe, Mooncake, and NVIDIA Dynamo) physically splits the inference pipeline across specialized hardware nodes:

┌─────────────────────────────────────────────────────────────────────────────┐
│                    DISAGGREGATED SERVING ARCHITECTURE                       │
└─────────────────────────────────────────────────────────────────────────────┘
                                       │ Incoming Prompt
                                       ▼
                         ┌───────────────────────────┐
                         │   Intelligent Router &    │
                         │   Cache-Aware Dispatcher  │
                         └─────────────┬─────────────┘
                                       │
                ┌──────────────────────┴──────────────────────┐
                ▼                                             ▼
┌───────────────────────────────┐             ┌───────────────────────────────┐
│     PREFILL WORKER POOL       │             │      DECODE WORKER POOL       │
│  (Compute-Optimized, TP=1/2)  │             │   (Memory-Optimized, TP=4/8)  │
│                               │             │                               │
│ • Runs prompt GEMM in FP8     │             │ • Runs iteration decode loop  │
│ • Generates initial KV Cache  │             │ • Zero prefill interference   │
│ • Emits Token 1               │             │ • Ultra-stable ITL (10-15ms)  │
└───────────────┬───────────────┘             └───────────────▲───────────────┘
                │                                             │
                └─────────── Transmit KV Cache ───────────────┘
                    (via RDMA / RoCE / NVLink-C2C Network)

How Disaggregation Works

1. Prompt Ingestion: The router receives an incoming prompt sequence.
2. Dedicated Prefill Execution: A prefill worker executes the full causal prompt pass in parallel, computing the prompt KV cache and sampling the first output token.
3. KV Cache Transfer: The prefill worker transfers the completed KV cache across the high-speed network fabric (InfiniBand, RoCE v2, or NVLink) directly into the decode worker's physical memory pool.
4. Isolated Decode Loop: The decode worker appends the first token, assumes ownership of the request, and performs continuous autoregressive decoding with zero compute interference.

Conditional Disaggregation

In production clusters, transferring small KV caches introduces network serialization overhead. Systems use Conditional Disaggregation:

• Short prompts ($< 256$ tokens): Executed locally on the decode worker to avoid network transfer latency.
• Long prompts ($\ge 256$ tokens): Dispatched to dedicated prefill workers to protect decode workers from head-of-line blocking.

3. Prefix Caching & Radix Tree Management

Modern agentic workflows, long-document RAG, and multi-turn conversations reuse massive amounts of prompt text. Re-running prefill on identical tokens burns expensive compute and spikes TTFT.

Prefix Caching retains the KV cache of previously processed tokens in memory across independent requests.

Request 1: [System Prompt (2,048 tokens)] + [Doc A (4,096 tokens)] + [Question 1]
                                 ▲
                                 │ Cached in Radix Tree!
Request 2: [System Prompt (2,048 tokens)] + [Doc A (4,096 tokens)] + [Question 2]
                                 ▲
                       (Skip 6,144 tokens!)
                       Pre-fill takes 5ms instead of 350ms!

RadixAttention (SGLang & vLLM APC)

Rather than storing independent linear prompt caches, state-of-the-art inference engines organize KV cache blocks as a Radix Tree (a space-optimized Trie):

                      [Root: System Prompt]
                            /        \
                           /          \
               [Tool Defs A]          [Tool Defs B]
                  /      \                  \
                 /        \                  \
          [Doc A-1]      [Doc A-2]          [Doc B-1]

• Prefix Sharing: Any node in the tree can be shared by thousands of active requests.
• LRU Eviction: When GPU memory pressure rises, an LRU (Least Recently Used) policy evicts leaf nodes while preserving high-frequency ancestor nodes (like core system prompts).
• Cache-Aware Routing: The cluster load balancer hashes the prompt prefix and routes incoming requests to the specific GPU worker whose local memory already contains that Radix branch.

4. Distributed Model Parallelism: TP vs PP vs EP

When frontier models exceed the memory capacity of a single GPU (e.g., DeepSeek-V3 with 671 billion parameters requires $>700$ GB for FP8 weights alone), inference must be partitioned across multiple GPUs.

┌─────────────────────────────────────────────────────────────────────────────┐
│                 INFERENCE PARALLELISM STRATEGIES COMPARED                   │
└─────────────────────────────────────────────────────────────────────────────┘

1. Tensor Parallelism (TP) - Intra-Node via NVLink (Low Latency):
   Matrix W1 sharded column-wise; Matrix W2 sharded row-wise.
   Requires an All-Reduce collective operation per Transformer layer.

2. Pipeline Parallelism (PP) - Inter-Node via InfiniBand (High Throughput):
   Layers 1-16 on Node 1 ──► Layers 17-32 on Node 2 ──► Layers 33-48 on Node 3.
   Introduces pipeline bubbles; suited for batch throughput, not single-user ITL.

3. Expert Parallelism (EP) - MoE Routing via All-to-All:
   Attention layers sharded via TP; MoE Feed-Forward Experts sharded across nodes.
   Tokens routed dynamically to expert GPUs via fast All-to-All collectives.

DeepSeek-V3 & Frontier MoE Inference Architecture

Frontier MoE models like DeepSeek-V3 (671B total parameters, 37B active parameters per token) require Dual-Topology Parallelism:

• Attention Layers: Replicated or sharded via Tensor Parallelism ($TP=4$) across dense intra-node NVLink.
• Expert Layers: Sharded via Expert Parallelism ($EP=64$ or $EP=128$) across multiple nodes connected via RoCE/InfiniBand.
• Multi-Head Latent Attention (MLA): Compresses the KV cache into a low-dimensional latent space ($d_c = 512$), reducing KV cache memory footprint by $5\times$ to $7\times$ compared to standard MHA/GQA.

5. Production Latency Metrics & Sizing Equation

In production inference engineering, system performance is governed by strict Service Level Agreements (SLAs).

Total Request Latency = TTFT + (Output Tokens × ITL)

1. Time-To-First-Token (TTFT): Latency from HTTP request arrival to the emission of the first token. Dominated by queueing delay and prefill GEMM speed.
2. Inter-Token Latency (ITL): Time between consecutive streamed tokens. Dominated by GPU memory bandwidth and decode Tensor Parallelism.
3. Time-Per-Output-Token (TPOT): Average duration per output token across the entire sequence.

GPU VRAM Allocation Sizing Formula

To determine the minimum number of GPUs required to serve a model under concurrent load:

$VRAM_{\text{req}} = \text{Model Weights} + \text{KV Cache Pool} + \text{Activation Working Memory}$

$\text{KV Cache Pool} = B_{\max} \times L_{\max} \times 2 \times n_{\text{layers}} \times n_{\text{kv\_heads}} \times d_{\text{head}} \times \text{bytes\_per\_elem}$

6. Python Simulation: Prefix Radix Tree Cache Simulator

Run this live in-browser Python implementation to simulate how a Radix Tree cache eliminates redundant prefill compute across multi-turn agent interactions: