Networking & Distributed Systems

Every inference API call, every distributed training run, every RAG system's vector DB query crosses a network — and networks fail in ways local function calls never do: partially, slowly, and unpredictably.

HTTP Basics

• Request/response: a client sends a request (method, URL, headers, optional body), a server sends back a response (status code, headers, body). Every LLM API call — POST /v1/chat/completions — is exactly this.
• Status codes: 2xx success, 3xx redirect, 4xx client error (400 bad request, 401 unauthorized, 429 rate limited — the one every LLM API integration needs to handle gracefully), 5xx server error.
• REST: an architectural convention for structuring HTTP APIs around resources (/users/123) and standard verbs (GET/POST/PUT/DELETE) — the default style for the model-serving APIs covered in Model Serving.

See APIs, HTTP & Communication Patterns for the full depth: methods and idempotency, status code families, REST vs. RPC vs. GraphQL, authentication/authorization (API keys, OAuth 2.0, JWTs), the five patterns covering every way two systems stay in sync (polling, webhooks, SSE, WebSockets — including what LLM token streaming actually is under the hood), and building an API in FastAPI.

TCP/IP vs. UDP

• TCP: connection-oriented, guarantees delivery and ordering (retransmits lost packets, reorders out-of-order ones) — the foundation under HTTP and virtually every API call, because losing or reordering part of a request/response silently would be far worse than the retransmission overhead.
• UDP: connectionless, no delivery guarantee, no ordering guarantee — but much lower overhead. Used where occasional loss is acceptable and latency matters more (real-time audio/video, some game networking) — rarely the choice for typical AI API traffic, but worth knowing exists as the alternative TCP is trading against.
• Latency vs. bandwidth: latency is how long one request takes round-trip; bandwidth is how much data can flow per second once a connection is established. A high-bandwidth, high-latency connection (satellite link) can still feel slow for many small requests — this distinction is exactly why LLM inference's TTFT (latency-bound) and tokens/sec (bandwidth-bound-ish) are reported as separate metrics, not one "speed" number.

Everything above (HTTP, REST, TCP) at the socket level, underneath the frameworks that normally hide it:

import socket

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:  # SOCK_STREAM = TCP
    s.connect(("api.example.com", 443))
    request = b"GET /v1/health HTTP/1.1\r\nHost: api.example.com\r\nConnection: close\r\n\r\n"
    s.sendall(request)
    response = b""
    while chunk := s.recv(4096):
        response += chunk

HTTP vs. gRPC vs. WebSockets for service-to-service communication specifically — three different tradeoffs, not just three transports:

// gRPC: a strict, code-generated contract via Protocol Buffers -- faster to (de)serialize than JSON,
// and the client/server stubs are generated, not hand-written
service InferenceService {
  rpc Predict (PredictRequest) returns (PredictResponse);
}
message PredictRequest { repeated float features = 1; }

# the generated client, used like a normal function call -- the network round-trip is hidden
response = stub.Predict(inference_pb2.PredictRequest(features=[5.1, 3.5, 1.4, 0.2]))

HTTP/REST (see APIs, HTTP & Communication Patterns) wins on universal tooling and human-readability (curl anything, no codegen step); gRPC wins on performance and a strict typed contract, common for internal service-to-service calls (e.g. a gateway calling an inference server) where every caller is also your own code; WebSockets win when the client needs to push data mid-connection too, not just receive it — none of the three is strictly better, the choice depends on who's calling and what they need.

DNS

Domain Name System resolution turns a hostname (api.example.com) into an IP address before a connection can even be opened — an often-invisible step that becomes very visible the moment it's slow or misconfigured (a DNS lookup adding 200ms to every cold request, or a stale DNS cache routing traffic to a decommissioned server).

Load Balancing

Distributing incoming requests across multiple backend instances so no single instance is overwhelmed and the system survives one instance failing:

• Round-robin: cycle through backends in order — simple, works well when requests are roughly uniform cost.
• Least-connections: route to whichever backend currently has the fewest active requests — better than round-robin when request costs vary a lot, which is exactly the case for LLM inference (a short completion vs. a long one occupy a backend for very different amounts of time).
• Consistent hashing: route requests with the same key (e.g. the same user, or — relevant to prefix caching — the same prompt prefix) to the same backend consistently, so that backend's warm cache/state keeps getting reused instead of being cold-started on every request.

The CAP Theorem

A distributed data system can provide at most two of three guarantees simultaneously: Consistency (every read sees the most recent write), Availability (every request gets a response, even if it might not reflect the latest write), and Partition tolerance (the system keeps working despite network partitions between nodes). Since real networks do partition (a link goes down, a data center loses connectivity), partition tolerance isn't really optional — in practice the meaningful choice is between CP (stay consistent, refuse requests during a partition) and AP (stay available, accept the risk of serving stale data during a partition). This is why "which database/vector store fits this system" is a genuine engineering question, not just "which one has more features" — see Databases.

Consensus (Raft/Paxos, Intuition Only)

How do multiple nodes agree on a single value (who's the leader, what's the latest committed state) when any node might crash or messages might be delayed? Consensus algorithms (Raft is the one most production systems actually implement, designed to be more understandable than the original Paxos) solve this by requiring a majority of nodes to agree before a value is considered committed — the same majority-quorum idea underlies leader election in distributed databases, distributed lock services, and orchestration systems like Kubernetes' etcd. You won't implement Raft yourself, but recognizing "this tool uses Raft under the hood" (etcd, many distributed databases) tells you a lot about its failure behavior for free.

Why Distributed Training and Serving Are Genuinely Hard

Everything above compounds once you have many machines doing one job together:

• Partial failure: in a single process, an error either happens or it doesn't. In a distributed system, one worker out of a thousand can fail while the rest keep running — "did the job succeed" stops being a yes/no question, and systems have to explicitly decide what a partial failure means (see GPU/AI Infrastructure & Distributed Training's discussion of checkpointing and fault tolerance).
• Stragglers: in synchronous distributed training (see Distributed AI's data/tensor/pipeline parallelism), every worker must finish a step before any can proceed to the next — one slow worker (a flaky GPU, a network hiccup) slows down the entire job, not just its own share of the work.
• Network partitions: a group of nodes losing connectivity to the rest doesn't look like a clean failure — it looks like both sides being unable to tell whether the other side crashed or is just unreachable, which is exactly the scenario the CAP theorem above is about.

Next: Linux, Git & Developer Tooling — the daily tools every AI engineering job assumes fluency in.