Evaluation & Serving

An LLM system isn't done when it works once — it needs measurable evaluation and a serving setup that's fast and affordable enough to actually run in production.

Evaluation

Perplexity: an intrinsic measure of how well a model predicts held-out text — lower is better, but it doesn't directly measure usefulness or correctness for real tasks, only how "unsurprised" the model is by real text.

Benchmark suites (MMLU, HellaSwag, and many others): standardized test sets covering knowledge and reasoning across many domains. Useful for comparing models broadly, but easy to over-index on — a model can score well on benchmarks while still failing on your specific use case, and benchmark contamination (test data leaking into training data) is an ongoing concern industry-wide.

LLM-as-judge: use a strong LLM to evaluate another model's outputs against criteria (helpfulness, correctness, tone) — scales far better than human evaluation for iterating quickly, though it inherits whatever biases the judge model has and works best combined with periodic human spot-checks.

Display order

judge's position bias0.5

Response A (shown first)

7.70

Response B

7.00

Standard mitigation: evaluate both orderings and check the verdict is consistent -- exactly what toggling the pill above simulates.

Real quality: A=7.2, B=7 -- A is genuinely (barely) better, always. Judge score with a real +0.50 first-position bonus: A=7.70, B=7.00. Judge's verdict: "A wins." Correct this time, but only because A happened to also be first -- try the other order.

Evaluating for hallucination, toxicity, bias: typically a mix of automated classifiers, curated adversarial test sets, and human review — this isn't a solved problem, and production systems generally layer several imperfect checks rather than relying on one.

Serving & Inference Optimization

Batching: process multiple requests' forward passes together to better utilize GPU compute — the core lever for serving throughput. Continuous batching (dynamically adding new requests to a running batch as others finish) further improves GPU utilization over static batching.

Batching strategy

Continuous (vLLM's core innovation): a request joins or leaves the running batch at EVERY decode step -- no idle slots, no waiting for a batch boundary.

Speculative decoding: use a small, fast "draft" model to propose several tokens ahead, then have the large model verify them all in a single forward pass — accepted tokens are free speedup, rejected ones fall back to normal generation. Effective because verifying is cheaper than generating token-by-token.

DRAFT MODEL proposes:

The

quick

brown

fox

jumped

✓ accepted: 3 tokens · ✗ rejected: 2 tokens (regenerated correctly by target model)

High acceptance rate = a meaningful wall-clock speedup, since verification is cheaper than generating one token at a time.

Target model verifies all 5 draft tokens in ONE forward pass -- accepts the matching prefix (3 tokens), rejects the rest and generates correctly from there. Net result: 3 tokens for the cost of ~1 target-model step instead of 3.

Paged Attention: manages the KV cache (see Foundation Model Internals) using memory paging techniques borrowed from operating systems — allocates KV cache memory in fixed-size blocks rather than one large contiguous buffer per request, dramatically reducing memory waste and allowing many more concurrent requests to be served on the same hardware.

KV cache allocation

● req 1● req 2● req 3

Paged: fixed-size blocks allocated on demand from a shared pool, non-contiguous -- no over-reservation, no fragmentation, far more concurrent requests fit in the same GPU memory.

Cost/Latency Tradeoffs

Production LLM systems constantly balance: model size (bigger = better quality, slower, more expensive), context length (longer = more cost, "lost in the middle" risk), and precision (quantization trades a small quality hit for large speed/cost wins). The right tradeoff depends entirely on the use case — a real-time chat assistant has very different latency tolerance than an overnight batch summarization job.

Reasoning models add a fourth axis: the internal "thinking" tokens a reasoning model generates before its visible answer are real billed output tokens, not free — see Prompt Engineering — Test-Time Compute Scaling. Evaluating one at scale costs meaningfully more per run than the visible response length suggests, since the hidden reasoning trace is often many times longer than the final answer.

Guardrails

Production systems typically layer safety on top of the model itself: input filtering (blocking known attack patterns before they reach the model), output filtering (scanning responses for policy violations before returning them to the user), and rate limiting/monitoring for abuse patterns — treating the model's own safety training as one layer of defense, not the only one.

LLMs & GenAI core content complete. Continue to: Multimodal & Generative Models for the vision/diffusion side, or on to Agents for how LLMs take action.