AI Cost Engineering

Every architecture decision covered across this site — which model size, which quantization level, which serving engine, how much redundancy — has a cost implication, and at GPU-driven AI infrastructure's price points, getting cost wrong is expensive in a way it rarely was for traditional web infrastructure. This page consolidates the cost-optimization techniques scattered across earlier pages into one place, and frames the actual tradeoff every one of them makes.

Training Cost

Dominated by GPU-hours — see GPU/AI Infrastructure & Distributed Training for the mechanics (mixed precision, gradient accumulation, parallelism strategies) that determine how efficiently those GPU-hours get used. The scaling laws and Chinchilla result are directly a cost-optimization finding at their core: training compute-optimally (right-sized model, right-sized data, together) gets more capability per dollar than training an oversized model on too little data for the same total spend.

Inference Cost

For most deployed AI products, inference cost — not training cost — dominates total spend over the system's lifetime, since training happens once (or periodically) while inference happens on every single request, indefinitely. Every technique in LLM Inference Optimization is, from a cost-engineering lens, a cost-per-token reduction technique: quantization shrinks memory and increases throughput per GPU, continuous batching increases GPU utilization (more requests served per GPU-hour), speculative decoding reduces wall-clock time per response. The standard unit to optimize and report is cost per 1M tokens — see LLM Inference Optimization — Inference Metrics Summary — because it's comparable across models, engines, and hardware choices in a way raw GPU-hour cost alone isn't.

GPU and Memory Utilization

Idle GPU capacity is pure waste — a GPU costs the same whether it's at 20% or 95% utilization. Low utilization traces back to specific, diagnosable causes covered elsewhere: a data-loading bottleneck starving training compute (see Python Engineering — Performance), insufficient batching during inference (see Inference Optimization — Batching), or simply over-provisioned capacity relative to actual demand (addressed by autoscaling or serverless inference for spiky workloads specifically). Cost engineering starts with measuring utilization, not just measuring spend — a high bill with low utilization is a fixable efficiency problem; a high bill with high utilization is a genuine scale/pricing problem, and the fix is different for each.

Caching

Avoiding redundant computation entirely is cheaper than computing it efficiently:

• Prompt/response caching: for requests that repeat exactly or nearly exactly (a common FAQ-style query, a repeated API call with identical parameters), cache the response and skip inference entirely for a cache hit — the cheapest possible inference cost is the one you never pay.
• Prefix caching: covered in LLM Inference Optimization — KV Cache — reusing computed KV cache for a shared prompt prefix (a system prompt, a few-shot template) across many requests, avoiding redundant computation on the shared portion of the input.

Batching and Quantization

Both covered in depth in LLM Inference Optimization — restated here specifically in cost terms: continuous batching increases the number of requests served per GPU-hour (more revenue-generating or product-serving work extracted from the same fixed GPU cost), and quantization (GGUF ecosystem, GPTQ/AWQ) both shrinks the hardware needed to serve a given model and increases throughput on whatever hardware is used — a rare case where a technique improves cost and latency simultaneously rather than trading one against the other.

Model Routing and Distillation

• Model routing: not every request needs the largest, most capable (and most expensive) model — routing simpler requests to a smaller, cheaper model and reserving the largest model for genuinely complex requests can cut average cost substantially with minimal quality impact on the easy majority of traffic. This requires a reliable way to classify request difficulty upfront — often itself a small, cheap classifier model, or confidence-based escalation (try the cheap model first, escalate to the expensive one only if its own confidence is low).
• Distillation: covered in Training Pipeline — Knowledge Distillation — training a smaller, cheaper "student" model to mimic a larger "teacher," producing a genuinely smaller model to deploy rather than routing between two separately-maintained models.

Prompt and Context Compression

• Prompt compression: shortening a prompt while preserving the information the model actually needs from it — directly reduces token cost (most LLM APIs charge per input token) and, for long prompts, reduces prefill latency (see LLM Inference Optimization — Prefill vs. Decode) as a side benefit.
• Context compression: for RAG and agent systems specifically, retrieving and passing only the genuinely necessary context (well-tuned chunk size and top-k, rather than defaulting to "retrieve more, just in case") directly controls both cost and the "lost in the middle" quality problem — a case where the cost-optimal choice and the quality-optimal choice point the same direction, not opposite ones.

The Central Tradeoff: Quality vs. Latency vs. Cost

Nearly every technique on this page (and the inference-optimization page it draws from) is a point on a three-way tradeoff surface, not a free win: a smaller/more-quantized model is cheaper and faster but somewhat lower quality; a larger batch size improves throughput/cost but can increase per-request latency; a cache hit is free but only helps for repeated requests. The actual engineering job is choosing where on that tradeoff surface a given product needs to sit — a real-time chat interface prioritizes latency more than a batch analytics job does, and a customer-facing feature has a different quality floor than an internal tool — rather than assuming any single technique is a universal improvement independent of what a specific system actually needs.

Next: Legal, Licensing & Governance — the non-technical constraints that shape which of these tradeoffs are even legally available to make.