LLM Serving and Fine-Tuning Interview Questions
GenAI interviews now go past "what is a transformer" into what it costs to run one and how to adapt it. These questions use a typical 8B-class model with grouped-query attention: 32 layers, 8 key-value heads, head dimension 128, served in bf16. The arithmetic is the same for any model, so it's worth doing once by hand.
23 checkable questions on this page. Each is checked when you answer, and the worked answer opens after.
Prefill, decode and memory
When a model generates tokens one at a time for a single user, what limits its speed on a GPU?
A 7B model in bf16 (14 GB of weights) runs on a GPU with 2 TB/s of memory bandwidth. What's the theoretical maximum decode speed for one sequence? (Tokens per second.)
Same GPU, same model, but quantised to int4 weights (3.5 GB). What's the new ceiling for one sequence?
With 32 layers, 8 KV heads of dimension 128, and bf16 (2 bytes), how much KV cache does one token take? (KiB.)
How much KV cache does one sequence with an 8,192-token context need? (GiB.)
After loading the weights, 40 GiB of GPU memory is left for the KV cache. How many 8k-token sequences can run at once?
Why do most recent open models use grouped-query attention (GQA)?
Throughput, latency and batching
With static batching, a batch of 16 requests finishes only when the longest answer finishes. What does continuous batching change?
Users say answers take a long time to *start*, but once text appears it streams quickly. Which metric is bad, and what are the usual fixes?
Every request starts with the same 3,000-token system prompt. Which optimisation targets that directly?
In speculative decoding, a small draft model proposes 4 tokens and the large model checks them in one pass. Each draft token is accepted with probability 0.8, independently, and checking stops at the first rejection. On average, how many tokens does each pass of the large model produce? (Two decimals.)
With temperature 0 in production, the same prompt sometimes gets slightly different answers. What's the usual explanation?
Fine-tuning
A support bot must answer from a policy manual that changes every week. Fine-tune, or use RAG?
What does LoRA train?
LoRA with rank 16 on one 4,096 × 4,096 projection matrix. How many trainable parameters does that add?
As a percentage of that matrix's 16,777,216 parameters, how many does the rank-16 adapter train? (Percent, three decimals.)
What does QLoRA add on top of LoRA?
You fine-tune on 10,000 examples averaging 1,500 tokens each, for 3 epochs. How many training tokens is that, in millions?
After fine-tuning on 5,000 legal Q&A pairs, the model is great at legal questions but worse at general instructions and basic maths. What happened, and what helps?
Validation loss on your fine-tuning data keeps going down, but when people try the model, its answers are getting worse. What's the likely gap?
A fine-tuned chat model works in your training notebook but produces rambling text when served. What do you check first?
How does DPO differ from RLHF with PPO?
You can fine-tune on 1,000 examples carefully written by experts, or on 100,000 scraped Q&A pairs of mixed quality. For teaching an instruction-following style, which is usually better?