Deep Learning Training Interview Questions

A freshly initialised 10-class classifier should predict roughly uniform probabilities. What cross-entropy loss should you see on the very first batch? (Natural log.)

Training loss sits at 2.30 for a 10-class problem and doesn't move for 1,000 steps. What's your first debugging step?

Training loss keeps falling. Validation loss falls until epoch 5, then rises every epoch. What's happening, and what do you do?

Loss jumps up and down by a lot from step to step and doesn't trend down. What do you change first?

Training in fp16, the loss suddenly becomes NaN after 200 steps. What's the most likely cause?

Evaluating the same checkpoint twice on the same validation set gives 81.2% and then 79.6%. What did you forget?

The learning rate warms up linearly from 0 to 3e-4 over the first 2,000 steps. What is it at step 500?

After warmup, the learning rate follows a cosine decay from 3e-4 down to 0 over the remaining steps. What is it exactly halfway through the decay?

Why do most transformer fine-tuning recipes use AdamW rather than Adam with an L2 penalty added to the loss?

You raise the batch size from 256 to 1,024 on the same SGD setup. What's a common starting point for the learning rate?

A transformer trains well but the loss spikes sharply every few thousand steps, then slowly recovers. What's a standard fix?

With He initialisation for a ReLU layer that has 512 inputs, what is the standard deviation of the initial weights? (Three decimals.)

You train a segmentation model with batch norm, and memory limits you to 2 images per GPU. Results are poor and noisy. Why, and what do you do?

A plain 50-layer network trains worse than an 18-layer one, even on the training set. What's the usual fix?

How many parameters are in a fully connected layer from 4,096 inputs to 4,096 outputs, with a bias?

A 3×3 convolution takes 64 input channels to 128 output channels, with a bias. How many parameters?

How much GPU memory do the weights alone of a 7-billion-parameter model take in bf16? (In GB, where 1 GB is 10⁹ bytes.)

Training that 7B model with Adam in mixed precision: bf16 weights and gradients (2 bytes each), plus fp32 master weights and Adam's two moments (4 bytes each). How much memory before activations? (GB, 10⁹ bytes.)

Training runs out of memory at batch size 32. You want to keep an effective batch of 32. What do you do?

Each GPU processes a micro-batch of 8, you accumulate gradients over 16 micro-batches, and you train on 4 GPUs with data parallelism. What's the effective batch size?

You need to train on 1 billion tokens. Each GPU processes 50,000 tokens a second, and you have 8 of them. How many minutes does one pass take, ignoring overheads?

Why do most large-model training runs now use bf16 rather than fp16?

An image classifier scores 98% on validation and 81% on new photos. You find that augmented copies of the same originals appear in both the training and validation sets. What's the fix?