Deep Learning Training
Training a neural network is not just "compute a gradient and take a step". The real difficulty is in mode details: regularization fights overfitting, dropout and normalization layers behave differently at training and inference, gradient accumulation lets you train with large batches on small memory, and permutation-invariant architectures handle unordered sets.
The traps here are subtle and almost always about the "train vs inference" phase. Dropout at inference does not drop neurons. BatchNorm at inference uses stored statistics, not the current batch. LayerNorm does not depend on the batch at all. Confuse any of these and you get a model that behaves differently at test time than in training. The full map is below.
Topic map
- Neural-network regularization — dropout, L2 weight decay, normalization, augmentation, and early stopping as ways to fight overfitting.
- Dropout — randomly zeroing neurons during training and matching the activation scale at inference.
- Normalization layers — the Batch/Layer/Instance/Group Norm family and which axis each normalizes over.
- BatchNorm — train vs inference — batch statistics versus running statistics and the contrast with LayerNorm.
- Gradient accumulation — emulating a large batch by summing gradients over micro-batches.
- Set architectures — permutation-invariant networks (Deep Sets, GAT) for unordered, variable-length inputs.
Common traps
| Mistake | Consequence |
|---|---|
| Leaving dropout active at inference | Non-deterministic, jittering predictions |
| Forgetting to rescale activations after dropout | The activation magnitude at test time does not match training |
| Thinking BatchNorm uses the current batch at inference | The output depends on batch neighbours — wrong and unstable |
| Confusing LayerNorm with BatchNorm on the normalization axis | LayerNorm normalizes per sample over features, the batch is irrelevant |
| Calling zero_grad after every micro-batch | The accumulation is wiped — a large batch is not emulated |
| Using an RNN or concatenation for an unordered set | The result depends on neighbour order, which it must not |
Interview relevance
The topic is asked to check whether you understand the train-vs-inference phase difference. A candidate who, unprompted, says "at inference dropout drops nothing, and activations are multiplied by (1−p)" immediately shows depth.
Typical checks:
- What regularization methods exist and why a bigger model is not regularization.
- What dropout does in each phase and why rescaling is needed.
- How BatchNorm differs from LayerNorm and why Transformers use LayerNorm.
- How to train "with a large batch" when memory is short.
Common wrong answer: "BatchNorm computes statistics on the current batch at inference too". In fact at inference it uses stored running statistics, so the output is deterministic and independent of batch composition.