Deep Learning Training

ReLU is the cheap default for hidden layers; sigmoid and tanh saturate and belong on outputs and gates. LeakyReLU and GELU keep a nonzero slope for negative inputs, GELU being the transformer default. A dying ReLU sits on negative input — zero out, zero gradient, no recovery.

Adam keeps running estimates of the first and second gradient moments per weight and divides the step by the root of the second, so every weight gets its own effective step plus momentum. That converges fast with little tuning, but the rescaling tends toward sharper minima, while tuned SGD with momentum finds flatter ones.

One forward pass caches each layer's activations, then the loss gradient travels backward and every layer multiplies it by its local Jacobian. An early layer's gradient is thus a long product of Jacobians — norms below or above one make it decay or explode exponentially with depth.

BatchNorm in training normalizes each mini-batch by its own mean/variance and updates running statistics; at inference it uses stored ones, so output is deterministic. LayerNorm normalizes per sample over features, without batch statistics, alike in both phases.

You watch the validation metric, not the training loss, and stop once it stops improving, restoring the weights saved at the best epoch. Patience is how many epochs without improvement you tolerate, so a noisy dip does not end a run early. It regularizes by capping the capacity the model gets to use.

A batch is the samples behind one gradient step, an iteration is that step, and an epoch is one full pass over the training set — so iterations per epoch equal dataset size divided by batch size. Doubling the batch halves iterations and lowers gradient noise, so you scale the learning rate up, with warmup.

Too high and the loss spikes, oscillates or diverges to NaN and never settles. Too low and it falls smoothly but far too slowly, flattening above what the model can reach. A range test finds a good value in one run: sweep the rate up exponentially and take the value just before the loss turns upward.

The main ones are dropout (randomly zeroing neurons during training) and L2 weight decay. A normalization family — Batch/Layer/Instance/Group Norm — also stabilizes and mildly regularizes training. Data augmentation and early stopping regularize too.

Vanishing means the gradient shrinks toward zero going back, so early layers barely update and the loss stalls. Exploding means it grows without bound, giving huge weight jumps and a loss that spikes or turns NaN. Deep plain stacks and RNNs without residual connections suffer most.

You clip when gradients occasionally explode — recurrent nets, transformers, reinforcement learning. Clipping by norm rescales the whole gradient vector once its global norm passes a threshold, so direction is preserved and only step length shrinks. Clipping by value truncates each coordinate on its own, changing the direction.

A bigger batch gives a less noisy gradient, and that noise is part of what regularizes SGD. Without it the optimizer settles into sharp minima, whose loss climbs steeply under small shifts in weights or data, while flat minima tolerate them and transfer better. Scaling the rate with warmup recovers much of it.

Go cheapest first: try to overfit a single batch — if that fails the bug is in the code, not the data. Then check labels line up with inputs, the loss matches the task, inputs are normalized, and the rate is not wildly off. Last, look for dead ReLUs and parameters the optimizer never got.

Step drops the rate by a factor at fixed epochs; cosine decays it smoothly toward zero; one-cycle ramps up then back down, often with momentum inverted. Warmup raises the rate from near zero over the first steps because Adam's second moments are still noisy, and a full step would destabilize the residual stack.

Exploding gradients — clip by global norm. An fp16 overflow — use a loss scaler or bf16. A hand-written log(0) — use the fused logits loss or add an epsilon. A rate that is too high — lower it or add warmup. NaN already in the batch — assert on inputs. Find the first bad step and log gradient norms.

First confirm the gap is real — a validation split big enough to trust, from the same distribution, with no leakage flattering earlier epochs. Then weigh model capacity against dataset size and re-read the rate schedule. Only then add regularization: early stopping first, then dropout and weight decay.

All-zero weights make every unit in a layer compute the same output and get the same gradient, so they stay identical and symmetry is never broken. Xavier scales initial variance by fan-in and fan-out to hold signal variance steady through tanh and sigmoid; He doubles it for ReLU, which zeroes half its inputs.

At inference NO neurons are dropped — the full network runs. To match the training activation magnitude, standard dropout multiplies activations by (1−p) at inference (p = drop probability). The common alternative, inverted dropout, scales kept activations by 1/(1−p) DURING training and does nothing at inference.

Use gradient accumulation. Run forward + loss.backward() on each micro-batch of 32, letting gradients sum in the .grad buffers, and call optimizer.step() + optimizer.zero_grad() only once every 2048/32 = 64 micro-batches. With mean-reduced losses, scale each loss by 1/64.

Model the neighborhood as an unordered SET (or graph) and use a permutation-invariant architecture: Deep Sets, attention pooling, or a Graph Attention Network (GAT). Embed each neighbor, then aggregate symmetrically (sum/mean or attention) — making the output order-invariant and accepting any neighbor count.

As capacity grows, test error follows the classical U up to the interpolation threshold, where the model just barely fits the training set and error peaks. Past it error falls again: among the many zero-training-error solutions the optimizer picks smoother, lower-norm ones. The U describes only the underparameterized regime.

Weights, optimizer state (Adam holds two moments), gradients and the activations cached for backward dominate; activations scale with batch times depth. Rank levers by cost — mixed precision, then gradient accumulation, activation checkpointing, ZeRO sharding, CPU offload.