Train loss falls while validation loss rises from epoch three. What do you check, and in what order?
Validation loss bottoms out at epoch 3 and climbs steadily afterwards, while training loss keeps falling. The team's first instinct is to raise dropout to 0.7 and retrain.
Constraints: do not change the architecture yet, and make the run keep the best model rather than the last.
X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.05, shuffle=True)
scaler = StandardScaler().fit(X) # fitted on everything
X_tr, X_val = scaler.transform(X_tr), scaler.transform(X_val)
best = None
for epoch in range(50):
train_one_epoch(model, X_tr, y_tr)
val = evaluate(model, X_val, y_val)
print(epoch, val)
torch.save(model.state_dict(), "final.pt")Name what you check before adding regularization, and fix the setup.
First confirm the gap is real — a validation split big enough to trust, from the same distribution, with no leakage flattering earlier epochs. Then weigh model capacity against dataset size and re-read the rate schedule. Only then add regularization: early stopping first, then dropout and weight decay.
- ✗Reaching for stronger regularization before validating the split and ruling out leakage
- ✗Fitting the scaler on all data, leaking validation statistics into training
- ✗Saving the final-epoch weights when the best model was several epochs earlier
- →Why does a 5% validation split make an early divergence hard to trust?
- →Which regularizer would you reach for first, and why that one?
Raising dropout is premature here — the signal has not been validated, and the code has two real bugs.
1. Leakage through StandardScaler. It is fitted on all of X, so the validation mean and variance flow into training. Early epochs look better than they are, and the "divergence" is partly manufactured. The scaler must be fitted on X_tr alone.
2. A 5% split with shuffle=True. On a validation set that small the curve is noisy enough that an epoch-3 minimum may be chance. Use 15–20%, and if the data has temporal structure, split by time instead of shuffling.
3. The final epoch is saved, not the best one. Even with genuine overfitting, the run must hand back the weights from the best epoch.
X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.2, shuffle=True, stratify=y)
scaler = StandardScaler().fit(X_tr) # TRAINING portion only
X_tr, X_val = scaler.transform(X_tr), scaler.transform(X_val)
best_val, best_state, patience, bad = float("inf"), None, 5, 0
for epoch in range(50):
train_one_epoch(model, X_tr, y_tr)
val = evaluate(model, X_val, y_val)
if val < best_val:
best_val, best_state, bad = val, copy.deepcopy(model.state_dict()), 0
else:
bad += 1
if bad >= patience:
break
model.load_state_dict(best_state)
torch.save(best_state, "best.pt")Order: split size and origin → leakage → capacity against data volume → learning-rate schedule → and only then dropout, weight decay and augmentation.
