Training loss sits flat and high from epoch one and will not move. How do you diagnose it?
A training run reports the same loss value every epoch and accuracy never leaves chance level. The data loader and the model itself were both verified in isolation and are known to be correct.
Constraints: change only this training setup, keep the architecture, and name the check that fires first.
model = MLP(in_dim=64, hidden=128, out_dim=10)
opt = torch.optim.SGD(model.parameters(), lr=1e-7)
loss_fn = torch.nn.CrossEntropyLoss()
for epoch in range(20):
for x, y in loader:
opt.zero_grad()
logits = model(x)
loss = loss_fn(torch.softmax(logits, dim=-1), y)
loss.backward()
opt.step()Find the reasons the loss cannot move and fix them.
Go cheapest first: try to overfit a single batch — if that fails the bug is in the code, not the data. Then check labels line up with inputs, the loss matches the task, inputs are normalized, and the rate is not wildly off. Last, look for dead ReLUs and parameters the optimizer never got.
- ✗Reading a flat loss as convergence instead of a broken training step
- ✗Adding capacity before verifying the model can overfit a single batch
- ✗Feeding softmax probabilities into a loss that already expects raw logits
- →Why is overfitting a single batch the cheapest possible first check?
- →How would you confirm gradients actually reach the earliest layer?
Two things are broken here, and both are visible without running anything.
1. Double softmax. PyTorch's CrossEntropyLoss applies log_softmax to raw logits itself. Feeding it an already-normalized torch.softmax(logits) runs softmax twice: the distribution goes nearly uniform, the gradient becomes tiny, and the loss freezes near ln(10) ≈ 2.30.
2. Learning rate 1e-7. Even with the loss fixed, the step is so small that 20 epochs move the weights by nothing observable.
model = MLP(in_dim=64, hidden=128, out_dim=10)
opt = torch.optim.SGD(model.parameters(), lr=1e-2, momentum=0.9)
loss_fn = torch.nn.CrossEntropyLoss()
# Check #1 — the cheapest one: can the model overfit a SINGLE batch?
x, y = next(iter(loader))
for _ in range(200):
opt.zero_grad()
loss = loss_fn(model(x), y) # raw logits, no softmax
loss.backward()
opt.step()
assert loss.item() < 0.01, "code is broken: could not learn one batch"
for epoch in range(20):
for x, y in loader:
opt.zero_grad()
loss = loss_fn(model(x), y)
loss.backward()
opt.step()Order of checks: overfit one batch → labels and loss match the task → input normalization → learning rate → dead ReLUs and parameters that never reached the optimizer.
