A fine-tune on 500 examples now parrots the training set and lost general ability — diagnose and fix it.
A 7B chat model was fully fine-tuned on 500 hand-written support replies. It now reproduces training answers almost verbatim and refuses unrelated tasks it handled before the tune.
Constraints: the 500 examples are all the labelled data you have this month, and one GPU is available. You may change the recipe but you may not swap the base model.
run=support-sft-v3 base=7B-chat examples=500 method=full lr=2e-5 epochs=8
epoch 1 train_loss=1.42 heldout_loss=1.51 general_bench=64.1
epoch 2 train_loss=0.83 heldout_loss=1.49 general_bench=61.7
epoch 4 train_loss=0.21 heldout_loss=1.88 general_bench=52.3
epoch 8 train_loss=0.02 heldout_loss=2.640 general_bench=38.9Name the ordered checks, say which line already localises the fault, and give the fix.
It is overfitting plus forgetting on a tiny set — train loss near zero while held-out loss climbs after the first epoch. Cut to one or two epochs with early stopping, lower the learning rate, switch to LoRA, and diversify the data with replay examples.
- ✗Reading a near-zero training loss as success instead of memorisation
- ✗Watching only the target metric while the general benchmark quietly collapses
- ✗Adding epochs or learning rate to a 500-example set instead of shrinking the update
- →How would you size a held-out split when only 500 labelled examples exist?
- →Why does a modest LoRA rank often beat a full fine-tune on data this small?
The epoch 2 line already localises it: held-out loss stops falling and turns up while general_bench starts sliding — everything after that is memorisation, not learning. Work from the curves to the recipe, never from the weights.
- Compare train and held-out loss per epoch: they diverge at epoch 2, the classic overfit shape.
- Read
general_benchseparately from the target metric: 64 down to 39 is forgetting. - Check the size of the update: 8 epochs at
lr=2e-5over 500 examples is a lot of passes. - Check data diversity: 500 similar replies are trivially memorised verbatim.
fix: method=lora rank=16 lr=1e-5 epochs=2 early_stop=heldout_loss replay=20%
epoch 1 train_loss=1.39 heldout_loss=1.44 general_bench=63.8
epoch 2 train_loss=1.11 heldout_loss=1.40 general_bench=63.2Early stopping on held-out loss, a modest LoRA rank and mixed-in replay data hold general ability steady while target quality still improves, without parroting the training set.
