A CNN at 98% on curated validation collapses on user phone photos — find and fix the cause.
A classifier scores 98% on the curated validation split but collapses on real user phone photos.
Constraints:
- the trained weights are fine; retraining from scratch is not the answer
- the snippet below is the whole evaluation and serving pipeline
- phone photos arrive rotated by EXIF, with drifting white balance and JPEG compression
train_tf = Compose([Resize((224, 224)), ToTensor(),
Normalize(MEAN_IMAGENET, STD_IMAGENET)])
infer_tf = Compose([Resize((256, 256)), CenterCrop(224), ToTensor()])
val_ds = ImageFolder("data/studio_shots", transform=train_tf)Find the causes and state the fix.
Validation is drawn only from studio shots, so it never sees the deployment distribution, and the inference transform skips the normalization used in training. Fix both — match the serving transform to training and validate on real phone photos.
- ✗Calling it overfitting when the evaluation set itself is unrepresentative
- ✗Thinking normalization only matters during training
- ✗Retuning a threshold on a split that shares the wrong distribution
- →How would you monitor input distribution shift in production?
- →Why must the serving transform be identical to the training transform?
There are two independent causes, and both are visible in the snippet.
First, infer_tf has no Normalize. The model was trained on inputs scaled to the ImageNet statistics but at serving receives raw ToTensor values in 0–1. That alone wrecks predictions. The resize path also diverges: Resize((224, 224)) in training against Resize((256, 256)) + CenterCrop(224) at inference.
Second, val_ds is built only from data/studio_shots. Validation measures the same curated photography the model was trained on, so 98% says nothing about production.
infer_tf = Compose([Resize((224, 224)), ToTensor(),
Normalize(MEAN_IMAGENET, STD_IMAGENET)])
val_ds = ImageFolder("data/phone_holdout", transform=infer_tf)Then: apply EXIF rotation on load, and add augmentations that mirror deployment conditions — JPEG compression, white-balance jitter, mild blur. That narrows the distribution shift rather than merely measuring it.
