Imbalanced Data
Strategies for class imbalance: resampling, SMOTE, class weights, threshold selection, the anomaly-vs-classification framing and calibration after resampling.
8 questions
JuniorTheoryVery commonAt 1:1000 class imbalance, why is accuracy misleading and what must change?
At 1:1000 class imbalance, why is accuracy misleading and what must change?
A model that always predicts the majority class reaches 99.9% accuracy at 1:1000 while catching zero positives, so accuracy really measures the negatives. Three things change — the metric (PR-AUC, recall), the loss (class weights), and the decision threshold.
Common mistakes
- ✗Reporting accuracy as the headline metric on a 1:1000 problem
- ✗Assuming a high accuracy means the minority class is detected
- ✗Changing only the metric while leaving the loss and threshold untouched
Follow-up questions
- →Why does PR-AUC react to rare positives while ROC-AUC stays optimistic?
- →Which is the better default under imbalance — class weights or resampling?
JuniorTheoryCommonHow do you choose a classification threshold, and why is 0.5 arbitrary?
How do you choose a classification threshold, and why is 0.5 arbitrary?
0.5 is only the library default; it is optimal solely for a calibrated model with equal error costs. Pick the threshold on validation data from what you actually own — a point on the precision-recall curve, a cost matrix, or the reviewers' daily capacity.
Common mistakes
- ✗Treating 0.5 as an optimum rather than a library default
- ✗Tuning the threshold on the test set instead of a validation split
- ✗Picking a threshold with no cost or capacity constraint behind it
Follow-up questions
- →How does an uncalibrated score change which threshold you should pick?
- →How do you turn a review team's daily capacity into a threshold?
MiddleTheoryCommonHow do oversampling, undersampling and class weights differ in effect?
How do oversampling, undersampling and class weights differ in effect?
All three move the boundary toward the majority class — recall up, precision down. Oversampling duplicates minority rows and can overfit, undersampling discards data, weights only reweight the loss. All three inflate predicted probabilities, so scores need recalibration.
Common mistakes
- ✗Believing class weights and oversampling leave predicted probabilities valid
- ✗Treating undersampling as free because the majority rows look redundant
- ✗Reporting resampled scores as probabilities without recalibrating them
Follow-up questions
- →When is undersampling preferable to class weights on a very large dataset?
- →How do you recalibrate a model that was trained with class weights?
MiddleDebuggingCommonCV score jumps after adding SMOTE, but production quality does not move
CV score jumps after adding SMOTE, but production quality does not move
You resampled before splitting: synthetic points built from training rows land in the validation fold, so CV grades near-copies of what the model saw and inflates. Resample inside each fold, on the training part only — put SMOTE in a pipeline CV refits per fold.
Open full question →Common mistakes
- ✗Calling
fit_resampleon the full dataset before the train/validation split - ✗Assuming synthetic rows cannot leak because they are new objects
- ✗Resampling the validation or test fold to match the training balance
Follow-up questions
- →Which class keeps a resampler inside the fold in the library
imbalanced-learn? - →Would class weights instead of SMOTE have avoided this leak entirely?
MiddleTheoryCommonHow does the oversampling technique SMOTE build points, and when does it hurt?
How does the oversampling technique SMOTE build points, and when does it hurt?
SMOTE places a synthetic point between a minority object and one of its k nearest minority neighbours. It fails when those neighbours are not similar — high dimensions flatten distances, categorical features have no midpoint, and noisy borders bleed into the majority.
Common mistakes
- ✗Thinking SMOTE duplicates rows rather than interpolating between neighbours
- ✗Applying SMOTE unchanged to one-hot or high-cardinality categorical features
- ✗Assuming SMOTE still helps where high dimensionality flattens distances
Follow-up questions
- →How does the categorical variant
SMOTE-NChandle non-numeric features? - →Why can SMOTE amplify label noise that sits near the class border?
MiddleDesignOccasionalA payment platform detects account takeovers. In the labelled history there is about 1 positive per 10000 sessions — roughly 400 confirmed cases against 4 million sessions — and labels arrive weeks late, produced by manual investigation. New attack patterns appear almost every month. You must decide whether to keep the task as supervised binary classification or reframe it as anomaly detection over normal behaviour. Say what actually decides this, what each framing needs from the data, and how you would validate the choice.
A payment platform detects account takeovers. In the labelled history there is about 1 positive per 10000 sessions — roughly 400 confirmed cases against 4 million sessions — and labels arrive weeks late, produced by manual investigation. New attack patterns appear almost every month. You must decide whether to keep the task as supervised binary classification or reframe it as anomaly detection over normal behaviour. Say what actually decides this, what each framing needs from the data, and how you would validate the choice.
Label count and stability decide it, not the ratio. 400 solid positives can train a supervised model, so keep classification while attacks repeat. Switch to anomaly detection when labels are too few, too late, or cover only known attacks — validate both on one labelled sample.
Common mistakes
- ✗Deciding on the imbalance ratio alone rather than on the label supply
- ✗Assuming anomaly detection needs no labels even for validation
- ✗Ignoring that late labels only describe already-known attack patterns
Follow-up questions
- →What does the unsupervised method
Isolation Forestassume about normal data? - →How do you compare an anomaly score and a supervised score on one metric?
MiddleTheoryOccasionalHow do you recalibrate after resampling, and does the threshold move?
How do you recalibrate after resampling, and does the threshold move?
Resampling raises the training prior, so scores overstate the positive probability. Either apply the analytic prior correction back to the true base rate, or fit Platt scaling or isotonic regression on a real-ratio holdout. Re-pick the threshold — it lives on the new scale.
Common mistakes
- ✗Assuming resampling only changes ranking and leaves probabilities valid
- ✗Fitting the calibrator on the resampled sample instead of a real-ratio holdout
- ✗Keeping the old threshold after the score scale has changed
Follow-up questions
- →When does the non-parametric method
isotonic regressionoverfit a small holdout? - →Do class weights distort predicted probabilities the same way resampling does?
SeniorDesignOccasionalA medical-device maker asks you to detect a rare failure mode. Two years of telemetry contain 300 confirmed positive cases against roughly 2 million healthy device-days; each new label costs an engineer half a day of investigation, and the remaining labelling budget is 200 cases. Failures cluster by device model and by firmware version, and the regulator will read the report. Design the labelling and modelling approach — how you spend the remaining budget, whether you lean on transfer learning, weak supervision or an anomaly-detection model, and what you validate on so the number you report is believable.
A medical-device maker asks you to detect a rare failure mode. Two years of telemetry contain 300 confirmed positive cases against roughly 2 million healthy device-days; each new label costs an engineer half a day of investigation, and the remaining labelling budget is 200 cases. Failures cluster by device model and by firmware version, and the regulator will read the report. Design the labelling and modelling approach — how you spend the remaining budget, whether you lean on transfer learning, weak supervision or an anomaly-detection model, and what you validate on so the number you report is believable.
Spend the remaining labels by active learning on uncertain cases rather than at random, and add weak labels from service tickets. Train a transfer or strongly regularised model; anomaly detection only supplements it. Validate on a real-ratio holdout grouped by device and firmware, reporting PR-AUC.
Common mistakes
- ✗Spending a scarce label budget on a uniformly random sample
- ✗Splitting randomly across device-days so one device leaks between the parts
- ✗Reporting ROC-AUC, which stays optimistic when positives are this rare
Follow-up questions
- →How does the labelling strategy
active learningchoose which cases to label? - →Why must the split be grouped by device rather than by individual device-day?
