Generalization & Validation
Bias–variance, the curse of dimensionality, double descent, train/validation/test discipline, cross-validation schemes and the four faces of data leakage.
11 questions
JuniorTheoryVery commonWhat is k-fold cross-validation, and when do you need the stratified variant?
What is k-fold cross-validation, and when do you need the stratified variant?
The data is cut into k equal folds; the model is refit k times, each run holding one fold out for scoring, and the k scores are averaged. Every row is scored once, so the estimate is steadier than one split. Stratified folds keep class shares equal.
Common mistakes
- ✗Averaging training-fold scores instead of the held-out fold scores
- ✗Using plain k-fold on a rare-class problem and getting folds without positives
- ✗Believing cross-validation removes the need for a final untouched test set
Follow-up questions
- →How does the choice of k trade estimate variance against compute cost?
- →Why does plain k-fold break when several rows belong to the same user?
JuniorTheoryVery commonWhat are the train, validation and test sets for, and in which order do you split and fit?
What are the train, validation and test sets for, and in which order do you split and fit?
Train fits model parameters, validation picks hyperparameters and early stopping, test gives one unbiased estimate at the end. Split first, before any fitting; shares are near 60/20/20. Every transform is fit on train alone.
Common mistakes
- ✗Tuning hyperparameters on the test set and reporting that same score
- ✗Fitting scalers or encoders on the full table before splitting
- ✗Re-splitting with a new random seed after every disappointing run
Follow-up questions
- →When would you replace a fixed validation split with k-fold cross-validation?
- →How do the split shares change when you have ten million labelled rows?
JuniorTheoryCommonWhat is the bias-variance trade-off, and where do a depth-2 and a depth-30 tree sit on it?
What is the bias-variance trade-off, and where do a depth-2 and a depth-30 tree sit on it?
Bias is error from a model too rigid to fit the signal; variance is how much the fit swings with the training sample drawn. A depth-2 tree is high-bias and underfits; a depth-30 tree is high-variance and memorises noise.
Common mistakes
- ✗Calling high training error variance rather than bias
- ✗Assuming a deeper tree always generalises better than a shallow one
- ✗Treating variance as a property of the labels instead of the fitted model
Follow-up questions
- →How does the ensembling method
baggingcut variance without moving bias much? - →Which term does an L2 penalty (ridge regularization) move, and in which direction?
MiddleTheoryCommonWhy do kNN and k-means degrade as dimension grows, and what does that mean for embeddings?
Why do kNN and k-means degrade as dimension grows, and what does that mean for embeddings?
Volume grows exponentially with dimension, so a sample turns sparse and distances concentrate: nearest and farthest neighbour become nearly equidistant, and a neighbourhood is no longer local. Embeddings survive because data sits on a low-dimensional manifold.
Common mistakes
- ✗Reading the curse as a compute problem rather than a geometry problem
- ✗Assuming a 768-dimensional embedding is as sparse as 768 raw one-hot columns
- ✗Expecting more features to keep helping a distance-based model indefinitely
Follow-up questions
- →Why does cosine similarity often behave better than Euclidean distance on embeddings?
- →How do approximate nearest-neighbour indexes such as
HNSWchange the picture?
MiddleTheoryCommonGroup, stratified and nested CV — when does plain k-fold overestimate model quality?
Group, stratified and nested CV — when does plain k-fold overestimate model quality?
Plain k-fold assumes rows are independent. When one user or near-duplicate spans folds, the model recognises the entity rather than the pattern, so grouped folds are needed. Stratified folds hold class shares; nested CV keeps tuning in the inner loop.
Common mistakes
- ✗Shuffling rows that share an entity and calling the result independent folds
- ✗Reporting a tuned cross-validation score without an outer loop around the search
- ✗Leaving near-duplicate rows in the data and trusting the averaged fold score
Follow-up questions
- →How do you build folds when data is grouped by user and also ordered in time?
- →What does the gap between an inner and an outer nested-CV score tell you?
MiddleDebuggingCommonCV says 0.92 and production says 0.71 — which leakage channels explain the gap?
CV says 0.92 and production says 0.71 — which leakage channels explain the gap?
Four channels: target leakage from a field filled only after the outcome, contamination from duplicate rows spanning folds, group leakage from one user on both sides, and temporal leakage from shuffling time-ordered data. Fixing one alone drops the score.
Open full question →Common mistakes
- ✗Blaming drift for a gap that a post-outcome feature fully explains
- ✗Assuming a shuffled split is safe on data that is ordered in time
- ✗Trusting an averaged fold score while duplicates span the folds
Follow-up questions
- →How would you rank the four channels by the damage each one usually does?
- →What audit would catch a post-outcome feature before any model is trained?
MiddleDebuggingCommonA scaler, imputer and selector are fit before the split — what exactly leaks, and how?
A scaler, imputer and selector are fit before the split — what exactly leaks, and how?
Fitting them on the whole table pushes held-out rows into the training statistics: the scaler carries held-out means and variances, the imputer fills from them, and selection reads held-out labels, which leaks worst. A pipeline refits each step per training fold.
Open full question →Common mistakes
- ✗Calling unsupervised transforms leak-free because they never read the target
- ✗Scaling once up front for convenience and splitting the array afterwards
- ✗Selecting features on the full table and then cross-validating the survivors
Follow-up questions
- →Why does target encoding need an inner split even inside a correct pipeline?
- →How do you apply the same discipline to a resampling step such as
SMOTE?
MiddleTheoryCommonWhy touch the test set once, and what does validation overfitting look like after 200 runs?
Why touch the test set once, and what does validation overfitting look like after 200 runs?
Every decision taken on a score leaks that set into the model, so a set reused for selection stops being unbiased. After 200 runs the best validation score is the maximum of 200 noisy draws, inflated by selection; the tell is a widening validation-to-test gap.
Common mistakes
- ✗Treating the maximum over many runs as an unbiased estimate of quality
- ✗Rescoring the test set between experiments to monitor progress
- ✗Assuming cross-validation alone removes hyperparameter selection bias
Follow-up questions
- →How does nested cross-validation restore an honest estimate under heavy tuning?
- →What changes if the search is a Bayesian optimizer rather than a random search?
SeniorTheoryCommonA model that holds up on random 80/20 splits fails on next month — build an honest protocol
A model that holds up on random 80/20 splits fails on next month — build an honest protocol
A random split scores interpolation between nearby days; deployment is extrapolation into an unseen period. Use rolling-origin splits that train on the past and score a strictly later window with a gap for label delay, then run adversarial validation.
Common mistakes
- ✗Reading a random-split score as an estimate of next month's performance
- ✗Adding a raw timestamp feature and expecting the model to extrapolate the trend
- ✗Blaming variance and re-running seeds when the real cause is distribution shift
Follow-up questions
- →How do you set the gap width when label delay and drift speed disagree?
- →What follow-up action does a high adversarial-validation AUC justify?
MiddleTheoryOccasionalDecompose expected test error into bias, variance and noise — which change moves which term?
Decompose expected test error into bias, variance and noise — which change moves which term?
Expected squared error at a point splits into squared bias, variance and irreducible noise. Capacity and boosting cut bias; averaging, regularization and more rows cut variance. Noise is a floor set by the data, moved only by better features or labels.
Common mistakes
- ✗Believing a better model or longer search can reduce irreducible noise
- ✗Swapping the levers — thinking capacity cuts variance and bagging cuts bias
- ✗Reading the three terms as the three data splits rather than error sources
Follow-up questions
- →How would you estimate the noise floor empirically from repeated labels?
- →Where does the double-descent curve sit relative to this decomposition?
SeniorDesignOccasionalYou own a card-fraud model. A transaction is labelled fraudulent only when the cardholder disputes it, and disputes arrive up to 60 days after the transaction, so the most recent two months of data carry labels that are still maturing — an unreported fraud currently looks exactly like an honest payment. Fraud rings also adapt within weeks, so the patterns that dominate this quarter were largely absent last quarter. The base rate is roughly 0.2% positives, one cardholder generates many transactions, and the model is retrained and redeployed every two weeks. Design the validation scheme you would use to decide whether a candidate model is fit to ship — the split, the fold construction, what you hold back and what you report — and be explicit about what this scheme costs you in data, in freshness and in the honesty of the estimate.
You own a card-fraud model. A transaction is labelled fraudulent only when the cardholder disputes it, and disputes arrive up to 60 days after the transaction, so the most recent two months of data carry labels that are still maturing — an unreported fraud currently looks exactly like an honest payment. Fraud rings also adapt within weeks, so the patterns that dominate this quarter were largely absent last quarter. The base rate is roughly 0.2% positives, one cardholder generates many transactions, and the model is retrained and redeployed every two weeks. Design the validation scheme you would use to decide whether a candidate model is fit to ship — the split, the fold construction, what you hold back and what you report — and be explicit about what this scheme costs you in data, in freshness and in the honesty of the estimate.
Split forward in time with a 60-day embargo before the validation window, so no immature label is scored. Roll that origin across several windows and fold by cardholder. Report precision at a fixed review capacity; the cost is a stale estimate and unused fresh data.
Common mistakes
- ✗Validating on the freshest window whose labels have not finished maturing
- ✗Treating a not-yet-disputed transaction as a confirmed honest label
- ✗Judging a fast-drifting model on one window instead of several rolling ones
Follow-up questions
- →How would you monitor once deployed, given the score only settles 60 days later?
- →What would adversarial validation add about the shift between train and deployment?
