Quality Metrics
Precision, recall and accuracy, the precision/recall threshold trade-off, AUC-ROC and its rank-based properties, and regression metrics under outliers.
17 questions
JuniorTheoryVery commonWhat is AUC-ROC?
What is AUC-ROC?
AUC-ROC is the area under the ROC curve — true-positive rate vs false-positive rate across thresholds. It equals the probability a random positive outranks a random negative, i.e. the fraction of correctly ordered pairs. Threshold-independent.
Common mistakes
- ✗Confusing AUC-ROC with accuracy at a fixed threshold
- ✗Not knowing the pairwise-ranking interpretation
- ✗Thinking AUC depends on the chosen decision threshold
Follow-up questions
- →Why is AUC unchanged if you duplicate every positive example?
- →Can a high AUC still give zero positives in the top-K predictions?
JuniorTheoryVery commonHow do you read TPR, FPR and specificity off a confusion matrix, and when is specificity the metric you defend?
How do you read TPR, FPR and specificity off a confusion matrix, and when is specificity the metric you defend?
TPR, or recall, is TP/(TP+FN) — the share of real positives caught. Specificity is TN/(TN+FP), the share of negatives correctly cleared, and FPR is one minus specificity. Defend specificity in mass screening, where false alerts flood a fixed review capacity.
Common mistakes
- ✗Confusing specificity with precision — they use different denominators
- ✗Computing FPR from the positive row instead of the negative row
- ✗Reporting only recall on a screening task where false alarms consume the budget
Follow-up questions
- →Which confusion-matrix row does each of these three rates come from?
- →How does class prevalence affect specificity compared with precision?
JuniorTheoryVery commonDefine precision, recall, and accuracy with their formulas.
Define precision, recall, and accuracy with their formulas.
Precision = TP/(TP+FP) — of predicted positives, the fraction truly positive. Recall = TP/(TP+FN) — of actual positives, the fraction found. Accuracy = (TP+TN)/total — the fraction correct. Accuracy misleads under class imbalance.
Common mistakes
- ✗Swapping the denominators of precision and recall (FP vs FN)
- ✗Reporting accuracy on an imbalanced set and calling the model good
- ✗Conflating the Russian «tochnost» (precision here) with accuracy
Follow-up questions
- →When does accuracy give a misleadingly high number?
- →Which metric matters more when false positives are very costly?
JuniorTheoryCommonWhat are Type I and Type II errors in ML, and which task makes each expensive?
What are Type I and Type II errors in ML, and which task makes each expensive?
A Type I error is a false positive — flagging a negative case; a Type II error is a false negative — missing a real positive. Precision guards the first, recall the second. False positives are expensive in spam filtering, false negatives in cancer screening.
Common mistakes
- ✗Swapping the two: calling a missed positive a Type I error
- ✗Assuming both errors are equally costly and defaulting to accuracy
- ✗Forgetting that the decision threshold trades one error type against the other
Follow-up questions
- →Which confusion-matrix cell holds each error type?
- →How does moving the decision threshold shift the balance between them?
JuniorTheoryCommonWhat is F1, and when is F0.5 or F2 the more honest choice?
What is F1, and when is F0.5 or F2 the more honest choice?
F1 is the harmonic mean of precision and recall, so it stays low unless both are decent. F-beta generalizes it — beta weights recall relative to precision. Pick F2 when a missed positive costs more, and F0.5 when a false alarm is the expensive error.
Common mistakes
- ✗Calling F1 an arithmetic mean, so a high precision hides a near-zero recall
- ✗Inverting the beta direction — thinking beta above 1 favours precision
- ✗Reporting F1 by default when the task's two error costs are clearly asymmetric
Follow-up questions
- →Why does the harmonic mean punish an imbalanced precision/recall pair?
- →What does F-beta ignore that a full cost matrix would capture?
JuniorTheoryCommonWhat do RMSE, MAE and MAPE each penalize, and when does MAPE break down?
What do RMSE, MAE and MAPE each penalize, and when does MAPE break down?
MAE averages absolute errors and treats every miss equally. RMSE squares before averaging, so large errors dominate — choose it when a big miss costs disproportionately more. MAPE divides by the actual, so it explodes near zero and is asymmetric.
Common mistakes
- ✗Believing RMSE and MAE always rank models the same way
- ✗Using MAPE on series with zeros or near-zero actuals
- ✗Missing that MAPE is asymmetric — it under-penalizes over-prediction
Follow-up questions
- →Why is RMSE never smaller than MAE on the same errors?
- →What alternative would you report instead of MAPE on intermittent demand?
MiddleTheoryCommonWhat is a reliability diagram, and how do calibration methods Platt scaling and isotonic regression differ?
What is a reliability diagram, and how do calibration methods Platt scaling and isotonic regression differ?
A reliability diagram bins predictions by score and plots each bin's observed positive rate against its mean score; a calibrated model sits on the diagonal. Platt scaling fits a sigmoid — parametric, safe on small data. Isotonic regression is flexible but overfits.
Common mistakes
- ✗Confusing a reliability diagram with a precision-recall or ROC curve
- ✗Swapping which method is parametric — Platt is the sigmoid, isotonic is not
- ✗Applying isotonic regression on a few hundred validation points and overfitting
Follow-up questions
- →How does bin count change what a reliability diagram appears to show?
- →Why must calibration be fitted on data the classifier never trained on?
MiddleTheoryCommonWhat do log-loss and Brier score capture that accuracy cannot, and why is log-loss harsher?
What do log-loss and Brier score capture that accuracy cannot, and why is log-loss harsher?
Both score the predicted probability rather than the thresholded label, so they reward honest confidence and expose miscalibration accuracy cannot see. Brier is the mean squared error of probabilities — bounded and gentle. Log-loss instead takes a logarithm.
Common mistakes
- ✗Treating them as accuracy variants that still threshold the probability
- ✗Inverting the boundedness — thinking log-loss saturates and Brier explodes
- ✗Assuming a monotone rescaling of scores leaves either metric unchanged
Follow-up questions
- →What does a predicted probability of exactly zero on a positive case do to log-loss?
- →Which of the two would you optimize when downstream code consumes the probability?
MiddleTheoryCommonAt a 1:1000 positive rate, what does precision-recall metric PR-AUC show that ROC-AUC hides?
At a 1:1000 positive rate, what does precision-recall metric PR-AUC show that ROC-AUC hides?
ROC-AUC is built on FPR, whose denominator is the enormous negative class, so thousands of false alerts barely move it and the model looks excellent. PR-AUC uses precision, whose denominator is the flagged set, so it collapses once most alerts are wrong.
Common mistakes
- ✗Shipping on a high ROC-AUC without checking precision at the operating point
- ✗Assuming both curves react the same way to class prevalence
- ✗Believing resampling makes ROC-AUC informative again at extreme imbalance
Follow-up questions
- →What does the PR curve's baseline equal at a 1:1000 positive rate?
- →Which denominator makes precision sensitive to prevalence while FPR is not?
MiddleDesignCommonA moderation model flags banned contacts in listing text. You set two thresholds — one to auto-reject a listing, one to route it to a fixed-capacity human moderation team. For each threshold, is precision or recall more important, and why? Spell out the trade-off and what bounds the manual-review threshold.
A moderation model flags banned contacts in listing text. You set two thresholds — one to auto-reject a listing, one to route it to a fixed-capacity human moderation team. For each threshold, is precision or recall more important, and why? Spell out the trade-off and what bounds the manual-review threshold.
Auto-reject → prioritize precision: a wrong auto-rejection harms a real user and floods support, so false positives must be rare. Manual review → prioritize recall, bounded by the routed volume moderators can handle.
Common mistakes
- ✗Optimizing recall for the auto-action threshold, causing wrongful auto-rejects
- ✗Ignoring the human-throughput ceiling that bounds the manual-review threshold
- ✗Treating both thresholds identically instead of by their different costs
Follow-up questions
- →What goes wrong if the manual-review recall is pushed past human capacity?
- →How would you re-tune the thresholds if moderator headcount doubled?
JuniorTheoryOccasionalWhat is R-squared, can it be negative, and what does a negative value tell you?
What is R-squared, can it be negative, and what does a negative value tell you?
R-squared is the share of target variance the model explains — one minus residual sum of squares over the total sum of squares around the mean. Yes, it goes negative on held-out data when the model predicts worse than always outputting the training mean.
Common mistakes
- ✗Claiming R-squared is bounded below by zero on any data
- ✗Reading it as a squared correlation rather than a comparison with the mean baseline
- ✗Ignoring that the baseline is the training mean, so it can lose on a shifted test set
Follow-up questions
- →Which baseline model corresponds exactly to an R-squared of zero?
- →How does adjusted R-squared change the picture when features are added?
MiddleTheoryOccasionalTwo models share an identical ROC-AUC but have very different PR curves — how, and which do you ship?
Two models share an identical ROC-AUC but have very different PR curves — how, and which do you ship?
ROC-AUC sums a whole ranking into one area, and different orderings integrate to the same value. One model may keep its errors at the very top of the ranking while the other spreads them lower, so precision at low recall differs sharply. Ship the better one there.
Common mistakes
- ✗Treating ROC-AUC as a complete description of the ranking
- ✗Choosing between models on a global area instead of the operating point
- ✗Explaining the gap as a calibration artefact rather than a ranking difference
Follow-up questions
- →Where in the ranking do errors hurt precision most, and why?
- →Which single number would you report instead of a whole-curve area?
MiddleTheoryOccasionalOn a multiclass task, which of macro, micro and weighted F1 hides a completely failing rare class?
On a multiclass task, which of macro, micro and weighted F1 hides a completely failing rare class?
Micro F1 pools all samples into one count, so frequent classes dominate and a dead rare class is invisible. Weighted F1 hides it the same way, weighting by support. Macro F1 averages per-class scores equally, so a failing rare class drags the number down.
Common mistakes
- ✗Reporting weighted F1 on an imbalanced multiclass task and calling it fair
- ✗Believing macro averaging dilutes rather than exposes a rare-class failure
- ✗Assuming micro F1 differs from overall accuracy in single-label settings
Follow-up questions
- →What does micro F1 reduce to when every sample carries exactly one label?
- →Which averaging would you pick if the rare class is also the costly one?
MiddleTheoryOccasionalWith outliers in a regression task, would you optimize MAE or MSE, and what is the trade-off?
With outliers in a regression task, would you optimize MAE or MSE, and what is the trade-off?
Prefer MAE. MSE squares each error, so outliers dominate the loss; MAE penalizes linearly and is more robust. Huber loss is a middle ground — quadratic near zero, linear far out. MAE is non-differentiable at 0, slowing training.
Common mistakes
- ✗Choosing MSE for outlier-heavy data because it 'pays more attention' to big errors
- ✗Not mentioning Huber loss as a robust compromise
- ✗Forgetting MAE's non-differentiability at 0 as its main practical cost
Follow-up questions
- →How does Huber loss interpolate between MAE and MSE?
- →What does MAE's non-differentiability at 0 mean for the optimizer?
SeniorTheoryOccasionalYou measured AUC-ROC = 0.92. How does it change if you duplicate each positive 7× and each negative 4×?
You measured AUC-ROC = 0.92. How does it change if you duplicate each positive 7× and each negative 4×?
It stays 0.92. AUC is the fraction of correctly-ordered positive/negative pairs. Duplicating ×7 and ×4 scales both that count and the total pair count by 7·4, so the ratio is unchanged. AUC is rank-based.
Common mistakes
- ✗Assuming more positive copies must raise AUC
- ✗Forgetting both numerator and denominator scale by the same factor
- ✗Not recognizing AUC as rank-based and resampling-invariant
Follow-up questions
- →Which other metrics would change under the same duplication?
- →Does this invariance hold for PR-AUC as well, and why or why not?
SeniorTheoryOccasionalA classifier scores 1M listings at AUC > 0.99. Review the top-1000 by score — could all 1000 be negatives?
A classifier scores 1M listings at AUC > 0.99. Review the top-1000 by score — could all 1000 be negatives?
Yes. AUC measures global ranking, not the top-K. Under extreme imbalance a ranking of [990k negatives, 1k positives, 9k negatives] still gives AUC > 0.99, yet the top 1000 are all negatives. AUC says nothing about precision@K.
Common mistakes
- ✗Equating a high AUC with high precision@K
- ✗Ignoring how extreme imbalance decouples global ranking from the top slice
- ✗Believing AUC near 1.0 guarantees a clean top-K
Follow-up questions
- →Which metric would you report instead to control the top-K screen?
- →How would you construct a concrete ranking that exhibits this failure?
SeniorDesignOccasionalLeadership wants ONE number to judge a fraud model. A missed fraud costs 200 dollars on average; a false alert costs 4 dollars of manual review plus some customer friction. The review team clears at most 3000 alerts a day out of 2 million transactions, and the model emits a score, not a label. Construct the single decision metric you would report. Explain how the cost matrix, the fixed review capacity and the shape of the score distribution each enter it, why plain F1 or ROC-AUC would not answer the question, and what you would still monitor alongside that one number so it does not quietly rot.
Leadership wants ONE number to judge a fraud model. A missed fraud costs 200 dollars on average; a false alert costs 4 dollars of manual review plus some customer friction. The review team clears at most 3000 alerts a day out of 2 million transactions, and the model emits a score, not a label. Construct the single decision metric you would report. Explain how the cost matrix, the fixed review capacity and the shape of the score distribution each enter it, why plain F1 or ROC-AUC would not answer the question, and what you would still monitor alongside that one number so it does not quietly rot.
Report expected daily net saving at the capacity-feasible threshold — the score cut whose alert volume fills the 3000 slots, valued at 200 per fraud caught minus 4 per false alert. Costs set the weights, capacity fixes the threshold. Monitor recall too.
Common mistakes
- ✗Reporting a threshold-free metric when the operating point is capacity-bound
- ✗Ignoring that review capacity, not the model, fixes the achievable threshold
- ✗Collapsing to one number without monitoring recall and drift beside it
Follow-up questions
- →What happens to the reported number if review capacity doubles next quarter?
- →How would you fold customer friction, which has no invoice, into the same figure?
