Quality Metrics
A metric answers "how good is the model", but the right answer depends on the task. Precision and recall look at the positive class from different sides, accuracy mixes everything and lies under imbalance, AUC-ROC measures ranking quality and is threshold-independent, and regression metrics react to outliers differently.
The topic seems textbook, but it is full of traps. Accuracy is useless under imbalance. AUC-ROC is the fraction of correctly ordered pairs, not accuracy at the 0.5 threshold, and a high AUC does not guarantee a clean top-K. Under outliers, MSE drags the fit toward the anomalies. The full map is in the layers below.
Topic map
- Classification metrics — precision, recall, accuracy, their formulas via TP/FP/FN/TN, and why accuracy misleads.
- Precision/recall trade-off — how the threshold shifts the balance and how to choose it by the cost of error.
- AUC-ROC — area under the ROC curve, the ranking interpretation, resampling invariance, and the gap with precision@K.
- Regression metrics — MAE, MSE, RMSE and Huber, robustness to outliers, and the cost of MAE's non-differentiability.
Common traps
| Mistake | Consequence |
|---|---|
| Swapping the denominators of precision and recall | FP versus FN confused — the metrics are computed wrong |
| Reporting accuracy on an imbalanced set | A model that always predicts the majority looks "good" |
| Confusing AUC-ROC with accuracy at the 0.5 threshold | AUC is about ranking, not a single operating point |
| Thinking a high AUC guarantees a clean top-K | Under imbalance the whole top-K can be negative |
| Choosing MSE under outliers | A few anomalies dominate the loss and drag the fit toward them |
| Conflating the Russian «точность» with accuracy | Here «точность» means precision, not the correct-prediction rate |
Interview relevance
Metrics are a common-sense filter. A candidate who spontaneously notes "accuracy lies under imbalance, so I look at precision/recall or AUC" immediately stands out from "well, 95% accuracy, so it is good".
Typical checks:
- The formulas for precision, recall, accuracy and when accuracy is misleading.
- That the threshold shifts the precision/recall balance — and how to choose it per task.
- What AUC-ROC is and why it is a ranking, not a threshold, metric.
- MAE versus MSE under outliers and why Huber loss exists.
Common wrong answer: "AUC-ROC is the fraction of correctly classified objects". In fact AUC is the probability that a random positive ranks above a random negative — the fraction of correctly ordered pairs — and it does not depend on the threshold.