Probability & Statistics for ML
Maximum likelihood, Bayes' theorem, correlation vs independence, the CLT, p-values, distributions, confidence intervals and sample-size/power.
14 questions
JuniorTheoryVery commonWhat exactly converges to a normal distribution under the Central Limit Theorem?
What exactly converges to a normal distribution under the Central Limit Theorem?
The CLT concerns the sampling distribution of the mean, not your raw data. Averages of independent finite-variance draws approach normality as n grows. The raw data can stay skewed forever; the theorem promises nothing about it.
Common mistakes
- ✗Claiming the raw data becomes normal once the sample is large
- ✗Forgetting the finite-variance condition that heavy tails violate
- ✗Applying the theorem to a single observation instead of an average
Follow-up questions
- →Which heavy-tailed distribution breaks the CLT, and why does it break it?
- →How does the CLT justify the error bar on an offline metric estimate?
JuniorTheoryVery commonWhich distributions show up in ML targets, and what does each one model?
Which distributions show up in ML targets, and what does each one model?
Bernoulli models one yes/no label; binomial counts successes over fixed trials. Poisson models event counts in a window, like orders per hour. Normal fits additive noise on a continuous target; log-normal fits skewed multiplicative quantities.
Common mistakes
- ✗Swapping Bernoulli and binomial, treating a single label as a count
- ✗Fitting a normal to a strictly positive, right-skewed target such as price
- ✗Assuming the target distribution is irrelevant to the loss you choose
Follow-up questions
- →Which loss follows from assuming a Poisson target, and where would you use it?
- →Why does log-transforming a right-skewed target often help a linear model?
JuniorTheoryVery commonWhat does a 95% confidence interval actually mean, and how does its width scale?
What does a 95% confidence interval actually mean, and how does its width scale?
The 95% describes the procedure, not one interval — repeat the experiment and 95% of intervals built this way contain the true parameter. Yours either holds it or does not. Width shrinks with root n, so halving it costs four times the data.
Common mistakes
- ✗Saying there is a 95% probability the parameter lies in this particular interval
- ✗Confusing a confidence interval with the spread of the raw observations
- ✗Expecting the width to halve when the sample size doubles
Follow-up questions
- →How would you turn that width rule into a sample-size estimate for an experiment?
- →How does the Bayesian counterpart, a credible interval, change the statement?
JuniorTheoryCommonWhy does zero Pearson correlation not imply independence, and when does it?
Why does zero Pearson correlation not imply independence, and when does it?
Pearson correlation measures only the linear part of a relationship. If Y equals X squared and X is symmetric about zero, the pair is strongly dependent yet the correlation is zero. Zero correlation implies independence only for jointly Gaussian variables.
Common mistakes
- ✗Reading a zero correlation coefficient as proof of independence
- ✗Forgetting that joint normality is what makes the implication hold
- ✗Assuming a rank coefficient detects any dependence, not just monotonic ones
Follow-up questions
- →What would the rank-based coefficient
Spearman's rhocatch that Pearson misses? - →Which test would you reach for to detect a non-monotonic dependence?
JuniorTheoryCommonHow do covariance and correlation differ, and when do you pick Spearman or Kendall?
How do covariance and correlation differ, and when do you pick Spearman or Kendall?
Covariance carries both variables' units, so its size is unreadable; correlation rescales it into minus one to one. Pearson assumes linearity and is outlier-sensitive; Spearman and Kendall work on ranks, resisting outliers and catching monotonic links.
Common mistakes
- ✗Comparing raw covariances across features with different units
- ✗Using Pearson on heavy-tailed data where a few outliers set the coefficient
- ✗Expecting a rank coefficient to reveal a non-monotonic relationship
Follow-up questions
- →How does the rank coefficient
Kendall's taudiffer from Spearman in practice? - →Why can correlation between two features matter before fitting a linear model?
JuniorTheoryCommonOn skewed salary data, do you report the mean or the median to the business?
On skewed salary data, do you report the mean or the median to the business?
Report the median. Salaries are right-skewed, so a few high earners drag the mean above what a typical person earns, while the median marks the middle person. Use the mean when the total matters — mean times headcount is payroll.
Common mistakes
- ✗Quoting the mean on right-skewed data as if it were typical
- ✗Believing the CLT makes the raw salary distribution symmetric
- ✗Assuming the median is above the mean under right skew
Follow-up questions
- →Which robust spread measure would you pair with the median, and why?
- →When would a trimmed mean be a better compromise than either one?
JuniorTheoryCommonHow do variance, standard deviation and standard error differ as data accumulates?
How do variance, standard deviation and standard error differ as data accumulates?
Variance is the mean squared deviation; standard deviation is its root, in data units. Both estimate fixed population traits, so more data cannot shrink them. Standard error is the standard deviation of the mean, sigma over root n, and it shrinks.
Common mistakes
- ✗Expecting the standard deviation to shrink as the sample grows
- ✗Using standard deviation and standard error interchangeably on a chart
- ✗Mixing up which quantity carries the data's units
Follow-up questions
- →Which of the two belongs on an error bar around a mean estimate, and why?
- →How does the standard error connect to the width of a confidence interval?
JuniorTheoryOccasionalWhat is a p-value, and name three things a p of 0.03 does not mean.
What is a p-value, and name three things a p of 0.03 does not mean.
A p-value is the probability of data at least as extreme as yours if the null hypothesis holds. A p of 0.03 is not a 3% chance the null is true, not a 97% chance the effect is real, and says nothing about the effect's size.
Common mistakes
- ✗Inverting the conditional and reading p as the probability the null is true
- ✗Treating a small p as evidence of a large or valuable effect
- ✗Reading p as the replication rate of the experiment
Follow-up questions
- →Why does peeking at a running experiment inflate the false-positive rate?
- →What does the confidence interval add that a p-value alone cannot give you?
MiddleTheoryOccasionalWith a 1% base rate and a 95%-accurate test, what is P(sick given positive)?
With a 1% base rate and a 95%-accurate test, what is P(sick given positive)?
About 16%. Of 10,000 people 100 are sick and 95 test positive; of the 9,900 healthy, 495 also test positive. That is 95 real out of 590. Intuition ignores the base rate — the healthy group's 5% error rate swamps the true positives.
Common mistakes
- ✗Reading test accuracy as the probability that a positive result is correct
- ✗Dropping the base rate once an individual result is observed
- ✗Forgetting that a large healthy group produces most of the false positives
Follow-up questions
- →How does this arithmetic explain precision collapsing on a rare positive class?
- →What happens to the answer if you raise specificity to 99% but keep the base rate?
MiddleTheoryOccasionalHow does maximum a posteriori (MAP) extend MLE, and which prior yields L2 versus L1?
How does maximum a posteriori (MAP) extend MLE, and which prior yields L2 versus L1?
MLE maximizes the likelihood alone; MAP adds a prior over the weights and maximizes the posterior. Taking logs turns the prior into an additive penalty — a zero-mean Gaussian prior yields ridge's L2, a Laplace prior lasso's L1, zeroing weights.
Common mistakes
- ✗Swapping the two priors, attributing sparsity to the Gaussian one
- ✗Thinking MAP discards the likelihood rather than weighting it by a prior
- ✗Missing that the log is what turns the prior into an additive penalty
Follow-up questions
- →How does the prior's variance map onto the regularization strength lambda?
- →Why does the sparsity-inducing penalty
L1drive weights to exactly zero?
SeniorTheoryOccasionalHow does a 95% confidence interval differ from a 95% credible interval?
How does a 95% confidence interval differ from a 95% credible interval?
A confidence interval is a claim about the procedure — 95% of intervals built this way cover the fixed parameter. A credible interval treats the parameter as random and puts 95% posterior probability inside. Bayes wins on small samples.
Common mistakes
- ✗Stating the frequentist interval as a probability about the parameter
- ✗Assuming the prior has no effect on the credible interval's bounds
- ✗Expecting the Bayesian framing to matter most when data is abundant
Follow-up questions
- →Where in an ML system does a posterior beat a point estimate outright?
- →How does a weak prior change a credible interval as the sample grows?
SeniorTheoryOccasionalHow do the law of large numbers and the CLT differ for a Monte-Carlo estimate?
How do the law of large numbers and the CLT differ for a Monte-Carlo estimate?
The law of large numbers says the sample mean converges to the true expectation — it justifies that a Monte-Carlo estimate is right at all, but not how far off it is. The CLT supplies the error's scale and shape, normal at sigma over root n.
Common mistakes
- ✗Treating the two theorems as interchangeable statements
- ✗Believing Monte-Carlo error shrinks linearly rather than as root n
- ✗Ignoring that both results need finite variance to apply
Follow-up questions
- →How many extra samples buy you one more decimal digit of precision?
- →Which variance-reduction technique would you use before simply adding samples?
SeniorTheoryOccasionalWhy is minimizing MSE maximum likelihood under Gaussian noise, and log-loss under Bernoulli?
Why is minimizing MSE maximum likelihood under Gaussian noise, and log-loss under Bernoulli?
Write the negative log-likelihood. If the target is the model output plus zero-mean Gaussian noise of fixed variance, the log-density's exponent is the squared error, so minimizing it is least squares. A Bernoulli label yields cross-entropy.
Common mistakes
- ✗Treating MSE and cross-entropy as arbitrary choices rather than noise assumptions
- ✗Swapping which noise model produces which loss
- ✗Believing maximum likelihood applies only to generative models
Follow-up questions
- →What loss follows if you assume heavy-tailed Laplace noise instead of Gaussian?
- →How does this framing explain modelling a count target with a Poisson loss?
SeniorTheoryRareHow can every segment improve while the pooled metric drops, and what does that mean for A/B reporting?
How can every segment improve while the pooled metric drops, and what does that mean for A/B reporting?
Simpson's paradox — a confounder shifts the segment mix. If the variant routes more traffic to a low-converting segment, each segment's rate can rise while the pooled rate falls, since the pooled number is a mix-weighted average. Report rates with the mix.
Common mistakes
- ✗Assuming the pooled rate is the unweighted average of segment rates
- ✗Blaming noise or multiple comparisons instead of a shifted segment mix
- ✗Reporting only the pooled metric with no view of the traffic composition
Follow-up questions
- →Which randomization mistake most often lets the segment mix drift between arms?
- →How would a pre-registered segmentation plan reduce the risk of this reversal?
