Sampling, the CLT and Standard Error

Why sample means fall into a bell curve: the central limit theorem, the standard error σ/√n and the 68–95–99.7 rule — with a simulator where samples pile up before your eyes.

Intuition

You measured a model's latency on 50 requests: 118 ms on average. A colleague measured it on another 50 — and got 123 ms. Who is right? Both and neither: you are looking at different samples from the same population — all the requests the model will ever receive. We never see the whole population: not all users, not all test examples, not all pizza deliveries. We see a piece and judge the whole by it.

Hence the main shift in thinking: the sample mean x̄ is itself a random variable. Take another sample — you get another number. The question of statistics is not “what is the mean equal to”, but “how much does it jump from sample to sample”. If it jumps by ±1 ms, the difference between 118 and 123 is real. If by ±10, it's noise.

How it works

Imagine we repeated the measurement a thousand times and built a histogram of a thousand means. This is the sampling distribution of the mean. Two things are known about it, and together they make up the central limit theorem (CLT):

• It is centered on the true population mean μ and tightens around it as n grows — this is also the law of large numbers.
• Its shape tends towards the normal distribution (Gaussian) — whatever the shape of the original data, as long as the variance is finite. Skewed, bimodal, discrete — still a bell.

The width of this bell is called the standard error (SE):

where σ is the standard deviation (SD) of individual observations and n is the sample size. The square root is the most important detail of the formula: to halve the error you need four times more data; to cut it tenfold, a hundred times more. Precision grows with data, but slowly.

And since the distribution is normal, the 68–95–99.7 rule applies: in 68% of samples the mean will fall within ±1 SE of μ, in 95% within ±2 SE (more precisely 1.96), in 99.7% within ±3 SE. This is exactly where confidence intervals and the p-value grow from: both constructions are simply ways of reading this bell.

Back to latency: if σ ≈ 30 ms, then with n = 50 we have SE ≈ 4.2 ms. The difference of 118 versus 123 is just over one standard error; this happens all the time with one and the same model. To reliably catch a 5 ms difference, you need samples of hundreds of requests.

When the CLT lets you down

The magic “n ≥ 30” from textbooks is a heuristic, not a law. How many observations you need depends on the shape of the data: for symmetric distributions the bell appears already at n = 5–10, for strongly skewed ones (latency with rare multi-second timeouts, incomes, text lengths) — at hundreds. Play with n in the interactive on the skewed distribution: at n = 3 the histogram of means is still noticeably skewed to the right, at n = 30 no trace of the skew remains.

Three situations where the formula SE = σ/√n is seriously wrong:

• Infinite variance. Distributions with very heavy tails (Cauchy, some power laws) have no finite σ — the mean over them doesn't converge at all, however much data you collect. For such data, look at the median and quantiles.
• Dependent observations. A thousand requests from ten users is not n = 1000. One person's requests are similar to each other, and the effective sample size is closer to ten. The formula assumes the observations are independent; time series, sessions, repeated measurements on one machine break this condition, and the SE turns out to be underestimated several times over.
• Estimating σ from a small sample. We don't know the true σ either, so we plug in the sample standard deviation s. With small n it is noisy itself — the t-distribution makes this correction; more on it in the topic on confidence intervals.

Where this hides in ML

• Test metrics. Accuracy is a mean of zeros and ones, so its standard error is √(p(1−p)/n). At accuracy 0.9 on 100 examples that's ±3 percentage points — a 1 pp difference between two models is indistinguishable from noise. An LLM benchmark of 200 questions suffers from the same problem.
• Mini-batches. The gradient over a batch of B examples is the sample mean of per-example gradients, so its noise falls as 1/√B. Hence the link between batch size and learning rate in the topic on training practice.
• Cross-validation. The spread of a metric across folds is a rough estimate of its standard error; if it is larger than the difference between candidates, the choice of hyperparameter is random.