Proof of Bessel's Correction

Let $X = \{ X_1, X_2, \dots, X_N\}$ be a random sample of $N$ i.i.d. random variables. Let $\bar{X}$ denote the sample mean,

$\bar{X} = \frac{1}{N} \sum_{n=1}^{N} X_n. \tag{1}$

When computing the sample variance $s^2$, students are told to divide by $N - 1$ rather than $N$:

$s^2 = \frac{1}{N-1} \sum_{n=1}^{N} (X_n - \bar{X})^2. \tag{2}$

When first learning about this fact, I was shown computer simulations but no mathematical proof of why this must hold. The goal of this post is to provide a quick proof of why this correction makes sense.

The proof outline is straightforward: we need to show that the estimator in Equation $4$ (below) is biased, and that we can correct this bias by dividing by $N - 1$ rather than $N$. For an estimator to be unbiased, the expectation of that estimator must equal the population parameter. In our case, if the sample variance is $s^2$ and the population variance is $\sigma^2$, we want

$\mathbb{E}[s^2] = \sigma^2. \tag{3}$

Let’s begin.

Proof

Let’s prove that the following estimator for the population variance is biased:

$s^2 = \frac{1}{N} \sum_{n=1}^{N} (X_n - \bar{X})^2. \tag{4}$

First, let’s take the expectation of this estimator and manipulate it:

$\begin{aligned} \mathbb{E}\left[\frac{1}{N} \sum_{n=1}^{N} (X_n - \bar{X})^2\right] &= \mathbb{E}\left[\frac{1}{N} \sum_{n=1}^{N} (X_n^2 - 2 X_n \bar{X} + \bar{X}^2) \right] \\ &= \mathbb{E}\left[\frac{1}{N} \sum_{n=1}^{N} X_n^2 - 2 \bar{X} \frac{1}{N} \sum_{n=1}^{N} X_n + \frac{1}{N} \sum_{n=1}^{N} \bar{X}^2 \right] \\ &\stackrel{\star}{=} \mathbb{E}\left[\frac{1}{N} \sum_{n=1}^{N} X_n^2 \right] - \mathbb{E}\left[2 \bar{X}^2\right] + \mathbb{E}\left[\bar{X}^2 \right] \\ &= \mathbb{E}\left[\frac{1}{N} \sum_{n=1}^{N} X_n^2\right] - \mathbb{E}\left[\bar{X}^2 \right] \\ &\stackrel{\dagger}{=} \mathbb{E}\left[ X_n^2 \right] - \mathbb{E} \left[ \bar{X}^2 \right]. \end{aligned} \tag{5}$

Note that step $\star$ holds because

$\sum_{n=1}^{N} X_n = N \bar{X}. \tag{6}$

while step $\dagger$ holds because the data are i.i.d., i.e.

$\mathbb{E}\left[\frac{1}{N} \sum_{n=1}^{N} X_n^2 \right] = \frac{1}{N} \sum_{n=1}^{N} \mathbb{E}\left[ X_n^2 \right] = \mathbb{E}\left[ X_n^2 \right]. \tag{7}$

Now note that since $X_n$ is an i.i.d. random variable, all $X_n \in X$ have the same variance. Furthermore, recall that for any random variable $Y$,

$\begin{aligned} \mathbb{V}[Y] &= \mathbb{E}[Y^2] - \mathbb{E}[Y]^2, \\ &\Downarrow \\ \mathbb{E}[Y^2] &= \mathbb{V}[Y] + \mathbb{E}[Y]^2. \end{aligned} \tag{8}$

So we can write

$\begin{aligned} \mathbb{E}\left[ X_n^2 \right] &= \mathbb{V}[X_n] + \mathbb{E}[X_n]^2 \\ &= \sigma^2 + \mu^2, \\\\ \mathbb{E} \left[ \bar{X}^2 \right] &= \mathbb{V}[\bar{X}] + \mathbb{E}[\bar{X}]^2 \\ &\stackrel{\star}{=} \frac{\sigma^2}{N} + \mu^2. \end{aligned} \tag{9}$

Step $\star$ holds because

$\begin{aligned} \mathbb{V}[\bar{X}] &= \mathbb{V}\left[\frac{1}{N} \sum_{n=1}^{N} X_n \right] \\ &\stackrel{\textsf{iid}}{=} \frac{1}{N^2} \sum_{n=1}^{N} \mathbb{V}[X_n] \\ &= \frac{1}{N^2} \sum_{n=1}^{N} \sigma^2 \\ &= \frac{\sigma^2}{N}. \end{aligned} \tag{10}$

Finally, let’s put everything together:

$\begin{aligned} \mathbb{E}[s^2] &= \sigma^2 + \mu^2 - \left(\frac{\sigma^2}{N} + \mu^2\right) \\ &= \sigma^2 \left(1 - \frac{1}{N} \right). \end{aligned} \tag{11}$

What we have shown is that our estimator is off by a constant, $\left(1 - \frac{1}{N} \right) = \left( \frac{N-1}{N} \right)$. If we want an unbiased estimator, we should multiply both sides of Equation $11$ by the inverse of the constant:

$\mathbb{E}\left[\left(\frac{N}{N-1}\right) s^2\right] = \mathbb{E}\left[\frac{1}{N-1} \sum_{n=1}^{N} (X_n - \bar{X})^2\right] = \sigma^2. \tag{12}$

And this new estimator is exactly what we wanted to prove. Bessel’s correction results in an unbiased estimator for the population variance.