The Gauss–Markov Theorem

Informally, the Gauss–Markov theorem states that, under certain conditions, the ordinary least squares (OLS) estimator is the best linear model we can use. This is a powerful claim. Formally, the theorem states the following:

Gauss–Markov theorem. In a linear regression with response vector $\mathbf{y}$ and design matrix $\mathbf{X}$, the least squares estimator $\hat{\boldsymbol{\beta}} \triangleq (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{y}$ is the minimum-variance linear unbiased estimator of the model parameter $\boldsymbol{\beta}$, under the ordinary least squares assumptions.

Here, we can see that “best” is defined as both minimum variance and unbiased, and that the regularity conditions are the assumptions of OLS. There is no guarantee that a nonlinear method, for example, will not be better for our data by some other metric, but if we want to use an unbiased linear model and if the OLS assumptions hold for our data, then we should just use OLS.

An estimator that is optimal in this way is sometimes referred to as “BLUE”, for best linear unbiased estimator. The Gauss–Markov theorem could be stated even more succinctly as: “Under the OLS assumptions, the OLS estimator is BLUE.”

Obviously, if the OLS assumptions do not hold, then the OLS estimator is not necessarily BLUE. If our data have heteroscedasticity, for example, then a least squares regression fit to our data will not necessarily be optimal as defined above.

Proof

This proof is from (Greene, 2003). Consider a second linear and unbiased estimator $\hat{\boldsymbol{\beta}}_0$:

$\hat{\boldsymbol{\beta}}_0 = \mathbf{C} \mathbf{y}. \tag{1}$

Here, $\mathbf{C}$ is a $P \times N$ matrix, and therefore $\hat{\boldsymbol{\beta}}_0$ is linear in that the predictions $\hat{\mathbf{y}}$ are linear functions of the response, since

$\hat{\mathbf{y}} = \mathbf{X} \hat{\boldsymbol{\beta}}_0 = \mathbf{X} \mathbf{C} \mathbf{y}. \tag{2}$

A linear function of a linear function is still a linear function, so $\mathbf{X} \mathbf{C} \mathbf{y}$ is simply a linear projection of our response variables $\mathbf{y}$ onto the space spanned by the columns of $\mathbf{X} \mathbf{C}$. Thus, this estimator adheres to the first assumption of OLS, linearity.

We assumed this new estimator $\hat{\boldsymbol{\beta}}_0$, like the OLS estimator $\hat{\boldsymbol{\beta}}$, is unbiased. (Although we proved, not assumed, that the original OLS estimator is unbiased.) This implies

$\begin{aligned} \boldsymbol{\beta} &= \mathbb{E}[\hat{\boldsymbol{\beta}}_0 \mid \mathbf{X}] \\ &= \mathbb{E}[\mathbf{C}\mathbf{y} \mid \mathbf{X}] \\ &= \mathbb{E}[\mathbf{C} (\mathbf{X}\boldsymbol{\beta} + \mathbf{y}) \mid \mathbf{X}] \\ &= \mathbb{E}[\mathbf{C} \mathbf{X} \boldsymbol{\beta} + \mathbf{C} \boldsymbol{\varepsilon} \mid \mathbf{X}]. \end{aligned} \tag{3}$

Thus, it must be true that

$\begin{aligned} \mathbf{C} \mathbf{X} &= \mathbf{I}, \\ \mathbf{C} \mathbb{E}[\boldsymbol{\varepsilon} \mid \mathbf{X}] &= \mathbf{0}, \\ \hat{\boldsymbol{\beta}}_0 &= \boldsymbol{\beta} + \mathbf{C} \boldsymbol{\varepsilon}. \end{aligned} \tag{4}$

In other words, the properties in Equation $4$ are true since we assume that $\hat{\boldsymbol{\beta}}_0$ is unbiased. Note that $\mathbb{E}[\boldsymbol{\varepsilon} \mid \mathbf{X}] = \mathbf{0}$ is the second assumption of OLS, strict exogeneity.

Now the variance of this new linear estimator is

$\begin{aligned} \mathbb{V}[\hat{\boldsymbol{\beta}}_0 \mid \mathbf{X}] &= \mathbb{V}[\boldsymbol{\beta} + \mathbf{C} \boldsymbol{\varepsilon} \mid \mathbf{X}] \\ &= \mathbb{V}[\mathbf{C} \boldsymbol{\varepsilon} \mid \mathbf{X}] \\ &= \mathbf{C} \mathbb{V}[ \boldsymbol{\varepsilon} \mid \mathbf{X}] \mathbf{C}^{\top} \\ &= \sigma^2 \mathbf{C} \mathbf{C}^{\top}. \end{aligned} \tag{5}$

Here, we have used the fact that both $\boldsymbol{\varepsilon}$ and $\mathbf{C}$ are non-random. Furthermore, notice that we assumed that

$\mathbb{V}[ \boldsymbol{\varepsilon} \mid \mathbf{X}] = \sigma^2 \mathbf{I}. \tag{6}$

This is the fourth assumption of OLS, spherical errors. Now let’s define two new matrices $\mathbf{A}$ and $\mathbf{D}$:

$\begin{aligned} \mathbf{A} &\triangleq (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top}, \\ \mathbf{D} &\triangleq \mathbf{C} - \mathbf{A}. \end{aligned} \tag{7}$

$\mathbf{A}$ is the OLS analog to $\mathbf{C}$, the linear map of the estimator that projects $\mathbf{y}$. Then we can rewrite Equation $5$ as:

$\begin{aligned} \mathbb{V}[\hat{\boldsymbol{\beta}}_0 \mid \mathbf{X}] &= \sigma^2 \mathbf{C} \mathbf{C}^{\top} \\ &= \sigma^2 \left(\mathbf{D} + \mathbf{A})(\mathbf{D} + \mathbf{A} \right)^{\top} \\ &= \sigma^2 \left(\mathbf{D}\mathbf{D}^{\top} + \mathbf{A}\mathbf{D}^{\top} + \mathbf{D}\mathbf{A}^{\top} + \mathbf{A}\mathbf{A}^{\top} \right) \\ &= \sigma^2 \left(\mathbf{D}\mathbf{D}^{\top} + \cancel{\mathbf{A}\mathbf{D}^{\top}} + \cancel{\mathbf{D}\mathbf{A}^{\top}} + \cancel{(\mathbf{X}^{\top} \mathbf{X})^{-1}} \cancel{\mathbf{X}^{\top} \mathbf{X}} (\mathbf{X}^{\top} \mathbf{X})^{-1} \right) \\ &= \sigma^2 \mathbf{D}\mathbf{D}^{\top} + \sigma^2 (\mathbf{X}^{\top}\mathbf{X})^{-1} \\ &\geq \mathbb{V}[\hat{\boldsymbol{\beta}} \mid \mathbf{X}]. \end{aligned} \tag{8}$

The inequality follows because $\mathbf{D}\mathbf{D}^{\top}$ is a positive definite matrix. So in words, the conditional variance of this new estimator $\hat{\boldsymbol{\beta}}_0$ is greater or equal to the variance of $\hat{\boldsymbol{\beta}}$. This proves that $\hat{\boldsymbol{\beta}}$ is the minimum-variance linear unbiased estimator or BLUE.

So why are the cross terms in Equation $8$ zero? Because

$\begin{aligned} \mathbf{D}\mathbf{X} &= (\mathbf{C} - \mathbf{A}) \mathbf{X} \\ &= \mathbf{C}\mathbf{X} - \mathbf{A}\mathbf{X} \\ &= \mathbf{C}\mathbf{X} - (\mathbf{X}^{\top}\mathbf{X})^{-1}\mathbf{X}^{\top} \mathbf{X} \\ &= \mathbf{I} - \mathbf{I} \\ &= \mathbf{0}. \end{aligned} \tag{9}$

Note that $\mathbf{C}\mathbf{X} = \mathbf{I}$ by the assumption that $\hat{\boldsymbol{\beta}}_0$ is unbiased. Clearly, if $\mathbf{D}\mathbf{X} = \mathbf{0}$, then $\mathbf{X}^{\top} \mathbf{D}^{\top} = \mathbf{0}^{\top}$—both $\mathbf{0}$ and $\mathbf{0}^{\top}$ are $P \times P$ matrices of all zeros. Therefore the cross terms can be each written as

$\begin{aligned} \mathbf{D}\mathbf{A}^{\top} &= \mathbf{D} \mathbf{X} (\mathbf{X}^{\top} \mathbf{X})^{-1} \\ &= \mathbf{0}, \\ \\ \mathbf{A} \mathbf{D}^{\top} &= (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{D}^{\top} \\ &= \mathbf{0}. \end{aligned} \tag{10}$

Notice that implicit in the existence of the inverse of $\mathbf{X}^{\top} \mathbf{X}$, we assume that $\mathbf{X}$ is full-rank. This is the second OLS assumption, no multicollinearity. Thus, the Gauss–Markov theorem holds when we adhere to the four assumptions of OLS: linearity, no multicollinearity, strict exogeneity, and spherical errors. If we make these four assumptions, then $\hat{\boldsymbol{\beta}}$ is BLUE, the best (minimum-variance) linear unbiased estimator.