Completing the Square

This operation, while useful in elementary algebra, also arises frequently when manipulating Gaussian random variables. I review and document both the univariate and multivariate cases.

Published

18 September 2019

In elementary algebra, we learned about completing the square. This operation has many uses. For most people, the first use of completing the square is finding roots of quadratic equations, and the operation is one way of re-deriving the quadratic formula. I have recently found completing the square useful because it also arises when manipulating Gaussian random variables, since every normal distribution is the exponential of a quadratic function. This post is a review of the univariate case and an extension to the multivariate case.

Univariate case

To review, given a variable x∈Rx \in \mathbb{R}, we want to write a quadratic polynomial

x2+bx+c(1) x^2 + bx + c \tag{1}

in the form

(x+d)2+k(2) (x + d)^2 + k \tag{2}

This second form is often easier to work with. For example, the real roots of the polynomial are immediately obvious from (2)(2). To see how to complete the square, let’s expand the squared term in (2)(2),

(x+d)2+k=x2+2dx+d2+k. (x + d)^2 + k = x^2 + 2dx + d^2 + k.

And now we just solve for dd and kk,

d=b2k=c−d2=c−b24. \begin{aligned} d &= \frac{b}{2} \\ k &= c - d^2 = c - \frac{b^2}{4}. \end{aligned}

The reason this trick is called “completing the square” is because it can be viewed as adding a term to (1)(1) such that you can square the equation. For example,

x2−4x=x2−4x+4−4=(x−2)2−4. \begin{aligned} x^2 - 4x &= x^2 - 4x + 4 - 4 \\ &= (x - 2)^2 - 4. \end{aligned}

Putting our results together, we get

x2+bx+c=(x+b2)2+(c−b24).(3) x^2 + bx + c = \Big(x + \frac{b}{2} \Big)^2 + \Big( c - \frac{b^2}{4} \Big). \tag{3}

Multivariate case

Now let’s consider the multivariate extension. Let x\mathbf{x} and b\mathbf{b} be dd-dimensional vectors, and let M∈Rd×dM \in \mathbb{R}^{d \times d} be a symmetric invertible matrix. Then

x⊤Mx−2b⊤x=(x−M−1b)⊤M(x−M−1b)−b⊤M−1b(4) \mathbf{x}^{\top} M \mathbf{x} - 2 \mathbf{b}^{\top} \mathbf{x} = (\mathbf{x} - M^{-1} \mathbf{b})^{\top} M (\mathbf{x} - M^{-1} \mathbf{b}) - \mathbf{b}^{\top} M^{-1} \mathbf{b} \tag{4}

We can easily verify this by multiplying out the quadratic form,

(x−M−1b)⊤M(x−M−1b)=(x⊤−b⊤M−1)M(x−M−1b)=(x⊤M−b⊤)(x−M−1b)=x⊤Mx−b⊤x−x⊤b+b⊤M−1b=x⊤Mx−2b⊤x+b⊤M−1b \begin{aligned} (\mathbf{x} - M^{-1} \mathbf{b})^{\top} M (\mathbf{x} - M^{-1} \mathbf{b}) &= (\mathbf{x}^{\top} - \mathbf{b}^{\top} M^{-1}) M (\mathbf{x} - M^{-1} \mathbf{b}) \\ &= (\mathbf{x}^{\top} M - \mathbf{b}^{\top}) (\mathbf{x} - M^{-1} \mathbf{b}) \\ &= \mathbf{x}^{\top} M \mathbf{x} - \mathbf{b}^{\top} \mathbf{x} - \mathbf{x}^{\top} \mathbf{b} + \mathbf{b}^{\top} M^{-1} \mathbf{b} \\ &= \mathbf{x}^{\top} M \mathbf{x} - 2 \mathbf{b}^{\top} \mathbf{x} + \mathbf{b}^{\top} M^{-1} \mathbf{b} \end{aligned}

Of course, the b⊤M−1b\mathbf{b}^{\top} M^{-1} \mathbf{b} term cancels with the one in (4)(4), and we’re done.