Symmetric matrices and orthogonal diagonalization

The last topic of the semester shows how “well-behaved” symmetric matrices are. A general matrix may not be diagonalizable, may have no real eigenvalues, and its eigenvectors may sit at skew angles to each other. None of this happens with a symmetric matrix: the eigenvalues are real, eigenvectors of distinct eigenvalues are perpendicular, and the matrix is always diagonalizable. Moreover, $P$ can be chosen so that its columns are mutually perpendicular unit vectors; then you don’t even need to compute $P^{-1}$, because $P^{-1} = P^T$. This result is called the spectral theorem and is used everywhere from statistics (PCA) to physics.

Golden rules:

1. An orthogonal matrix: the columns have unit length and are pairwise perpendicular. Its inverse is its transpose.
2. In a symmetric matrix, eigenvectors of distinct eigenvalues are automatically perpendicular.
3. Apply Gram-Schmidt in the eigenspace of a repeated eigenvalue.
4. At the end normalize every vector. This is the step that gets forgotten.

Orthogonal vectors and orthogonal matrices

From week 6: if $\mathbf{u} \cdot \mathbf{v} = 0$, then $\mathbf{u}$ and $\mathbf{v}$ are orthogonal. A set made of pairwise perpendicular vectors is an orthogonal set, and if all have unit length it is an orthonormal set. An orthogonal set containing no zero vector is always linearly independent.

A square matrix $Q$ is an orthogonal matrix if it satisfies $Q^TQ = I$. Since the $(i, j)$ entry of $Q^TQ$ is the dot product of column $i$ with column $j$:

$Q^TQ = I \iff \text{the columns are orthonormal} \iff Q^{-1} = Q^T.$

The last row is a classic trap: perpendicular columns are not enough, they must also be unit length. $\frac{1}{\sqrt2}\begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix}$ is orthogonal.

Orthogonal matrices preserve length and angle: $\|Q\mathbf{x}\| = \|\mathbf{x}\|$, because $\|Q\mathbf{x}\|^2 = \mathbf{x}^TQ^TQ\mathbf{x} = \mathbf{x}^T\mathbf{x}$. Geometrically, only rotations and reflections.

Three properties of symmetric matrices

If $A$ is real and symmetric ($A^T = A$):

1. All its eigenvalues are real.
2. Eigenvectors belonging to distinct eigenvalues are perpendicular. Proof: $A\mathbf{u} = \lambda\mathbf{u}$, $A\mathbf{v} = \mu\mathbf{v}$, $\lambda \neq \mu$. $\lambda(\mathbf{u} \cdot \mathbf{v}) = (A\mathbf{u})^T\mathbf{v} = \mathbf{u}^TA^T\mathbf{v} = \mathbf{u}^TA\mathbf{v} = \mu(\mathbf{u} \cdot \mathbf{v})$. $(\lambda - \mu)(\mathbf{u} \cdot \mathbf{v}) = 0$, so $\mathbf{u} \cdot \mathbf{v} = 0$. ($A^T = A$ is used exactly here.)
3. For every eigenvalue, geometric multiplicity = algebraic multiplicity. Consequence: it is always diagonalizable.

Spectral theorem. Let $A$ be an $n \times n$ real matrix. The following two statements are equivalent:

• $A$ is symmetric.
• There exist an orthogonal $P$ and a diagonal $D$ with $P^TAP = D$ (orthogonal diagonalization).

So the answer to “is it orthogonally diagonalizable?” is at a glance: is the matrix symmetric?

The recipe for orthogonal diagonalization

1. Solve the characteristic equation.
2. Find a basis of the eigenspace for each eigenvalue.
3. If an eigenspace has dimension 2 or more, apply Gram-Schmidt to its basis to make it orthogonal. (Single-dimensional eigenspaces, and orthogonality between different eigenspaces, come for free.)
4. Normalize every vector: $\mathbf{v}/\|\mathbf{v}\|$.
5. The columns of $P$ are these orthonormal vectors, with the eigenvalues in $D$ in the same order. $A = PDP^T$.

Gram-Schmidt (two vectors). If $\{\mathbf{v}_1, \mathbf{v}_2\}$ is independent,

$\mathbf{u}_1 = \mathbf{v}_1, \qquad \mathbf{u}_2 = \mathbf{v}_2 - \frac{\mathbf{v}_2 \cdot \mathbf{u}_1}{\mathbf{u}_1 \cdot \mathbf{u}_1}\mathbf{u}_1.$

$\mathbf{u}_2$ removes from $\mathbf{v}_2$ the component along $\mathbf{u}_1$; what remains is the perpendicular part. With three vectors, both the $\mathbf{u}_1$ and $\mathbf{u}_2$ components are removed from $\mathbf{u}_3$.

Example 1 (2×2). $A = \begin{bmatrix} 3 & 1 \\ 1 & 3 \end{bmatrix}$. Trace 6, det 8: $\lambda^2 - 6\lambda + 8 = (\lambda - 4)(\lambda - 2)$.

$\lambda = 4: (1, 1), \qquad \lambda = 2: (1, -1), \qquad P = \frac{1}{\sqrt2}\begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix}, \quad D = \begin{bmatrix} 4 & 0 \\ 0 & 2 \end{bmatrix}.$

The eigenvectors came out perpendicular by themselves; the only job was to normalize.

Example 2 (3×3, repeated eigenvalue). $A = \begin{bmatrix} 2 & 1 & 1 \\ 1 & 2 & 1 \\ 1 & 1 & 2 \end{bmatrix}$.

The characteristic polynomial is $\lambda^3 - 6\lambda^2 + 9\lambda - 4 = (\lambda - 1)^2(\lambda - 4)$. Check: trace $6 = 1 + 1 + 4$.

• $\lambda = 4$: the rows of $A - 4I$ are $(-2, 1, 1), (1, -2, 1), (1, 1, -2)$; the solution is $x = y = z$: $(1, 1, 1)$.
• $\lambda = 1$: all rows of $A - I$ are $(1, 1, 1)$: a single equation $x + y + z = 0$. A basis is $\mathbf{v}_1 = (-1, 1, 0)$, $\mathbf{v}_2 = (-1, 0, 1)$. These are not perpendicular ($\mathbf{v}_1 \cdot \mathbf{v}_2 = 1$).

Gram-Schmidt (within the eigenspace):

u1 = (-1, 1, 0)
v2·u1 = 1 + 0 + 0 = 1,   u1·u1 = 2
u2 = (-1, 0, 1) - (1/2)(-1, 1, 0) = (-1/2, -1/2, 1)   → multiply by 2: (-1, -1, 2)
check: u1·u2 = 1 - 1 + 0 = 0;  A u2 = (-2-1+2, -1-2+2, -1-1+4) = (-1, -1, 2) = 1·u2

$(1, 1, 1)$ is already perpendicular to both (a different eigenvalue). Normalize: the lengths are $\sqrt2$, $\sqrt6$, $\sqrt3$.

$P = \begin{bmatrix} -\frac{1}{\sqrt2} & -\frac{1}{\sqrt6} & \frac{1}{\sqrt3} \\ \frac{1}{\sqrt2} & -\frac{1}{\sqrt6} & \frac{1}{\sqrt3} \\ 0 & \frac{2}{\sqrt6} & \frac{1}{\sqrt3} \end{bmatrix}, \qquad D = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 4 \end{bmatrix}.$

PITFALL · skipping Gram-Schmidt for a repeated eigenvalue. Two eigenvectors from the same eigenspace need not be perpendicular. In Example 2, $(-1, 1, 0)$ and $(-1, 0, 1)$ are both correct eigenvectors, but the angle between them is $60^\circ$. If you normalize them directly and put them in $P$, you get $P^TP \neq I$.

TIP · check. To check $P^TP = I$, look at the dot product of each pair of columns (it must be 0) and the length of each column (it must be 1). Much shorter than multiplying out $P^TAP$ completely.

Application: quadratic forms

The expression $Q(x, y) = ax^2 + bxy + cy^2$ is a quadratic form and is written with a symmetric matrix:

$Q(\mathbf{x}) = \mathbf{x}^TA\mathbf{x}, \qquad A = \begin{bmatrix} a & b/2 \\ b/2 & c \end{bmatrix}.$

If $A = PDP^T$, then with the change of variable $\mathbf{x} = P\mathbf{y}$, $Q = \lambda_1y_1^2 + \lambda_2y_2^2$: no cross term. The columns of $P$ are the principal axes. In Q11, $3x^2 + 4xy + 3y^2 = 5x'^2 + y'^2$; the directions $(1, 1)$ and $(-1, 1)$ are the axes of the ellipse. The signs of the eigenvalues tell whether the form is always positive (positive definite).

Semester summary: the diagonalization hierarchy

Common mistakes