Singular Value Decomposition - Properties of Symmetric Matrices

In a previous post, you have seen that the eigenvectors of a symmetric matrix are perpendicular to each other. This is not a coincidence and is an important property of symmetric matrices.

An important property of symmetric matrices is that an $n\times n$ symmetric matrix has $n$ linearly independent and orthogonal eigenvectors, and it has $n$ real eigenvalues corresponding to those eigenvectors. It is important to note that these eigenvalues are not necessarily unique; some of them can be identical. Another important property of symmetric matrices is that they are orthogonally diagonalizable. Next, let’s unpack orthogonally diagonalizable.

Eigendecomposition ¶

A symmetric matrix is orthogonally diagonalizable. It means that for an $n\times n$ symmetric matrix $\mathbf{A}$, we can decompose it as

$$\mathbf{A}=\mathbf{P}\mathbf{D}{\mathbf{P}}^{\mathrm{\top }}$$

in which $\mathbf{D}$ is an $n\times n$ diagonal matrix comprised of the $n$ eigenvalues of $\mathbf{A}$ on its diagonal. $\mathbf{P}$ is also an $n\times n$ matrix, and the columns of $\mathbf{P}$ are the $n$ linearly independent eigenvectors of $\mathbf{A}$ that correspond to those eigenvalues in $\mathbf{D}$ respectively. For example, if ${\mathbf{u}}_1,{\mathbf{u}}_2,\dots ,{\mathbf{u}}_n$ are the eigenvectors of $\mathbf{A}$, and ${\lambda }_1,{\lambda }_2,\dots ,{\lambda }_n$ are their corresponding eigenvalues, then $\mathbf{A}$ can be written as

$$\mathbf{A}=\left[\begin{array}{cccc}{\mathbf{u}}_1& {\mathbf{u}}_2& \cdots & {\mathbf{u}}_n\end{array}\right]\left[\begin{array}{cccc}{\lambda }_1& 0& \dots & 0\\ 0& {\lambda }_2& \dots & 0\\ ⋮& ⋮& ⋮& ⋮\\ 0& 0& \dots & {\lambda }_n\end{array}\right]{\left[\begin{array}{cccc}{\mathbf{u}}_1& {\mathbf{u}}_2& \dots & {\mathbf{u}}_n\end{array}\right]}^{\mathrm{\top }}$$

This can also be written as

$$\mathbf{A}=\left[\begin{array}{cccc}{\mathbf{u}}_1& {\mathbf{u}}_2& \cdots & {\mathbf{u}}_n\end{array}\right]\left[\begin{array}{cccc}{\lambda }_1& 0& \dots & 0\\ 0& {\lambda }_2& \dots & 0\\ ⋮& ⋮& ⋮& ⋮\\ 0& 0& \dots & {\lambda }_n\end{array}\right]\left[\begin{array}{c}{\mathbf{u}}_1^{\mathrm{\top }}\\ {\mathbf{u}}_2^{\mathrm{\top }}\\ \dots \\ {\mathbf{u}}_n^{\mathrm{\top }}\end{array}\right]$$

This factorization of $\mathbf{A}$ is called the eigendecomposition of $\mathbf{A}$.

Let’s see an example. Suppose that

$$\mathbf{A}=\left[\begin{array}{cc}3& 1\\ 1& 2\end{array}\right]$$

It has two eigenvectors:

$${\mathbf{u}}_1=\left[\begin{array}{c}-0.85\\ -0.53\end{array}\right]{\mathbf{u}}_2=\left[\begin{array}{c}0.53\\ -0.85\end{array}\right]$$

and the corresponding eigenvalues are:

$${\lambda }_1=3.62,{\lambda }_2=1.38$$

Therefore, $\mathbf{D}$ can be defined as

$$\mathbf{D}=\left[\begin{array}{cc}{\lambda }_1& 0\\ 0& {\lambda }_2\end{array}\right]=\left[\begin{array}{cc}3.62& 0\\ 0& 1.38\end{array}\right]$$

Likewise, columns of $\mathbf{P}$ are the eigenvectors of $\mathbf{A}$ corresponding to those eigenvalues in $\mathbf{D}$,

$$\mathbf{P}=\left[\begin{array}{cc}{\mathbf{u}}_1& {\mathbf{u}}_2\end{array}\right]=\left[\begin{array}{cc}-0.85& 0.53\\ -0.53& -0.85\end{array}\right]$$

The transpose of $\mathbf{P}$ is

$${\mathbf{P}}^{\mathrm{\top }}={\left[\begin{array}{cc}{\mathbf{u}}_1& {\mathbf{u}}_2\end{array}\right]}^{\mathrm{\top }}=\left[\begin{array}{c}{\mathbf{u}}_1^{\mathrm{\top }}\\ {\mathbf{u}}_2^{\mathrm{\top }}\end{array}\right]=\left[\begin{array}{cc}-0.85& -0.53\\ 0.53& -0.85\end{array}\right]$$

And finally, barring some round error, $\mathbf{A}$ can be written as

$$\mathbf{A}=\left[\begin{array}{cc}3& 1\\ 1& 2\end{array}\right]=\left[\begin{array}{cc}-0.85& 0.53\\ -0.53& -0.85\end{array}\right]\left[\begin{array}{cc}3.62& 0\\ 0& 1.38\end{array}\right]\left[\begin{array}{cc}-0.85& -0.53\\ 0.53& -0.85\end{array}\right]$$

It is neat to be able to re-write a symmetric matrix as the product of three matrices. But to understand its implication, we need to look at it geometrical interpretation, which will be the topic of the next post.