Singular Value Decomposition - Transpose and Dot Product

In this third post of the series, we will introduce two additional building blocks.

Transpose ¶

The transpose of a column vector $\mathbf{u}$, ${\mathbf{u}}^{\mathrm{\top }}$, is the row vector of $\mathbf{u}$. The transpose of an $m\times n$ matrix $\mathbf{A}$ is an $n\times m$ matrix whose columns are formed from the corresponding rows of $\mathbf{A}$. For example, if we have

$$\mathbf{C}=\left[\begin{array}{ccc}5& 4& 2\\ 7& 1& 9\end{array}\right]$$

then the transpose of $\mathbf{C}$ is:

$$\mathbf{C}=\left[\begin{array}{cc}5& 7\\ 4& 1\\ 2& 9\end{array}\right]$$

The transpose of a row vector becomes a column vector with the same elements and vice versa. The element in the $i^{\text{th}}$ row and $j^{\text{th}}$, $x_{\text{ij}}$, is equal to the element in the $j^{\text{th}}$ row and $i^{\text{th}}$ column, $x_{\text{ji}}$, of the original matrix. Therefore,

$${\mathbf{A}}_{\text{ij}}^{\mathrm{\top }}={\mathbf{A}}_{\text{ji}}$$

The transpose has some important properties. First, the transpose of a transpose is itself.

$${\mathbf{A}}_{\text{ij}}^{\mathrm{\top }}={\mathbf{A}}_{\text{ji}}$$

In addition, the transpose of a product is the product of transpose in the reverse order.

$$(\mathbf{A}\mathbf{B}{)}^{\mathrm{\top }}={\mathbf{B}}^{\mathrm{\top }}{\mathbf{A}}^{\mathrm{\top }}$$

Dot product ¶

If we have two vectors $\mathbf{u}$ and $\mathbf{v}$:

$$\mathbf{u}=\left[\begin{array}{c}{u}_1\\ u_2\\ ⋮\\ u_n\end{array}\right]$$

$$\mathbf{v}=\left[\begin{array}{c}{v}_1\\ v_2\\ ⋮\\ v_n\end{array}\right]$$

The dot product (aka inner product) of these vectors is defined as the transpose of $\mathbf{u}$ multipled by $\mathbf{v}$ :

$$\mathbf{u}\cdot \mathbf{v}={\mathbf{u}}^T\mathbf{v}=\left[\begin{array}{cccc}{u}_1& u_2& \cdots & u_n\end{array}\right]\left[\begin{array}{c}{v}_1\\ v_2\\ ⋮\\ v_n\end{array}\right]=u_1{v}_1+u_2{v}_2+\cdots +u_n{v}_n$$

Based on this definition, the dot product is commutative:

$\mathbf{u}\cdot \mathbf{v}=\mathbf{v}\cdot \mathbf{u}$

In the next post, we will return to the topic of eigenvalues and eigenvectors.