Karhunen-Loeve Transform

Intutively, we want to transform the samples into a different coordinate system where the new axes are uncorrelated, and most of the information is concentrated in just a few axes.

Let’s say you have a random vector $x\in {\mathbb{R}}^D$. The covariance matrix of the data is:

$$\Psi =E[(x-\mu )(x-\mu {)}^T]=E[x{x}^T]$$

The covariance matrix reflects how spread out the data is, and how correlated one dimension with another.

We have to perform eigen-decomposition $\Psi$ next.

$$\Psi u_i={\lambda }_i{u}_i$$

where $u_i$ is the eigenvector and ${\lambda }_i$ is the eigenvalue. Next, we project the original data $x$ to new axes:

$$c_i=u_i^{T}x$$

Following Feature Extraction, we should only keep the first $d$ coefficient with the largest eigenvalues, to get a compressed representation:

$$\tilde{x}=\sum _{i=1}^d{c}_i{u}_i$$