Feature Extraction

Fisher Criterion is basically projecting into a 1D line for separability. The solution is

$${\mathbf{w}}^{\ast }=S_w^{-1}({\mathbf{m}}_1-{\mathbf{m}}_2)$$

which we gets from maximizing

$$\max J_F(\mathbf{w})=\frac{{\mathbf{w}}^T{\mathbf{S}}_b\mathbf{w}}{{\mathbf{w}}^T{\mathbf{S}}_w\mathbf{w}}$$

We can generalize this. If we have separability criteria $J_1,J_2,J_3,J_4,J_5$, all the criteria can be mazimized by using eigenvectors corresponding to the largest eigenvalues of $S_w^{-1}{S}_b$. Hence, the feature extraction matrix

$$W=[u_1,u_2,\dots ,u_d]$$

contains the top $d$ eigenvectors.

It is a good time to introduce Karhunen-Loeve Transform

$$\mathbf{w}\ast =S_w^{-1}({\mathbf{m}}_1-{\mathbf{m}}_2)$$ $$\max J_F(\mathbf{w})=({\mathbf{w}}^T{\mathbf{S}}_b)$$