Fisher Criterion

Following LDA, we can then formalize Fisher’s Criterion for best separation

$$\underset{\mathbf{w}}{\max }{J}_F(\mathbf{w})=\frac{({\tilde{m}}_1-{\tilde{m}}_2{)}^2}{{\tilde{S}}_1+{\tilde{S}}_2}$$

where $y_i={\mathbf{w}}^T{\mathbf{x}}_i$

So the optimal w, ${\mathbf{w}}^{\ast }=\arg {\max }_{\mathbf{w}}{J}_F(\mathbf{w})$

$$J_F(\mathbf{w})=\frac{{\mathbf{w}}^T{\mathbf{S}}_b\mathbf{w}}{{\mathbf{w}}^T{\mathbf{S}}_w\mathbf{w}}$$

Note that the derivation comes from substituting $y_i={\mathbf{w}}^T{\mathbf{x}}_i$

Some things to note with the value of $\max J_F(\mathbf{w})$: - Not unique, so if we change the scale of $\mathbf{w}$, the value of $J_F(\mathbf{w})$ won’t change. - Of course, we can fix the denominator ${\mathbf{w}}^T{\mathbf{S}}_w\mathbf{w}=c\ne 0$, and then maximise the numerator ${\mathbf{w}}^T{\mathbf{S}}_b\mathbf{w}$

We define a Langragian function:

$$L(\mathbf{w},\lambda )={\mathbf{w}}^T{\mathbf{S}}_b\mathbf{w}-\lambda ({\mathbf{w}}^T{\mathbf{S}}_w\mathbf{w}-c)$$

Let $\frac{\partial L}{\partial w}=0$,

$${\mathbf{S}}_w^{-1}{\mathbf{S}}_b{\mathbf{w}}^{\ast }=\lambda {\mathbf{w}}^{\ast }$$

which means that ${\mathbf{w}}^{\ast }$ is the eigenvector of matrix ${\mathbf{S}}_w^{-1}{\mathbf{S}}_b$.

Substituting ${\mathbf{S}}_b=({\mathbf{m}}_1-{\mathbf{m}}_2)({\mathbf{m}}_1-{\mathbf{m}}_2{)}^T$, we eventually will have

$${\mathbf{w}}^{\ast }=S_w^{-1}({\mathbf{m}}_1-{\mathbf{m}}_2)$$

Remember that $w^{\ast }$is the direction of projection.

Note that this is the binary linear discriminant. The multiclass version is slighty different.