Linear Discriminant Analysis

This is for classification tasks. We want to know how to best separate features.

Let’s say we have two features. Sample set: $\mathcal{X}=\{x_1,...,x_N\}$ where, Class 1$({\omega }_1):{\mathcal{X}}_1=\{x_1^1,...,x_{N_1}^1\}$ and Class 2$({\omega }_2):{\mathcal{X}}_2=\{x_1^2,...,x_{N_2}^2\}$

Projection is

$$\mathcal{X}\to \mathcal{Y}:y_i={\mathbf{w}}^T{\mathbf{x}}_i,i=1,...,N$$

You need to find the value of $\mathbf{w}$ that is best separate the two classes, where within class : as close as possible, and between class: as far as possible.

In the $\mathcal{X}$ space,

1. Class mean: ${\mathbf{m}}_i=\frac{1}{N_i}{\sum }_{x_j\in {\mathcal{X}}_i}{\mathbf{x}}_{j}i=1,2$
2. within class scatter matrix: ${\mathbf{S}}_i={\sum }_{x_j\in {\mathcal{X}}_i}({\mathbf{x}}_j-{\mathbf{m}}_i)({\mathbf{x}}_j-{\mathbf{m}}_i{)}^{T}i=1,2$
   • Total within class scatter matrix: ${\mathbf{S}}_{\mathbf{w}}={\mathbf{S}}_1+{\mathbf{S}}_2$
3. between class scatter matrix: ${\mathbf{S}}_b=({\mathbf{m}}_1-{\mathbf{m}}_2)({\mathbf{m}}_1-{\mathbf{m}}_2{)}^T$

In the $\mathcal{Y}$ space,

1. Class mean: ${\tilde{m}}_i=\frac{1}{N_i}{\sum }_{y_j\in {\mathcal{Y}}_i}{y}_j$
2. within class scatter matrix: ${\tilde{S}}_i={\sum }_{y_j\in {\mathcal{Y}}_i}(y_j-{\tilde{m}}_i)(y_j-{\tilde{m}}_i{)}^T$
   • total within class scatter matrix: ${\tilde{S}}_w={\tilde{S}}_i+{\tilde{S}}_2$
3. between class scatter matrix: ${\tilde{S}}_b=({\tilde{m}}_1-{\tilde{m}}_2{)}^2$