Classification

Classification is similar to linear regression, insofar we are making a line in a high dimensional space. But instead of using this line to predict, we are using this line to segregate different classes.

The classification equation is

$$y=\text{sgn}\left(\sum _{i=1}^n{w}_i{x}_i+w_0\right)=\text{sgn}({\mathbf{w}}^T\mathbf{x}+w_0)$$

where if $y=1$ then $\mathbf{x}\in {\omega }_1$ and vice versa.

In this course, we learn to derive the optimal equation using Linear Discriminant Analysis and Fisher Criterion.

However, both of them only explains how to get the optimal direction of projection (i.e. the value of $\mathbf{w}$), while we still need the value of $w_0$.

We have a few option for the value of $w_0$,

1. $w_0=-\frac{1}{2}({\tilde{m}}_1+{\tilde{m}}_2)$
2. $w_0=-\tilde{m}$
3. $w_0=-\frac{1}{2}({\tilde{m}}_1+{\tilde{m}}_2)+\frac{1}{N_1+N_2-2}\ln \frac{P({\omega }_1)}{P({\omega }_2)}$

You have to choose based on (1) Domain situation, (2) Threshold with the ROC curve.

Note that you can’t just classify everything and anything — it depends on whether they are Linearly Separable or not. Using the Normalized Augmented Feature Vector, we can derive the cost function for a Perceptron to help with classification.

So what should we do if we have linearly non-separable cases? We have to use models with the least Minimum Squared Error.