Separability of Classes

If we want to do feature selection, we can’t just do trial and error to find the features that correspond to the lowest error ⇒ too expensive!

So, we have to determine the separability of the features. The more separable they are, the better classification work and therefore higher accuracy. Doing this requires some metrics, and there are a bunch of metrics.

Metrics based on distributions

To measure the overlapping of two distributions:

$$J_p(\cdot )=\int g[p(\mathbf{x},{\omega }_1),p(\mathbf{x},{\omega }_2),P_1,P_2]d\mathbf{x}$$

$\mathbf{x}$ is the feature vector; ${\omega }_1,{\omega }_2$ are the two classes; $P_i$ is the probability of class ${\omega }_i$; and $g[\cdot ]$ is the function that computes the overlap between the distributions.

There are many equations, but unnecessary for now.

Metrics based on information theory

We have Shannon entropy, for feature $x$:

$$H=-\sum _{i=1}^{c}P({\omega }_i|x){\log }_{2}P({\omega }_i|x)$$