t-SNE

This is developed since PCA cannot handle non-linear data. The logic is that we want to create probability of the high-dimensional data, and then find the most similar probability for the low-dimensional data.

1. Model the neighborhood of the high dimensional data as distribution
   • We use t-distribution here (that’s where the t in t-SNE comes from)
   $$p_{j|i}=\frac{\exp (-{\Vert \begin{array}{c}{x}_i-x_j\end{array}\Vert }^2/2{\sigma }_i^2)}{{\sum }_{k\ne i}\exp (-{\Vert \begin{array}{c}{x}_i-x_k\end{array}\Vert }^2/2{\sigma }_i^2)}$$
   • The value of ${\sigma }_i$ is chosen by hand
   • The value of $k$ is the number of local neighbor to care about, or the perplexity
2. Model the neighborhood of the low dimensional data as distribution

$$q_{j|i}=\frac{\exp (-{\Vert \begin{array}{c}{y}_i-y_j\end{array}\Vert }^2)}{{\sum }_{k\ne i}\exp (-{\Vert \begin{array}{c}{y}_i-y_k\end{array}\Vert }^2)}$$

3. Find the cost function, which is the KL-divergence

$$\text{KL-Div}=\sum _j{p}_{j|i}\log \frac{p_{j|i}}{q_{j|i}}$$

4. Gradient dexcent to find the distribution of low dimensional data