Multi-Layer Perceptron

The main breakthrough here, compared to previous methods is the addition of activation function.

In here, the activation function is the sigmoid function:

$$f(\alpha )=\frac{1}{1+e^{-1(\alpha -\theta )}}$$

Based on the Kolmogorov Theorem,

Kolmogorov Theorem

Any continuous function $g(\mathbf{x})$ defined on the unit hypercube $I^n$ where $I=[0,1]$ and $n\ge 2$ can be represented in the form

$$g(\mathbf{x})=\sum _{j=1}^{2n+1}{\Xi }_j\left(\sum _{i=1}^d{\psi }_{ij}(x_i)\right)$$

for a properly chosen function ${\Xi }_j$ and ${\psi }_{ij}$.

As such,

Universal Approximation Theorem

$$g_k(\mathbf{x})\equiv y_k=f\left(\sum _j{w}_{jk}f\left(\sum _i{w}_{ij}{x}_i+w_{j0}\right)+w_{k0}\right)$$

Which means, MLP is capable of “implementing” any continuous function from input to output, given sufficient number of hidden units, proper activation function and weights.

Definition of MLP

The model:

$$g(\mathbf{x})=f\left(\sum _j{w}_{jk}f\left(\sum _i{w}_{ij}{x}_i+w_{j_0}\right)+w_{k_0}\right)$$

The training data:

$$\{({\mathbf{x}}_1,y_1),\dots ,({\mathbf{x}}_N,y_N)\},{\mathbf{x}}_j\in {\mathbb{R}}^{d+1},y_j\in \mathbb{R}$$

Objective function:

$$\min E=\frac{1}{2}\sum _{j=1}^N(g({\mathbf{x}}_j)-y_j{)}^2$$

And finally, the learning algorithm:

$$\mathbf{w}(k+1)=\mathbf{w}(k)-{\rho }_k▽E$$

Of course, to get $▽E$ we need to differentiate wrt $\mathbf{w}$. In the following equation, $k$ refers to the output layer, while $z$ refers to the value before activation function and $a$ is after.

For the output layer, we have ${\delta }_k$ which is the value of $\frac{\partial E}{\partial z_k}$. We assume $f(x)$ is sigmoid.

$${\delta }_k=(a_k-y_k)\cdot f^{\prime }(z_k)=(a_k-y_k)a_k(1-a_k)$$

For the hidden layers, we have

$$\begin{array}{rl}{\delta }_j& =\frac{\partial L}{\partial z_j}\\ & =\sum _k\frac{\partial L}{\partial z_k}\frac{\partial z_k}{\partial a_j}\frac{\partial a_j}{\partial z_j}\\ & =\sum _k{\delta }_k{w}_{kj}{f}^{\prime }(z_j)\\ & =f^{\prime }(z_j)\sum _k{w}_{kj}{\delta }_k\\ & =a_j(1-a_j)\sum _k{w}_{kj}{\delta }_k\end{array}$$

Note that in multi-class classification setting, we usually use cross-entropy loss as the objective function instead.

$$E_i=-\sum _{i=1}^C{y}_i\log ({\hat{y}}_i)$$

where ${\hat{y}}_i$ is the softmax output of the last layer,

$${\hat{y}}_i=\frac{e^{z_i}}{{\sum }_j{e}^{z_j}}$$