Minimum Squared Error

Instead of finding ${\mathbf{\alpha }}^T\mathbf{y}>0$ for all samples, we find for as many samples as possible.

One method to do this is by introducing an error / bias term. Basically, we change from the cost function of Perceptron

$$J_P(\mathbf{\alpha })=\sum _{{\mathbf{y}}_j\in {\mathcal{Y}}^k}(-{\mathbf{\alpha }}^T{\mathbf{y}}_j)$$

to

$$J_s(\mathbf{\alpha })=\sum _{i=1}^N({\mathbf{\alpha }}^T{\mathbf{y}}_i-b_i{)}^2$$

We then perform differentiation for gradient descent to get

$$▽J_s(\mathbf{\alpha })=2{\mathbf{Y}}^T(\mathbf{Y}\mathbf{\alpha }-\mathbf{b})$$

and so we update the weights for every example with

$$\mathbf{\alpha }(k+1)=\mathbf{\alpha }(k)+{\rho }_k(b_k-\mathbf{\alpha }(k{)}^T{\mathbf{y}}^k){\mathbf{y}}^k$$

This is the ADALINE / LMS algorithm.