Logistic Regression

The model is

$$h(\mathbf{x})=\theta ({\mathbf{w}}^T\mathbf{x})$$

Then the training data is

$$\left\{\begin{array}{c}({\mathbf{x}}_1,y_1),\dots ,({\mathbf{x}}_N,y_N)\end{array}\right\},{\mathbf{x}}_j\in {\mathbb{R}}^{d+1},y_j\in \{-1,1\}$$

For objective function, We get the likelihood of $({\mathbf{x}}_j,y_j)$ (trust me):

$$P(y_j|{\mathbf{x}}_j)=\theta (y_j{\mathbf{w}}^T{\mathbf{x}}_j)$$

Then we do maximum likelihood estimation…

$$L(\mathbf{w})=\prod _{j=1}^{N}P(y_j|{\mathbf{x}}_j)=\prod _{j=1}^N\theta (y_j{\mathbf{w}}^T{\mathbf{x}}_j)$$ $$\begin{array}{rl}\min E(\mathbf{w})& =-\frac{1}{N}\ln (L(\mathbf{w}))=-\frac{1}{N}\ln \left(\prod _{j=1}^N\theta (y_j{\mathbf{w}}^T{\mathbf{x}}_j)\right)\\ & \\ & =\frac{1}{N}\sum _{j=1}^N\ln \left(\frac{1}{\theta (y_j{\mathbf{w}}^T{\mathbf{x}}_j)}\right)=\frac{1}{N}\sum _{j=1}^N\ln (1+e^{-y_n{\mathbf{w}}^T{\mathbf{x}}_j})\end{array}$$

Then we differentiate wrt $\mathbf{w}$.

$$▽E=-\frac{1}{N}\sum _{j=1}^N\frac{y_j{\mathbf{x}}_j}{1+e^{y_j\mathbf{w}(k{)}^T{\mathbf{x}}_j}}$$

And so, the learning algorithm is

$$\mathbf{w}(k+1)=\mathbf{w}(k)-{\rho }_k▽E$$