Linear Regression

‘The basic formula:

$$f(\mathbf{x})=\sum _{i=0}^d{w}_i{x}_i={\mathbf{w}}^T\mathbf{x}$$

The training data:

$$\begin{gathered} \{({\mathbf{x}}_1,y_1),...,({\mathbf{x}}_N,y_N)\}, \\ {\mathbf{x}}_j\in R^{d+1},y_j\in R \end{gathered}$$

Objective function:

$$\min E=\frac{1}{N}\sum _{j=1}^N(f({\mathbf{x}}_j)-y_j{)}^2$$

In here, $E$ is the mean squared error.

We get the optimal weight $w^{\ast }$ when we differentiate $E$ w.r.t. $w$. Let’s expand $E$ first.

$$\begin{array}{rl}E& =\frac{1}{N}||\mathbf{Xw}-\mathbf{y}|{|}^2\\ & =\frac{1}{N}(\mathbf{Xw}-\mathbf{y}{)}^T(\mathbf{Xw}-\mathbf{y})\\ & =\frac{1}{N}({\mathbf{w}}^T{\mathbf{X}}^T\mathbf{Xw}+\underset{\text{scalar}}{\underbrace{{\mathbf{w}}^T{\mathbf{X}}^T\mathbf{y}-{\mathbf{y}}^T\mathbf{Xw}}}+{\mathbf{y}}^{\mathbf{T}}\mathbf{y})\\ & =\frac{1}{N}({\mathbf{w}}^T{\mathbf{X}}^T\mathbf{Xw}+2{\mathbf{y}}^T\mathbf{Xw}+{\mathbf{y}}^{\mathbf{T}}\mathbf{y})\end{array}$$

Next, we have to use Standard Results of Matrix Calculus, equation (4) and (1).

$$\begin{array}{rl}\frac{\partial E}{\partial w}& =\frac{1}{N}(\underset{\text{equation(4), }\mathbf{A}={\mathbf{X}}^{\mathbf{T}}\mathbf{X}}{\underbrace{\mathbf{w}({\mathbf{X}}^T\mathbf{X}+({\mathbf{X}}^{\mathbf{T}}\mathbf{X}{)}^{\mathbf{T}})}}-\underset{\text{equation(1)}}{\underbrace{(2{\mathbf{y}}^T\mathbf{X}{)}^T}})\\ & =\frac{1}{N}(\underset{\text{Since }{\mathbf{X}}^X\mathbf{X}\text{ is symmetric}}{\underbrace{2{\mathbf{X}}^T\mathbf{Xw}}}-2{\mathbf{X}}^T\mathbf{y})\\ & =\frac{2}{N}{\mathbf{X}}^T(\mathbf{Xw}-\mathbf{y})\end{array}$$

We set $\frac{\partial E}{\partial w}=0$.

$$<spanstyle="color:rgb(0,32,96)">\begin{array}{rl}<br>{\mathbf{X}}^T\mathbf{Xw}& ={\mathbf{X}}^T\mathbf{y}\\ <br>{\mathbf{w}}^{\ast }& =({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{y}<br>\end{array}</span>$$

Note that the last line is only applicable if ${\mathbf{X}}^T\mathbf{X}$ is invertible.

To determine the fitness of the model, we have R2 Goodness of Fit.