Lecture 1 -- Introduction to Deep Learning

Linear Regression

Crucial components:

• Input vector: $x=[x_1,x_2,\dots ,x_n]$
• Weight vector (one per feature): $W=[w_1,w_2,\dots ,w_n]$
  • Note that the weight vector tells how important each feature are.
• And it outputs a predicted class $\hat{y}\in \{0,1\}$

$${\hat{y}}_i={\mathbf{x}}_i^T\mathbf{w}$$

• Objective function: minimize the residual sum of squares (RSS)

$$\begin{array}{rl}J(\theta )& =\arg \underset{\mathbf{w}}{\min }\sum _{i=1}^N(y_i-{\hat{y}}_i{)}^2\\ & =\sum _{i=1}^N(y_i-{\mathbf{x}}_i^T\mathbf{w}{)}^2=(\mathbf{y}-\mathbf{Xw}{)}^T(\mathbf{y}-\mathbf{Xw})\end{array}$$

TODO: I should look through the derivation of the derivative and try it by hand.

Logistic Regression (Single Class)

• We do the same as above, but we add a sigmoid function → $\sigma (z_i)$

$$\sigma (z_i)=\frac{e^{z_i}}{e^{z_i}+1}$$

• We set the threshold at 0.5

$$\hat{y}=\{\begin{array}{ll}1& \text{if }\sigma (\mathbf{w}x+b)>0.5\\ 0& \text{otherwise}\end{array}$$

• Objective function: Cross Entropy error function

$${\mathbf{w}}^{\ast }=\arg \underset{w}{\min }-\sum _{i=1}^N(y_i\log {\mu }_i+(1-y_i)\log (1-{\mu }_i))$$

Logistic Regression (Multi-Classs)

• Instead of sigmoid function, we use softmax

$$\mathcal{P}(y_i=c|{\mathbf{x}}_{i,\mathbf{W}})=\frac{\exp ({\mathbf{w}}_c^T{\mathbf{x}}_i)}{{\sum }_{c^{\prime }=1}^C\exp ({\mathbf{w}}_{c^{\prime }}^T{\mathbf{x}}_i)}$$

• So our objective function is

$$\begin{array}{rl}{\mathbf{W}}^{\ast }& =\sum _{i=1}^N\sum _{c=1}^C{y}_{ic}\log (\text{softmax}({\mathbf{w}}_c^T{\mathbf{x}}_i))\\ & =\arg \underset{\mathbf{W}}{\max }\sum _{i=1}^N\left[\left(\sum _{c=1}^C{y}_{ic}{\mathbf{w}}_c^T{\mathbf{x}}_i\right)-\log \left(\sum _{c^{\prime }=1}^C\exp ({\mathbf{w}}_{c^{\prime }}^T{\mathbf{x}}_i)\right)\right]\end{array}$$

Gradient Descent

1. Batch gradient descent: uses the entire dataset. $\theta \leftarrow \theta -\eta \cdot {▽}_{\theta }\frac{1}{N}{\sum }_{i=1}^N{J}_i$
2. Stochastic gradient descent: uses one data point. $\theta \leftarrow \theta -\eta \cdot {▽}_{\theta }{J}_{\pi (i)}$
3. Mini-batch gradient descent combines both. $\theta \leftarrow \eta \cdot {▽}_{\theta }\frac{1}{S}{\sum }_{i=b_1}^{b_S}{J}_i$

Backpropagation

$$\frac{\partial z}{\partial x}=\sum _{i=1}^n\frac{\partial z}{\partial y_i}\frac{\partial y_i}{\partial x}$$

• Just slowly go back from the last layer
• Remember that the weights are between $a$ and $z$

Regularization

1. Dropout — make the model learns a sparse representation, has an ensemble effect
2. L1/2 regularization — discourages the weights from being too large

$$\arg \underset{\theta }{\min }\text{Loss}(\mathbf{\theta })+\lambda {\Vert \begin{array}{c}\mathbf{\theta }\end{array}\Vert }_p^p$$

- where $\lambda$ is the penalty hyperparameter and $p = 2$ or $1$, where

$${\Vert \begin{array}{c}\mathbf{\theta }\end{array}\Vert }_1=\sum _{j=1}^d|{\theta }_j|$$

3. Gradient clipping, we put a threshold on the gradient update. If it is above the threshold, we scale it down.