Lecture 4 -- Attention & Transformer

• In RNNs, we have a bottleneck in the encoder.
• This is because the decoder never sees the input text, but only the final vector
• This many-to-one compression results in many lost information

Attention

Core idea: We use direct connection to encoder to focus on the relevant part of the source sentence

• We have encoder hidden states $h_1,\dots ,h_N\in {\mathbb{R}}^h$
• On timestep $t$, we have decoder hidden state $s_t\in {\mathbb{R}}^h$
• We get the attention scores $e^t$ for this timestep:

$${\mathbf{e}}^t=[s_t^T{h}_1,\dots ,s_t^T{h}_N]\in {\mathbb{R}}^N$$

• We take softmax to get the attention distribution ${\alpha }^t$ for this step

$${\alpha }^t=\text{softmax}({\mathbf{e}}^t)\in {\mathbb{R}}^N$$

• We use ${\alpha }^t$ to take a weighted sum of the encoder hidden states to get the attention output ${\mathbf{a}}_t$

$${\mathbf{a}}_t=\sum _{i=1}^N{\alpha }_i^t{\mathbf{h}}_i\in {\mathbb{R}}^h$$

• Concatenate the attention output ${\mathbf{a}}_t$ with the decoder hidden state, then compute ${\hat{y}}_1$ as before

$${\hat{y}}_t=\text{softmax}(W_y\cdot [a_t;s_t]+b_y)$$

Why attention?

1. Attention improves MT performance significantly
2. It solves the bottleneck problem
3. It helps with vanishing gradient problem
4. It provides interpretability in terms of attention patterns

Attention as QKV computation

• From the above example,
  • The encoder hidden states $h_1,\dots ,h_N\in {\mathbb{R}}^h$ are the keys/values
  • The decoder hidden state $s_t\in {\mathbb{R}}^h$ are is the query

Self-attention

• Usually attentions are applied between encoder and decoder for seq2seq learning
• For one single sequence only, we have self-attention
• In RNNs, we do things in sequence, which is unparallelizable
• With self-attention, we can parallelize per layer, as all words interact at every layer

Let ${\mathbf{w}}_{i:n}$ be the sequence of words in vocabulary $V$. For each ${\mathbf{w}}_i$, let ${\mathbf{x}}_i=E{\mathbf{w}}_i$, where $E\in {\mathbb{R}}^{d\times |V|}$ is an embedding matrix.

1. Transform each word embedding with weight matrices $Q,K,V$, each in ${\mathbb{R}}^{d\times d}$

$$q_i=Q{\mathbf{x}}_i{k}_i=K{\mathbf{x}}_i{v}_i=V{\mathbf{x}}_i$$

2. Compute pairwise similarities between keys and queries, normalize with softmax

$$e_{ij}=q_i^T{k}_j{\alpha }_{ij}=\frac{\exp ({\mathbf{e}}_{ij})}{{\sum }_{j^{\prime }}\exp ({\mathbf{e}}_{i{j}^{\prime }})}$$

3. Compute output for each word as weighted sum of values

$$o_i=\sum _j{\alpha }_{ij}{v}_j$$

Transformer

Needs to add:

1. Positional embeddings
2. Non-linearity
3. Single to multi-head self attention
4. Multiple layers

Positional embeddings

• We have positional embedding ${\mathbf{p}}_i\in {\mathbb{R}}^d$, and we just add this to the embedding

$${\mathbf{x}}_i={\mathbf{x}}_i+{\mathbf{p}}_i$$

• We can make these positional embedding learned parameter, but in doing so, we can’t extrapolate to indices outside $1,\dots ,n$.

Non-linearity

• Self-attention is just re-averaging the value vectors, there’s no non-linearity here
• We add a feed-forward network to post-process each output vector

$$m_i=\text{MLP}(o_i)=W_2\times \text{ReLU}(W_1{o}_1+b_1)+b_2$$

Multi-head self-attention

• We just do self-attention multiple times, but with different projection matrices $Q,K,V$
• After we get each head’s attention score, we concatenate everything to get the overall score
• Each head should focus on different “feature” of the sentence

Add & Norm

• We add as a residual connection to help the gradient flows better
  • We let $X^{(i)}=X^{(i-1)}+\text{Layer}(X^{(i-1)})$
• We also do layer norm to help models train faster
  • Idea: cut down uninformative variation in hidden vector values by normalizing to unit mean and standard dev within each layer