Attention Mechanism
Seq-to-Seq Model
The encoder processes each item in the input sequence, it compiles the information it captures into a vector (called the context) where, the context is basically a vector (an array of numbers). After processing the entire input sequence, the encoder sends the context over to the decoder, which begins producing the output sequence item by item.
Attention
The context vector turned out to be a bottleneck for these types of models. It made it challenging for the models to deal with long sentences. Attention allows the model to focus on the relevant parts of the input sequence as needed.
An attention model differs from a classic seq-to-seq model in two main ways:
First, the encoder passes a lot more data to the decoder. Instead of passing the last hidden state of the encoding stage, the encoder passes all the hidden states to the decoder.
Second, an attention decoder does an extra step before producing its output. In order to focus on the parts of the input that are relevant to this decoding time step, the decoder does the following:
Look at the set of encoder hidden states it received – each encoder hidden state is most associated with a certain word in the input sentence.
Give each hidden state a score and multiply each hidden state by its soft-maxed score. Thus, amplifying hidden states with high scores, and drowning out hidden states with low scores. This scoring exercise is done at each time step on the decoder side.
The entire attention process at decoder:
The attention decoder RNN takes in the embedding of the <END> token, and an initial decoder hidden state.
The RNN processes its inputs, producing an output and a new hidden state vector (h4). The output is discarded.
Attention Step: We use the encoder hidden states and the h4 vector to calculate a context vector (C4) for this time step.
We concatenate h4 and C4 into one vector.
We pass this vector through a feedforward neural network (one trained jointly with the model).
The output of the feedforward neural networks indicates the output word of this time step.
Repeat for the next time steps.
