Abstract
We propose a novel framework based on the attention mechanism to
identify the sentiment of a movie review document. Previous efforts on
deep neural networks with attention mechanisms focus on encoder and
decoder with fixed numbers of multi-head attention. Therefore, we need a
mechanism to stop the attention process automatically if no more useful
information can be read from the memory.In this paper, we propose an
adaptive multi-head attention architecture (AdaptAttn) which varies the
number of attention heads based on length of sentences. AdaptAttn has a
data preprocessing step where each document is classified into any one
of the three bins small, medium or large based on length of the
sentence. The document classified as small goes through two heads in
each layer, the medium group passes four heads and the large group is
processed by eight heads. We examine the merit of our model on the
Stanford large movie review dataset. The experimental results show that
the F1 score from our model is on par with the baseline model.