pytorch-attention-mechanism

my codes for learning attention mechanism

CNN with attention

Apply spatial attention to CIFAR100 dataset

Usage

Train the model:

$ python cnn-with-attention.py --train

Visualize attention map:

$ python cnn-with-attention.py --visualize

RNN with attention

Apply temporal attention to sequential data

e.g. A sequence of length 20, the output is only related to the 5th position and the 13th position

Usage

Train the model:

$ python rnn-with-attention.py --train

Visualize attention map:

$ python rnn-with-attention.py --visualize

Todos

• CNN+attention
• RNN+attention

References

• Learn to Pay Attention! Trainable Visual Attention in CNNs

• Attention in Neural Networks and How to Use It

• Attention? Attention!

• Tutorial - Visual Attention for Action Recognition

• BahdanauAttention与LuongAttention注意力机制简介

• 目前主流的attention方法都有哪些？

• Keras Attention Mechanism