Attention Mechanisms

An implementation of multiple notable attention mechanisms using TensorFlow 2.

ReZero

from attention_mechanism.utils import ReZero

Reversible Sequence

Part of Reformer

from attention_mechanism.sequence import ReversibleSequence

Self-attention

Traditional Multi head attention with scaled dot product

from attention_mechanism.attention import MultiHeadAttention

Standalone self-attention module with linear complexity with respect to sequence length using FAVOR+

from attention_mechanism.attention import SelfAttention

Performer

The module PerformerLM is designed to work with Language Model, and Performer can be applied to more general high input dimensions such as images

from attention_mechanism.performer import Performer, PerformerLM

Lambda Layer for Lambda Networks

from attention_mechanism.lamda_layer import LambdaLayer

Vision Transformer

from attention_mechanism.vision_transformer import VisionTransformer

Fast Autoregressive

Incoming...