A transformer model based on sliding kernel self attention mechanism. This model is based on a implementation of Swin Transformer. See Swin Transformer repository for the original implementation.
Model | Params | Val. Acc. |
---|---|---|
Swin Transformer (tiny) | 26,598,166 | 82.19% @200eps |
Swin Transformer (tiny) | 26,598,166 | 83.34% @300eps |
Kernel Transformer (tiny) | 26,600,362 | 85.83% @300eps |
![kernel_vs_swin](https://private-user-images.githubusercontent.com/89094576/273986235-4d5581c2-bc09-4ac0-bd23-6f188dd011f1.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYxNzM2NjUsIm5iZiI6MTcyNjE3MzM2NSwicGF0aCI6Ii84OTA5NDU3Ni8yNzM5ODYyMzUtNGQ1NTgxYzItYmMwOS00YWMwLWJkMjMtNmYxODhkZDAxMWYxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEyVDIwMzYwNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTJmMjQ0MGY0YzMwMmUxNTc0ZGJlYTczOGQ2M2U0MDUxNjJiYThhYTEzNTk4ODk1YjNhZGU5ZTUyYzM2YmZlNjcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Z68XvJKEk8zqcbp5gR8GesD4OEs7Ci7knZ7csxRwbXU)