This report summarizes the findings of the original Squeeze-and-Excitation Networks paper and shows a reproduction of the results with Tensorflow [1][6]. Convolutional neural networks are a building block of all state-of-the-art models for image classification, including the well-known ResNet architecture. This report investigates the effectiveness of Squeeze-and-Excitation blocks whose aim is to strengthen the inter-channel relationship by performing channel-wise feature recalibration. The original paper showed that it consistently increased the classification accuracy with various data sets such as Image Net. This paper is going to depict how we were able to increase the accuracy of CNN models with the Squeeze-and-Excitation method. Because of technical barriers, we came up with slightly different models and data sets. We also tried to analyze the effect of the key parameters of Squeeze-and-Excitation blocks on our models, matching the paper’s findings with varying success.
[1] J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu, Squeeze-and-Excitation Networks. 2019.
Models are ordered by test category folders (link below)
Drive directory tree
full_models/
├── ablation/
│ ├── SE_identity.h5
│ ├── SE_post.h5
│ └── SE_pre.h5
├── cifar10/
│ ├── inception weights/
│ │ ├── inception_v3_custom_cifar10.ckpt.data-00000-of-00001
│ │ └── inception_v3_custom_cifar10.ckpt.index
│ ├── resnet_best.h5
│ ├── SE-inception weights/
│ │ ├── se_inception_v3_custom_cifar10.ckpt.data-00000-of-00001
│ │ └── se_inception_v3_custom_cifar10.ckpt.index
│ └── SE_resnet_best.h5
├── cifar100/
│ ├── resnet_best.h5
│ └── SE_resnet_best.h5
├── ratio/
│ ├── se_resnet_ratio_1.h5
│ ├── se_resnet_ratio_2.h5
│ ├── se_resnet_ratio_32.h5
│ ├── se_resnet_ratio_4.h5
│ └── se_resnet_ratio_8.h5
├── stage/
│ ├── stage2.h5
│ ├── stage3.h5
│ └── stage4.h5
└── tinyImageNet/
├── resnet_18_aug_best.h5
├── resnet_18_best.h5
├── resnet_34_aug_best.h5
├── resnet_34_best.h5
├── resnet_50_aug_best.h5
├── resnet_50_best.h5
├── SE-resnet_18_aug_best.h5
├── SE-resnet_18_best.h5
├── SE-resnet_18best.h5
├── SE-resnet_34_aug_best.h5
├── SE-resnet_34_best.h5
├── SE-resnet_50_aug_best.h5
└── SE-resnet_50_best.h5
Lion drive link (https://drive.google.com/drive/folders/15kpxrvAyOuMmiqZfGMf_imL12vHBefLx?usp=sharing)
- Load and Test Models.ipynb loads all the saved models and computes the top-1,3 and 5 accuracy on the associated test data sets.
- ResNet with CIFAR10.ipynb Shows the training process and results of ResNet et SE-Resnet models on CIFAR-10
- ResNeXt with CIFAR10.ipynb Shows the training process and results of ResNeXt et SE-ResneXt models on CIFAR-10
- InceptionV3 with CIFAR10-100.ipynb Shows the training process and results of InceptionV3 et SE-InceptionV3 models on CIFAR-10
- ResNet with CIFAR100.ipynb Shows the training process and results of ResNet et SE-Resnet models on CIFAR-100
- ResNeXt with CIFAR100.ipynb Shows the training process and results of ResNeXt et SE-ResneXt models on CIFAR-100
- InceptionV3 with CIFAR10-100.ipynb Shows the training process and results of InceptionV3 et SE-InceptionV3 models on CIFAR-100
- ResNet18 with tinyImageNet.ipynb Shows the training process and results of ResNet-18 et SE-Resnet-18 models on Tiny ImageNet with and without data augmentation
- ResNet34 with tinyImageNet.ipynb Shows the training process and results of ResNet-34 et SE-Resnet-34 models on Tiny ImageNet with and without data augmentation
- ResNet50 with tinyImageNet.ipynb Shows the training process and results of ResNet-50 et SE-Resnet-50 models on Tiny ImageNet with and without data augmentation
- analysis_ablation.ipynb shows the ablation study tests (Different SE block integrations)
- analysis_activation.ipynb shows the different activation distributions accross channels for all stages and block ids
- analysis_inference.ipynb shows the inference speed performance of different ResNet et SE-ResNet models with Tiny ImageNet
- analysis_ratio.ipynb shows the effect of the ratio parameter on the accuracies
- analysis_stage.ipynb shows the impact of each stage where SE blocks are added.
Description is also in the ./utils directory
- custom_resnet.py: builds custom ResNet models with the specified input, output sizes, stages, block multiplicity, and kernel sizes.
- custom_ResNeXt.py: builds custom ResNeXt models
- SE_resnet.py: build custom SE-ResNet models with the specified input, output sizes, stages, block multiplicity, and kernel sizes.
- SE_ResNeXt.py: build custom SE-ResNeXt models
- add_SE.py: adds SE blocks to any models by entering a list of layers where the SE blocks go.
- ablation_resnet.py: builds custom SE-Resnet models with different SE-block integration methods: Standard, POST, PREO and Identity
- evaluate_model.py: evaluate model accuracy with Top-n accuray parameters
- train_xxx.py: all the training functions for the CIFAR, Tiny ImageNet and ratio tests on different models.
Contains graphs of the models of the ablation test and stages test
Contains miscellaneous figures such as the GCP screenshots
- CIFAR-10-100 https://www.cs.toronto.edu/~kriz/cifar.html or use the Tensorflow included functions.
- Tiny ImageNet http://cs231n.stanford.edu/tiny-imagenet-200.zip
Linux commands:
wget http://cs231n.stanford.edu/tiny-imagenet-200.zip
unzip tiny-imagenet-200.zip
rm tiny-imagenet-200.zip
/
├── analysis_ablation.ipynb
├── analysis_activation.ipynb
├── analysis_inference.ipynb
├── analysis_ratio.ipynb
├── analysis_stage.ipynb
├── figures/
│ ├── inceptionv3.png
│ ├── screen_GCP_1.PNG
│ ├── screen_GCP_2.PNG
│ └── screen_GCP_3.PNG
├── img/
│ ├── ablation_identity.png
│ ├── ablation_post.png
│ ├── ablation_pre.png
│ ├── ablation_standard.png
│ ├── stage_2.png
│ ├── stage_3.png
│ └── stage_4.png
├── InceptionV3 with CIFAR10-100.ipynb
├── Load and Test Models.ipynb
├── README.md
├── ResNet with CIFAR10.ipynb
├── ResNet with CIFAR100.ipynb
├── ResNet18 with tinyImageNet.ipynb
├── ResNet34 with tinyImageNet.ipynb
├── ResNet50 with tinyImageNet.ipynb
├── ResNeXt with CIFAR10.ipynb
├── ResNeXt with CIFAR100.ipynb
├── Untitled.ipynb
├── utils/
│ ├── ablation_resnet.py
│ ├── add_SE.py
│ ├── custom_resnet.py
│ ├── custom_ResNeXt.py
│ ├── evaluate_model.py
│ ├── README.md
│ ├── SE_inception_v3.py
│ ├── SE_resnet.py
│ ├── SE_ResNeXt.py
│ ├── tiny_imageNet.py
│ ├── train_CIFAR_all.py
│ ├── train_CIFAR_ResNet.py
│ ├── train_CIFAR_ResNeXt.py
│ ├── train_ratio.py
│ └── train_TinyImageNet_SE_ResNet.py
└── Wael InceptionV3.ipynb