NNs Reimplementations in keras
The purpose of this project is to offer flexible customizable options for modern NNs in keras.
Another problem statement is to suggest best practices for training ConvNets with limited amount of resources. For example, (Bello et al., 2021) showed differential strengths of variable layer settings when trainig length is considered as the optimizing variable. Likewise, some ConvNets may work better for certain training regime that are constrained by resource availability --- Wider and shorter ResNets may work better when training for shorter time than deeper ResNets.
- Add patch extraction method in CVT --- Feed Convolutional layer with kernel size K and strides K to reduce image resolution R to (R//K, R//K, C), where C= embedding dimension and projection dimension.
- Residual Blocks and BottleNeck Structure:
- Deep Residual Learning for Image Recognition (Kaiming He et al., 2015)
- Bag of Tricks for Image Classification with Convolutional Neural Networks (Tong He et al., 2018)
- Identity Mappings in Deep Residual Networks (Kaiming He et al., 2016)
- ResNeXt: Aggregated Residual Transformations for Deep Neural Networks (Saining Xie et al., 2016)
- ReXNet: Rethinking Channel Dimensions for Efficient Model Design (Dongyoon Han et al., 2021)
- ResNet strikes back: An improved training procedure in timm (Ross Wightman et al., 2021)
- Revisiting ResNets: Improved Training and Scaling Strategies (Irwan Bello et al., 2021)
- Inverted Residual Blocks and Linear BottleNeck
- MobileNetV2: Inverted Residuals and Linear Bottlenecks (Mark Sandler et al., 2018)
- Searching for MobileNetV3
- Other structures
- SqueezeNext: Hardware-Aware Neural Network Design
- ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks
- Squeeze-and-Excitation Networks
- Initialization
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification: HeNormal
- How Does Batch Normalization Help Optimization?
- Compact ConvNets
- MobileViT: Light-weight, General_purpose, and Mobile-friendly Vision Transformer
- Separable Self-attention for Mobile Vision Transformers
- Escaping the Big Data Paradigm with Compact Transformers
- Efficient Transformers
pip install git+https://github.com/johnypark/ConvNets-keras@main
ResNet() allows customizing number of channels, bottleneck layers, and number of blocks.
Official pytorch implementation: https://github.com/clovaai/rexnet
ResNet() allows customizing number of channels, bottleneck layers, and number of blocks.
-
ConvBlock: Basic convolutional layer followed by batch normalization and activaiton function.
-
BN_Res_Block: Building unit of ResNet, with BottleNeck structure first descirbed in He et al., (2015).
-
Inverted_BN_Block: Building unit of ReXNet, with a modified version of inverted BottleNeck structure described in Han et al. (2021), originally invented in Snadler et al. (2018).
Usage example building ResNet-50
import NeuralNets_keras as NNs
rs50 = NNs.ResNet(classes = 1000,
input_shape = (224, 224, 3),
N_filters = [256, 512, 1024, 2048],
N_BottleNecks = {256: 64, 512:128, 1024:256, 2048:512},
N_blocks = {256:3, 512:4, 1024:6, 2048:3},
stem_channels = 64,
stem_kernel = 7,
ResNetType = "C",
pooling = "average",
)Result:
BN_Residual_16_2_batch_norm (B (None, 7, 7, 512) 2048 ['BN_Residual_16_2_3x3conv_ch512[
atchNormalization) 0][0]']
BN_Residual_16_2_act (Activati (None, 7, 7, 512) 0 ['BN_Residual_16_2_batch_norm[0][
on) 0]']
BN_Residual_16_3_1x1conv_ch204 (None, 7, 7, 2048) 1050624 ['BN_Residual_16_2_act[0][0]']
8 (Conv2D)
BN_Residual_16_3_batch_norm (B (None, 7, 7, 2048) 8192 ['BN_Residual_16_3_1x1conv_ch2048
atchNormalization) [0][0]']
BN_Residual_16_3_act (Activati (None, 7, 7, 2048) 0 ['BN_Residual_16_3_batch_norm[0][
on) 0]']
add_15 (Add) (None, 7, 7, 2048) 0 ['add_14[0][0]',
'BN_Residual_16_3_act[0][0]']
global_average_pooling2d (Glob (None, 2048) 0 ['add_15[0][0]']
alAveragePooling2D)
dense (Dense) (None, 1000) 2049000 ['global_average_pooling2d[0][0]'
]
==================================================================================================
Total params: 25,656,136
Trainable params: 25,602,888
Non-trainable params: 53,248
__________________________________________________________________________________________________