Added QARepVGGBlock #562
Merged
Added QARepVGGBlock #562
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ofrimasad
reviewed
Dec 28, 2022
There was a problem hiding this comment.
This PR breaks the logic we have built for quantization.
instead of having a block and registering a quantized block, we define here a block that knows how to be quantized. I don't like this and I am not sure exactly how that will work with the rest of the quantization code.
Instead, lets define a block without the quantization part and (no if qat_mode) and override some logics in a second block that is registered as his Quantized version
…/super-gradients into feature/implement-qarepvgg-block
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
QARepVGG (S4) block from 'Make RepVGG Greater Again: A Quantization-aware Approach' (https://arxiv.org/pdf/2212.01593.pdf)
It consists of three branches:
3x3: a branch of a 3x3 Convolution + BatchNorm
1x1: a branch of a 1x1 Convolution with bias
identity: a Residual branch
BatchNorm is applied after summation of all three branches.
In contrast to our implementation of RepVGGBlock, SE is applied AFTER NONLINEARITY in order to fuse Conv+Act in inference frameworks.
This module converts to Conv+Act in a PTQ-friendly way by calling QARepVGGBlock.fuse_block_residual_branches().
Has the same API as RepVGGBlock and is designed to be a plug-and-play replacement but is not compatible parameter-wise.
Has less trainable parameters than RepVGGBlock because it has only 2 BatchNorms instead of 3.