Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/sg 573 Integrate new EMA decay schedules #647

Merged
merged 18 commits into from
Jan 30, 2023

Conversation

BloodAxe
Copy link
Collaborator

@BloodAxe BloodAxe commented Jan 23, 2023

This PR brings option to use additional EMA decay schedules:

  • constant - Fixed EMA decay value (Former config: exp_activation: False)
  • exp - EMA decay warmup using exp rule decay * (1 - math.exp(-x * self.beta)) (Former config: exp_activation: True)
  • threshold - New schedule used in PP-Yolo training min(decay, (1 + step) / (10 + step))

Examples:

ema: True
ema_params:
  decay_type: constant
  decay: 0.9999
ema: True
ema_params:
  decay: 0.9999
  decay_type: exp
  beta: 15
ema: True
ema_params:
  decay_type: threshold
  decay: 0.9999

Note: We still keep backward compatibility with old params and emit a deprecation warning showing what should be updated:

    def test_train_with_old_ema_params(self):
        self._init_model()
        self._train({"decay": 0.9999, "exp_activation": True, "beta": 10})

Emits the following warning message:

[2023-01-24 10:33:37] WARNING - sg_trainer.py - Parameter `exp_activation` is deprecated for EMA model. Please update your config to use decay_type: str (constant|exp|threshold) instead:
ema: True
ema_params: 
  decay: 0.9999
  decay_type: exp
  beta: 15

ema: True
ema_params: 
  decay: 0.9999
  decay_type: constant

ema: True
ema_params: 
  decay: 0.9999
  decay_type: threshold
In the next major release of SG this warning will become an error.

@dagshub
Copy link

dagshub bot commented Jan 23, 2023

@BloodAxe BloodAxe marked this pull request as ready for review January 23, 2023 14:55
Copy link
Collaborator

@ofrimasad ofrimasad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great.
I think we should use deprecate library for this deprecation. see my comments inline

src/super_gradients/training/kd_trainer/kd_trainer.py Outdated Show resolved Hide resolved
src/super_gradients/training/utils/ema.py Show resolved Hide resolved
src/super_gradients/training/utils/ema_decay_schedules.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@ofrimasad ofrimasad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@BloodAxe BloodAxe merged commit 69a82bc into master Jan 30, 2023
@BloodAxe BloodAxe deleted the feature/SG-573-Integrate-EMA branch January 30, 2023 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants