Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify exclude_from_weight_decay implementation #2676

Merged
merged 1 commit into from
Feb 24, 2022

Conversation

lgeiger
Copy link
Contributor

@lgeiger lgeiger commented Feb 21, 2022

Description

This PR changes the handling of exclude_from_weight_decay to ensure that the regex is only evaluated once per step. Currently it gets called within a distributed merge_call which might lead to problems.
I implemented this when trying to debug a performance regression which turned out to be unrelated to this, but I think this still is a nice cleanup to have since it consolidates the handling of decay_var_list and exclude_from_weight_decay.

Type of change

Checklist:

  • I've properly formatted my code according to the guidelines
    • By running Black + Flake8
    • By running pre-commit hooks
  • This PR addresses an already submitted issue for TensorFlow Addons
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • This PR contains modifications to C++ custom-ops

How Has This Been Tested?

I adapted the unittests that where directly relying on the internal functions that had been modified.

@bot-of-gabrieldemarmiesse

@PhilJd

You are owner of some files modified in this pull request.
Would you kindly review the changes whenever you have the time to?
Thank you very much.

@fsx950223 fsx950223 merged commit da14c3b into tensorflow:master Feb 24, 2022
@lgeiger lgeiger deleted the fix-weight-decay-performance branch February 24, 2022 08:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants