Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save exclude_from_weight_decay in config for LAMB #2619

Merged
merged 1 commit into from
Dec 14, 2021
Merged

Save exclude_from_weight_decay in config for LAMB #2619

merged 1 commit into from
Dec 14, 2021

Conversation

leondgarse
Copy link
Contributor

Before:

import tensorflow_addons as tfa
mm = keras.models.Sequential([keras.layers.Input([32, 32, 3]), keras.layers.Flatten(), keras.layers.Dense(10)])
mm.compile(optimizer=tfa.optimizers.LAMB(learning_rate=0.1, weight_decay_rate=0.02, exclude_from_weight_decay=['/gamma', '/beta']))
mm.save('aa.h5')
bb = keras.models.load_model('aa.h5')
print(bb.optimizer.exclude_from_weight_decay, bb.optimizer.exclude_from_layer_adaptation)
# None, None

After:

import tensorflow_addons as tfa
mm = keras.models.Sequential([keras.layers.Input([32, 32, 3]), keras.layers.Flatten(), keras.layers.Dense(10)])
mm.compile(optimizer=tfa.optimizers.LAMB(learning_rate=0.1, weight_decay_rate=0.02, exclude_from_weight_decay=['/gamma', '/beta']))
mm.save('aa.h5')
bb = keras.models.load_model('aa.h5')
print(bb.optimizer.exclude_from_weight_decay, bb.optimizer.exclude_from_layer_adaptation)
# ['/gamma', '/beta'] ['/gamma', '/beta']

Description

Brief Description of the PR:

Fixes # (issue)

Type of change

Checklist:

  • I've properly formatted my code according to the guidelines
    • By running Black + Flake8
    • By running pre-commit hooks
  • This PR addresses an already submitted issue for TensorFlow Addons
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • This PR contains modifications to C++ custom-ops

How Has This Been Tested?

If you're adding a bugfix or new feature please describe the tests that you ran to verify your changes:
*

Before:
```py
import tensorflow_addons as tfa
mm = keras.models.Sequential([keras.layers.Input([32, 32, 3]), keras.layers.Flatten(), keras.layers.Dense(10)])
mm.compile(optimizer=tfa.optimizers.LAMB(learning_rate=0.1, weight_decay_rate=0.02, exclude_from_weight_decay=['/gamma', '/beta']))
mm.save('aa.h5')
bb = keras.models.load_model('aa.h5')
print(bb.optimizer.exclude_from_weight_decay)
# None, None
```
After:
```py
import tensorflow_addons as tfa
mm = keras.models.Sequential([keras.layers.Input([32, 32, 3]), keras.layers.Flatten(), keras.layers.Dense(10)])
mm.compile(optimizer=tfa.optimizers.LAMB(learning_rate=0.1, weight_decay_rate=0.02, exclude_from_weight_decay=['/gamma', '/beta']))
mm.save('aa.h5')
bb = keras.models.load_model('aa.h5')
print(bb.optimizer.exclude_from_weight_decay, bb.optimizer.exclude_from_layer_adaptation)
# ['/gamma', '/beta'] ['/gamma', '/beta']
```
@bot-of-gabrieldemarmiesse

@junjiek

You are owner of some files modified in this pull request.
Would you kindly review the changes whenever you have the time to?
Thank you very much.

@fsx950223 fsx950223 merged commit b303f23 into tensorflow:master Dec 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants