-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Question] How to pass a varying gamma to DQN or PPO during training? #1889
Comments
Hello, |
Thanks for your quick response. So if I understand, you’re suggesting to augment the replay buffer to collect time-varying gammas with each rollout, then in the PPO loss function, use the gammas from the replay buffer? |
correct, that would be actually a gamma per timesteps to make it simpler to use (and make it work with |
Got it. Yes absolutely that gamma would be the discount factor for a “step” or discrete decision epoch. |
If I augment the replay buffer, does all of is content get passed to the learn function? Meaning I dont need to modify the input for the training update functions, I just need to extract the gammas from the batch of step buffer data and use them in the GAE discount? |
you need to modify the named tuple that represent a transition and modify the GAE computation accordingly yes |
❓ Question
Reinforcement learning and the SB3 implementations apply the typical constant gamma for discounting future values when learning. This is fine for discrete time environments where for each action the future value is discounted as a constant for each step.
I have a custom gym environment where my environment steps in discrete decision epochs, but each action takes a different amount of time. Discounting future values at a constant rate is therefore incorrect. What I need to do is discount future values by a gamma that is a function of the time it takes to conduct the action in the environment.
Is there anyway to pass in gamma as a function or as tensors that map to each (s, a, s’, r) tuple during learning? Maybe possible with existing features or callbacks? I’d like to avoid forking the repository if possible.
Any input would be appreciated as I’ve been stuck on this for some time. Thanks in advance!
Checklist
The text was updated successfully, but these errors were encountered: