Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] User-Selected Normalisation Values #65

Open
EdanToledo opened this issue Jul 4, 2024 · 0 comments
Open

[Feature] User-Selected Normalisation Values #65

EdanToledo opened this issue Jul 4, 2024 · 0 comments

Comments

@EdanToledo
Copy link
Contributor

Description

Hey! A super useful feature would be allowing users to set the values that are used for normalisation of tasks. For example then one could easily do PPO-Normalisation or Human-Performance-Normalisation etc. Additionally, one could simply set the true max and min values of a task. This is really a much needed feature since the way normalisation is done here can actually cause serious undesired effects. Through the addition of new algorithms and more data thereby changing the normalisation, you can actually change the outcome severely of previous experiments. For example, if you ran PPO and SAC and compared performance, you might see that one is statistically significant in its improvement, however as soon as you add a new algorithm this changes the normalisation and can completely change this result. Addiyionally, if you are making two separate plots of different subsets of algorithms, these scores can be normalised completely differently leading to misleading results when looking across the graphs.

Desired Outcome

I imagine the best way would simply be to allow the user to input a dictionary mapping env suite and task to min and max score.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant