-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
inconvenience during training #5
Comments
I have trained two dueling agents, one without double and the other with double, both without prioritized replay and in both the training was successful, I am almost sure that the problem I experienced before was due to using experienzed replay, at least this is using dueling. The problem is most likely in the sum tree, although the training initially works perfectly, it is hyper efficient much more than when it is not used, there comes a point where performance declines and never recovers, I don't know what could be being the cause, I played with changing alpha and beta and the behavior of epsilon greedy and all without success. I will try to make Beta grow gradually, like epsilon does, which decreases with steps, and make Beta take the value of 1 as the episodes pass. It may be something else, the truth is I don't know very well how Priorized replay works. dueling without double took 3361 episodes to achieve 500 average reward dueling with double did it faster, at 2874. all of this was with only one agent at a time. |
This should be fixed in the latest release, where I have a minimum value for the probabilities. |
I was doing training using dueling networks and after episode 582 an error occurred and the training continued as if nothing had happened but I lost all the progress and the agent as if the weights had been restored
episode 582 ep score 141.0 average score 181.4 n steps 111244
/home/edo/projects/protorl/protorl/examples/protorl/memory/sum_tree.py:104: RuntimeWarning: divide by zero encountered in double_scalars
weights = np.array([(1 / self.counter * 1 / prob)**self.beta
/home/edo/projects/protorl/protorl/examples/protorl/memory/sum_tree.py:106: RuntimeWarning: invalid value encountered in multiply
weights *= 1 / max(weights)
episode 583 ep score 161.0 average score 181.0 n steps 111405
The training was going well, it reached +350 average reward in episode 292 but then it started to drop in performance, something that is normal but then that happened
there was a division by zero there and then another error in a multiplication
source code
from protorl.agents.dueling import DuelingDQNAgent as Agent
from protorl.actor.dueling import DuelingDQNActor as Actor
from protorl.learner.dueling import DuelingDQNLearner as Learner
from protorl.loops.single import EpisodeLoop
from protorl.policies.epsilon_greedy import EpsilonGreedyPolicy
from protorl.utils.network_utils import make_dqn_networks
from protorl.wrappers.common import make_env
from protorl.memory.generic import initialize_memory
def main():
env_name = 'CartPole-v1'
# env_name = 'PongNoFrameskip-v4'
use_prioritization = True
use_double = True
use_dueling = True
use_atari = False
layers=[32]
env = make_env(env_name, use_atari=use_atari)
n_games = 1500
bs = 64
# 0.3, 0.5 works okay for cartpole
# 0.25, 0.25 doesn't seem to work
# 0.25, 0.75 doesn't work
memory = initialize_memory(max_size=100_000,
obs_shape=env.observation_space.shape,
batch_size=bs,
n_actions=env.action_space.n,
action_space='discrete',
prioritized=use_prioritization,
alpha=0.3,
beta=0.5
)
if name == 'main':
main()
The text was updated successfully, but these errors were encountered: