[question] HER and prioritized experience replay

[johannes-dornheim]

Hi

in the stable-baselines implementation, HER does not support prioritized replay buffer. In the HER Paper they state that: "Prioritized experience replay (....) is orthogonal to our work and both approaches can be easily combined". So my question is: Are there 'deeper reasons' for the lack of support or is it just a currently missing feature?

Best Regards,
Johannes

[Miffyli]

I believe there are no bigger reasons to lack of support, other than lack of implementation. It would require coming up with prioritizes for the samples in the buffer, and then updating the replay_buffer.py in HER. I am not too familiar with HER to know how easy of a feat this would be. On the first glance it does not sound as straight-forward as with DQNs.

[RyanRizzo96]

Actually, PER has been shown not to improve performance over HER, hence there is no real motivation to imlpement. Not only does PER not improve performance, but it actually increases computational time substantially.

PER works by prioritising transitions with higher TD-error, which means that the TD-error must be computed for each transition, hence the expensive computational time.

Prioritised Sequence Experience Replay (PSER) outperforms PER but has not been imlpemented with HER.

There are other methods which improve the sampling efficiency of HER (such as Energy Based Prioritisation), but PER is not one of them. I have put my name forward to implement this in the new PyTorch version.

[Miffyli]

Ok, thanks for the info! Indeed any such new features would be things for the PyTorch version :).

[araffin]

I added it to possible features for Stable-Baselines3 1.1+ in DLR-RM/stable-baselines3#1