Decrease gamma for SAC/TQC on Swimmer-v4

[JacobHA]

some experiments indicate that gamma=0.999 is better for swimmer, in agreement with https://arxiv.org/pdf/2208.07587 Fig. 4

Description

I've simply decreased gamma for SAC on Swimmer, motivated by experiments.

Motivation and Context

The current value of gamma fails for Swimmer-v4 on SAC.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)

Checklist:

• I've read the CONTRIBUTION guide (required)
• I have updated the changelog accordingly (required).
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.
• I have reformatted the code using make format (required)
• I have checked the codestyle using make check-codestyle and make lint (required)
• I have ensured make pytest and make type both pass. (required)

Note: we are using a maximum length of 127 characters per line

[araffin]

Experiments ongoing at https://wandb.ai/openrlbenchmark/sbx/reports/SAC-MuJoCo-Swimmer-v4--Vmlldzo3NzM5OTk2

[araffin]

So, the results look good.
I'm doing some run with CrossQ (cc @danielpalen) and it seems that it also now solves the env consistently with gamma=0.999, although it is not more sample efficient as SAC.

cc @osigaud for the results with SAC/TQC/CrossQ (maybe we should update the paper after)

EDIT: I added the result for TQC, I guess we need to update its hyperparameters too

[araffin]

LGTM, thanks =)