@Miffyli All algorithms match performances except A2C (I ran it with the pytorch rmsprop, just wanted to check), which is a good sign after #110 (table created using the zoo) :

current a2c graphs (trained and plotted using the zoo, those are the deterministic evaluations):
a2c.pdf
a2c_ant.pdf
a2c_half.pdf
a2c_hopper.pdf
a2c_walker.pdf

I will run A2C with the tensorflow RmsProp and also do some run of all algorithm with gSDE enabled (to check that it matches the paper).

As expected changing the optimizer to tf rmsprop closes the gap.
Will now try to replicate gSDE results.

Results for gSDE are replicated (in fact, after solving PPO issue in #110 , it is even better, I will need to update the paper).
closing this issue.

@araffin Thanks for paying so much attention to performance checking. It makes us feel more confident about using SB3.

I assume that for DDPG, TD3 and SAC, you are using the default parameters given in the documentation / paper. Just to give an example, in SB3 doc, for DDPG and TD3, the learning rates are 1e-3 and the batch sizes are 100; for SAC, the learning rate is 3e-4 and the batch size is 256. Personally, I find these hyper-parameter differences unjustified from an algorithmic standpoint. Although I'm aware that this is an effort to match the respective original publications, these are pretty similar algorithms.

Is it possible to test them with shared hyper-parameters? Also, just to double check, where can I find the hyper-parameters you used for this exact replication (gSDE paper or the current SB3 doc)?

Thanks again!

I assume that for DDPG, TD3 and SAC, you are using the default parameters given in the documentation / paper.

actually, slightly different ones as I'm training on PyBullet envs (different from MuJoCo ones, used in the paper).

You have instructions in the doc ;) I'm using the RL zoo: https://github.com/DLR-RM/rl-baselines3-zoo.

Instructions: https://stable-baselines3.readthedocs.io/en/master/modules/sac.html#how-to-replicate-the-results

Personally, I find these hyper-parameter differences unjustified from an algorithmic standpoint. Although I'm aware that this is an effort to match the respective original publications, these are pretty similar algorithms.

You are completely right. In fact, the original code of TD3 now shares SAC hyperparams (https://github.com/sfujim/TD3)
And you can easily do that in the zoo.

Is it possible to test them with shared hyper-parameters? Also, just to double check, where can I find the hyper-parameters you used for this exact replication (gSDE paper or the current SB3 doc)?

yes you can (but you need to deactivate gSDE for SAC, as gSDE for TD3 is no longer supported).

Also, just to double check, where can I find the hyper-parameters you used for this exact replication (gSDE paper or the current SB3 doc)?

In the RL Zoo. You can even check the learning curves from the saved logs: https://github.com/DLR-RM/rl-trained-agents

I feel amazed how well documented SB3 is. Thank you so much for the link to instructions, I should have noticed them myself while reading the doc.

@araffin Just to double-check, I see that you used a linear decay learning rate scheduler for HopperBulletEnv-v0 and Walker2DBulletEnv-v0?

• https://github.com/DLR-RM/rl-baselines3-zoo/blob/master/hyperparams/sac.yml#L117
• https://github.com/DLR-RM/rl-baselines3-zoo/blob/master/hyperparams/sac.yml#L132

I see that you used a linear decay learning rate scheduler for HopperBulletEnv-v0 and Walker2DBulletEnv-v0?

yes (also mentioned in the gSDE paper), there are important to stabilize the training.

https://github.com/DLR-RM/rl-trained-agents/blob/master/sac/Walker2DBulletEnv-v0_1/Walker2DBulletEnv-v0/config.yml#L15

For DDPG, you wrote that you

Using the same hyperparameters as TD3.

Does this mean that, for running DDPG for this performance check, you didn't used the hyper-params specified here:

https://github.com/DLR-RM/rl-trained-agents/tree/master/ddpg

but instead you used the hyper-params specified here:

https://github.com/DLR-RM/rl-trained-agents/tree/master/td3

?

Does this mean that, for running DDPG for this performance check, you didn't used the hyper-params specified here:

They are the same ...
see https://github.com/DLR-RM/rl-trained-agents/blob/master/ddpg/HalfCheetahBulletEnv-v0_1/HalfCheetahBulletEnv-v0/config.yml and https://github.com/DLR-RM/rl-trained-agents/blob/master/td3/HalfCheetahBulletEnv-v0_1/HalfCheetahBulletEnv-v0/config.yml

Also in https://github.com/DLR-RM/rl-baselines3-zoo/blob/master/hyperparams/ddpg.yml

They are not the same for Ant, for example:

https://github.com/DLR-RM/rl-trained-agents/blob/master/ddpg/AntBulletEnv-v0_1/AntBulletEnv-v0/config.yml

https://github.com/DLR-RM/rl-trained-agents/blob/master/td3/AntBulletEnv-v0_1/AntBulletEnv-v0/config.yml

They are not the same for Ant, for example:

oh true, I probably did some minor adjustments (reduce lr) to improve stability. I will update my comment.

(I did not spend much time on DDPG as we treat it as a special case of TD3, cf code)

Hi there,
are the SAC Actor and Critic Loss plots also available somewhere? I'd like to compare mine with the "official" SB3 loss plots.

I'm using a custom environment for robotic grasping and for some reason my critic loss increases after a while. My reward starts to decrease as well and I believe this is connected to the critic loss going up.

I tried various hyper-params already (gradient_steps, train_freq, learning_rate, batch_size) but saw the same behavior throughout. I am keeping ent_coef = 0.01, the auto adjustment only lead to a larger drop in reward unfortunately.

Maybe a LR scheduler helps?

@axkoenig I believe these are (still) the correct instructions and settings to replicate things: https://stable-baselines3.readthedocs.io/en/master/modules/sac.html#how-to-replicate-the-results (@araffin please correct me if I am wrong).

Ok thanks, I was just wondering whether they were posted somewhere s.t. I don't need to train the model myself until the end

@axkoenig I believe these are (still) the correct instructions and settings to replicate things: https://stable-baselines3.readthedocs.io/en/master/modules/sac.html#how-to-replicate-the-results (@araffin please correct me if I am wrong).

Ok thanks, I was just wondering whether they were posted somewhere s.t. I don't need to train the model myself until the end

yes, the RL Zoo is the place to go to replicate results. I saved the training/evaluation reward and the trained agent but not the rest of the metrics (although you can easily reproduce the run normally).
Your issue is probably related to https://discord.com/channels/765294874832273419/767403892446593055/866702257499668492

Dead @araffin,
I used SB3 for my custom environment, but no matter which algorithm I use, I get the same false result. Could you please help me with which part is wrong? I am pretty sure that my custom environment code is true.
Thanks

@vahidqo Answering you #672.