[Feature] PPO Trainer Updates #3190

vmoens · 2025-10-14T08:41:01Z

Stack from ghstack (oldest at bottom):

[ghstack-poisoned]

pytorch-bot · 2025-10-14T08:41:05Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3190

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 New Failures, 11 Pending, 1 Unrelated Failure

As of commit 7759c3d with merge base 92c20cd ():

NEW FAILURES - The following jobs have failed:

Generate documentation / build-docs (3.9, 12.8) / linux-job (gh)
RuntimeError: Command docker exec -t ac4b6a91b1a51bd483d4f65ec6ca4de504b698d9344f7d9458afd579ced1b402 /exec failed with exit code 1
Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
RuntimeError: Command docker exec -t b02a593a792fcc823c076892c63433ff441a74abbb2a8d944ba0640d7b04f7e5 /exec failed with exit code 1
Libs Tests on Linux / unittests-gym (3.9, 12.8) / linux-job (gh)
RuntimeError: Command docker exec -t 870f3e6b9e40bea1196dfc2cae7d72934e1894904f6167c97b30741e5f5f0318 /exec failed with exit code 1
Libs Tests on Linux / unittests-sklearn (3.10, 12.8) / linux-job (gh)
RuntimeError: Command 8000 docker exec -t 57f8fa579329fb15faa91fd95092ef5bb53a7bc92d38b193802675eb237add8c /exec failed with exit code 2
SOTA Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
RuntimeError: Command docker exec -t 4d26e409239bee0cf63f285d7324cd7d75c639d05632cbd5ebf156b522ae2991 /exec failed with exit code 1

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

Unit-tests on Windows / unittests-cpu (3.10, windows.4xlarge, cpu) / windows-job (gh) (trunk failure)
Process completed with exit code 2.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[ghstack-poisoned]

github-actions · 2025-10-20T02:02:02Z

$\color{#D29922}\textsf{\Large&#x26A0;\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 154. Improved: $\large\color{#35bf28}37$. Worsened: $\large\color{#d91a1a}7$.

Expand to view detailed results

8000

Name	Max	Mean	Ops	Ops on Repo `HEAD`	Change
test_tensor_to_bytestream_speed[pickle]	86.6361μs	85.0253μs	11.7612 KOps/s	11.6156 KOps/s	$\color{#35bf28}+1.25\%$
test_tensor_to_bytestream_speed[torch.save]	0.1461ms	0.1448ms	6.9038 KOps/s	6.9046 KOps/s	$\color{#d91a1a}-0.01\%$
test_tensor_to_bytestream_speed[untyped_storage]	0.1158s	0.1153s	8.6712 Ops/s	8.4553 Ops/s	$\color{#35bf28}+2.55\%$
test_tensor_to_bytestream_speed[numpy]	2.8479μs	2.8400μs	352.1127 KOps/s	351.4165 KOps/s	$\color{#35bf28}+0.20\%$
test_tensor_to_bytestream_speed[safetensors]	46.7575μs	46.4553μs	21.5261 KOps/s	21.7998 KOps/s	$\color{#d91a1a}-1.26\%$
test_simple	0.6708s	0.5740s	1.7423 Ops/s	1.7286 Ops/s	$\color{#35bf28}+0.79\%$
test_transformed	1.2297s	1.1380s	0.8788 Ops/s	0.8771 Ops/s	$\color{#35bf28}+0.19\%$
test_serial	1.6578s	1.6492s	0.6063 Ops/s	0.5917 Ops/s	$\color{#35bf28}+2.47\%$
test_parallel	1.1489s	1.0864s	0.9204 Ops/s	0.9189 Ops/s	$\color{#35bf28}+0.17\%$
test_step_mdp_speed[True-True-True-True-True]	0.1989ms	44.9175μs	22.2630 KOps/s	21.7731 KOps/s	$\color{#35bf28}+2.25\%$
test_step_mdp_speed[True-True-True-True-False]	64.9720μs	25.4718μs	39.2591 KOps/s	38.0651 KOps/s	$\color{#35bf28}+3.14\%$
test_step_mdp_speed[True-True-True-False-True]	61.9810μs	25.5610μs	39.1222 KOps/s	37.9317 KOps/s	$\color{#35bf28}+3.14\%$
test_step_mdp_speed[True-True-True-False-False]	78.2110μs	14.2068μs	70.3889 KOps/s	70.1918 KOps/s	$\color{#35bf28}+0.28\%$
test_step_mdp_speed[True-True-False-True-True]	89.1120μs	48.4364μs	20.6456 KOps/s	19.9589 KOps/s	$\color{#35bf28}+3.44\%$
test_step_mdp_speed[True-True-False-True-False]	61.4410μs	28.1748μs	35.4927 KOps/s	34.7097 KOps/s	$\color{#35bf28}+2.26\%$
test_step_mdp_speed[True-True-False-False-True]	63.1110μs	28.0632μs	35.6338 KOps/s	34.2495 KOps/s	$\color{#35bf28}+4.04\%$
test_step_mdp_speed[True-True-False-False-False]	43.0010μs	16.8134μs	59.4764 KOps/s	57.4968 KOps/s	$\color{#35bf28}+3.44\%$
test_step_mdp_speed[True-False-True-True-True]	84.1410μs	50.5097μs	19.7982 KOps/s	18.9513 KOps/s	$\color{#35bf28}+4.47\%$
test_step_mdp_speed[True-False-True-True-False]	61.6510μs	30.9020μs	32.3604 KOps/s	30.7852 KOps/s	$\textbf{\color{#35bf28}+5.12\%}$
test_step_mdp_speed[True-False-True-False-True]	54.2610μs	28.1911μs	35.4722 KOps/s	33.4733 KOps/s	$\textbf{\color{#35bf28}+5.97\%}$
test_step_mdp_speed[True-False-True-False-False]	45.7610μs	16.8644μs	59.2965 KOps/s	57.6341 KOps/s	$\color{#35bf28}+2.88\%$
test_step_mdp_speed[True-False-False-True-True]	91.5510μs	53.0710μs	18.8427 KOps/s	17.9191 KOps/s	$\textbf{\color{#35bf28}+5.15\%}$
test_step_mdp_speed[True-False-False-True-False]	64.1710μs	33.1232μs	30.1903 KOps/s	28.3419 KOps/s	$\textbf{\color{#35bf28}+6.52\%}$
test_step_mdp_speed[True-False-False-False-True]	65.0920μs	30.4528μs	32.8378 KOps/s	30.9870 KOps/s	$\textbf{\color{#35bf28}+5.97\%}$
test_step_mdp_speed[True-False-False-False-False]	50.1710μs	19.3256μs	51.7447 KOps/s	48.9319 KOps/s	$\textbf{\color{#35bf28}+5.75\%}$
test_step_mdp_speed[False-True-True-True-True]	91.0120μs	50.2146μs	19.9145 KOps/s	18.6206 KOps/s	$\textbf{\color{#35bf28}+6.95\%}$
test_step_mdp_speed[False-True-True-True-False]	64.9510μs	30.4002μs	32.8946 KOps/s	31.3606 KOps/s	$\color{#35bf28}+4.89\%$
test_step_mdp_speed[False-True-True-False-True]	2.3964ms	32.3089μs	30.9512 KOps/s	29.3537 KOps/s	$\textbf{\color{#35bf28}+5.44\%}$
test_step_mdp_speed[False-True-True-False-False]	46.1610μs	18.3966μs	54.3578 KOps/s	52.2088 KOps/s	$\color{#35bf28}+4.12\%$
test_step_mdp_speed[False-True-False-True-True]	88.9810μs	52.1858μs	19.1623 KOps/s	18.0854 KOps/s	$\textbf{\color{#35bf28}+5.95\%}$
test_step_mdp_speed[False-True-False-True-False]	64.6520μs	32.8887μs	30.4055 KOps/s	28.5571 KOps/s	$\textbf{\color{#35bf28}+6.47\%}$
test_step_mdp_speed[False-True-False-False-True]	63.0220μs	34.0723μs	29.3493 KOps/s	27.7909 KOps/s	$\textbf{\color{#35bf28}+5.61\%}$
test_step_mdp_speed[False-True-False-False-False]	48.2010μs	20.6323μs	48.4677 KOps/s	46.0253 KOps/s	$\textbf{\color{#35bf28}+5.31\%}$
test_step_mdp_speed[False-False-True-True-True]	0.1033ms	55.4313μs	18.0404 KOps/s	17.4983 KOps/s	$\color{#35bf28}+3.10\%$
test_step_mdp_speed[False-False-True-True-False]	70.2120μs	35.7867μs	27.9434 KOps/s	26.9747 KOps/s	$\color{#35bf28}+3.59\%$
test_step_mdp_speed[False-False-True-False-True]	63.5210μs	34.1284μs	29.3011 KOps/s	28.2068 KOps/s	$\color{#35bf28}+3.88\%$
test_step_mdp_speed[False-False-True-False-False]	61.1820μs	20.7276μs	48.2448 KOps/s	46.4110 KOps/s	$\color{#35bf28}+3.95\%$
test_step_mdp_speed[False-False-False-True-True]	91.4620μs	57.5426μs	17.3784 KOps/s	16.7969 KOps/s	$\color{#35bf28}+3.46\%$
test_step_mdp_speed[False-False-False-True-False]	92.6420μs	37.6022μs	26.5942 KOps/s	25.1582 KOps/s	$\textbf{\color{#35bf28}+5.71\%}$
test_step_mdp_speed[False-False-False-False-True]	65.3520μs	36.4791μs	27.4129 KOps/s	26.0960 KOps/s	$\textbf{\color{#35bf28}+5.05\%}$
test_step_mdp_speed[False-False-False-False-False]	47.6510μs	23.1913μs	43.1195 KOps/s	40.8811 KOps/s	$\textbf{\color{#35bf28}+5.48\%}$
test_values[generalized_advantage_estimate-True-True]	10.4320ms	9.7022ms	103.0692 Ops/s	106.8403 Ops/s	$\color{#d91a1a}-3.53\%$
test_values[vec_generalized_advantage_estimate-True-True]	21.0627ms	17.6384ms	56.6945 Ops/s	89.6976 Ops/s	$\textbf{\color{#d91a1a}-36.79\%}$
test_values[td0_return_estimate-False-False]	0.2280ms	0.1253ms	7.9779 KOps/s	7.8408 KOps/s	$\color{#35bf28}+1.75\%$
test_values[td1_return_estimate-False-False]	27.7201ms	26.2634ms	38.0758 Ops/s	37.0909 Ops/s	$\color{#35bf28}+2.66\%$
test_values[vec_td1_return_estimate-False-False]	21.3636ms	17.6763ms	56.5730 Ops/s	89.4358 Ops/s	$\textbf{\color{#d91a1a}-36.74\%}$
test_values[td_lambda_return_estimate-True-False]	43.2932ms	40.0770ms	24.9520 Ops/s	24.8286 Ops/s	$\color{#35bf28}+0.50\%$
test_values[vec_td_lambda_return_estimate-True-False]	18.5049ms	17.5855ms	56.8651 Ops/s	89.6251 Ops/s	$\textbf{\color{#d91a1a}-36.55\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512]	8.4789ms	8.3188ms	120.2094 Ops/s	122.6817 Ops/s	$\color{#d91a1a}-2.02\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	1.8527ms	1.5168ms	659.2972 Ops/s	670.4556 Ops/s	$\color{#d91a1a}-1.66\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.4803ms	0.4054ms	2.4665 KOps/s	2.4847 KOps/s	$\color{#d91a1a}-0.73\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	34.5721ms	33.9497ms	29.4553 Ops/s	36.1731 Ops/s	$\textbf{\color{#d91a1a}-18.57\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	2.0615ms	1.7033ms	587.0909 Ops/s	581.0589 Ops/s	$\color{#35bf28}+1.04\%$
test_dqn_speed[False-None]	6.4528ms	1.3843ms	722.3641 Ops/s	715.6940 Ops/s	$\color{#35bf28}+0.93\%$
test_dqn_speed[False-backward]	1.9702ms	1.8963ms	527.3298 Ops/s	531.1618 Ops/s	$\color{#d91a1a}-0.72\%$
test_dqn_speed[True-None]	0.5874ms	0.5270ms	1.8975 KOps/s	1.9047 KOps/s	$\color{#d91a1a}-0.38\%$
test_dqn_speed[True-backward]	0.9913ms	0.9627ms	1.0387 KOps/s	846.5447 Ops/s	$\textbf{\color{#35bf28}+22.70\%}$
test_dqn_speed[reduce-overhead-None]	0.8698ms	0.5118ms	1.9540 KOps/s	1.8866 KOps/s	$\color{#35bf28}+3.57\%$
test_dqn_speed[reduce-overhead-backward]	1.0021ms	0.9652ms	1.0361 KOps/s	936.4331 Ops/s	$\textbf{\color{#35bf28}+10.64\%}$
test_ddpg_speed[False-None]	3.2526ms	2.8392ms	352.2128 Ops/s	348.2019 Ops/s	$\color{#35bf28}+1.15\%$
test_ddpg_speed[False-backward]	4.9786ms	4.0591ms	246.3576 Ops/s	247.4203 Ops/s	$\color{#d91a1a}-0.43\%$
test_ddpg_speed[True-None]	1.7617ms	1.4074ms	710.5395 Ops/s	703.7756 Ops/s	$\color{#35bf28}+0.96\%$
test_ddpg_speed[True-backward]	2.4086ms	2.3492ms	425.6724 Ops/s	405.2802 Ops/s	$\textbf{\color{#35bf28}+5.03\%}$
test_ddpg_speed[reduce-overhead-None]	1.5948ms	1.3873ms	720.8472 Ops/s	713.6236 Ops/s	$\color{#35bf28}+1.01\%$
test_ddpg_speed[reduce-overhead-backward]	2.4261ms	2.3539ms	424.8235 Ops/s	356.8013 Ops/s	$\textbf{\color{#35bf28}+19.06\%}$
test_sac_speed[False-None]	8.1980ms	7.7356ms	129.2727 Ops/s	129.5142 Ops/s	$\color{#d91a1a}-0.19\%$
test_sac_speed[False-backward]	11.3624ms	10.8824ms	91.8914 Ops/s	91.6538 Ops/s	$\color{#35bf28}+0.26\%$
test_sac_speed[True-None]	2.4919ms	2.1207ms	471.5408 Ops/s	467.5595 Ops/s	$\color{#35bf28}+0.85\%$
test_sac_speed[True-backward]	4.1285ms	4.0020ms	249.8748 Ops/s	246.4226 Ops/s	$\color{#35bf28}+1.40\%$
test_sac_speed[reduce-overhead-None]	2.5376ms	2.1233ms	470.9539 Ops/s	461.7465 Ops/s	$\color{#35bf28}+1.99\%$
test_sac_speed[reduce-overhead-backward]	4.2124ms	4.0604ms	246.2833 Ops/s	204.7631 Ops/s	$\textbf{\color{#35bf28}+20.28\%}$
test_redq_speed[False-None]	10.9175ms	10.1689ms	98.3388 Ops/s	96.5688 Ops/s	$\color{#35bf28}+1.83\%$
test_redq_speed[False-backward]	22.8109ms	17.7874ms	56.2196 Ops/s	56.2183 Ops/s	$+0.00\%$
test_redq_speed[True-None]	4.7744ms	4.4104ms	226.7388 Ops/s	222.4940 Ops/s	$\color{#35bf28}+1.91\%$
test_redq_speed[True-backward]	10.3435ms	9.9269ms	100.7360 Ops/s	102.3627 Ops/s	$\color{#d91a1a}-1.59\%$
test_redq_speed[reduce-overhead-None]	4.7302ms	4.3857ms	228.0140 Ops/s	240.0743 Ops/s	$\textbf{\color{#d91a1a}-5.02\%}$
test_redq_speed[reduce-overhead-backward]	10.3008ms	10.0026ms	99.9742 Ops/s	102.8552 Ops/s	$\color{#d91a1a}-2.80\%$
test_redq_deprec_speed[False-None]	11.4785ms	10.9137ms	91.6284 Ops/s	91.2089 Ops/s	$\color{#35bf28}+0.46\%$
test_redq_deprec_speed[False-backward]	16.1693ms	15.6386ms	63.9444 Ops/s	63.1769 Ops/s	$\color{#35bf28}+1.21\%$
test_redq_deprec_speed[True-None]	4.1646ms	3.6434ms	274.4653 Ops/s	268.2475 Ops/s	$\color{#35bf28}+2.32\%$
test_redq_deprec_speed[True-backward]	7.9125ms	7.6360ms	130.9590 Ops/s	130.2571 Ops/s	$\color{#35bf28}+0.54\%$
test_redq_deprec_speed[reduce-overhead-None]	3.9055ms	3.6360ms	275.0270 Ops/s	269.3396 Ops/s	$\color{#35bf28}+2.11\%$
test_redq_deprec_speed[reduce-overhead-backward]	7.7723ms	7.5453ms	132.5322 Ops/s	129.3137 Ops/s	$\color{#35bf28}+2.49\%$
test_td3_speed[False-None]	8.0488ms	7.8465ms	127.4446 Ops/s	128.0987 Ops/s	$\color{#d91a1a}-0.51\%$
test_td3_speed[False-backward]	11.5305ms	10.7188ms	93.2936 Ops/s	92.4448 Ops/s	$\color{#35bf28}+0.92\%$
test_td3_speed[True-None]	1.9165ms	1.8403ms	543.4039 Ops/s	535.6486 Ops/s	$\color{#35bf28}+1.45\%$
test_td3_speed[True-backward]	4.3677ms	3.7288ms	268.1827 Ops/s	250.5538 Ops/s	$\textbf{\color{#35bf28}+7.04\%}$
test_td3_speed[reduce-overhead-None]	2.0553ms	1.8451ms	541.9760 Ops/s	559.7816 Ops/s	$\color{#d91a1a}-3.18\%$
test_td3_speed[reduce-overhead-backward]	3.7841ms	3.6667ms	272.7230 Ops/s	256.7408 Ops/s	$\textbf{\color{#35bf28}+6.23\%}$
test_cql_speed[False-None]	28.5662ms	25.7218ms	38.8775 Ops/s	39.5343 Ops/s	$\color{#d91a1a}-1.66\%$
test_cql_speed[False-backward]	38.6850ms	35.0138ms	28.5602 Ops/s	28.4232 Ops/s	$\color{#35bf28}+0.48\%$
test_cql_speed[True-None]	15.6256ms	12.5193ms	79.8769 Ops/s	77.1941 Ops/s	$\color{#35bf28}+3.48\%$
test_cql_speed[True-backward]	19.2224ms	18.6191ms	53.7083 Ops/s	53.8755 Ops/s	$\color{#d91a1a}-0.31\%$
test_cql_speed[reduce-overhead-None]	12.9250ms	12.5309ms	79.8029 Ops/s	79.5958 Ops/s	$\color{#35bf28}+0.26\%$
test_cql_speed[reduce-overhead-backward]	18.9768ms	18.5557ms	53.8919 Ops/s	55.6709 Ops/s	$\color{#d91a1a}-3.20\%$
test_a2c_speed[False-None]	5.7610ms	5.3473ms	187.0098 Ops/s	183.7037 Ops/s	$\color{#35bf28}+1.80\%$
test_a2c_speed[False-backward]	12.1320ms	11.7856ms	84.8496 Ops/s	85.0031 Ops/s	$\color{#d91a1a}-0.18\%$
test_a2c_speed[True-None]	3.8826ms	3.7242ms	268.5145 Ops/s	262.0660 Ops/s	$\color{#35bf28}+2.46\%$
test_a2c_speed[True-backward]	8.9041ms	8.7261ms	114.5991 Ops/s	100.6577 Ops/s	$\textbf{\color{#35bf28}+13.85\%}$
test_a2c_speed[reduce-overhead-None]	4.0776ms	3.7293ms	268.1442 Ops/s	270.0176 Ops/s	$\color{#d91a1a}-0.69\%$
test_a2c_speed[reduce-overhead-backward]	9.1824ms	8.9227ms	112.0732 Ops/s	112.2902 Ops/s	$\color{#d91a1a}-0.19\%$
test_ppo_speed[False-None]	6.3323ms	5.8720ms	170.2986 Ops/s	170.0659 Ops/s	$\color{#35bf28}+0.14\%$
test_ppo_speed[False-backward]	12.8976ms	12.4439ms	80.3606 Ops/s	81.2607 Ops/s	$\color{#d91a1a}-1.11\%$
test_ppo_speed[True-None]	3.9759ms	3.6688ms	272.5655 Ops/s	270.7703 Ops/s	$\color{#35bf28}+0.66\%$
test_ppo_speed[True-backward]	8.7823ms	8.5492ms	116.9697 Ops/s	116.4863 Ops/s	$\color{#35bf28}+0.42\%$
test_ppo_speed[reduce-overhead-None]	3.8351ms	3.6505ms	273.9375 Ops/s	271.4793 Ops/s	$\color{#35bf28}+0.91\%$
test_ppo_speed[reduce-overhead-backward]	8.9139ms	8.7045ms	114.8837 Ops/s	112.9708 Ops/s	$\color{#35bf28}+1.69\%$
test_reinforce_speed[False-None]	4.9415ms	4.5345ms	220.5329 Ops/s	217.6426 Ops/s	$\color{#35bf28}+1.33\%$
test_reinforce_speed[False-backward]	7.5794ms	7.3359ms	136.3157 Ops/s	135.8036 Ops/s	$\color{#35bf28}+0.38\%$
test_reinforce_speed[True-None]	3.3292ms	2.9313ms	341.1407 Ops/s	341.9613 Ops/s	$\color{#d91a1a}-0.24\%$
test_reinforce_speed[True-backward]	8.1193ms	7.7915ms	128.3456 Ops/s	126.8428 Ops/s	$\color{#35bf28}+1.18\%$
test_reinforce_speed[reduce-overhead-None]	3.2499ms	2.8718ms	348.2153 Ops/s	333.1398 Ops/s	$\color{#35bf28}+4.53\%$
test_reinforce_speed[reduce-overhead-backward]	8.2599ms	7.9332ms	126.0532 Ops/s	125.5922 Ops/s	$\color{#35bf28}+0.37\%$
test_iql_speed[False-None]	20.4163ms	19.6920ms	50.7820 Ops/s	50.5293 Ops/s	$\color{#35bf28}+0.50\%$
test_iql_speed[False-backward]	30.7358ms	30.1232ms	33.1970 Ops/s	32.6285 Ops/s	$\color{#35bf28}+1.74\%$
test_iql_speed[True-None]	9.0040ms	8.5528ms	116.9209 Ops/s	115.7854 Ops/s	$\color{#35bf28}+0.98\%$
test_iql_speed[True-backward]	17.2862ms	16.8045ms	59.5079 Ops/s	57.9285 Ops/s	$\color{#35bf28}+2.73\%$
test_iql_speed[reduce-overhead-None]	8.8775ms	8.5629ms	116.7823 Ops/s	113.4219 Ops/s	$\color{#35bf28}+2.96\%$
test_iql_speed[reduce-overhead-backward]	17.7036ms	17.2659ms	57.9175 Ops/s	56.1357 Ops/s	$\color{#35bf28}+3.17\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	7.3203ms	6.0287ms	165.8723 Ops/s	167.9673 Ops/s	$\color{#d91a1a}-1.25\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.6066ms	0.3418ms	2.9253 KOps/s	3.0040 KOps/s	$\color{#d91a1a}-2.62\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.6640ms	0.2723ms	3.6718 KOps/s	3.1035 KOps/s	$\textbf{\color{#35bf28}+18.31\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.9726ms	5.7257ms	174.6511 Ops/s	177.7893 Ops/s	$\color{#d91a1a}-1.77\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	1.9300ms	0.2830ms	3.5339 KOps/s	3.2400 KOps/s	$\textbf{\color{#35bf28}+9.07\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.6331ms	0.2516ms	3.9743 KOps/s	3.4282 KOps/s	$\textbf{\color{#35bf28}+15.93\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000]	1.4892ms	1.2219ms	818.3660 Ops/s	771.7175 Ops/s	$\textbf{\color{#35bf28}+6.04\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000]	1.3842ms	1.1266ms	887.6005 Ops/s	826.3830 Ops/s	$\textbf{\color{#35bf28}+7.41\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	9.3935ms	5.9999ms	166.6707 Ops/s	172.3434 Ops/s	$\color{#d91a1a}-3.29\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	2.0641ms	0.5027ms	1.9892 KOps/s	2.1661 KOps/s	$\textbf{\color{#d91a1a}-8.17\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8281ms	0.4658ms	2.1467 KOps/s	2.2026 KOps/s	$\color{#d91a1a}-2.54\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	5.9386ms	5.7892ms	172.7361 Ops/s	173.4440 Ops/s	$\color{#d91a1a}-0.41\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.8798ms	0.3297ms	3.0334 KOps/s	2.9693 KOps/s	$\color{#35bf28}+2.16\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.6225ms	0.2809ms	3.5603 KOps/s	3.2159 KOps/s	$\textbf{\color{#35bf28}+10.71\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.9190ms	5.7326ms	174.4420 Ops/s	174.7205 Ops/s	$\color{#d91a1a}-0.16\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	1.7240ms	0.2808ms	3.5610 KOps/s	3.0095 KOps/s	$\textbf{\color{#35bf28}+18.33\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.4237ms	0.2530ms	3.9521 KOps/s	3.2144 KOps/s	$\textbf{\color{#35bf28}+22.95\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	5.9656ms	5.8753ms	170.2042 Ops/s	170.6371 Ops/s	$\color{#d91a1a}-0.25\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	1.9477ms	0.4283ms	2.3347 KOps/s	2.1188 KOps/s	$\textbf{\color{#35bf28}+10.19\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.6853ms	0.4069ms	2.4579 KOps/s	2.2148 KOps/s	$\textbf{\color{#35bf28}+10.98\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	6.5951ms	5.0323ms	198.7146 Ops/s	195.1876 Ops/s	$\color{#35bf28}+1.81\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	9.8830ms	2.3335ms	428.5482 Ops/s	443.5484 Ops/s	$\color{#d91a1a}-3.38\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	5.8698ms	1.1561ms	864.9810 Ops/s	834.5695 Ops/s	$\color{#35bf28}+3.64\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.5268s	15.4608ms	64.6798 Ops/s	54.3948 Ops/s	$\textbf{\color{#35bf28}+18.91\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	3.8160ms	1.7114ms	584.3324 Ops/s	484.6850 Ops/s	$\textbf{\color{#35bf28}+20.56\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	10.6302ms	1.2382ms	807.6324 Ops/s	900.0473 Ops/s	$\textbf{\color{#d91a1a}-10.27\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	7.7886ms	5.1966ms	192.4326 Ops/s	183.2439 Ops/s	$\textbf{\color{#35bf28}+5.01\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	7.9024ms	2.1378ms	467.7799 Ops/s	460.4177 Ops/s	$\color{#35bf28}+1.60\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	8.3382ms	1.3641ms	733.0680 Ops/s	725.6832 Ops/s	$\color{#35bf28}+1.02\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True]	35.8233ms	32.6332ms	30.6436 Ops/s	30.2095 Ops/s	$\color{#35bf28}+1.44\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False]	19.0821ms	17.2958ms	57.8176 Ops/s	58.4455 Ops/s	$\color{#d91a1a}-1.07\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True]	36.9051ms	33.5192ms	29.8337 Ops/s	29.4884 Ops/s	$\color{#35bf28}+1.17\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False]	19.5759ms	17.6452ms	56.6725 Ops/s	57.6174 Ops/s	$\color{#d91a1a}-1.64\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True]	37.2802ms	35.5745ms	28.1100 Ops/s	21.1487 Ops/s	$\textbf{\color{#35bf28}+32.92\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False]	20.7180ms	19.0651ms	52.4518 Ops/s	53.6329 Ops/s	$\color{#d91a1a}-2.20\%$

[ghstack-poisoned]

Update

5d33073

[ghstack-poisoned]

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 14, 2025

vmoens added 7 commits October 16, 2025 15:00

Update

8557795

[ghstack-poisoned]

Update

8e5cce3

[ghstack-poisoned]

Update

3c22f43

[ghstack-poisoned]

Update

308cd5f

[ghstack-poisoned]

Update

7db41a4

[ghstack-poisoned]

Update

dc7193a

[ghstack-poisoned]

Update

717bc3f

[ghstack-poisoned]

vmoens mentioned this pull request Oct 18, 2025

[Quality] Fix flaky test #3211

Open

vmoens added 4 commits October 18, 2025 19:00

Update

6072106

[ghstack-poisoned]

Update

7356e29

[ghstack-poisoned]

Update

7f4c006

[ghstack-poisoned]

Update

ddcc7df

[ghstack-poisoned]

vmoens added the enhancement New feature or request label Oct 20, 2025

Update

8ef7b80

[ghstack-poisoned]

vmoens mentioned this pull request Oct 20, 2025

[CI] Fix benchmarks for LLMs #3212

Merged

Update

5a672ad

[ghstack-poisoned]

vmoens mentioned this pull request Oct 20, 2025

[BugFix] Fix GRPO tests and runs #3213

Merged

Update

91e6c02

[ghstack-poisoned]

vmoens added 6 commits October 19, 2025 20:56

Update

09a93f3

[ghstack-poisoned]

Update

f523683

[ghstack-poisoned]

Update

c3b011f

[ghstack-poisoned]

Update

647fba0

[ghstack-poisoned]

Update

0978ca8

[ghstack-poisoned]

Update

7759c3d

[ghstack-poisoned]

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Feature] PPO Trainer Updates #3190

[Feature] PPO Trainer Updates #3190

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

[Feature] PPO Trainer Updates #3190

Are you sure you want to change the base?

[Feature] PPO Trainer Updates #3190

Conversation

Uh oh!

Uh oh!

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3190

❌ 5 New Failures, 11 Pending, 1 Unrelated Failure

Uh oh!

Uh oh!

$\color{#D29922}\textsf{\Large&amp;#x26A0;\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 154. Improved: $\large\color{#35bf28}37$. Worsened: $\large\color{#d91a1a}7$.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests