From 10ecd397f21f78ac29cbc19b4ac9ec6e608d5082 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Thu, 2 Apr 2026 03:07:23 +0200 Subject: [PATCH] =?UTF-8?q?fix:=20batch=5Fsize=3D128=20in=20production=20T?= =?UTF-8?q?OML=20=E2=80=94=201024=20causes=20training=20hang?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit batch_size=1024 has never been tested on H100. It was always overridden by the CLI default (128) in all previous runs. Our changes exposed this latent bug by conditional batch_size override logic. Root cause of the hang is unknown (likely CUDA Graph capture with batch_size=1024 buffers). Setting to known-working 128 while we investigate. The batch_size=1024 hang investigation is tracked in the mega-graph refactor plan. Co-Authored-By: Claude Opus 4.6 (1M context) --- config/training/dqn-production.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/training/dqn-production.toml b/config/training/dqn-production.toml index de299a7a0..36ccf4d75 100644 --- a/config/training/dqn-production.toml +++ b/config/training/dqn-production.toml @@ -8,7 +8,7 @@ [training] epochs = 200 -batch_size = 1024 +batch_size = 128 learning_rate = 0.0001 gamma = 0.99 weight_decay = 0.0001