gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

perf: dynamic batch sizing — AutoBatchSizer drives batch_size on H100

Browse Source 
Set batch_size=0 in config/gpu/h100.toml as sentinel for auto-compute.
The constructor now treats batch_size==0 as "let AutoBatchSizer drive":
it uses the VRAM-computed ceiling capped at 8192, instead of the static
1024 that was wasting >90% of H100's 80GB VRAM bandwidth.

Previously AutoBatchSizer computed the optimal batch (e.g. 2085808) but
the profile's batch_size=1024 always won. Now with batch_size=0 the
sizer's result flows through, enabling full SM occupancy on H100.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-04-01 22:08:01 +02:00
parent 7e04dd0c0f
commit 5363185721
2 changed files with 8 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
config/gpu/h100.toml
View File

@@ -1,6 +1,6 @@
# H100 PCIe/SXM (80GB VRAM) -- full production
[training]
batch_size = 1024
batch_size = 0 # 0 = auto-compute from VRAM via AutoBatchSizer
num_atoms = 51
buffer_size = 500_000
hidden_dim_base = 256

15
crates/ml/src/trainers/dqn/trainer/constructor.rs
View File

@@ -30,14 +30,6 @@ use super::DQNTrainer;
impl DQNTrainer {
pub(crate) fn new_internal(mut hyperparams: DQNHyperparameters, debug_logging: bool, override_device: Option<MlDevice>) -> Result<Self> {
// Validate batch size is non-zero
if hyperparams.batch_size == 0 {
return Err(anyhow::anyhow!(
"Batch size must be greater than 0, got: {}",
hyperparams.batch_size
));
}
// WAVE 26 P2.2: Validate gradient_accumulation_steps > 0
if hyperparams.gradient_accumulation_steps == 0 {
return Err(anyhow::anyhow!(
@@ -105,6 +97,13 @@ impl DQNTrainer {
}
};
// batch_size == 0 is the sentinel for auto-compute from VRAM.
// Let AutoBatchSizer drive the batch size, capped at 8192.
if hyperparams.batch_size == 0 {
hyperparams.batch_size = max_safe_batch.min(STATIC_MAX_BATCH_SIZE);
info!("AutoBatchSizer: batch_size auto-computed to {}", hyperparams.batch_size);
}
// Cap to VRAM ceiling from AutoBatchSizer (no separate scale-UP —
// AutoBatchSizer already accounts for model size and available VRAM)
if hyperparams.batch_size > max_safe_batch {