Auto-sizer inflated replay buffer to 15.8M entries on H100 (45% of 80GB VRAM).
The PER prefix scan inside the CUDA graph processes ALL 15.8M entries every step,
dominating step time. With buffer_size=500K, the scan processes 500K entries instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
At 16384, each GEMM saturates 97% of 132 SMs — no room for multi-stream
parallelism. At 8192, GEMMs use ~50% SMs, allowing branch streams and
aux parallelism to fill idle SMs. 2x more steps/epoch but each ~2x faster.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The warp-parallel (32,1,1) change to c51_loss_reduce caused the same
hang as isv_feature_gate: changing block_dim inside graph_mega alters
CUDA Graph node structure on Hopper's TMA scheduler. Must stay (1,1,1).
Also: batch_size 8192→16384, gpu_n_episodes 1024→4096, num_atoms 51→52
to align h100.toml with dqn-production.toml.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
dqn-production.toml batch_size=8192 was overriding GPU profile batch_size=64
on RTX 3050 via apply_to(), causing OOM/hang. GPU profile is the sole
authority for batch_size — training profile controls learning params only.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
gpu_n_episodes was manually overridden in GPU profiles, training configs,
test files, and hyperopt — all set to 0 or small fixed values that
bypassed the auto-scaling logic, causing a div-by-zero crash in
train_baseline_rl.
Now: single auto-scaling path via optimal_n_episodes() from VRAM/SM
count. No manual override field. Cap at 16384 (consistent with
AutoBatchSizer's 8192 cap pattern). Floor at 32 for small GPUs.
Removed gpu_n_episodes from:
- DQNHyperparameters, PpoHyperparameters structs
- All 4 GPU profiles (rtx3050, h100, a100, default)
- Training profiles (smoketest, localdev)
- ExperienceProfile struct + serde
- Hyperopt adapter
- All test overrides
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Set batch_size=0 in config/gpu/h100.toml as sentinel for auto-compute.
The constructor now treats batch_size==0 as "let AutoBatchSizer drive":
it uses the VRAM-computed ceiling capped at 8192, instead of the static
1024 that was wasting >90% of H100's 80GB VRAM bandwidth.
Previously AutoBatchSizer computed the optimal batch (e.g. 2085808) but
the profile's batch_size=1024 always won. Now with batch_size=0 the
sizer's result flows through, enabling full SM occupancy on H100.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Created config/gpu/{default,rtx3050,h100,a100}.toml with all GPU-specific
parameters: batch_size, num_atoms, buffer_size, hidden_dim_base,
replay_buffer_vram_fraction, gpu_n_episodes, gpu_timesteps_per_episode,
cuda_stack_bytes.
GpuProfile::load() auto-detects GPU by device name, falls back to
embedded defaults (include_str!). Override via FOXHUNT_GPU_PROFILE env.
Removed dead code:
- detect_vram_mb(), vram_scaled_hidden_dims(), vram_scaled_base_dim(),
resolve_hidden_dim_base() + 18 tests for these functions
All callers updated: train_baseline_rl, DQNTrainer constructor,
PPO trainer, smoke tests, pipeline tests.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace scattered VRAM-based if/else chains with a declarative TOML profile
system. GPU profiles (rtx3050, a100, h100, default) are selected by device
name and embedded at compile time via include_str! for zero-filesystem
fallback in CI/containers, with filesystem and env var overrides.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
