jgrusewski fe223b3843 fix(dqn): eliminate OOM in hyperopt by gating GPU features on small GPUs ...

Three root causes fixed:
1. GPU PER + experience collector (cudarc) created dual CUDA allocator
   fragmentation on GPUs ≤8 GB, making training impossible even at
   batch_size=1. Added `use_gpu_replay_buffer` config flag; both GPU PER
   and experience collector now disabled when VRAM ≤8192 MB.

2. Search space bounds were WIDENED instead of capped — max_batch_size
   returning 4096 replaced the original 512 upper bound, and
   max_hidden_dim_base_full returning 3072 replaced the original 1024.
   Fixed with min() to only narrow, never widen.

3. VRAM estimator assumed GPU features always active, overcharging when
   they're disabled on small GPUs. Now conditional: when
   replay_buffer_capacity=0 (proxy for GPU PER disabled), collector/cudarc
   costs are zero and fragmentation multiplier drops from 3× to 1.5×.

Additional small-GPU guard: GPUs ≤8 GB get clamped search space
(batch≤128, hidden≤512, atoms≤51, buffer≤50K) to fit 3 regime heads
+ C51 + noisy nets + dueling in limited VRAM.

Validated: 7/7 trials complete on RTX 3050 Ti 4 GB, zero OOM, best trial
Sharpe 9.8 with 50.6% win rate. Previous runs had 100% OOM failure rate.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-09 16:35:38 +01:00

26 KiB

Raw Blame History

View Raw