gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: production v_min/v_max ±15→±50 for gamma=0.99 + PopArt GPU buffer reset

Browse Source 
v_min/v_max ±15 with gamma=0.99 only covers 15% of theoretical Q range
(Q_max = 1/(1-0.99) = 100 for unit-variance PopArt rewards). C51 top atom
saturates on sustained winners, degrading distributional learning. ±50
covers 95th percentile.

PopArt GPU buffers (popart_mean, popart_var, popart_count) were never
zeroed between walk-forward folds — fold 2's reward normalization was
contaminated by fold 1's statistics. Now reset alongside Adam state in
reset_adam_state().

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-04-12 06:56:37 +02:00
parent e18ab46e13
commit dab4478b47
3 changed files with 18 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
config/training/dqn-production.toml
View File

@@ -36,6 +36,10 @@ trades_data_dir = "/data/futures-baseline-trades"
[distributional]
num_atoms = 51
# gamma=0.99 → Q_max = R/(1-gamma) = 100 for unit-variance PopArt rewards.
# ±50 covers the 95th percentile of Q distribution (±15 default only covers 15%).
v_min = -50.0
v_max = 50.0
[exploration]
epsilon_start = 0.3

4
config/training/dqn-smoketest.toml
View File

@@ -30,6 +30,10 @@ imbalance_bar_ewma_alpha = 0.1
[distributional]
num_atoms = 51
# gamma=0.95 → Q_max = R/(1-gamma) = 20 for PopArt-normalized rewards.
# ±15 covers 75% of theoretical Q range — adequate for smoke test.
v_min = -15.0
v_max = 15.0
[exploration]
epsilon_start = 0.3

11
crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs
View File

@@ -942,7 +942,16 @@ impl GpuDqnTrainer {
.map_err(|e| MLError::ModelError(format!("reset iqn t: {e}")))?;
self.iqn_trunk_adam_step = 0;
tracing::info!("Adam optimizer state reset for new fold");
// Reset PopArt running statistics — prevents fold 1's reward distribution
// from contaminating fold 2's normalization.
self.stream.memset_zeros(&mut self.popart_mean)
.map_err(|e| MLError::ModelError(format!("reset popart_mean: {e}")))?;
self.stream.memset_zeros(&mut self.popart_var)
.map_err(|e| MLError::ModelError(format!("reset popart_var: {e}")))?;
self.stream.memset_zeros(&mut self.popart_count)
.map_err(|e| MLError::ModelError(format!("reset popart_count: {e}")))?;
tracing::info!("Adam optimizer + PopArt state reset for new fold");
Ok(())
}
}