3 Commits

Author	SHA1	Message	Date
jgrusewski	12fdd18223	refactor: remove entire CPU training path — 5,307 lines of dead code ... Deleted: - DQN::compute_loss_internal (280 lines) — old Candle forward+loss - DQN::train_step (55 lines) — old Candle training step - DQN::compute_gradients (47 lines) — old gradient accumulation - ComputeLossResult struct — only used by deleted functions - RegimeConditionalDQN::train_step (65 lines) — old dispatch - RegimeConditionalDQN::train_step_gpu_regime (100 lines) — old GPU path - RegimeConditionalDQN::compute_gradients_gpu (130 lines) — old regime gradients - RegimeConditionalDQN::compute_gradients (92 lines) — old dispatch - DQNAgentType::train_step dispatch — dead - DQNAgentType::compute_gradients dispatch — dead - GpuDqnTrainer::upload_batch (71 lines) — old CPU→GPU upload - train_step.rs (500 lines) — entire module including ensure_fused_ctx - dqn_benchmark.rs — used old train_step - examples.rs — used old train_step - validation/adapters.rs (289 lines) — used old train_step - dqn/trainable_adapter.rs — used old train_step - gpu_smoketest.rs — tested old train_step - Gradient accumulation path in training_loop.rs (144 lines) - IQN d_h_s2().clone() → raw pointer (zero alloc) - Causal intervention format! string alloc removed - Dead HER relabel functions (320 lines) Kept: - ensure_fused_ctx logic inlined into training_loop.rs - set_noise_sigma_scale re-added to RegimeConditionalDQN Fixed: - GpuReplayBuffer max_batch_size wired from batch_size parameter (was hardcoded 1024, blocking batch_size=8192) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-02 09:06:23 +02:00
jgrusewski	35d417f08e	chore: remove accidental local test artifacts from ml/ dir ... Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-03-23 00:40:01 +01:00
jgrusewski	0b37ff77b0	fix: reward v2 + dynamic C51 support — root cause of Q-value collapse ... ROOT CAUSE: 5 interlocking bugs made learning impossible: 1. DSR denominator floor 1e-12 produced values in millions → drowned all signal 2. Global [-1,+1] clamp destroyed Bellman equation signal (can't distinguish catastrophic loss from mild loss) 3. v_range=20 exactly equals V_max for gamma=0.95 → Bellman target pins at ceiling → Q-values saturate → Q-gap collapses to 0.0000 4. num_atoms=11 over 40-unit range = 4.0 per atom (C51 paper min is 51) 5. 6/7 reward components were penalties → mean_reward=-0.311 regardless of action FIXES: - DSR denominator floor: 1e-12 → 0.01 (prevents million-scale spikes) - Each component individually clamped BEFORE weighting (DSR to [-1,+1], z-score to [-3,+3], drawdown to [0,1], time decay to [0,0.3]) - Removed global [-1,+1] clamp (no longer needed with bounded components) - profit_take_bonus: 0.1 → 0.01 (was 100x too large, caused reward hacking) - Removed confidence scaling (positive feedback loop destabilized learning) - Removed regime scaling (non-stationary reward confused the model) - Dynamic v_range from gamma: v_range = 2.5/(1-gamma)*1.2 (always covers Q range) - num_atoms minimum: 11 → 51 (C51 paper standard) - gamma default: 0.99 → 0.95 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-03-23 00:39:39 +01:00