diff --git a/crates/ml-dqn/src/dqn.rs b/crates/ml-dqn/src/dqn.rs index eb502b169..7705af353 100644 --- a/crates/ml-dqn/src/dqn.rs +++ b/crates/ml-dqn/src/dqn.rs @@ -1943,6 +1943,33 @@ impl DQN { self.count_bonus.reset(); } + /// Reset state that should not persist across walk-forward fold + /// boundaries. Pairs with `DQNTrainer::reset_for_fold`. + /// + /// What's reset (and why): + /// - `gradient_collapse_counter` — counts consecutive low-grad steps; + /// carrying across folds caused fold 1 to trip patience after a + /// single collapse-tier step (counter starts already near patience). + /// - `training_steps` — the warmup gate `past_warmup = training_steps + /// > warmup_steps` becomes always-true after fold 0 completes, + /// removing the per-fold warmup grace period for data distribution + /// shifts at fold boundaries. Resetting gives fold 1+ a clean + /// warmup window before the gradient-collapse check fires. + /// + /// What's NOT reset (model + agent state): + /// - Model parameters (the whole point of walk-forward; fold N+1 + /// bootstraps from fold N's learned weights) + /// - count_bonus state (Plan C T2 amendment removed direct dependence + /// on UCB count bonus; the count_bonus persistence is handled by + /// `reset_count_bonus` called explicitly elsewhere if needed) + /// - target_network (handled by `reset_target_network` if explicitly + /// wanted; per project_fold_boundary_q_drift_resolved.md the IQN + /// sync_target_from_online is the principled mechanism) + pub fn reset_for_fold(&mut self) { + self.gradient_collapse_counter = 0; + self.training_steps = 0; + } + /// Get count bonuses for all actions (UCB exploration). /// Returns a Vec of length `num_actions` with exploration bonuses. pub fn get_count_bonuses(&self) -> Vec { diff --git a/crates/ml/src/trainers/dqn/config.rs b/crates/ml/src/trainers/dqn/config.rs index 5d7b7efc6..9ffc62872 100644 --- a/crates/ml/src/trainers/dqn/config.rs +++ b/crates/ml/src/trainers/dqn/config.rs @@ -112,6 +112,13 @@ impl DQNAgentType { self.agent.reset_count_bonus(); } + /// Reset DQN state that should not persist across walk-forward folds. + /// Delegates to `DQN::reset_for_fold` (zeros `gradient_collapse_counter` + /// and `training_steps`). Pairs with `DQNTrainer::reset_for_fold`. + pub fn reset_for_fold(&mut self) { + self.agent.reset_for_fold(); + } + /// Get count bonuses for all actions (UCB exploration). /// Returns a Vec of length `num_actions` with exploration bonuses. pub fn get_count_bonuses(&self) -> Vec { diff --git a/crates/ml/src/trainers/dqn/trainer/mod.rs b/crates/ml/src/trainers/dqn/trainer/mod.rs index eac57f083..379c088a5 100644 --- a/crates/ml/src/trainers/dqn/trainer/mod.rs +++ b/crates/ml/src/trainers/dqn/trainer/mod.rs @@ -1749,6 +1749,20 @@ impl DQNTrainer { self.best_epoch = 0; self.best_val_loss = f64::INFINITY; + // Plan C T11 follow-up A.3 (2026-04-29): reset DQN agent state that's + // keyed to fold lifecycle (gradient_collapse_counter, training_steps). + // Pairs with the prev_epoch_q_mean reset above — see DQN::reset_for_fold + // for the contract. Without this, smoke smoke-test-vh9bj fold 1 ep1 + // tripped the gradient-collapse patience instantly because fold 0's + // late-epoch near-zero-grad steps had pushed the counter near patience, + // and `past_warmup = training_steps > warmup_steps` was always-true in + // fold 1+ (no per-fold warmup grace period for data distribution shifts). + // Same fold-boundary state-reset gap pattern as A.1. + { + let mut agent = self.agent.write().await; + agent.reset_for_fold(); + } + // Plan 1 Task 3 + Plan 2 Task 4: registry-driven fold-boundary reset. // FoldReset entries are cleared/zeroed; SoftReset entries are written // to their bootstrap values (CPU-born input, not adaptive computation) diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index 4fe7eb56e..71047856a 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -2198,3 +2198,5 @@ Plan C Phase 2 follow-up H (2026-04-29): replace brittle `ratio = |q_mean| / |pr Plan C Phase 2 follow-up F (2026-04-29): wire `q_mag_bin_means_reduce` and `q_dir_bin_means_reduce` per-step alongside `update_isv_signals` in `fused_training.rs` (both the captured `adam_update` child graph at ~line 2228 AND the ungraphed step-0 fallback at ~line 1465). **Root cause for the smoke-test-n9xzr `ISV[16]=0.0500, ISV[21]=0.0500` floor stuck at the cold-start `max(0.5, 3 × 0.05) = 0.5` value**: the producers ran ONLY inside `reduce_current_q_stats` (epoch boundary cadence) at `gpu_dqn_trainer.rs:15223`, while the per-step captured `update_isv_signals` consumed scratch buffers that were therefore zero-initialized for all of epoch 0 and stale-by-one-epoch thereafter. With α=0.05 EMA and one effective producer fire per epoch, ISV[16,21] climbed glacially (0.05 → 0.0975 → 0.143 over three epochs vs the `q_mean=0.7647` magnitude actually being observed), so the adaptive floor never graduated above the 0.5 cold-start hard-floor. The kill criterion's `kill_floor = max(0.5, 3 × max(ISV[16], ISV[21]))` collapsed to the literal `0.5` constant for the entire smoke. The fix runs both `launch_q_mag_bin_means_reduce` and `launch_q_dir_bin_means_reduce` immediately before `update_isv_signals` in BOTH paths so per-step graph replays read fresh `q_out_buf` (already populated by `forward_child` earlier in the parent-graph topological order) and feed live scalars into the per-step ISV EMA. Epoch 1 onward then sees ISV[16,21] saturate to the policy's actual q_abs_ref scale within ~60 steps (95% saturation at α=0.05). Per `pearl_cold_path_no_exception_to_gpu_drives.md`: even cold-path EMAs stay on GPU and fire alongside their consumers — once-per-epoch was the wrong cadence. Per `feedback_no_partial_refactor.md`: graphed and ungraphed paths both migrate in this commit (same producer-consumer contract). Layout fingerprint unchanged — no kernel signature change, no ISV slot add, no param-tensor change. Plan C Phase 2 follow-up (2026-04-29): bonus reward optimism-coupling break in `experience_kernels.cu`. The C.4 timing bonus (~line 2521) and D.4b regime penalty (~line 2581) both used the trade-cumulative `|final_pnl|` / `|reward|` as their unbounded multiplicand. By the time bonus shaping runs in `experience_env_step`, `reward` is the Layer 2 vol-normalised return PLUS Layer 4-9 credits PLUS the C.4 timing bonus PLUS the D.4a persistence bonus — i.e. it is itself a function of all prior shaping inflation. Across trades this compounds into a multi-trade optimism feedback loop: bonus inflates → Q-target inflates (Bellman bootstraps off shaped reward) → next trade's reward inflates → bonus inflates further. Researcher reproduction (a25f669e9df953174, 2026-04-29) found the a52d99613 baseline ALSO hits bonus EMA=235 by F2 ep2 — the pathology is structural and pre-dates Plan C Phase 2. Per `pearl_one_unbounded_signal_per_reward.md` the rule is exactly ONE unbounded multiplicand per reward term, and per `feedback_isv_for_adaptive_bounds.md` adaptive bounds live in the ISV signal bus. **Surgical fix**: replace `|final_pnl|` (C.4) and `|reward|` (D.4b) with `min(|x|/ISV[Q_DIR_ABS_REF_INDEX=21], 1) × ISV[21]` — the multiplicand is now the direction-branch Q-scale EMA (already produced gradient-decoupled by `q_stats_kernel.cu` since Plan 1) rather than the trade-cumulative shaping output. Cap is adaptive (matches Q magnitude as it evolves) and breaks cross-trade compounding; |reward| can still drive the cap to its max (= ISV[21]) so directional information is preserved, just bounded. **Kernel signature**: `experience_env_step` gains a final `int q_dir_abs_ref_idx` parameter (after the existing `trade_target_rate_idx`); single Rust call site `gpu_experience_collector.rs:3800` passes `Q_DIR_ABS_REF_INDEX as i32`. `experience_action_select` not affected (it never read `|reward|`/`|final_pnl|` as a multiplicand). **Other shaping sites NOT touched** because they were already bounded — D.4a persist (multiplicand `drawdown_depth ≤ ~0.1` gates the term, `tanh(reward/dd)` is bounded), B.2 novelty (vol_proxy ≤ 0.01 gates, kl_amp ≤ 1.0), D.4c stable (vol_proxy gates + bounded stability/conviction). Per `feedback_no_partial_refactor.md` consistency: both fixed sites use the SAME ISV[21] bound and the SAME `(unit, capped)` two-step formula — symmetric migration. Predicted impact: bonus EMA O(100-256) → O(0.05-2.0); rc[5] → Bellman-target optimism loop broken; Plan C smoke F0 ep2 Q-drift kill should no longer fire on this pathway; a52d99613 baseline gets the same fix automatically (validates pre-existing pathology, not Plan-C-specific). Layout fingerprint unchanged — kernel signature change but no ISV slot add and no param-tensor change. + +Plan C T11 follow-up A.3 (2026-04-29): added `DQN::reset_for_fold` zeroing `gradient_collapse_counter` + `training_steps`. Wired from `DQNTrainer::reset_for_fold` next to the A.1 `prev_epoch_q_mean` reset. Discovered when `smoke-test-vh9bj`'s fold-0 succeeded (F+H fix worked) but fold-1 failed immediately at epoch 1 with "Gradient collapse detected for 5 consecutive" — the counter had carried over from fold 0's near-zero-grad late-epoch steps, plus the `past_warmup` gate was always-true in fold 1+ because `training_steps` accumulated. Resetting both restores per-fold warmup semantics. Same fold-boundary state-reset gap pattern as A.1.