fix(dqn): Plan C K — Adam shrink-and-perturb + adaptive fold-warmup ISV

Closes the F1 ep1 catastrophic-overshoot gap exposed by smoke-test-s9h4h
(F0 succeeded with Best Sharpe = 36.03; F1 ep1 grad_norm = 355,009 — 5
orders of magnitude larger than F0 steady-state ~10 — leading to NaN
propagation and grad-clamp-to-zero early stop). Combined fix: (1) Adam
shrink-and-perturb at fold boundary (replace m=0/v=0 with m*=0.1, v*=0.01
to preserve direction while damping magnitude), and (2) a single
adaptive ISV signal driving BOTH lr_eff and clip_eff dampening over
the fold's first ~50 steps.

K — Adam shrink-and-perturb in `GpuDqnTrainer::reset_adam_state`:
- m *= 0.1, v *= 0.01 via existing `dqn_scale_f32_kernel` (loaded as
  `scale_f32_ungraphed`); t_pinned still zeroed so bias correction
  restarts. Architectural constants (preserve direction / lose magnitude
  history) per `feedback_isv_for_adaptive_bounds.md` Invariant 1
  carve-out — not tuned. Composes with existing param shrink-and-perturb
  (`alpha=0.8`) in `FusedTrainingCtx::reset_for_fold`. Root cause for
  F1 overshoot: m=0,v=0 → first Adam step ≈ lr × g / ε → 6 OoM
  amplification.

New CUDA kernel `fold_warmup_factor_kernel.cu`:
- Single-block single-thread cold-path producer mirroring
  `q_drift_rate_ema_kernel.cu` / `moe_lambda_eff_kernel.cu` shape.
- Reads two grad-norm EMAs (fast α=0.1, slow α=0.001) plus host-passed
  step counter; writes ISV[FOLD_WARMUP_FACTOR_INDEX=130] = clamp(fast/slow, 0, 1).
- Bootstrap branches (steps_observed < 200, slow EMA < 1e-6) emit
  factor=1.0 (no damping during cold-start). No atomicAdd; no DtoH.

New ISV slot Q_DRIFT_RATE_INDEX → FOLD_WARMUP_FACTOR_INDEX = 130:
- ISV_TOTAL_DIM 130 → 131; layout fingerprint shifts (checkpoint-
  incompatible per `feedback_no_legacy_aliases.md`, expected for a
  real architecture change).
- FoldReset entries: `isv_fold_warmup_factor` → 0.0 and companion
  `isv_grad_norm_fast_ema` → 0.0 (lockstep reset per
  `feedback_no_partial_refactor.md`); slow EMA persists across folds
  as the cross-fold steady-state baseline.
- Two new mapped-pinned scalars on GpuDqnTrainer (grad_norm_fast_ema_pinned,
  grad_norm_slow_ema_pinned) fed by `update_adaptive_clip` from the
  same `gr.raw_grad_norm` observation source as the existing adaptive
  clip EMA.

Two consumers, both monotone (only dampen, never excite):
- lr_eff   = cosine_effective_lr × max(MIN_WARMUP_LR_FRAC=0.05, factor)
            via `set_lr` per-step. New `cosine_effective_lr_base` field
            on DQNTrainer composes the cosine schedule's per-epoch
            baseline with the warmup factor's per-step damping (rather
            than overriding the cosine schedule).
- clip_eff = clip_base × (MIN_CLIP_FRAC=0.1 + 0.9 × factor) via new
            `set_active_clip` setter on FusedTrainingCtx + GpuDqnTrainer.
            Composes with the EMA-derived `clip_base = grad_norm_ema × 2`
            that `update_adaptive_clip` just wrote to the pinned slot.
            Numerical-stability bounds 0.05 / 0.1 are Invariant 1
            carve-outs.

Steady-state behaviour unchanged: factor=1 → lr_eff=lr_base,
clip_eff=clip_base. Fold-boundary behaviour: factor starts at 0 →
lr_eff = 0.05 × lr_base, clip_eff ≈ 0.1 × clip_base; rises to 1 over
~50 steps as the fast EMA catches up to the slow steady-state EMA.

Predicted impact on Plan C smoke F1: 355,009-magnitude transient grad
clipped to ~clip_base × 0.1 ≈ 1.0 (vs 10), Adam state shrunk instead
of zeroed → first step update bounded; grad recovers normally over
~50 steps. Companion to A.1 (prev_epoch_q_mean reset), A.2 (adaptive
Polyak-tau), A.3 (gradient_collapse_counter reset), F+H (kill-criterion
robustness) — completes the fold-boundary state-reset family.

Per `pearl_adaptive_moe_lambda.md` (kernel + ISV slot + bootstrap +
reset + observability template), `pearl_cold_path_no_exception_to_gpu_drives.md`
(GPU-stays-on-GPU even at cold-path cadence),
`pearl_blend_formulas_must_have_permanent_floor.md` (lr/clip floors
are permanent minimums), `feedback_adaptive_not_tuned.md` (lr+clip
ISV-driven), `feedback_isv_for_adaptive_bounds.md` (factor IS the
bound; consumers compose at runtime), `feedback_no_atomicadd.md`
(single-thread reduce), `feedback_cudarc_f64_f32_abi.md` (slot index
passed as i32), `feedback_no_partial_refactor.md` (kernel + slot +
reset + producer + 2 consumers all land together),
`feedback_no_quickfixes.md` (replaces brittle full-reset with
adaptive damping; not threshold relaxation).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-04-29 20:24:36 +02:00
parent ef243e771d
commit 4ef1d8ebb7
9 changed files with 611 additions and 6 deletions

View File

@@ -2200,3 +2200,27 @@ Plan C Phase 2 follow-up F (2026-04-29): wire `q_mag_bin_means_reduce` and `q_di
Plan C Phase 2 follow-up (2026-04-29): bonus reward optimism-coupling break in `experience_kernels.cu`. The C.4 timing bonus (~line 2521) and D.4b regime penalty (~line 2581) both used the trade-cumulative `|final_pnl|` / `|reward|` as their unbounded multiplicand. By the time bonus shaping runs in `experience_env_step`, `reward` is the Layer 2 vol-normalised return PLUS Layer 4-9 credits PLUS the C.4 timing bonus PLUS the D.4a persistence bonus — i.e. it is itself a function of all prior shaping inflation. Across trades this compounds into a multi-trade optimism feedback loop: bonus inflates → Q-target inflates (Bellman bootstraps off shaped reward) → next trade's reward inflates → bonus inflates further. Researcher reproduction (a25f669e9df953174, 2026-04-29) found the a52d99613 baseline ALSO hits bonus EMA=235 by F2 ep2 — the pathology is structural and pre-dates Plan C Phase 2. Per `pearl_one_unbounded_signal_per_reward.md` the rule is exactly ONE unbounded multiplicand per reward term, and per `feedback_isv_for_adaptive_bounds.md` adaptive bounds live in the ISV signal bus. **Surgical fix**: replace `|final_pnl|` (C.4) and `|reward|` (D.4b) with `min(|x|/ISV[Q_DIR_ABS_REF_INDEX=21], 1) × ISV[21]` — the multiplicand is now the direction-branch Q-scale EMA (already produced gradient-decoupled by `q_stats_kernel.cu` since Plan 1) rather than the trade-cumulative shaping output. Cap is adaptive (matches Q magnitude as it evolves) and breaks cross-trade compounding; |reward| can still drive the cap to its max (= ISV[21]) so directional information is preserved, just bounded. **Kernel signature**: `experience_env_step` gains a final `int q_dir_abs_ref_idx` parameter (after the existing `trade_target_rate_idx`); single Rust call site `gpu_experience_collector.rs:3800` passes `Q_DIR_ABS_REF_INDEX as i32`. `experience_action_select` not affected (it never read `|reward|`/`|final_pnl|` as a multiplicand). **Other shaping sites NOT touched** because they were already bounded — D.4a persist (multiplicand `drawdown_depth ≤ ~0.1` gates the term, `tanh(reward/dd)` is bounded), B.2 novelty (vol_proxy ≤ 0.01 gates, kl_amp ≤ 1.0), D.4c stable (vol_proxy gates + bounded stability/conviction). Per `feedback_no_partial_refactor.md` consistency: both fixed sites use the SAME ISV[21] bound and the SAME `(unit, capped)` two-step formula — symmetric migration. Predicted impact: bonus EMA O(100-256) → O(0.05-2.0); rc[5] → Bellman-target optimism loop broken; Plan C smoke F0 ep2 Q-drift kill should no longer fire on this pathway; a52d99613 baseline gets the same fix automatically (validates pre-existing pathology, not Plan-C-specific). Layout fingerprint unchanged — kernel signature change but no ISV slot add and no param-tensor change.
Plan C T11 follow-up A.3 (2026-04-29): added `DQN::reset_for_fold` zeroing `gradient_collapse_counter` + `training_steps`. Wired from `DQNTrainer::reset_for_fold` next to the A.1 `prev_epoch_q_mean` reset. Discovered when `smoke-test-vh9bj`'s fold-0 succeeded (F+H fix worked) but fold-1 failed immediately at epoch 1 with "Gradient collapse detected for 5 consecutive" — the counter had carried over from fold 0's near-zero-grad late-epoch steps, plus the `past_warmup` gate was always-true in fold 1+ because `training_steps` accumulated. Resetting both restores per-fold warmup semantics. Same fold-boundary state-reset gap pattern as A.1.
Plan C Phase 2 follow-up K (2026-04-29): combined Adam shrink-and-perturb + adaptive fold-boundary warmup factor. Closes the F1 ep1 catastrophic-overshoot gap exposed by smoke-test-s9h4h (F0 successful with Best Sharpe = 36.03; F1 ep1 grad_norm = 355,009 — 5 OoM larger than F0 steady-state ~10 — leading to NaN propagation and grad-clamp-to-zero).
* **Adam shrink-and-perturb** in `GpuDqnTrainer::reset_adam_state` replaces the prior full-zero reset of `m_buf` / `v_buf` with multiplicative shrink (`m *= 0.1`, `v *= 0.01`). Preserves directional information (sign+ratio across tensors) while damping magnitude; composes with the existing param shrink-and-perturb (`alpha=0.8` in `FusedTrainingCtx::reset_for_fold`). Shrink launches use the existing `dqn_scale_f32_kernel` (loaded as `scale_f32_ungraphed` — same module / same launch path as `scale_adam_momentum`). `t_pinned` (Adam step counter) IS still zeroed so the bias correction `1 - β^t` restarts properly. Per `feedback_isv_for_adaptive_bounds.md` Invariant 1 carve-out: shrink factors `0.1` / `0.01` are STRUCTURAL ("preserve direction / lose magnitude history") not tuned constants. Root cause for the F1 overshoot: `m=0, v=0` → first Adam step is `lr × m̂ / (sqrt(v̂) + ε) ≈ lr × g / ε` (a 6 OoM amplification when `ε = 1e-8`).
* **New CUDA kernel** `crates/ml/src/cuda_pipeline/fold_warmup_factor_kernel.cu` (single-block single-thread cold-path producer mirroring `q_drift_rate_ema_kernel.cu` / `moe_lambda_eff_kernel.cu` shape). Reads two grad-norm EMAs (fast α=0.1 ~10-step horizon, slow α=0.001 ~1000-step horizon) plus a host-passed step counter; writes `clamp(fast/slow, 0, 1)` to `ISV[FOLD_WARMUP_FACTOR_INDEX=130]`. Bootstrap branches for the cold-start window (`steps_observed < WARMUP_FACTOR_MIN_STEPS=200`) and the slow-EMA-near-zero edge case both emit `factor = 1.0` (no damping). Registered in `crates/ml/build.rs` kernel list (count grew by 1).
* **New ISV slot** `FOLD_WARMUP_FACTOR_INDEX = 130` (tail-appended after `Q_DRIFT_RATE_INDEX = 129`, raising `ISV_TOTAL_DIM` 130→131). Cold-start 0.0 (constructor `*sig_ptr.add(FOLD_WARMUP_FACTOR_INDEX) = 0.0_f32` is satisfied by the existing zero-init of the mapped-pinned ISV bus); FoldReset 0.0 via new `isv_fold_warmup_factor` registry entry in `state_reset_registry.rs` + dispatch arm in `training_loop.rs::reset_named_state`. Companion FoldReset entry `isv_grad_norm_fast_ema` resets the kernel's NUMERATOR (host-side mapped-pinned scalar) so the ratio starts at 0 (fast=0 over slow≈baseline) → factor=0 → heavy damping for the new fold's first steps. Slow EMA `isv_grad_norm_slow_ema` deliberately persists across folds (cross-fold steady-state baseline). Layout fingerprint shifts (one slot added + ISV_TOTAL_DIM bump) — checkpoint-incompatible per `feedback_no_legacy_aliases.md`, expected for a real architecture change.
* **Two new mapped-pinned scalars** on `GpuDqnTrainer`: `grad_norm_fast_ema_pinned` (α=0.1) and `grad_norm_slow_ema_pinned` (α=0.001). Both updated by `update_adaptive_clip` (the same `gr.raw_grad_norm` observation source that already feeds the legacy `grad_norm_ema` field driving the existing adaptive clip). Step counter `grad_norm_emas_step_count` gates the kernel's bootstrap window. Mapped-pinned via `cuMemHostAlloc DEVICEMAP` per `feedback_no_htod_htoh_only_mapped_pinned.md`. `Drop` impl frees both alongside `adaptive_clip_pinned`.
* **Rust orchestrator wrapper** `GpuDqnTrainer::launch_fold_warmup_factor()` in `gpu_dqn_trainer.rs` — same pattern as `launch_q_drift_rate_ema` / `launch_h_s2_rms_ema` (debug_assert non-zero `isv_signals_dev_ptr`, single-thread launch config, kernel-launch error mapped to `MLError::ModelError`).
* **Per-step producer launch** in `training_loop.rs` alongside `launch_h_s2_rms_ema` — fires once per training step, after `update_adaptive_clip` has fed the fast/slow EMAs. Cold-path cadence; no atomicAdd; no DtoH.
* **LR consumer wire-up** in `training_loop.rs` per-step block: read `ISV[FOLD_WARMUP_FACTOR_INDEX]`, derive `lr_eff = cosine_effective_lr_base × max(MIN_WARMUP_LR_FRAC=0.05, factor)`, write via `set_lr`. New `cosine_effective_lr_base: f32` field on `DQNTrainer` records the cosine-scheduled per-epoch baseline so the per-step warmup damping multiplies the cosine schedule rather than overriding it. Floor `0.05` is a numerical-stability bound (Invariant 1 carve-out per `feedback_isv_for_adaptive_bounds.md`).
* **Clip consumer wire-up** in same block: after `update_adaptive_clip` writes `clip_base = grad_norm_ema × 2` (floored at 1.0) to the pinned slot, scale by `clip_warmup_scale = MIN_CLIP_FRAC=0.1 + (1 - MIN_CLIP_FRAC) × factor` and write the result via the new `FusedTrainingCtx::set_active_clip` setter (forwarded to `GpuDqnTrainer::set_active_clip`, which writes `*adaptive_clip_pinned` directly — same zero-graph-recapture contract as `set_lr`). Floor `0.1` is a numerical-stability bound.
* **Both consumers are MONOTONE** (factor ∈ [0, 1] structurally, so `lr_eff ≤ lr_base` and `clip_eff ≤ clip_base` always). Healthy runs (steady-state factor=1) are unaffected — `lr_eff = lr_base × max(0.05, 1) = lr_base` and `clip_eff = clip_base × (0.1 + 0.9) = clip_base`. Per `pearl_blend_formulas_must_have_permanent_floor.md`: `MIN_WARMUP_LR_FRAC` and `MIN_CLIP_FRAC` are permanent floors — even at factor=0 the consumers never reach zero lr/clip.
Predicted impact on Plan C smoke fold 1: factor starts at 0 → `lr_eff = 0.05 × lr_base`, `clip_eff = 0.1 × clip_base ≈ 1.0` (vs `clip_base = 10`). The 355,009-magnitude transient grad gets clipped tightly to ~1, Adam state shrunk (m × 0.1, v × 0.01) instead of zeroed → first step update is bounded by `0.05 × lr × m̂/(sqrt(v̂) + ε)` on a non-degenerate denominator. Over ~50 steps the fast EMA catches up to the slow steady-state baseline → factor → 1 → full lr/clip restore. Steady-state (mid-fold) behavior unchanged.
Per `pearl_adaptive_moe_lambda.md` — canonical "EMA-tracked diagnostic drives a controller" pattern: kernel + ISV slot + bootstrap (0.0 cold-start) + reset (0.0 fold boundary, fast EMA) + observability (HEALTH_DIAG via slot 130 mirror, plus `grad_norm_fast_ema_value` / `grad_norm_slow_ema_value` accessors on the trainer). Per `pearl_cold_path_no_exception_to_gpu_drives.md` — even a cold-path scalar arithmetic stays on GPU when its inputs already live in mapped-pinned host memory the device can read directly (no DtoH for the EMAs). Per `feedback_adaptive_not_tuned.md` — both lr_eff and clip_eff are now ISV-driven, not constant. Per `feedback_isv_for_adaptive_bounds.md` — the warmup factor IS the bound; consumers read ISV[130] at runtime and compose with their respective architectural floors. Per `feedback_no_partial_refactor.md` — both halves of the warmup-factor input contract (factor slot + fast EMA companion) reset together; both consumers (lr + clip) wire together; producer launch + consumer reads land in the same commit.