From bdc5cb8bb2ac14fc6ea8fc61fccc7f209fc2ed40 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Tue, 5 May 2026 08:10:35 +0200 Subject: [PATCH] =?UTF-8?q?feat(sp13):=20P0b=20=E2=80=94=20aux=5Fw=20defic?= =?UTF-8?q?it+stagnation=20controller=20+=20Hold=20cost=20lift?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit P0a smoke (train-67gqb on f934ea171) returned PARTIAL: aux_dir_acc climbed 0.149 → 0.483 in 10 epochs (signal exists), but val_win_rate stuck at 0.4638 and observed_hold_rate climbed to 0.479 despite controller saturating at 2.4× base. Two findings drive P0b: (1) aux_w=0.05 (SP11-era clamp) starves the aux head of gradient. Replace inverted formula at training_loop.rs SP11 site with deficit+stagnation: deficit = max(0, target - short_ema) improve = max(0, short_ema - long_ema) stag = (deficit > 0.005) ? clamp(1 - improve/deficit, 0, 1) : 0 aux_w = base × (1 + 5 × deficit) × (1 - 0.7 × stag), clamped [0.3×base, 3.0×base] base 0.05 → 0.5 (10× lift). Stagnation decay prevents permanent destabilisation in data-limited case. Formula extracted as host helper `compute_aux_w_p0b(target, short, long)` for unit-test coverage. (2) HOLD_COST_BASE=0.001 was too weak — max cost 0.005/bar × 30-bar hold = 0.15 cumulative vs ±5-10 reward range = <3% of magnitude. Lift to 0.005 (max 0.025/bar × 30 bars = 0.75 cumulative ≈ 10-15% of capped reward). Genuinely deters lazy Hold without crippling MFT use. Constructor static-init unchanged: it still writes the (now-lifted) HOLD_COST_BASE constant to slot 380 at fold boundary. Per pearl_event_driven_reward_density_alignment tension already addressed in P0a spec; the lift doesn't change the architecture, just the calibration. Per feedback_isv_for_adaptive_bounds: base/gain/decay/floor/ceil are numerical anchors; target/short/long EMAs read from ISV. Tests: 3 new unit tests for controller formula (aux_w_at_target_returns_base, aux_w_stagnation_decays_to_floor, aux_w_improving_amplifies_above_base). 14 SP13 GPU oracle tests + 14 SP12 reward-math tests still green; no kernel changes. Audit-doc updated with P0b section per Invariant 7. Co-Authored-By: Claude Opus 4.7 (1M context) --- crates/ml/src/cuda_pipeline/sp13_isv_slots.rs | 27 +++++- crates/ml/src/trainers/dqn/trainer/tests.rs | 94 +++++++++++++++++++ .../src/trainers/dqn/trainer/training_loop.rs | 89 ++++++++++++++---- docs/dqn-wire-up-audit.md | 60 ++++++++++++ 4 files changed, 251 insertions(+), 19 deletions(-) diff --git a/crates/ml/src/cuda_pipeline/sp13_isv_slots.rs b/crates/ml/src/cuda_pipeline/sp13_isv_slots.rs index 955a811ec..5a3363366 100644 --- a/crates/ml/src/cuda_pipeline/sp13_isv_slots.rs +++ b/crates/ml/src/cuda_pipeline/sp13_isv_slots.rs @@ -60,12 +60,37 @@ pub const SKILL_BONUS_CAP_RATIO_DEFAULT: f32 = 0.3; /// (~10 ticks of price for ES.FUT at ~0.25 tick value) but cumulative across /// long Hold runs becomes meaningful relative to the SP12 capped trade reward /// (±5..10). -pub const HOLD_COST_BASE: f32 = 0.001; +/// +/// P0b lift (2026-05-04): `HOLD_COST_BASE` raised 0.001 → 0.005 after P0a smoke +/// (`train-67gqb` on `f934ea171`) showed `observed_hold_rate` climbing to 0.479 +/// despite the controller saturating at 2.4× base — model rationally absorbed +/// the cost because it was <3% of reward magnitude (0.001 × 5 ceiling × 30-bar +/// hold = 0.15 cumulative vs ±5..10 reward range). Lifted base × 5 ceiling now +/// yields 0.025/bar × 30 bars = 0.75 cumulative ≈ 10–15% of capped reward — +/// genuinely deters lazy Hold without crippling MFT use. Per-bar slot wiring +/// unchanged: constructor static-init still writes `HOLD_COST_BASE` to slot 380 +/// at fold boundary; controller now scales the (lifted) base. +pub const HOLD_COST_BASE: f32 = 0.005; pub const HOLD_RATE_TARGET_DEFAULT: f32 = 0.20; pub const HOLD_COST_CONTROLLER_GAIN: f32 = 5.0; pub const HOLD_COST_FLOOR_RATIO: f32 = 0.5; // never below 0.5 × base pub const HOLD_COST_CEIL_RATIO: f32 = 5.0; // never above 5.0 × base +/// SP13 P0b: aux_w deficit + stagnation controller anchors. Replaces the +/// SP11-era inverted formula `aux_w = 0.1 × health × (1 - sharpe).clamp(0.05, 0.3)` +/// at `training_loop.rs` (which LOWERED aux when sharpe was high — exactly +/// wrong for the failure mode where high sharpe + low WR is the structural- +/// extraction trap). The new controller drives aux_w from the deficit +/// `target_dir_acc - short_ema` and decays back to floor when the short EMA +/// fails to pull ahead of the long EMA (data-limited stagnation). Per +/// `feedback_isv_for_adaptive_bounds.md`: base/gain/decay/floor/ceil are +/// numerical anchors; target/short/long EMAs read from ISV. Spec §"Change 5". +pub const AUX_W_BASE: f32 = 0.5; // 10× lift from SP11 0.05 floor +pub const AUX_W_DEFICIT_GAIN: f32 = 5.0; // aux_w = base × (1 + 5 × deficit) +pub const AUX_W_STAGNATION_DECAY: f32 = 0.7; // (1 - 0.7 × stagnation) +pub const AUX_W_HARD_FLOOR_RATIO: f32 = 0.3; // never below 0.3 × base +pub const AUX_W_HARD_CEIL_RATIO: f32 = 3.0; // never above 3.0 × base + /// Sentinel for both dir-acc EMAs at fold boundary: 0.5 is the random-guessing /// baseline for binary directional accuracy. First observation replaces this /// directly per `pearl_first_observation_bootstrap` so the EMA tracks the new diff --git a/crates/ml/src/trainers/dqn/trainer/tests.rs b/crates/ml/src/trainers/dqn/trainer/tests.rs index 94fd52d6e..c146e67b4 100644 --- a/crates/ml/src/trainers/dqn/trainer/tests.rs +++ b/crates/ml/src/trainers/dqn/trainer/tests.rs @@ -617,3 +617,97 @@ fn test_14d_roundtrip_preserves_intensities() { "ensemble_intensity must survive roundtrip" ); } + +/// SP13 P0b — aux_w deficit + stagnation controller unit tests. +/// +/// The kernel-tested machinery (apply_pearls EMAs, dir_acc reduction) lives in +/// `sp13_phase0_oracle_tests.rs`. These tests cover the host-side controller +/// formula `compute_aux_w_p0b(target, short, long)` extracted in P0b for +/// testability, exercising the two boundary regimes the spec calls out: +/// 1. at-target: deficit=0 → aux_w collapses to base +/// 2. fully-stagnant: short below target with no improvement → decays to floor +mod aux_w_controller_tests { + use crate::cuda_pipeline::sp13_isv_slots::{ + AUX_W_BASE, AUX_W_HARD_FLOOR_RATIO, + }; + use super::super::training_loop::compute_aux_w_p0b; + + /// At-target steady state: short = target, no deficit, no stagnation + /// activation → aux_w = base × 1 × 1 = base. + #[test] + fn aux_w_at_target_returns_base() { + let target = 0.55_f32; + let short = 0.55_f32; + let long = 0.50_f32; + let aux_w = compute_aux_w_p0b(target, short, long); + assert!( + (aux_w - AUX_W_BASE).abs() < 1e-6, + "expected base={}, got {}", + AUX_W_BASE, aux_w, + ); + } + + /// Stagnation regime: short stuck below target with zero improvement + /// (short == long) → stagnation = 1.0 → (1 - 0.7 × 1) = 0.3 decay factor + /// drives aux_w_raw close to floor without crippling Q-head training. + /// With deficit=0.05: aux_w_raw = 0.5 × 1.25 × 0.3 = 0.1875. The hard floor + /// 0.15 = base × 0.3 prevents further suppression even at extreme stagnation + /// (the 0.3 decay factor cannot push below floor unless deficit→0, in which + /// case the deficit-gain term collapses anyway). The contract: under + /// stagnation, aux_w must be much closer to floor than the matching + /// improving regime would give. + #[test] + fn aux_w_stagnation_decays_to_floor() { + let target = 0.55_f32; + let short = 0.50_f32; + let long = 0.50_f32; // no improvement → stagnation = 1 + let aux_w_stagnant = compute_aux_w_p0b(target, short, long); + let floor = AUX_W_BASE * AUX_W_HARD_FLOOR_RATIO; + + // Sanity: never below the hard floor. + assert!( + aux_w_stagnant >= floor - 1e-6, + "must respect floor={}; got {}", + floor, aux_w_stagnant, + ); + + // Same deficit but with full improvement (long < short) → stagnation = 0. + let long_improving = 0.45_f32; + let aux_w_improving = compute_aux_w_p0b(target, short, long_improving); + + // Stagnation must drag aux_w toward floor: ≥3× drop versus improving. + assert!( + aux_w_improving / aux_w_stagnant >= 3.0, + "stagnation factor 0.3 must decay aux_w by ≥3× (improving={}, stagnant={})", + aux_w_improving, aux_w_stagnant, + ); + + // Concrete value check: 0.5 × 1.25 × 0.3 = 0.1875. + let expected = 0.1875_f32; + assert!( + (aux_w_stagnant - expected).abs() < 1e-5, + "stagnant value should be {}, got {}", + expected, aux_w_stagnant, + ); + } + + /// Improving below target: short pulled ahead of long, no stagnation, but + /// deficit-gain still amplifies. With target=0.55, short=0.50, long=0.45: + /// deficit=0.05, improvement=0.05, stagnation=0.0 + /// aux_w_raw = 0.5 × (1 + 5 × 0.05) × 1.0 = 0.5 × 1.25 = 0.625 + /// Within ceiling 1.5, so result = 0.625. + #[test] + fn aux_w_improving_amplifies_above_base() { + let target = 0.55_f32; + let short = 0.50_f32; + let long = 0.45_f32; + let aux_w = compute_aux_w_p0b(target, short, long); + let expected = 0.625_f32; + assert!( + (aux_w - expected).abs() < 1e-5, + "improving regime should amplify to {}, got {}", + expected, aux_w, + ); + assert!(aux_w > AUX_W_BASE, "must rise above base when improving with deficit"); + } +} diff --git a/crates/ml/src/trainers/dqn/trainer/training_loop.rs b/crates/ml/src/trainers/dqn/trainer/training_loop.rs index 2ea6f882f..6741d0ebe 100644 --- a/crates/ml/src/trainers/dqn/trainer/training_loop.rs +++ b/crates/ml/src/trainers/dqn/trainer/training_loop.rs @@ -93,6 +93,47 @@ fn shannon_entropy_normalized(dist: &[f32]) -> f32 { h / (dist.len() as f32).ln() } +/// SP13 P0b aux_w deficit + stagnation controller (extracted for testability). +/// +/// Replaces the SP11-era inverted formula +/// `aux_w = (0.1 × learning_health × (1 - tanh(0.1 × sharpe))).clamp(0.05, 0.3)` +/// which lowered aux_w when sharpe was high (exact wrong direction for the +/// failure mode where high sharpe + low WR signals structural-extraction trap). +/// +/// Mechanism per spec §"Change 5": +/// ```text +/// deficit = max(0, target - short) +/// improvement = max(0, short - long) +/// stagnation = (deficit > 0.005) ? clamp(1 - improvement/deficit, 0, 1) : 0 +/// aux_w_raw = AUX_W_BASE × (1 + 5 × deficit) × (1 - 0.7 × stagnation) +/// aux_w = clamp(aux_w_raw, [0.3 × base, 3.0 × base]) +/// ``` +/// +/// The 0.005 deficit-floor on the stagnation gate avoids divide-by-near-zero +/// when at/above target — at that point we are satisfied, not stagnant, so +/// stagnation is forced to 0 and the (1 + 5 × deficit) term is also ≈ 1, so +/// `aux_w` collapses cleanly to the base. +pub(crate) fn compute_aux_w_p0b(target: f32, short: f32, long: f32) -> f32 { + use crate::cuda_pipeline::sp13_isv_slots::{ + AUX_W_BASE, AUX_W_DEFICIT_GAIN, AUX_W_STAGNATION_DECAY, + AUX_W_HARD_FLOOR_RATIO, AUX_W_HARD_CEIL_RATIO, + }; + let deficit = (target - short).max(0.0); + let improvement = (short - long).max(0.0); + let stagnation = if deficit > 0.005 { + (1.0 - improvement / deficit).clamp(0.0, 1.0) + } else { + 0.0 + }; + let aux_w_raw = AUX_W_BASE + * (1.0 + AUX_W_DEFICIT_GAIN * deficit) + * (1.0 - AUX_W_STAGNATION_DECAY * stagnation); + aux_w_raw.clamp( + AUX_W_BASE * AUX_W_HARD_FLOOR_RATIO, + AUX_W_BASE * AUX_W_HARD_CEIL_RATIO, + ) +} + /// Metrics returned from `process_epoch_boundary` for downstream logging. pub(crate) struct EpochBoundaryMetrics { pub avg_loss: f32, @@ -4163,25 +4204,37 @@ impl DQNTrainer { } } - // Plan 4 Task 6 Commit B: refresh the aux-loss weight before the - // next training-step graph capture picks it up. - // - // aux_weight = clamp(0.1 × ISV[LEARNING_HEALTH] × (1 - tanh(SHARPE_SCALE × sharpe_ema)), - // 0.05, 0.3) - // - // SHARPE_SCALE = 0.1 — matches Plan 3's controller convention for - // sharpe-driven α adaptation. Numerical-stability bounds [0.05, 0.3] - // per Invariant 1 carve-out. Coefficient `0.1` is the standard - // aux-loss base weight per spec §4.E.6 (only tuned multiplicand). - // The setter is idempotent and does NOT invalidate the captured - // graph — the new value takes effect on the next graph re-capture - // (fold boundary, lr change, etc.). + // SP13 P0b (2026-05-04): aux_w deficit + stagnation controller. + // Replaces SP11-era inverted formula + // aux_w = (0.1 × learning_health × (1 - tanh(0.1 × sharpe_ema))).clamp(0.05, 0.3) + // which LOWERED aux when sharpe was high — exactly wrong for the + // failure mode (high sharpe + low WR is the structural-extraction + // trap; aux head needs MORE gradient when policy is over-fitting + // a noise structure, not less). New mechanism per spec §"Change 5": + // deficit = max(0, target_dir_acc - short_ema) + // improvement = max(0, short_ema - long_ema) + // stagnation = (deficit > 0.005) ? clamp(1 - improvement/deficit, 0, 1) : 0 + // aux_w_raw = base × (1 + 5 × deficit) × (1 - 0.7 × stagnation) + // aux_w = clamp(aux_w_raw, [0.3 × base, 3.0 × base]) + // Stagnation decay backs aux_w to floor when the short EMA fails + // to pull ahead of the long EMA — "we tried, it didn't help, stop + // destabilising Q." Hard floor 0.3× base prevents permanent + // suppression (the SP11 inversion bug). The setter is idempotent + // and does NOT invalidate the captured graph — new value takes + // effect on the next re-capture (fold boundary, lr change, etc.). if let Some(ref mut fused) = self.fused_ctx { - use crate::cuda_pipeline::gpu_dqn_trainer::LEARNING_HEALTH_INDEX; - let learning_health = fused.read_isv_signal_at(LEARNING_HEALTH_INDEX); - const SHARPE_SCALE: f32 = 0.1; - let sharpe_tanh = (SHARPE_SCALE * self.training_sharpe_ema).tanh(); - let aux_w = (0.1_f32 * learning_health * (1.0 - sharpe_tanh)).clamp(0.05, 0.3); + use crate::cuda_pipeline::sp13_isv_slots::{ + TARGET_DIR_ACC_INDEX, AUX_DIR_ACC_SHORT_EMA_INDEX, + AUX_DIR_ACC_LONG_EMA_INDEX, + }; + let target = fused.read_isv_signal_at(TARGET_DIR_ACC_INDEX); + let short_ema = fused.read_isv_signal_at(AUX_DIR_ACC_SHORT_EMA_INDEX); + let long_ema = fused.read_isv_signal_at(AUX_DIR_ACC_LONG_EMA_INDEX); + let aux_w = compute_aux_w_p0b(target, short_ema, long_ema); + tracing::debug!( + "aux_w controller: target={:.3} short={:.3} long={:.3} aux_w={:.3}", + target, short_ema, long_ema, aux_w, + ); fused.set_aux_weight(aux_w); } diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index 3f0614b6d..c1eec8c0f 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -5801,3 +5801,63 @@ Per-bar Hold cost is per-bar reward shaping, which `pearl_event_driven_reward_de ### Files (atomic P0a commit) 15 files / 2316 lines added: 5 new kernels + 9 modifications + 1 mod.rs wire-up. Per `feedback_no_partial_refactor`: all consumers of the new ISV slots wired in this single commit. + +--- + +## SP13 P0b — aux_w deficit+stagnation controller + Hold cost lift (2026-05-04) + +**Spec**: `docs/superpowers/specs/2026-05-04-sp13-redefine-success-for-predictive-skill.md` v3 §"Change 5" +**Plan**: `docs/superpowers/plans/2026-05-04-sp13-redefine-success-for-predictive-skill.md` v3 §"Phase 0b" +**Trigger**: P0a smoke `train-67gqb` on `f934ea171` returned PARTIAL — `aux_dir_acc` climbed 0.149 → 0.483 over 10 epochs (signal exists), but `val_win_rate` stuck at 0.4638 (Q-head not consuming aux signal) and `observed_hold_rate` climbed to 0.479 (Hold cost too weak; controller saturated at 2.4× base, model rationally absorbed it). + +### Decision A — corrected aux_w controller (replaces SP11 inverted formula) + +`training_loop.rs` SP11-era site (post Plan 4 Task 6 Commit B): +```rust +// REPLACED: +let aux_w = (0.1 × learning_health × (1 - tanh(0.1 × sharpe))).clamp(0.05, 0.3); +``` +The SP11 formula LOWERED aux_w when sharpe was high — exactly wrong for the failure mode where high sharpe + low WR signals the structural-extraction trap (aux head needs MORE gradient when policy is over-fitting noise structure, not less). + +Replaced with deficit+stagnation controller per spec §"Change 5": +```rust +deficit = max(0, target_dir_acc - short_ema) +improvement = max(0, short_ema - long_ema) +stagnation = (deficit > 0.005) ? clamp(1 - improvement/deficit, 0, 1) : 0 +aux_w_raw = AUX_W_BASE × (1 + AUX_W_DEFICIT_GAIN × deficit) × (1 - AUX_W_STAGNATION_DECAY × stagnation) +aux_w = clamp(aux_w_raw, [AUX_W_BASE × 0.3, AUX_W_BASE × 3.0]) +``` +Reads `target_dir_acc` from ISV[372], `short_ema` from ISV[373], `long_ema` from ISV[374] (all P0a slots). `AUX_W_BASE = 0.5` is a 10× lift from the SP11 0.05 floor — the lift is the fix, not a confounder, because the SP11 formula was structurally wrong (hard floor was acting as the dominant suppression mechanism). Stagnation decay prevents permanent destabilisation in the data-limited case ("we tried, it didn't help, stop destabilising Q"). + +Formula extracted as a top-level `pub(crate) fn compute_aux_w_p0b(target, short, long) -> f32` helper (`training_loop.rs`) for unit-test coverage. The callsite reads ISV slots and invokes the helper; setter `fused.set_aux_weight()` is unchanged (idempotent, no graph re-capture invalidation). + +### Decision B — `HOLD_COST_BASE` lift 0.001 → 0.005 + +The P0a Hold-pricing controller wired the per-bar Hold cost in `experience_kernels.cu` reading from ISV[380]. The constant lift is a single-line change in `sp13_isv_slots.rs` — the controller scales `1 + 5 × max(0, observed - target)` clamped to `[0.5×, 5.0×]`, so with `BASE = 0.001` the maximum cost is 0.005/bar × 30-bar hold = 0.15 cumulative vs ±5-10 reward range = <3% of magnitude. Model rationally absorbed the cost. Lift to `BASE = 0.005`: max 0.025/bar × 30 bars = 0.75 cumulative ≈ 10–15% of capped reward, genuinely deters lazy Hold without crippling MFT use. Constructor static-init unchanged: it still writes the (now-lifted) `HOLD_COST_BASE` constant to slot 380 at fold boundary; controller now scales the lifted base. + +### Constants added (`sp13_isv_slots.rs`) + +5 new aux_w controller anchors per `feedback_isv_for_adaptive_bounds.md` (numerical anchors only; bounds derived from ISV signals): + +| Constant | Value | Role | +|---|---|---| +| `AUX_W_BASE` | 0.5 | aux_w at target (steady-state). 10× SP11 floor. | +| `AUX_W_DEFICIT_GAIN` | 5.0 | `aux_w = base × (1 + 5 × deficit)` — deficit-driven amplification | +| `AUX_W_STAGNATION_DECAY` | 0.7 | `(1 - 0.7 × stagnation)` — pulls back when not improving | +| `AUX_W_HARD_FLOOR_RATIO` | 0.3 | Floor = base × 0.3 = 0.15 — prevents complete suppression | +| `AUX_W_HARD_CEIL_RATIO` | 3.0 | Ceil = base × 3.0 = 1.5 — prevents Q destabilisation | + +`HOLD_COST_BASE` lifted in-place; no slot index changes; constructor write site unchanged. + +### Test coverage + +3 new host-side unit tests in `crates/ml/src/trainers/dqn/trainer/tests.rs::aux_w_controller_tests`: +- `aux_w_at_target_returns_base` — short = target → aux_w = AUX_W_BASE +- `aux_w_stagnation_decays_to_floor` — short stuck below target with no improvement → aux_w drops ≥3× vs improving regime; respects floor +- `aux_w_improving_amplifies_above_base` — short pulled ahead of long → aux_w = 0.625 (above base, deficit-amplified) + +14 SP13 GPU oracle tests + 14 SP12 reward-math tests still green; no kernel changes (P0b is a constants + host-side controller change only). + +### Files (atomic P0b commit) + +3 files: `sp13_isv_slots.rs` (constants), `training_loop.rs` (controller + helper extraction), `tests.rs` (3 unit tests). No new kernels, no new ISV slots, no fingerprint change.