From a225926e5f71dcae1de5545ee0cf7df7ba768562 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Fri, 8 May 2026 20:35:29 +0200 Subject: [PATCH] plan(sp17): allocate ISV slots [474..483) for dueling-Q diagnostics MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 9 slots: A_var_ema × 4 branches (per-branch advantage variance EMA) + V_share × 4 branches (V/(V+A) magnitude share) + adaptive advantage_clip_bound. All Pearl-A first-observation bootstrap with fold-reset registry entries + dispatch arms in trainer/training_loop.rs::reset_named_state. Bump ISV_TOTAL_DIM 474 → 483 + layout fingerprint seed. Per `feedback_isv_for_adaptive_bounds`: advantage_clip_bound is producer-tracked from p99(|A_centered|) × 1.5 safety factor, never hardcoded. Bounds [0.1, 100.0] are Category-1 dimensional safety floors. No consumer change in this commit (additive infrastructure). Phase 1 mean-centering producer (atomic across compute_expected_q + c51_loss + c51_grad + mag_concat_qdir) lands in the next 4 commits. Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md Co-Authored-By: Claude Opus 4.7 (1M context) --- .../ml/src/cuda_pipeline/gpu_dqn_trainer.rs | 13 ++- crates/ml/src/cuda_pipeline/sp14_isv_slots.rs | 83 +++++++++++++++++++ .../src/trainers/dqn/state_reset_registry.rs | 56 +++++++++++++ .../src/trainers/dqn/trainer/training_loop.rs | 81 ++++++++++++++++++ docs/dqn-wire-up-audit.md | 46 ++++++++++ 5 files changed, 277 insertions(+), 2 deletions(-) diff --git a/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs b/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs index 8be3db537..d2ad2e11b 100644 --- a/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs +++ b/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs @@ -2520,7 +2520,7 @@ const ISV_NETWORK_DIM: usize = 23; /// (shifted 112→116 in Plan 4 Task 6 Commit A). /// Written by the constructor; checked at checkpoint load. Fail-fast only — no migration /// path exists. See spec §4.A.2 and `LAYOUT_FINGERPRINT_CURRENT` for structural-hash rationale. -pub(crate) const ISV_TOTAL_DIM: usize = 474; // SP5 + Layer D D1+D2+D3 + SP7 + SP8 + SP9 + SP10 + SP11 + SP13 + SP13 v3 + SP14 (post-C.1) + SP15 + SP14-C aux trunk control plane + SP14-C Phase C.4b aux horizon + Class A P0-A adaptive reward caps + Class A P1-Producer adaptive Bayesian Kelly priors + Class A audit-fix Batch 4-A adaptive DD saturation floor + Class A audit-fix Batch 4-B adaptive plan_threshold floor + adaptive MIN_HOLD_TEMPERATURE + SP16 Phase 2 adaptive Hold cost scale + SP16 T3 Wiener-optimal adaptive α Welford accumulators. Bumped 462 → 474 by SP16 T3 (2026-05-08): added 12 slots [462..474) — 6 Welford accumulators per producer kernel (T1 MIN_HOLD_TEMPERATURE + T2 HOLD_COST_SCALE). Replaces the hardcoded `alpha = 0.05f` EMA constant in both producer kernels (`min_hold_temperature_update_kernel.cu` + `hold_cost_scale_update_kernel.cu`) with a signal-driven Wiener-optimal α per `pearl_wiener_optimal_adaptive_alpha`: α = diff_var / (diff_var + sample_var + ε). Per train-multi-seed-hjzss validation, the SP16 T1+T2 chain was structurally landed but BEHAVIORALLY INERT in 5-epoch smoke (bit-identical to pfh9n baseline through epoch 3) because α=0.05 needs ~60 epochs from cold start to converge. Wiener-optimal α naturally falls in [0, 1] (clamped to [0.01, 0.95] for numerical safety): cold-start (signal jumps 1.0 → 6.4 → 7.0) → diff_var dominates → α ≈ 0.6+ → near-bootstrap responsiveness in epochs 1-3; steady-state → diff_var drops → α decays naturally → smoothing emerges without hardcoded constant. Each producer maintains 6 Welford accumulators contiguously: [HCS_TARGET_MEAN, HCS_TARGET_M2, HCS_DIFF_MEAN, HCS_DIFF_M2, HCS_PREV_TARGET, HCS_SAMPLE_COUNT] at [462..468) and [MHT_TARGET_MEAN, MHT_TARGET_M2, MHT_DIFF_MEAN, MHT_DIFF_M2, MHT_PREV_TARGET, MHT_SAMPLE_COUNT] at [468..474). FoldReset sentinel 0.0 — combined with the cold-start fallback (sample_count < 3 → α = 1.0 REPLACE), the first observation of each fold replaces blended directly, the second seeds diff statistics, the third stabilises Wiener α. Per `feedback_isv_for_adaptive_bounds.md` — α was an adaptive bound being hardcoded; lifting to ISV-driven Wiener-optimal closes that violation. Bumped 461 → 462 by SP16 Phase 2 (2026-05-08): added 1 slot [461..462) — HOLD_COST_SCALE_INDEX (461). Replaces the static `ISV[HOLD_COST_INDEX=380]` per-bar Hold-cost magnitude (~0.006) when over-holding is observed. Producer kernel `hold_cost_scale_update_kernel` reads `ISV[HOLD_RATE_OBSERVED_EMA_INDEX=382]` and `ISV[HOLD_RATE_TARGET_INDEX=381]` (same SP13 hold-pricing chain feeding SP16-P1's MIN_HOLD_TEMPERATURE — same cadence, same input pattern), maps overrun into a [1, 25] multiplier via `target_scale = SCALE_MIN + (SCALE_MAX − SCALE_MIN) × clamp(max(0, observed - target) / max(target, 0.01), 0, 1)`, and Pearl-A-bootstrap + α=0.05 mid-cadence EMA blends into slot 461. Consumer: `experience_kernels.cu` per-bar Hold cost subtractions (3 sites: segment_complete branch line 3089, per-bar positioned-Hold branch line 3553, per-bar flat-Hold branch line 3617) multiply `isv[ISV_HOLD_COST_IDX=380]` by `isv[HOLD_COST_SCALE_INDEX=461]`. Cold-start fallback: when slot 461 is at sentinel 0.0 OR outside [1, 25], the consumer uses scale=1.0 (bit-identical to pre-Phase-2 v3-P0a Hold-cost magnitude). At observed=2× target → scale=25× → effective cost ~0.15/bar competitive with per-bar reward magnitudes ~0.01-0.1 but well below SP12 asymmetric cap [-10, +5]. Bounds [1, 25] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds.md`. Bumped 459 → 461 by Class A audit-fix Batch 4-B (2026-05-08): added 2 slots [459..461) — PLAN_THRESHOLD_FLOOR_ADAPTIVE (459) + MIN_HOLD_TEMPERATURE_ADAPTIVE (460). PLAN_THRESHOLD_FLOOR_ADAPTIVE replaces the hardcoded `0.1f` floor in `plan_threshold_update_kernel.cu:44` (`fmaxf(0.1f, 0.5f * ema)`) — Design Y inline producer (no new kernel; same kernel writes both slot 49 and the slow-EMA-shadow floor at slot 459 in the same launch, then uses the freshly-written floor as the lower bound). Pearl-A bootstrap (sentinel 0.1 matches pre-fix hardcoded value for bit-identical cold-start) + α=0.005 slow EMA (per-fold cadence — the floor is a long-term shadow of `0.5 × readiness_ema` and must move much slower than the readiness EMA itself). Bounds [0.05, 0.50] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds` (probability units). MIN_HOLD_TEMPERATURE_ADAPTIVE replaces the deleted epoch-driven anneal schedule `T_end + (T_start − T_end) × exp(-epoch / decay)` (formerly state_layout.cuh::MIN_HOLD_TEMPERATURE_{START=50, END=5, DECAY=20} + training_loop.rs::min_hold_temperature_for_epoch). Producer kernel `min_hold_temperature_update_kernel` reads `ISV[AUX_DIR_ACC_SHORT_EMA_INDEX=373]` (sp13 fast EMA of binary directional accuracy), maps to a [5, 50] temperature via `temp = TEMP_MIN + (TEMP_MAX − TEMP_MIN) × confusion` where `confusion = 1 − clamp((short_ema − 0.5)/0.5, 0, 1)` (so dir_acc ≈ 0.5 plateau → temp HIGH/permissive exit ramp; dir_acc → 1.0 → temp LOW/strict informed-commitment), and slow-EMA-blends into slot 460. HIGH T = forgiving (`deficit/(deficit+T)` saturation factor → 0); LOW T = sharp (→ 1) per `compute_min_hold_penalty` at trade_physics.cuh:575. Pearl-A bootstrap (sentinel 50.0 matches the deleted MIN_HOLD_TEMPERATURE_START=50 anchor for bit-identical cold-start under both regimes) + α=0.05 mid-cadence EMA. Bounds [5, 50] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds` (matches the original schedule range). Decouples temperature from epoch number — when WR plateaus, the old schedule pinned LOW temp (sharp) at end of training (max punishment exactly when most needed forgiveness); the signal-driven version reacts to actual realized skill. Per `feedback_no_legacy_aliases.md` the legacy schedule constants + helper function are deleted atomically. Bumped 458 → 459 by Class A audit-fix Batch 4-A (2026-05-08): added 1 adaptive-DD-saturation-floor slot [458..459) — DD_SATURATION_FLOOR_ADAPTIVE (458). Replaces the hardcoded `0.25f` saturation floor in `trade_physics.cuh::apply_margin_cap` (the upper end of the linear position-size scaling ramp dd_scale = max(0.05, 1.0 − dd_frac / saturation_floor)). Distinct semantic role from `SP15_DD_THRESHOLD_INDEX=421` (the SP15 quadratic DD-penalty *trigger* threshold, a *lower* bound). Producer kernel `dd_saturation_floor_update_kernel` aggregates per-env DD_MAX from the existing `dd_state_per_env[env*6 + 1]` tile (same source as the SP15 dd_state_kernel writes to), computes p75 estimator via Welford rule (mean + Z_75 × sigma) × safety_factor=1.5, and slow-EMA blends. Pearl-A bootstrap (sentinel 0.25 matches pre-fix hardcoded value for bit-identical cold-start) + α=0.01 slow EMA (DD distribution is a slow-moving fold-volatility property, shouldn't track per-epoch noise). Bounds [0.10, 0.50] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds`. Atomically deleted alongside this slot addition: legacy `compute_drawdown_penalty` device function in trade_physics.cuh + its single caller at experience_kernels.cu:3822 + the `w_dd` and `dd_threshold` kernel args (Item 2 Case A — SP15's quadratic asymmetric DD penalty in `compute_sp15_final_reward_kernel.cu:154` is the production-grade replacement; layering both creates double-counting per `feedback_no_partial_refactor`). Bumped 454 → 458 by Class A P1-Producer (2026-05-08): added 4 adaptive-Bayesian-prior slots [454..458) — KELLY_PRIOR_WINS (454) + KELLY_PRIOR_LOSSES (455) + KELLY_PRIOR_SUM_WINS (456) + KELLY_PRIOR_SUM_LOSSES (457). Per Class A audit, hardcoded `prior_wins=2.0f / prior_losses=2.0f / prior_sum_wins=0.01f / prior_sum_losses=0.01f` (kelly_cap_update_kernel.cu:39-42 + trade_physics.cuh::kelly_position_cap:304-307) was a static "I don't know" Bayesian prior frozen at sprint-1 defaults — the cold-start of every new fold ignored what prior folds had learned about the realized Kelly distribution. Producer kernel `kelly_bayesian_priors_update_kernel` aggregates realized Kelly stats from the existing `portfolio_state[n_envs, PS_STRIDE]` buffer (same source as kelly_cap_update consumes) and slow-EMA blends into the four slots. Pearl-A bootstrap (sentinels 2.0/2.0/0.01/0.01 match pre-P1-Producer hardcoded values for bit-identical cold-start) + α=0.005 slow EMA (per-fold cadence; the Bayesian prior is a *prior belief* and should change slowly across folds). Bounds counts∈[0.5, 100], sums∈[0.001, 1.0] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds`. Bumped 452 → 454 by Class A P0-A (2026-05-08): added 2 adaptive-reward-cap slots [452..454) — REWARD_POS_CAP_ADAPTIVE (452) + REWARD_NEG_CAP_ADAPTIVE (453). Per Class A audit, hardcoded `REWARD_POS_CAP=+5.0f` / `REWARD_NEG_CAP=-10.0f` (state_layout.cuh:266-267) was structurally clipping the upper tail of realized alpha — controller signals (sharpe EMA, var_q, q_gap) all derive from the CAPPED buffer, so the controller cannot select for trades it cannot see. The state_layout comment explicitly deferred Phase 2 (ISV-driven) "IF Phase 1 validation reveals adaptive need"; 11 superprojects of WR-stuck-at-46-48% revealed adaptive need. Producer kernel `reward_cap_update_kernel` computes p99(winning_returns) × safety_factor=1.5 → POS cap and NEG = -2 × POS (preserves Kahneman 2:1 loss-aversion asymmetry per `pearl_audit_unboundedness_for_implicit_asymmetry` — moved from hardcoded scalar to producer-time multiplier, single source of truth). Pearl-A bootstrap (sentinels 5.0/-10.0) + Wiener-α=0.01 slow EMA. Bounds POS∈[1, 50], NEG∈[-100, -2] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds`, NOT tuning. Bumped 450 → 452 by SP14 Layer C Phase C.4b (2026-05-08): added 2 aux-prediction-horizon slots [450..452) — AUX_PRED_HORIZON_BARS (450) + AVG_WIN_HOLD_TIME_BARS (451). Aux's original label was (p_{t+1} > p_t), pure HFT-scale microstructure noise — pivoted to (p_{t+H} > p_t) with H read from ISV[450], adaptively driven from observed avg winning hold time via Pearl-A bootstrap + Wiener-α EMA. Case B: existing `hold_at_exit_per_sample` + `trade_profitable_per_sample` per-sample buffers, no aggregate slot — added new aggregate slot 451 to expose the EMA target. Bumped 444 → 450 by SP14 Layer C Phase C.1 (2026-05-08): atomically deleted 9 α-machinery slots in [385..396) AND added 6 aux-trunk control-plane slots [444..450) — AUX_TRUNK_LR (444) + AUX_TRUNK_BETA1 (445) + AUX_TRUNK_BETA2 (446) + AUX_TRUNK_EPS (447) + AUX_TRUNK_GRAD_CLIP (448) + H_S2_AUX_RMS_EMA (449). Per `feedback_no_legacy_aliases` and `feedback_no_partial_refactor`. Deleted slots left as RESERVED gap (NOT compacted) to preserve checkpoint layout fingerprint compatibility — the surviving SP14 q_disagreement_* diagnostic slots (383, 384, 389) keep their original indices. Pre-C.1 base: 173 + 210 (164 SP5 slots @ 174..278/280..340, with 2-slot gap before cross-fold-persistent Kelly block; Layer D D1 PnL outputs at [286..290); Layer D D2 health composition outputs at [290..294); Layer D D3 training metrics EMA at [294..297); SP7 T1 loss-balance Wiener stats at [297..313); SP7 activation-flag fix per-(head,branch) flags at [313..321); SP8 Fix 36 train_active_frac canary @ [321..322) + LB_MAX_BUDGET per-(head,branch) at [322..330); SP9 Fix 37 Kelly warmup floor at [330..331) + Q_VAR_MAG_EMA at [331..332) + INTENT_EVAL_DIVERGENCE at [332..333) + 3 EMA targets at [333..336) + 3 eval_dist mag bins at [336..339); SP10 Fix 38 EVAL_THOMPSON_TEMP at [339..340); SP11 Fix 39 reward-subsystem controller at [340..367) — 6 component weights [340..346) + 4 controller scalar outputs [346..350) + 2 val-sharpe canaries [350..352) + 6 mag-ratio canaries [352..358) + saboteur engagement [358] + PnL magnitude EMA [359] + popart-component magnitude EMA [360, B1b fix-up] + per-component variance EMAs [361..367, B1b smoke-recovery z-score normalization]; SP13 directional-skill instrumentation at [372..380) — TARGET_DIR_ACC [372] + AUX_DIR_ACC_SHORT/LONG_EMA [373..375) + AUX_DIR_PREDICTION [375] + DIR_SKILL_BONUS_ALPHA/BETA [376..378) + LUCK_WIN_DISCOUNT [378] + SKILL_BONUS_CAP_RATIO [379]; SP13 v3 P0a.T3 Hold-pricing controller at [380..383) — HOLD_COST [380] + HOLD_RATE_TARGET [381] + HOLD_RATE_OBSERVED_EMA [382]; SP14 q_disagreement diagnostic at [383..385), [389] — Q_DISAGREEMENT_SHORT/LONG_EMA + Q_DISAGREEMENT_VARIANCE_EMA (slots [385..389) and [390..396) RESERVED gap from C.1 deletion); SP15 [397..444) per Phase 1; SP14-C [444..450) per Phase C.1. Intentional 5-slot boundary gap at [367..372)) +pub(crate) const ISV_TOTAL_DIM: usize = 483; // SP5 + Layer D D1+D2+D3 + SP7 + SP8 + SP9 + SP10 + SP11 + SP13 + SP13 v3 + SP14 (post-C.1) + SP15 + SP14-C aux trunk control plane + SP14-C Phase C.4b aux horizon + Class A P0-A adaptive reward caps + Class A P1-Producer adaptive Bayesian Kelly priors + Class A audit-fix Batch 4-A adaptive DD saturation floor + Class A audit-fix Batch 4-B adaptive plan_threshold floor + adaptive MIN_HOLD_TEMPERATURE + SP16 Phase 2 adaptive Hold cost scale + SP16 T3 Wiener-optimal adaptive α Welford accumulators + SP17 dueling-Q identifiability diagnostics. Bumped 474 → 483 by SP17 (2026-05-08): added 9 slots [474..483) for the dueling-Q identifiability diagnostic chain. With mean-zero advantage projection `A_centered[a, z] = A[a, z] − mean_a A[a, z]` (Phase 1-3 of SP17), V and A become uniquely identifiable: V_share ≈ 0.5 + Var(A_centered) > 0 indicates a healthy dueling decomposition; V_share → 1.0 + Var(A_centered) → 0 signals regression to the pre-SP17 V-dominated regime that the optimizer was free to learn under naive `Q = V + A`. Slots: A_VAR_EMA × 4 branches at [474..478) — per-branch EMA of the over-actions variance of the centered advantage; V_SHARE × 4 branches at [478..482) — per-branch |V| / (|V| + |A_centered|) magnitude share; ADVANTAGE_CLIP_BOUND at [482] — adaptive |A_centered| upper bound producer-tracked from p99(|A_centered|) × safety_factor=1.5 (mirrors SP14 P0-A REWARD_POS_CAP producer pattern; α=0.01 slow per-fold cadence). All Pearl-A first-observation bootstraps with sentinels SENTINEL_A_VAR_EMA=0.0, SENTINEL_V_SHARE=0.5 (cold-start 50/50 split), SENTINEL_ADVANTAGE_CLIP_BOUND=1.0 (cold-start "no effective clipping"). Bounds [ADVANTAGE_CLIP_BOUND_MIN=0.1, ADVANTAGE_CLIP_BOUND_MAX=100.0] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds`: floor 0.1 below which centered advantage is essentially noise (clipping would erase the action signal); ceiling 100.0 above which a single adversarial outlier dominates per-batch EMAs. This commit is additive infrastructure — no consumer change yet; the Phase 1 mean-centering producer (atomic across `compute_expected_q` + `c51_loss` + `c51_grad` + `mag_concat_qdir`) lands in the next 4 commits per `feedback_no_partial_refactor`. Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md. Bumped 462 → 474 by SP16 T3 (2026-05-08): added 12 slots [462..474) — 6 Welford accumulators per producer kernel (T1 MIN_HOLD_TEMPERATURE + T2 HOLD_COST_SCALE). Replaces the hardcoded `alpha = 0.05f` EMA constant in both producer kernels (`min_hold_temperature_update_kernel.cu` + `hold_cost_scale_update_kernel.cu`) with a signal-driven Wiener-optimal α per `pearl_wiener_optimal_adaptive_alpha`: α = diff_var / (diff_var + sample_var + ε). Per train-multi-seed-hjzss validation, the SP16 T1+T2 chain was structurally landed but BEHAVIORALLY INERT in 5-epoch smoke (bit-identical to pfh9n baseline through epoch 3) because α=0.05 needs ~60 epochs from cold start to converge. Wiener-optimal α naturally falls in [0, 1] (clamped to [0.01, 0.95] for numerical safety): cold-start (signal jumps 1.0 → 6.4 → 7.0) → diff_var dominates → α ≈ 0.6+ → near-bootstrap responsiveness in epochs 1-3; steady-state → diff_var drops → α decays naturally → smoothing emerges without hardcoded constant. Each producer maintains 6 Welford accumulators contiguously: [HCS_TARGET_MEAN, HCS_TARGET_M2, HCS_DIFF_MEAN, HCS_DIFF_M2, HCS_PREV_TARGET, HCS_SAMPLE_COUNT] at [462..468) and [MHT_TARGET_MEAN, MHT_TARGET_M2, MHT_DIFF_MEAN, MHT_DIFF_M2, MHT_PREV_TARGET, MHT_SAMPLE_COUNT] at [468..474). FoldReset sentinel 0.0 — combined with the cold-start fallback (sample_count < 3 → α = 1.0 REPLACE), the first observation of each fold replaces blended directly, the second seeds diff statistics, the third stabilises Wiener α. Per `feedback_isv_for_adaptive_bounds.md` — α was an adaptive bound being hardcoded; lifting to ISV-driven Wiener-optimal closes that violation. Bumped 461 → 462 by SP16 Phase 2 (2026-05-08): added 1 slot [461..462) — HOLD_COST_SCALE_INDEX (461). Replaces the static `ISV[HOLD_COST_INDEX=380]` per-bar Hold-cost magnitude (~0.006) when over-holding is observed. Producer kernel `hold_cost_scale_update_kernel` reads `ISV[HOLD_RATE_OBSERVED_EMA_INDEX=382]` and `ISV[HOLD_RATE_TARGET_INDEX=381]` (same SP13 hold-pricing chain feeding SP16-P1's MIN_HOLD_TEMPERATURE — same cadence, same input pattern), maps overrun into a [1, 25] multiplier via `target_scale = SCALE_MIN + (SCALE_MAX − SCALE_MIN) × clamp(max(0, observed - target) / max(target, 0.01), 0, 1)`, and Pearl-A-bootstrap + α=0.05 mid-cadence EMA blends into slot 461. Consumer: `experience_kernels.cu` per-bar Hold cost subtractions (3 sites: segment_complete branch line 3089, per-bar positioned-Hold branch line 3553, per-bar flat-Hold branch line 3617) multiply `isv[ISV_HOLD_COST_IDX=380]` by `isv[HOLD_COST_SCALE_INDEX=461]`. Cold-start fallback: when slot 461 is at sentinel 0.0 OR outside [1, 25], the consumer uses scale=1.0 (bit-identical to pre-Phase-2 v3-P0a Hold-cost magnitude). At observed=2× target → scale=25× → effective cost ~0.15/bar competitive with per-bar reward magnitudes ~0.01-0.1 but well below SP12 asymmetric cap [-10, +5]. Bounds [1, 25] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds.md`. Bumped 459 → 461 by Class A audit-fix Batch 4-B (2026-05-08): added 2 slots [459..461) — PLAN_THRESHOLD_FLOOR_ADAPTIVE (459) + MIN_HOLD_TEMPERATURE_ADAPTIVE (460). PLAN_THRESHOLD_FLOOR_ADAPTIVE replaces the hardcoded `0.1f` floor in `plan_threshold_update_kernel.cu:44` (`fmaxf(0.1f, 0.5f * ema)`) — Design Y inline producer (no new kernel; same kernel writes both slot 49 and the slow-EMA-shadow floor at slot 459 in the same launch, then uses the freshly-written floor as the lower bound). Pearl-A bootstrap (sentinel 0.1 matches pre-fix hardcoded value for bit-identical cold-start) + α=0.005 slow EMA (per-fold cadence — the floor is a long-term shadow of `0.5 × readiness_ema` and must move much slower than the readiness EMA itself). Bounds [0.05, 0.50] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds` (probability units). MIN_HOLD_TEMPERATURE_ADAPTIVE replaces the deleted epoch-driven anneal schedule `T_end + (T_start − T_end) × exp(-epoch / decay)` (formerly state_layout.cuh::MIN_HOLD_TEMPERATURE_{START=50, END=5, DECAY=20} + training_loop.rs::min_hold_temperature_for_epoch). Producer kernel `min_hold_temperature_update_kernel` reads `ISV[AUX_DIR_ACC_SHORT_EMA_INDEX=373]` (sp13 fast EMA of binary directional accuracy), maps to a [5, 50] temperature via `temp = TEMP_MIN + (TEMP_MAX − TEMP_MIN) × confusion` where `confusion = 1 − clamp((short_ema − 0.5)/0.5, 0, 1)` (so dir_acc ≈ 0.5 plateau → temp HIGH/permissive exit ramp; dir_acc → 1.0 → temp LOW/strict informed-commitment), and slow-EMA-blends into slot 460. HIGH T = forgiving (`deficit/(deficit+T)` saturation factor → 0); LOW T = sharp (→ 1) per `compute_min_hold_penalty` at trade_physics.cuh:575. Pearl-A bootstrap (sentinel 50.0 matches the deleted MIN_HOLD_TEMPERATURE_START=50 anchor for bit-identical cold-start under both regimes) + α=0.05 mid-cadence EMA. Bounds [5, 50] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds` (matches the original schedule range). Decouples temperature from epoch number — when WR plateaus, the old schedule pinned LOW temp (sharp) at end of training (max punishment exactly when most needed forgiveness); the signal-driven version reacts to actual realized skill. Per `feedback_no_legacy_aliases.md` the legacy schedule constants + helper function are deleted atomically. Bumped 458 → 459 by Class A audit-fix Batch 4-A (2026-05-08): added 1 adaptive-DD-saturation-floor slot [458..459) — DD_SATURATION_FLOOR_ADAPTIVE (458). Replaces the hardcoded `0.25f` saturation floor in `trade_physics.cuh::apply_margin_cap` (the upper end of the linear position-size scaling ramp dd_scale = max(0.05, 1.0 − dd_frac / saturation_floor)). Distinct semantic role from `SP15_DD_THRESHOLD_INDEX=421` (the SP15 quadratic DD-penalty *trigger* threshold, a *lower* bound). Producer kernel `dd_saturation_floor_update_kernel` aggregates per-env DD_MAX from the existing `dd_state_per_env[env*6 + 1]` tile (same source as the SP15 dd_state_kernel writes to), computes p75 estimator via Welford rule (mean + Z_75 × sigma) × safety_factor=1.5, and slow-EMA blends. Pearl-A bootstrap (sentinel 0.25 matches pre-fix hardcoded value for bit-identical cold-start) + α=0.01 slow EMA (DD distribution is a slow-moving fold-volatility property, shouldn't track per-epoch noise). Bounds [0.10, 0.50] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds`. Atomically deleted alongside this slot addition: legacy `compute_drawdown_penalty` device function in trade_physics.cuh + its single caller at experience_kernels.cu:3822 + the `w_dd` and `dd_threshold` kernel args (Item 2 Case A — SP15's quadratic asymmetric DD penalty in `compute_sp15_final_reward_kernel.cu:154` is the production-grade replacement; layering both creates double-counting per `feedback_no_partial_refactor`). Bumped 454 → 458 by Class A P1-Producer (2026-05-08): added 4 adaptive-Bayesian-prior slots [454..458) — KELLY_PRIOR_WINS (454) + KELLY_PRIOR_LOSSES (455) + KELLY_PRIOR_SUM_WINS (456) + KELLY_PRIOR_SUM_LOSSES (457). Per Class A audit, hardcoded `prior_wins=2.0f / prior_losses=2.0f / prior_sum_wins=0.01f / prior_sum_losses=0.01f` (kelly_cap_update_kernel.cu:39-42 + trade_physics.cuh::kelly_position_cap:304-307) was a static "I don't know" Bayesian prior frozen at sprint-1 defaults — the cold-start of every new fold ignored what prior folds had learned about the realized Kelly distribution. Producer kernel `kelly_bayesian_priors_update_kernel` aggregates realized Kelly stats from the existing `portfolio_state[n_envs, PS_STRIDE]` buffer (same source as kelly_cap_update consumes) and slow-EMA blends into the four slots. Pearl-A bootstrap (sentinels 2.0/2.0/0.01/0.01 match pre-P1-Producer hardcoded values for bit-identical cold-start) + α=0.005 slow EMA (per-fold cadence; the Bayesian prior is a *prior belief* and should change slowly across folds). Bounds counts∈[0.5, 100], sums∈[0.001, 1.0] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds`. Bumped 452 → 454 by Class A P0-A (2026-05-08): added 2 adaptive-reward-cap slots [452..454) — REWARD_POS_CAP_ADAPTIVE (452) + REWARD_NEG_CAP_ADAPTIVE (453). Per Class A audit, hardcoded `REWARD_POS_CAP=+5.0f` / `REWARD_NEG_CAP=-10.0f` (state_layout.cuh:266-267) was structurally clipping the upper tail of realized alpha — controller signals (sharpe EMA, var_q, q_gap) all derive from the CAPPED buffer, so the controller cannot select for trades it cannot see. The state_layout comment explicitly deferred Phase 2 (ISV-driven) "IF Phase 1 validation reveals adaptive need"; 11 superprojects of WR-stuck-at-46-48% revealed adaptive need. Producer kernel `reward_cap_update_kernel` computes p99(winning_returns) × safety_factor=1.5 → POS cap and NEG = -2 × POS (preserves Kahneman 2:1 loss-aversion asymmetry per `pearl_audit_unboundedness_for_implicit_asymmetry` — moved from hardcoded scalar to producer-time multiplier, single source of truth). Pearl-A bootstrap (sentinels 5.0/-10.0) + Wiener-α=0.01 slow EMA. Bounds POS∈[1, 50], NEG∈[-100, -2] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds`, NOT tuning. Bumped 450 → 452 by SP14 Layer C Phase C.4b (2026-05-08): added 2 aux-prediction-horizon slots [450..452) — AUX_PRED_HORIZON_BARS (450) + AVG_WIN_HOLD_TIME_BARS (451). Aux's original label was (p_{t+1} > p_t), pure HFT-scale microstructure noise — pivoted to (p_{t+H} > p_t) with H read from ISV[450], adaptively driven from observed avg winning hold time via Pearl-A bootstrap + Wiener-α EMA. Case B: existing `hold_at_exit_per_sample` + `trade_profitable_per_sample` per-sample buffers, no aggregate slot — added new aggregate slot 451 to expose the EMA target. Bumped 444 → 450 by SP14 Layer C Phase C.1 (2026-05-08): atomically deleted 9 α-machinery slots in [385..396) AND added 6 aux-trunk control-plane slots [444..450) — AUX_TRUNK_LR (444) + AUX_TRUNK_BETA1 (445) + AUX_TRUNK_BETA2 (446) + AUX_TRUNK_EPS (447) + AUX_TRUNK_GRAD_CLIP (448) + H_S2_AUX_RMS_EMA (449). Per `feedback_no_legacy_aliases` and `feedback_no_partial_refactor`. Deleted slots left as RESERVED gap (NOT compacted) to preserve checkpoint layout fingerprint compatibility — the surviving SP14 q_disagreement_* diagnostic slots (383, 384, 389) keep their original indices. Pre-C.1 base: 173 + 210 (164 SP5 slots @ 174..278/280..340, with 2-slot gap before cross-fold-persistent Kelly block; Layer D D1 PnL outputs at [286..290); Layer D D2 health composition outputs at [290..294); Layer D D3 training metrics EMA at [294..297); SP7 T1 loss-balance Wiener stats at [297..313); SP7 activation-flag fix per-(head,branch) flags at [313..321); SP8 Fix 36 train_active_frac canary @ [321..322) + LB_MAX_BUDGET per-(head,branch) at [322..330); SP9 Fix 37 Kelly warmup floor at [330..331) + Q_VAR_MAG_EMA at [331..332) + INTENT_EVAL_DIVERGENCE at [332..333) + 3 EMA targets at [333..336) + 3 eval_dist mag bins at [336..339); SP10 Fix 38 EVAL_THOMPSON_TEMP at [339..340); SP11 Fix 39 reward-subsystem controller at [340..367) — 6 component weights [340..346) + 4 controller scalar outputs [346..350) + 2 val-sharpe canaries [350..352) + 6 mag-ratio canaries [352..358) + saboteur engagement [358] + PnL magnitude EMA [359] + popart-component magnitude EMA [360, B1b fix-up] + per-component variance EMAs [361..367, B1b smoke-recovery z-score normalization]; SP13 directional-skill instrumentation at [372..380) — TARGET_DIR_ACC [372] + AUX_DIR_ACC_SHORT/LONG_EMA [373..375) + AUX_DIR_PREDICTION [375] + DIR_SKILL_BONUS_ALPHA/BETA [376..378) + LUCK_WIN_DISCOUNT [378] + SKILL_BONUS_CAP_RATIO [379]; SP13 v3 P0a.T3 Hold-pricing controller at [380..383) — HOLD_COST [380] + HOLD_RATE_TARGET [381] + HOLD_RATE_OBSERVED_EMA [382]; SP14 q_disagreement diagnostic at [383..385), [389] — Q_DISAGREEMENT_SHORT/LONG_EMA + Q_DISAGREEMENT_VARIANCE_EMA (slots [385..389) and [390..396) RESERVED gap from C.1 deletion); SP15 [397..444) per Phase 1; SP14-C [444..450) per Phase C.1. Intentional 5-slot boundary gap at [367..372)) /// Legacy alias preserved for call sites that haven't been audited for the /// network-vs-total split. New code should pick `ISV_NETWORK_DIM` (for weight /// tensor sizing) or `ISV_TOTAL_DIM` (for the broadcast bus buffer). @@ -3602,7 +3602,16 @@ const fn layout_fingerprint_seed() -> &'static [u8] { HCS_DIFF_M2=465;HCS_PREV_TARGET=466;HCS_SAMPLE_COUNT=467;\ MHT_TARGET_MEAN=468;MHT_TARGET_M2=469;MHT_DIFF_MEAN=470;\ MHT_DIFF_M2=471;MHT_PREV_TARGET=472;MHT_SAMPLE_COUNT=473;\ - ISV_TOTAL_DIM=474;\ + SLOT_474_A_VAR_EMA_DIR=474;\ + SLOT_475_A_VAR_EMA_MAG=475;\ + SLOT_476_A_VAR_EMA_ORD=476;\ + SLOT_477_A_VAR_EMA_URG=477;\ + SLOT_478_V_SHARE_DIR=478;\ + SLOT_479_V_SHARE_MAG=479;\ + SLOT_480_V_SHARE_ORD=480;\ + SLOT_481_V_SHARE_URG=481;\ + SLOT_482_ADVANTAGE_CLIP_BOUND=482;\ + ISV_TOTAL_DIM=483;\ SP14_C_AUX_TRUNK_CONTROL_PLANE=sp14_c_phase_1;\ ALPHA_MACHINERY_DELETED=sp14_c_phase_1;\ TRUNK_INPUT_DD_PCT=sp15_wave_4_1a;\ diff --git a/crates/ml/src/cuda_pipeline/sp14_isv_slots.rs b/crates/ml/src/cuda_pipeline/sp14_isv_slots.rs index 91caae8ed..0180f79f6 100644 --- a/crates/ml/src/cuda_pipeline/sp14_isv_slots.rs +++ b/crates/ml/src/cuda_pipeline/sp14_isv_slots.rs @@ -633,6 +633,53 @@ pub const WELFORD_EPS: f32 = 1.0e-6; pub const SP16_T3_WIENER_SLOT_BASE: usize = 462; pub const SP16_T3_WIENER_SLOT_END: usize = 474; +// ── SP17 (2026-05-08): dueling Q-network identifiability diagnostics ─── +// Adds 9 ISV slots driving the V/A diagnostic chain that observes +// whether the post-SP17 mean-centered advantage is doing real work and +// detects regression to the pre-SP17 V-dominated regime. Per +// `feedback_isv_for_adaptive_bounds`, all signal-driven; the +// `advantage_clip_bound` is producer-tracked from p99(|A_centered|), +// never hardcoded. +// +// Mathematical rationale: with mean-zero identifiability +// `A_centered[a, z] = A[a, z] − (1/|branch|) Σ_a A[a, z]`, the per-branch +// variance of `A_centered` over actions measures how much the policy +// discriminates between actions. Var(A) → 0 = "all actions equivalent, +// V dominates" = pre-SP17 pathology re-emerging. Var(A) > 0 with V_share +// stable around 0.5 = healthy dueling decomposition. +// +// Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md +pub const A_VAR_EMA_DIR_INDEX: usize = 474; // Var(A_centered) over dir actions, EMA +pub const A_VAR_EMA_MAG_INDEX: usize = 475; +pub const A_VAR_EMA_ORD_INDEX: usize = 476; +pub const A_VAR_EMA_URG_INDEX: usize = 477; +pub const V_SHARE_DIR_INDEX: usize = 478; // |V_contribution| / (|V| + |A_centered|) +pub const V_SHARE_MAG_INDEX: usize = 479; +pub const V_SHARE_ORD_INDEX: usize = 480; +pub const V_SHARE_URG_INDEX: usize = 481; +pub const ADVANTAGE_CLIP_BOUND_INDEX: usize = 482; // adaptive |A_centered| upper bound + +// Sentinels — Pearl-A first-observation bootstrap. +pub const SENTINEL_A_VAR_EMA: f32 = 0.0; +pub const SENTINEL_V_SHARE: f32 = 0.5; // healthy 50/50 split as cold-start +pub const SENTINEL_ADVANTAGE_CLIP_BOUND: f32 = 1.0; // 1.0 → no effective clipping pre-first-obs + +// Category-1 dimensional safety bounds per `feedback_isv_for_adaptive_bounds`. +// Floor 0.1: below this any centered advantage is essentially noise; clip +// would erase the action signal. Ceiling 100.0: above this a single +// adversarial outlier can dominate per-batch EMAs. +pub const ADVANTAGE_CLIP_BOUND_MIN: f32 = 0.1; +pub const ADVANTAGE_CLIP_BOUND_MAX: f32 = 100.0; + +// Producer constants. Safety factor 1.5× p99 mirrors the SP14 P0-A +// REWARD_POS_CAP producer pattern. EMA α slow per-fold cadence (the bound +// shouldn't track per-batch noise). +pub const ADVANTAGE_CLIP_SAFETY_FACTOR: f32 = 1.5; +pub const ADVANTAGE_CLIP_EMA_ALPHA: f32 = 0.01; + +pub const SP17_DUELING_SLOT_BASE: usize = 474; +pub const SP17_DUELING_SLOT_END: usize = 483; + #[cfg(test)] mod tests { use super::*; @@ -971,4 +1018,40 @@ mod tests { SP16_T3_WIENER_SLOT_END, ISV_TOTAL_DIM, ); } + + /// Lock SP17 (2026-05-08) dueling-Q identifiability diagnostic slot layout. + /// 9 contiguous slots [474..483) — A_var_ema per branch (4) + V_share + /// per branch (4) + adaptive advantage_clip_bound (1). + #[test] + fn sp17_dueling_slot_layout_locked() { + assert_eq!(SP17_DUELING_SLOT_BASE, 474); + assert_eq!(SP17_DUELING_SLOT_END, 483); + assert_eq!(A_VAR_EMA_DIR_INDEX, 474); + assert_eq!(A_VAR_EMA_MAG_INDEX, 475); + assert_eq!(A_VAR_EMA_ORD_INDEX, 476); + assert_eq!(A_VAR_EMA_URG_INDEX, 477); + assert_eq!(V_SHARE_DIR_INDEX, 478); + assert_eq!(V_SHARE_MAG_INDEX, 479); + assert_eq!(V_SHARE_ORD_INDEX, 480); + assert_eq!(V_SHARE_URG_INDEX, 481); + assert_eq!(ADVANTAGE_CLIP_BOUND_INDEX, 482); + // Pearl-A bootstrap sentinels. + assert_eq!(SENTINEL_A_VAR_EMA, 0.0); + assert_eq!(SENTINEL_V_SHARE, 0.5); // initial 50/50 split + assert_eq!(SENTINEL_ADVANTAGE_CLIP_BOUND, 1.0); // 1.0 = "no effective clipping" cold-start + // Category-1 dimensional safety bounds per `feedback_isv_for_adaptive_bounds`. + assert_eq!(ADVANTAGE_CLIP_BOUND_MIN, 0.1); + assert_eq!(ADVANTAGE_CLIP_BOUND_MAX, 100.0); + } + + #[test] + fn all_sp17_slots_fit_within_isv_total_dim() { + use crate::cuda_pipeline::gpu_dqn_trainer::ISV_TOTAL_DIM; + assert!( + SP17_DUELING_SLOT_END <= ISV_TOTAL_DIM, + "SP17_DUELING_SLOT_END={} exceeds ISV_TOTAL_DIM={}; \ + bump ISV_TOTAL_DIM in gpu_dqn_trainer.rs (and update layout_fingerprint_seed()).", + SP17_DUELING_SLOT_END, ISV_TOTAL_DIM, + ); + } } diff --git a/crates/ml/src/trainers/dqn/state_reset_registry.rs b/crates/ml/src/trainers/dqn/state_reset_registry.rs index 7f7f1f3ff..f8929792c 100644 --- a/crates/ml/src/trainers/dqn/state_reset_registry.rs +++ b/crates/ml/src/trainers/dqn/state_reset_registry.rs @@ -1836,6 +1836,62 @@ impl StateResetRegistry { category: ResetCategory::FoldReset, description: "ISV[TRAINING_SHARPE_EMA_INDEX=294..LOW_DD_RATIO_INDEX+1=297) — Layer D D3 training-metrics EMA outputs (training_sharpe_ema / max_dd_ema / low_dd_ratio). SP5 Pearl A sentinel 0 at fold boundary so the new fold's first `launch_training_metrics_ema` fires the first-observation replacement (Task D3, 2026-05-02). Producer is `training_metrics_ema_kernel.cu` — single-block 3-thread arithmetic kernel (one thread per metric) reproducing the host-side adaptive-α Sharpe EMA + fixed-α=0.1 max_dd EMA + fixed-α=0.15 low_dd_ratio EMA from `training_loop.rs:5039-5113` bit-for-bit. Consumer wires atomically in D4 (per `feedback_no_partial_refactor.md`). The underlying host scalars (`training_sharpe_ema` + `_initialized`, `max_dd_ema`, `low_dd_ratio` on `DQNTrainer`) reset at fold boundary alongside the ISV slots — D4 ties the host-side fold-reset call into the same path. Wiener-state companion: bulk memset of the entire `wiener_state_buf[0..(71 + SP5_PRODUCER_COUNT) × 3)` at fold boundary covers the D3 triples at offsets [213+(294-174)*3 .. 213+(297-174)*3) = [573..582). reset_for_fold's existing bulk memset of the whole buffer length covers these triples atomically — no per-slot wiener entry is needed. Buffer size derives from SP5_PRODUCER_COUNT to avoid drift on subsequent SPs. Per `feedback_trust_code_not_docs.md` the third metric is `low_dd_ratio` (the binary low-drawdown indicator EMA at `training_loop.rs:5052`), not the speculative `gamma_blend` referenced in the SP5 plan §D Task D3 — that field does not exist in the codebase.", }, + // ── SP17 (2026-05-08): dueling-Q identifiability diagnostics ────── + // Nine slots [474..483) driving the V/A diagnostic chain that + // observes whether the post-SP17 mean-centered advantage is + // doing real work and detects regression to the pre-SP17 + // V-dominated regime. Producer + consumer kernels land in the + // 4-commit Phase 1-3 atomic migration (compute_expected_q + + // c51_loss + c51_grad + mag_concat_qdir mean-centering) per + // `feedback_no_partial_refactor`; this commit is additive + // infrastructure only. All Pearl-A first-observation + // bootstraps. Plan: + // docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md. + RegistryEntry { + name: "sp17_a_var_ema_dir", + category: ResetCategory::FoldReset, + description: "ISV[A_VAR_EMA_DIR_INDEX=474] — SP17 per-branch EMA of the over-actions variance of the centered advantage `A_centered[a, z] = A[a, z] − mean_a A[a, z]` for the direction branch (4 actions: Long/Short/Hold/Flat). Diagnostic only; signals dueling-Q identifiability health (Var → 0 = `all actions equivalent, V dominates` = pre-SP17 pathology re-emerging; Var > 0 with stable V_share ≈ 0.5 = healthy decomposition). FoldReset sentinel SENTINEL_A_VAR_EMA=0.0 — Pearl-A first-observation bootstrap per `pearl_first_observation_bootstrap.md`. Producer + consumer wire in Phase 1-3 of SP17 (atomic across compute_expected_q + c51_loss + c51_grad + mag_concat_qdir mean-centering) — not yet wired in this commit. Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md.", + }, + RegistryEntry { + name: "sp17_a_var_ema_mag", + category: ResetCategory::FoldReset, + description: "ISV[A_VAR_EMA_MAG_INDEX=475] — SP17 per-branch EMA of the over-actions variance of `A_centered` for the magnitude branch (Quarter/Half/Full). Same contract/sentinel as slot 474. Producer + consumer wire in Phase 1-3 of SP17 (atomic) — not yet wired. Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md.", + }, + RegistryEntry { + name: "sp17_a_var_ema_ord", + category: ResetCategory::FoldReset, + description: "ISV[A_VAR_EMA_ORD_INDEX=476] — SP17 per-branch EMA of the over-actions variance of `A_centered` for the order branch. Same contract/sentinel as slot 474. Producer + consumer wire in Phase 1-3 of SP17 (atomic) — not yet wired. Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md.", + }, + RegistryEntry { + name: "sp17_a_var_ema_urg", + category: ResetCategory::FoldReset, + description: "ISV[A_VAR_EMA_URG_INDEX=477] — SP17 per-branch EMA of the over-actions variance of `A_centered` for the urgency branch. Same contract/sentinel as slot 474. Producer + consumer wire in Phase 1-3 of SP17 (atomic) — not yet wired. Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md.", + }, + RegistryEntry { + name: "sp17_v_share_dir", + category: ResetCategory::FoldReset, + description: "ISV[V_SHARE_DIR_INDEX=478] — SP17 per-branch magnitude share `|V_contribution| / (|V_contribution| + |A_centered|)` for the direction branch. Diagnostic only; signals dueling-Q identifiability health (V_share → 1.0 with Var(A_centered) → 0 = pre-SP17 V-dominated pathology). FoldReset sentinel SENTINEL_V_SHARE=0.5 — Pearl-A bootstrap interpreted as cold-start `healthy 50/50 split` (rather than 0.0) so the diagnostic chain doesn't flag the very first launch as pathological before any observation has landed. Producer + consumer wire in Phase 1-3 of SP17 (atomic) — not yet wired. Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md.", + }, + RegistryEntry { + name: "sp17_v_share_mag", + category: ResetCategory::FoldReset, + description: "ISV[V_SHARE_MAG_INDEX=479] — SP17 per-branch magnitude share for the magnitude branch. Same contract/sentinel as slot 478. Producer + consumer wire in Phase 1-3 of SP17 (atomic) — not yet wired. Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md.", + }, + RegistryEntry { + name: "sp17_v_share_ord", + category: ResetCategory::FoldReset, + description: "ISV[V_SHARE_ORD_INDEX=480] — SP17 per-branch magnitude share for the order branch. Same contract/sentinel as slot 478. Producer + consumer wire in Phase 1-3 of SP17 (atomic) — not yet wired. Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md.", + }, + RegistryEntry { + name: "sp17_v_share_urg", + category: ResetCategory::FoldReset, + description: "ISV[V_SHARE_URG_INDEX=481] — SP17 per-branch magnitude share for the urgency branch. Same contract/sentinel as slot 478. Producer + consumer wire in Phase 1-3 of SP17 (atomic) — not yet wired. Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md.", + }, + RegistryEntry { + name: "sp17_advantage_clip_bound", + category: ResetCategory::FoldReset, + description: "ISV[ADVANTAGE_CLIP_BOUND_INDEX=482] — SP17 adaptive |A_centered| upper bound, producer-tracked from `p99(|A_centered|) × ADVANTAGE_CLIP_SAFETY_FACTOR=1.5` (mirrors SP14 P0-A REWARD_POS_CAP producer pattern). FoldReset sentinel SENTINEL_ADVANTAGE_CLIP_BOUND=1.0 — Pearl-A bootstrap interpreted as cold-start `no effective clipping` (1.0 ≫ typical |A_centered| early in training when A → 0; clip is a no-op until the first valid observation lands a real bound). α=ADVANTAGE_CLIP_EMA_ALPHA=0.01 slow per-fold cadence — the bound shouldn't track per-batch noise. Bilateral clamp to `[ADVANTAGE_CLIP_BOUND_MIN=0.1, ADVANTAGE_CLIP_BOUND_MAX=100.0]` per `pearl_symmetric_clamp_audit.md` — Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds`: floor 0.1 below which centered advantage is essentially noise (clipping would erase the action signal); ceiling 100.0 above which a single adversarial outlier dominates per-batch EMAs. NEVER hardcoded — adaptive bound. Producer + consumer wire in Phase 1-3 of SP17 (atomic) — not yet wired. Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md.", + }, ]; Self { entries } } diff --git a/crates/ml/src/trainers/dqn/trainer/training_loop.rs b/crates/ml/src/trainers/dqn/trainer/training_loop.rs index 1cff31db3..a4f41b996 100644 --- a/crates/ml/src/trainers/dqn/trainer/training_loop.rs +++ b/crates/ml/src/trainers/dqn/trainer/training_loop.rs @@ -8720,6 +8720,87 @@ impl DQNTrainer { fused.trainer().write_isv_signal_at(MHT_SAMPLE_COUNT_INDEX, SENTINEL_WELFORD_ZERO); } } + // ── SP17 (2026-05-08): dueling-Q identifiability diagnostic slots + // [474..483). Pearl-A first-observation bootstrap. Producers / + // consumers wire in Phase 1-3 atomically per + // `feedback_no_partial_refactor`; this PP.2 commit is additive + // infrastructure only. Plan: + // docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md. + "sp17_a_var_ema_dir" => { + if let Some(ref fused) = self.fused_ctx { + use crate::cuda_pipeline::sp14_isv_slots::{ + A_VAR_EMA_DIR_INDEX, SENTINEL_A_VAR_EMA, + }; + fused.trainer().write_isv_signal_at(A_VAR_EMA_DIR_INDEX, SENTINEL_A_VAR_EMA); + } + } + "sp17_a_var_ema_mag" => { + if let Some(ref fused) = self.fused_ctx { + use crate::cuda_pipeline::sp14_isv_slots::{ + A_VAR_EMA_MAG_INDEX, SENTINEL_A_VAR_EMA, + }; + fused.trainer().write_isv_signal_at(A_VAR_EMA_MAG_INDEX, SENTINEL_A_VAR_EMA); + } + } + "sp17_a_var_ema_ord" => { + if let Some(ref fused) = self.fused_ctx { + use crate::cuda_pipeline::sp14_isv_slots::{ + A_VAR_EMA_ORD_INDEX, SENTINEL_A_VAR_EMA, + }; + fused.trainer().write_isv_signal_at(A_VAR_EMA_ORD_INDEX, SENTINEL_A_VAR_EMA); + } + } + "sp17_a_var_ema_urg" => { + if let Some(ref fused) = self.fused_ctx { + use crate::cuda_pipeline::sp14_isv_slots::{ + A_VAR_EMA_URG_INDEX, SENTINEL_A_VAR_EMA, + }; + fused.trainer().write_isv_signal_at(A_VAR_EMA_URG_INDEX, SENTINEL_A_VAR_EMA); + } + } + "sp17_v_share_dir" => { + if let Some(ref fused) = self.fused_ctx { + use crate::cuda_pipeline::sp14_isv_slots::{ + V_SHARE_DIR_INDEX, SENTINEL_V_SHARE, + }; + fused.trainer().write_isv_signal_at(V_SHARE_DIR_INDEX, SENTINEL_V_SHARE); + } + } + "sp17_v_share_mag" => { + if let Some(ref fused) = self.fused_ctx { + use crate::cuda_pipeline::sp14_isv_slots::{ + V_SHARE_MAG_INDEX, SENTINEL_V_SHARE, + }; + fused.trainer().write_isv_signal_at(V_SHARE_MAG_INDEX, SENTINEL_V_SHARE); + } + } + "sp17_v_share_ord" => { + if let Some(ref fused) = self.fused_ctx { + use crate::cuda_pipeline::sp14_isv_slots::{ + V_SHARE_ORD_INDEX, SENTINEL_V_SHARE, + }; + fused.trainer().write_isv_signal_at(V_SHARE_ORD_INDEX, SENTINEL_V_SHARE); + } + } + "sp17_v_share_urg" => { + if let Some(ref fused) = self.fused_ctx { + use crate::cuda_pipeline::sp14_isv_slots::{ + V_SHARE_URG_INDEX, SENTINEL_V_SHARE, + }; + fused.trainer().write_isv_signal_at(V_SHARE_URG_INDEX, SENTINEL_V_SHARE); + } + } + "sp17_advantage_clip_bound" => { + if let Some(ref fused) = self.fused_ctx { + use crate::cuda_pipeline::sp14_isv_slots::{ + ADVANTAGE_CLIP_BOUND_INDEX, SENTINEL_ADVANTAGE_CLIP_BOUND, + }; + fused.trainer().write_isv_signal_at( + ADVANTAGE_CLIP_BOUND_INDEX, + SENTINEL_ADVANTAGE_CLIP_BOUND, + ); + } + } // SP15 Phase 1.2 (2026-05-06): cost-net sharpe slots. // OFI_IMPACT_LAMBDA_INDEX=407 is an Invariant-1 anchor (NOT // a stateful EMA) — rewrite the constructor's value at fold diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index 3219e4a69..1392fe419 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -2,6 +2,52 @@ **Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7. +## 2026-05-08 — SP17 Task PP.2: dueling-Q identifiability ISV slots [474..483) + +Allocates the 9 ISV slots that drive the SP17 dueling-Q identifiability diagnostic chain. **Additive infrastructure only — no consumer wiring in this commit.** The Phase 1 mean-centering producer (atomic across `compute_expected_q` + `c51_loss` + `c51_grad` + `mag_concat_qdir`) lands in the next 4 commits per `feedback_no_partial_refactor`. + +### Why SP17 + +The codebase has separate V-head (`on_v_logits_buf`) and per-branch A-head (`on_b_logits_buf`) buffers, but combines them naively as `logit = v_row[z] + adv_a[z]` — no `A − mean_a A` subtraction. This is over-parameterization without identifiability: the optimizer is mathematically free to learn the degenerate `V ≈ Q(Flat), A ≈ Q(a) − Q(Flat)` decomposition, which IS the Q(Flat) attractor we have been fighting since SP11. The fix is small and surgical: insert mean-zero projection at 4 existing aggregation sites (Phase 1-3). PP.2 lays the ISV slot infrastructure for the diagnostics that observe whether the projection is doing real work. + +### Slot inventory (9 slots, [474..483)) + +| Slot | Name | Role | Sentinel | Bounds | +|------|------|------|----------|--------| +| 474 | `A_VAR_EMA_DIR_INDEX` | Var(A_centered) over dir actions, EMA | 0.0 (Pearl-A) | — | +| 475 | `A_VAR_EMA_MAG_INDEX` | Var(A_centered) over mag actions, EMA | 0.0 (Pearl-A) | — | +| 476 | `A_VAR_EMA_ORD_INDEX` | Var(A_centered) over ord actions, EMA | 0.0 (Pearl-A) | — | +| 477 | `A_VAR_EMA_URG_INDEX` | Var(A_centered) over urg actions, EMA | 0.0 (Pearl-A) | — | +| 478 | `V_SHARE_DIR_INDEX` | `\|V\| / (\|V\| + \|A_centered\|)` for dir | 0.5 (cold-start 50/50) | — | +| 479 | `V_SHARE_MAG_INDEX` | same for mag | 0.5 | — | +| 480 | `V_SHARE_ORD_INDEX` | same for ord | 0.5 | — | +| 481 | `V_SHARE_URG_INDEX` | same for urg | 0.5 | — | +| 482 | `ADVANTAGE_CLIP_BOUND_INDEX` | adaptive `\|A_centered\|` upper bound | 1.0 (no effective clipping) | [0.1, 100.0] | + +**Producer (Phase 1, not yet landed):** `advantage_clip_bound_update_kernel` computes `p99(|A_centered|) × ADVANTAGE_CLIP_SAFETY_FACTOR=1.5` (mirrors SP14 P0-A REWARD_POS_CAP producer pattern), bilateral-clamps to `[ADVANTAGE_CLIP_BOUND_MIN=0.1, ADVANTAGE_CLIP_BOUND_MAX=100.0]` per `pearl_symmetric_clamp_audit`, slow-EMA blends with `ADVANTAGE_CLIP_EMA_ALPHA=0.01`. Per `feedback_isv_for_adaptive_bounds`: producer-tracked, NEVER hardcoded. + +**Consumers (Phase 1-3, not yet landed):** mean-zero projection at 4 sites in atomic migration — `compute_expected_q`, `c51_loss`, `c51_grad`, `mag_concat_qdir`. The 9 diagnostic slots are read by HEALTH_DIAG (Phase 5). + +### Files modified in PP.2 + +- `crates/ml/src/cuda_pipeline/sp14_isv_slots.rs` — 9 slot constants + sentinels + bounds + 2 lock-tests appended after SP16 T3 block. +- `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` — `ISV_TOTAL_DIM` 474 → 483 + `layout_fingerprint_seed` extended with 9 new `SLOT__=` lines. +- `crates/ml/src/trainers/dqn/state_reset_registry.rs` — 9 `RegistryEntry { ResetCategory::FoldReset, ... }` entries appended at the end of `new()`. +- `docs/dqn-wire-up-audit.md` — this section. + +### Why no `state_layout.cuh` mirror update + +`state_layout.cuh::ISV_TOTAL_DIM` is a stale historical marker (currently 383) per its own comment block at line 260-266 — `kernels take const float* pointers and use raw indices, no .cu/.cuh file dimensions an array with this macro. The Rust constant is the single source of truth for the bus size.` No-op for this PP.2. + +### Verification + +``` +SQLX_OFFLINE=true cargo test -p ml --lib sp17_dueling_slot_layout_locked all_sp17_slots_fit_within_isv_total_dim → PASS (2/2) +SQLX_OFFLINE=true CUDA_COMPUTE_CAP=86 cargo check --workspace → clean +``` + +Plan: `docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md` Task PP.2. + ## 2026-05-08 — Class A audit-fix Batch 4-B: plan_threshold floor adaptive + MIN_HOLD_TEMPERATURE ISV-driven Per Class A audit-fix Batch 4-B (final 2 of 4 deferred items from P1-wiring/P1-producer). Completes the 8-commit WR-plateau intervention chain. Validation deferred to next L40S smoke.