From 16100912c3239d99573b6ffba408ac2f567c3588 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sat, 9 May 2026 00:34:13 +0200 Subject: [PATCH] plan(sp18 v2): allocate 22 ISV slots [483..505) for D-leg + B-leg MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds combined SP18 D-leg Hold-reward adaptive-cap slots [483..493) + B-leg TD(λ) Q(s') bootstrap diagnostics + PopArt reset flag at [493..505) per spec DD6 (D-leg 10 slots) + B-DD13 (B-leg 12 slots). D-leg [483..493) — 10 slots: 483 HOLD_REWARD_POS_CAP (sentinel 5.0, bounds [0.5, 50]) 484 HOLD_REWARD_NEG_CAP (sentinel -10.0) 485 HOLD_REWARD_DECOMP_DIAG (sentinel 0.0) 486 HOLD_OPP_COST_FIRE_RATE_EMA (sentinel 0.0) 487..493 HRC_* Welford accumulators (mirrors SP16 T3 HCS_* pattern) B-leg [493..505) — 12 slots: 493 TD_ERROR_MAG_EMA (HEALTH_DIAG B-DD9 ratio gate input) 494 Q_NEXT_TARGET_P99 (target-Q bootstrap bound check) 495 Q_NEXT_MINUS_REWARD_P99 (sanity: should be O(1) post-fix) 496 V_SHARE_TREND_DIAG (B-leg synergy probe) 497 POPART_RESET_FLAG (sentinel 1.0 — one-shot, B-DD11) 498..504 TDB_* Welford accumulators (mirrors HRC_* pattern) 504 RESERVED (B-leg follow-up) Pearl-A first-observation bootstrap sentinels match position-side SP14 P0-A REWARD_POS_CAP_ADAPTIVE pattern (POS=5.0, NEG=-10.0). Slot 497 POPART_RESET_FLAG sentinel = 1.0 per B-DD11 — host writes 1.0 once at first SP18 epoch, kernel zeroes after consuming, gating the per-fold PopArt slot 63 EMA reset at the SP18 deployment boundary. ISV_TOTAL_DIM bumped 483 → 505; layout_fingerprint_seed updated with all 22 new slot names; state_layout.cuh C-side mirror in lockstep (continues SP14-P0A/P1/audit-fix-4A/4B mirror precedent — SP16/SP17 slots intentionally not mirrored per existing pattern, only SP18 gets fresh mirror entries). `feedback_no_partial_refactor`: both legs share an ISV section + a single fingerprint bump; SP13 [380..383) and SP16 [461..474) slots remain ALLOCATED but RETIRED in PP.4 (sentinel 0.0, no producer launch — RESERVED-gap pattern from SP14-C.1 preserves checkpoint compatibility). Audit doc updated per Invariant 7 with Pre-Phase PP.2 entry. Spec: docs/superpowers/specs/2026-05-08-sp18-reward-shape-hold-attractor-design.md Plan: docs/superpowers/plans/2026-05-08-sp18-reward-shape-hold-attractor.md Co-Authored-By: Claude Opus 4.7 (1M context) --- .../ml/src/cuda_pipeline/gpu_dqn_trainer.rs | 26 +- crates/ml/src/cuda_pipeline/sp14_isv_slots.rs | 223 ++++++++++++++++++ crates/ml/src/cuda_pipeline/state_layout.cuh | 86 +++++++ docs/dqn-wire-up-audit.md | 40 ++++ 4 files changed, 373 insertions(+), 2 deletions(-) diff --git a/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs b/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs index 1e0fb46fe..e9b2a7b52 100644 --- a/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs +++ b/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs @@ -2568,7 +2568,7 @@ const ISV_NETWORK_DIM: usize = 23; /// (shifted 112→116 in Plan 4 Task 6 Commit A). /// Written by the constructor; checked at checkpoint load. Fail-fast only — no migration /// path exists. See spec §4.A.2 and `LAYOUT_FINGERPRINT_CURRENT` for structural-hash rationale. -pub(crate) const ISV_TOTAL_DIM: usize = 483; // SP5 + Layer D D1+D2+D3 + SP7 + SP8 + SP9 + SP10 + SP11 + SP13 + SP13 v3 + SP14 (post-C.1) + SP15 + SP14-C aux trunk control plane + SP14-C Phase C.4b aux horizon + Class A P0-A adaptive reward caps + Class A P1-Producer adaptive Bayesian Kelly priors + Class A audit-fix Batch 4-A adaptive DD saturation floor + Class A audit-fix Batch 4-B adaptive plan_threshold floor + adaptive MIN_HOLD_TEMPERATURE + SP16 Phase 2 adaptive Hold cost scale + SP16 T3 Wiener-optimal adaptive α Welford accumulators + SP17 dueling-Q identifiability diagnostics. Bumped 474 → 483 by SP17 (2026-05-08): added 9 slots [474..483) for the dueling-Q identifiability diagnostic chain. With mean-zero advantage projection `A_centered[a, z] = A[a, z] − mean_a A[a, z]` (Phase 1-3 of SP17), V and A become uniquely identifiable: V_share ≈ 0.5 + Var(A_centered) > 0 indicates a healthy dueling decomposition; V_share → 1.0 + Var(A_centered) → 0 signals regression to the pre-SP17 V-dominated regime that the optimizer was free to learn under naive `Q = V + A`. Slots: A_VAR_EMA × 4 branches at [474..478) — per-branch EMA of the over-actions variance of the centered advantage; V_SHARE × 4 branches at [478..482) — per-branch |V| / (|V| + |A_centered|) magnitude share; ADVANTAGE_CLIP_BOUND at [482] — adaptive |A_centered| upper bound producer-tracked from p99(|A_centered|) × safety_factor=1.5 (mirrors SP14 P0-A REWARD_POS_CAP producer pattern; α=0.01 slow per-fold cadence). All Pearl-A first-observation bootstraps with sentinels SENTINEL_A_VAR_EMA=0.0, SENTINEL_V_SHARE=0.5 (cold-start 50/50 split), SENTINEL_ADVANTAGE_CLIP_BOUND=1.0 (cold-start "no effective clipping"). Bounds [ADVANTAGE_CLIP_BOUND_MIN=0.1, ADVANTAGE_CLIP_BOUND_MAX=100.0] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds`: floor 0.1 below which centered advantage is essentially noise (clipping would erase the action signal); ceiling 100.0 above which a single adversarial outlier dominates per-batch EMAs. This commit is additive infrastructure — no consumer change yet; the Phase 1 mean-centering producer (atomic across `compute_expected_q` + `c51_loss` + `c51_grad` + `mag_concat_qdir`) lands in the next 4 commits per `feedback_no_partial_refactor`. Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md. Bumped 462 → 474 by SP16 T3 (2026-05-08): added 12 slots [462..474) — 6 Welford accumulators per producer kernel (T1 MIN_HOLD_TEMPERATURE + T2 HOLD_COST_SCALE). Replaces the hardcoded `alpha = 0.05f` EMA constant in both producer kernels (`min_hold_temperature_update_kernel.cu` + `hold_cost_scale_update_kernel.cu`) with a signal-driven Wiener-optimal α per `pearl_wiener_optimal_adaptive_alpha`: α = diff_var / (diff_var + sample_var + ε). Per train-multi-seed-hjzss validation, the SP16 T1+T2 chain was structurally landed but BEHAVIORALLY INERT in 5-epoch smoke (bit-identical to pfh9n baseline through epoch 3) because α=0.05 needs ~60 epochs from cold start to converge. Wiener-optimal α naturally falls in [0, 1] (clamped to [0.01, 0.95] for numerical safety): cold-start (signal jumps 1.0 → 6.4 → 7.0) → diff_var dominates → α ≈ 0.6+ → near-bootstrap responsiveness in epochs 1-3; steady-state → diff_var drops → α decays naturally → smoothing emerges without hardcoded constant. Each producer maintains 6 Welford accumulators contiguously: [HCS_TARGET_MEAN, HCS_TARGET_M2, HCS_DIFF_MEAN, HCS_DIFF_M2, HCS_PREV_TARGET, HCS_SAMPLE_COUNT] at [462..468) and [MHT_TARGET_MEAN, MHT_TARGET_M2, MHT_DIFF_MEAN, MHT_DIFF_M2, MHT_PREV_TARGET, MHT_SAMPLE_COUNT] at [468..474). FoldReset sentinel 0.0 — combined with the cold-start fallback (sample_count < 3 → α = 1.0 REPLACE), the first observation of each fold replaces blended directly, the second seeds diff statistics, the third stabilises Wiener α. Per `feedback_isv_for_adaptive_bounds.md` — α was an adaptive bound being hardcoded; lifting to ISV-driven Wiener-optimal closes that violation. Bumped 461 → 462 by SP16 Phase 2 (2026-05-08): added 1 slot [461..462) — HOLD_COST_SCALE_INDEX (461). Replaces the static `ISV[HOLD_COST_INDEX=380]` per-bar Hold-cost magnitude (~0.006) when over-holding is observed. Producer kernel `hold_cost_scale_update_kernel` reads `ISV[HOLD_RATE_OBSERVED_EMA_INDEX=382]` and `ISV[HOLD_RATE_TARGET_INDEX=381]` (same SP13 hold-pricing chain feeding SP16-P1's MIN_HOLD_TEMPERATURE — same cadence, same input pattern), maps overrun into a [1, 25] multiplier via `target_scale = SCALE_MIN + (SCALE_MAX − SCALE_MIN) × clamp(max(0, observed - target) / max(target, 0.01), 0, 1)`, and Pearl-A-bootstrap + α=0.05 mid-cadence EMA blends into slot 461. Consumer: `experience_kernels.cu` per-bar Hold cost subtractions (3 sites: segment_complete branch line 3089, per-bar positioned-Hold branch line 3553, per-bar flat-Hold branch line 3617) multiply `isv[ISV_HOLD_COST_IDX=380]` by `isv[HOLD_COST_SCALE_INDEX=461]`. Cold-start fallback: when slot 461 is at sentinel 0.0 OR outside [1, 25], the consumer uses scale=1.0 (bit-identical to pre-Phase-2 v3-P0a Hold-cost magnitude). At observed=2× target → scale=25× → effective cost ~0.15/bar competitive with per-bar reward magnitudes ~0.01-0.1 but well below SP12 asymmetric cap [-10, +5]. Bounds [1, 25] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds.md`. Bumped 459 → 461 by Class A audit-fix Batch 4-B (2026-05-08): added 2 slots [459..461) — PLAN_THRESHOLD_FLOOR_ADAPTIVE (459) + MIN_HOLD_TEMPERATURE_ADAPTIVE (460). PLAN_THRESHOLD_FLOOR_ADAPTIVE replaces the hardcoded `0.1f` floor in `plan_threshold_update_kernel.cu:44` (`fmaxf(0.1f, 0.5f * ema)`) — Design Y inline producer (no new kernel; same kernel writes both slot 49 and the slow-EMA-shadow floor at slot 459 in the same launch, then uses the freshly-written floor as the lower bound). Pearl-A bootstrap (sentinel 0.1 matches pre-fix hardcoded value for bit-identical cold-start) + α=0.005 slow EMA (per-fold cadence — the floor is a long-term shadow of `0.5 × readiness_ema` and must move much slower than the readiness EMA itself). Bounds [0.05, 0.50] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds` (probability units). MIN_HOLD_TEMPERATURE_ADAPTIVE replaces the deleted epoch-driven anneal schedule `T_end + (T_start − T_end) × exp(-epoch / decay)` (formerly state_layout.cuh::MIN_HOLD_TEMPERATURE_{START=50, END=5, DECAY=20} + training_loop.rs::min_hold_temperature_for_epoch). Producer kernel `min_hold_temperature_update_kernel` reads `ISV[AUX_DIR_ACC_SHORT_EMA_INDEX=373]` (sp13 fast EMA of binary directional accuracy), maps to a [5, 50] temperature via `temp = TEMP_MIN + (TEMP_MAX − TEMP_MIN) × confusion` where `confusion = 1 − clamp((short_ema − 0.5)/0.5, 0, 1)` (so dir_acc ≈ 0.5 plateau → temp HIGH/permissive exit ramp; dir_acc → 1.0 → temp LOW/strict informed-commitment), and slow-EMA-blends into slot 460. HIGH T = forgiving (`deficit/(deficit+T)` saturation factor → 0); LOW T = sharp (→ 1) per `compute_min_hold_penalty` at trade_physics.cuh:575. Pearl-A bootstrap (sentinel 50.0 matches the deleted MIN_HOLD_TEMPERATURE_START=50 anchor for bit-identical cold-start under both regimes) + α=0.05 mid-cadence EMA. Bounds [5, 50] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds` (matches the original schedule range). Decouples temperature from epoch number — when WR plateaus, the old schedule pinned LOW temp (sharp) at end of training (max punishment exactly when most needed forgiveness); the signal-driven version reacts to actual realized skill. Per `feedback_no_legacy_aliases.md` the legacy schedule constants + helper function are deleted atomically. Bumped 458 → 459 by Class A audit-fix Batch 4-A (2026-05-08): added 1 adaptive-DD-saturation-floor slot [458..459) — DD_SATURATION_FLOOR_ADAPTIVE (458). Replaces the hardcoded `0.25f` saturation floor in `trade_physics.cuh::apply_margin_cap` (the upper end of the linear position-size scaling ramp dd_scale = max(0.05, 1.0 − dd_frac / saturation_floor)). Distinct semantic role from `SP15_DD_THRESHOLD_INDEX=421` (the SP15 quadratic DD-penalty *trigger* threshold, a *lower* bound). Producer kernel `dd_saturation_floor_update_kernel` aggregates per-env DD_MAX from the existing `dd_state_per_env[env*6 + 1]` tile (same source as the SP15 dd_state_kernel writes to), computes p75 estimator via Welford rule (mean + Z_75 × sigma) × safety_factor=1.5, and slow-EMA blends. Pearl-A bootstrap (sentinel 0.25 matches pre-fix hardcoded value for bit-identical cold-start) + α=0.01 slow EMA (DD distribution is a slow-moving fold-volatility property, shouldn't track per-epoch noise). Bounds [0.10, 0.50] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds`. Atomically deleted alongside this slot addition: legacy `compute_drawdown_penalty` device function in trade_physics.cuh + its single caller at experience_kernels.cu:3822 + the `w_dd` and `dd_threshold` kernel args (Item 2 Case A — SP15's quadratic asymmetric DD penalty in `compute_sp15_final_reward_kernel.cu:154` is the production-grade replacement; layering both creates double-counting per `feedback_no_partial_refactor`). Bumped 454 → 458 by Class A P1-Producer (2026-05-08): added 4 adaptive-Bayesian-prior slots [454..458) — KELLY_PRIOR_WINS (454) + KELLY_PRIOR_LOSSES (455) + KELLY_PRIOR_SUM_WINS (456) + KELLY_PRIOR_SUM_LOSSES (457). Per Class A audit, hardcoded `prior_wins=2.0f / prior_losses=2.0f / prior_sum_wins=0.01f / prior_sum_losses=0.01f` (kelly_cap_update_kernel.cu:39-42 + trade_physics.cuh::kelly_position_cap:304-307) was a static "I don't know" Bayesian prior frozen at sprint-1 defaults — the cold-start of every new fold ignored what prior folds had learned about the realized Kelly distribution. Producer kernel `kelly_bayesian_priors_update_kernel` aggregates realized Kelly stats from the existing `portfolio_state[n_envs, PS_STRIDE]` buffer (same source as kelly_cap_update consumes) and slow-EMA blends into the four slots. Pearl-A bootstrap (sentinels 2.0/2.0/0.01/0.01 match pre-P1-Producer hardcoded values for bit-identical cold-start) + α=0.005 slow EMA (per-fold cadence; the Bayesian prior is a *prior belief* and should change slowly across folds). Bounds counts∈[0.5, 100], sums∈[0.001, 1.0] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds`. Bumped 452 → 454 by Class A P0-A (2026-05-08): added 2 adaptive-reward-cap slots [452..454) — REWARD_POS_CAP_ADAPTIVE (452) + REWARD_NEG_CAP_ADAPTIVE (453). Per Class A audit, hardcoded `REWARD_POS_CAP=+5.0f` / `REWARD_NEG_CAP=-10.0f` (state_layout.cuh:266-267) was structurally clipping the upper tail of realized alpha — controller signals (sharpe EMA, var_q, q_gap) all derive from the CAPPED buffer, so the controller cannot select for trades it cannot see. The state_layout comment explicitly deferred Phase 2 (ISV-driven) "IF Phase 1 validation reveals adaptive need"; 11 superprojects of WR-stuck-at-46-48% revealed adaptive need. Producer kernel `reward_cap_update_kernel` computes p99(winning_returns) × safety_factor=1.5 → POS cap and NEG = -2 × POS (preserves Kahneman 2:1 loss-aversion asymmetry per `pearl_audit_unboundedness_for_implicit_asymmetry` — moved from hardcoded scalar to producer-time multiplier, single source of truth). Pearl-A bootstrap (sentinels 5.0/-10.0) + Wiener-α=0.01 slow EMA. Bounds POS∈[1, 50], NEG∈[-100, -2] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds`, NOT tuning. Bumped 450 → 452 by SP14 Layer C Phase C.4b (2026-05-08): added 2 aux-prediction-horizon slots [450..452) — AUX_PRED_HORIZON_BARS (450) + AVG_WIN_HOLD_TIME_BARS (451). Aux's original label was (p_{t+1} > p_t), pure HFT-scale microstructure noise — pivoted to (p_{t+H} > p_t) with H read from ISV[450], adaptively driven from observed avg winning hold time via Pearl-A bootstrap + Wiener-α EMA. Case B: existing `hold_at_exit_per_sample` + `trade_profitable_per_sample` per-sample buffers, no aggregate slot — added new aggregate slot 451 to expose the EMA target. Bumped 444 → 450 by SP14 Layer C Phase C.1 (2026-05-08): atomically deleted 9 α-machinery slots in [385..396) AND added 6 aux-trunk control-plane slots [444..450) — AUX_TRUNK_LR (444) + AUX_TRUNK_BETA1 (445) + AUX_TRUNK_BETA2 (446) + AUX_TRUNK_EPS (447) + AUX_TRUNK_GRAD_CLIP (448) + H_S2_AUX_RMS_EMA (449). Per `feedback_no_legacy_aliases` and `feedback_no_partial_refactor`. Deleted slots left as RESERVED gap (NOT compacted) to preserve checkpoint layout fingerprint compatibility — the surviving SP14 q_disagreement_* diagnostic slots (383, 384, 389) keep their original indices. Pre-C.1 base: 173 + 210 (164 SP5 slots @ 174..278/280..340, with 2-slot gap before cross-fold-persistent Kelly block; Layer D D1 PnL outputs at [286..290); Layer D D2 health composition outputs at [290..294); Layer D D3 training metrics EMA at [294..297); SP7 T1 loss-balance Wiener stats at [297..313); SP7 activation-flag fix per-(head,branch) flags at [313..321); SP8 Fix 36 train_active_frac canary @ [321..322) + LB_MAX_BUDGET per-(head,branch) at [322..330); SP9 Fix 37 Kelly warmup floor at [330..331) + Q_VAR_MAG_EMA at [331..332) + INTENT_EVAL_DIVERGENCE at [332..333) + 3 EMA targets at [333..336) + 3 eval_dist mag bins at [336..339); SP10 Fix 38 EVAL_THOMPSON_TEMP at [339..340); SP11 Fix 39 reward-subsystem controller at [340..367) — 6 component weights [340..346) + 4 controller scalar outputs [346..350) + 2 val-sharpe canaries [350..352) + 6 mag-ratio canaries [352..358) + saboteur engagement [358] + PnL magnitude EMA [359] + popart-component magnitude EMA [360, B1b fix-up] + per-component variance EMAs [361..367, B1b smoke-recovery z-score normalization]; SP13 directional-skill instrumentation at [372..380) — TARGET_DIR_ACC [372] + AUX_DIR_ACC_SHORT/LONG_EMA [373..375) + AUX_DIR_PREDICTION [375] + DIR_SKILL_BONUS_ALPHA/BETA [376..378) + LUCK_WIN_DISCOUNT [378] + SKILL_BONUS_CAP_RATIO [379]; SP13 v3 P0a.T3 Hold-pricing controller at [380..383) — HOLD_COST [380] + HOLD_RATE_TARGET [381] + HOLD_RATE_OBSERVED_EMA [382]; SP14 q_disagreement diagnostic at [383..385), [389] — Q_DISAGREEMENT_SHORT/LONG_EMA + Q_DISAGREEMENT_VARIANCE_EMA (slots [385..389) and [390..396) RESERVED gap from C.1 deletion); SP15 [397..444) per Phase 1; SP14-C [444..450) per Phase C.1. Intentional 5-slot boundary gap at [367..372)) +pub(crate) const ISV_TOTAL_DIM: usize = 505; // SP5 + Layer D D1+D2+D3 + SP7 + SP8 + SP9 + SP10 + SP11 + SP13 + SP13 v3 + SP14 (post-C.1) + SP15 + SP14-C aux trunk control plane + SP14-C Phase C.4b aux horizon + Class A P0-A adaptive reward caps + Class A P1-Producer adaptive Bayesian Kelly priors + Class A audit-fix Batch 4-A adaptive DD saturation floor + Class A audit-fix Batch 4-B adaptive plan_threshold floor + adaptive MIN_HOLD_TEMPERATURE + SP16 Phase 2 adaptive Hold cost scale + SP16 T3 Wiener-optimal adaptive α Welford accumulators + SP17 dueling-Q identifiability diagnostics + SP18 v2 combined Hold-attractor fix (D-leg structural Hold opportunity-cost reward + B-leg TD(λ) Q(s') bootstrap diagnostics + PopArt reset gate). Bumped 483 → 505 by SP18 v2 Pre-Phase Task PP.2 (2026-05-08): added 22 slots [483..505) split across two atomic legs per `feedback_no_partial_refactor`. D-leg [483..493) — 10 slots: HOLD_REWARD_POS_CAP (483) + HOLD_REWARD_NEG_CAP (484) + HOLD_REWARD_DECOMP_DIAG (485) + HOLD_OPP_COST_FIRE_RATE_EMA (486) + 6 HRC_* Welford accumulators [487..493) for Wiener-α (mirrors SP16 T3 HCS_* layout). The two caps replace the SP13 P0a + SP16 P2 + SP16 T3 reactive per-bar Hold-cost chain (DD7=c) with a structural opportunity-cost reward at the 3 Hold-cost subtraction sites in `experience_kernels.cu`: each Hold-action bar pays `compute_sp18_hold_opportunity_cost(-2 × vol_normalized_next_log_return, neg_cap, pos_cap, shaping_scale)` — counterfactual cost in realised next-bar log-return units, asymmetric-capped via the `hold_reward_cap_update_kernel` adaptive producer (Phase 3) reading `p99(|step_ret|)` over Long/Short trade closes (DD3=b) × HOLD_REWARD_CAP_SAFETY_FACTOR=1.5 → POS, with NEG = -2 × POS (DD5=b mirrored 2:1 loss-aversion ratio matching SP14 P0-A REWARD_POS/NEG_CAP). Wiener-optimal α from Welford accumulators in [487..493) per `pearl_wiener_optimal_adaptive_alpha`. Pearl-A first-observation bootstrap sentinels POS=5.0/NEG=-10.0 match SP14 P0-A REWARD_POS/NEG_CAP_ADAPTIVE for bit-identical cold-start. Bounds [HOLD_REWARD_POS_CAP_MIN_BOUND=0.5, HOLD_REWARD_POS_CAP_MAX_BOUND=50.0] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds`. SP13/SP16 slots [380..383) and [461..474) remain ALLOCATED but RETIRED — sentinel 0.0, no producer launch — preserving checkpoint compatibility (RESERVED-gap pattern from SP14-C.1). B-leg [493..505) — 12 slots: TD_ERROR_MAG_EMA (493) + Q_NEXT_TARGET_P99 (494) + Q_NEXT_MINUS_REWARD_P99 (495) + V_SHARE_TREND_DIAG (496) + POPART_RESET_FLAG (497) + 6 TDB_* Welford accumulators [498..504) + 1 RESERVED (504). The B-leg replaces the `q_next = rewards` self-bootstrap at `gpu_experience_collector.rs:4143` with a real per-branch DDQN `Q_target(s', argmax_a Q_online(s', a))` computed by a target-net forward pass on `next_states` at experience-collection time (Phase 4-5). Reuses existing `target_params_buf` (SP14 P4.T1B-iv Polyak-tracked) — no new target-net infra (B-DD1). HEALTH_DIAG observability (B-DD12): td_error_mag_ema is the B-DD9 ratio-gate input (`avg(|TD-error|) ratio post-fix / pre-fix ∈ [0.5, 5.0]`); q_next_target_p99 is the bootstrap bound check; q_next_minus_reward_p99 sanity-checks magnitude (should be O(1) — proves the fix is doing something AND not blowing up); v_share_trend_diag observes the B-leg synergy hypothesis (bootstrap-bias predicts v_share monotone HD0→HD4). POPART_RESET_FLAG (497, sentinel 1.0) gates the per-fold PopArt slot 63 EMA reset at the SP18 deployment boundary per B-DD11 — switching `q_next` source from rewards-distributed to Q-distributed shifts the PopArt input distribution; cheap insurance is to reset PopArt EMA to identity normalization at the first SP18 epoch. NOT a FoldReset; host writes 1.0 once at deploy, kernel zeroes after consuming. Sentinels for diagnostic slots = 0.0 (Pearl-A direct-replace on first valid observation). 6 TDB_* Welford accumulators mirror HRC_* layout for q_next_target Wiener-α blending. Slot 504 RESERVED for B-leg follow-up. PER priority recomputation NOT migrated (per B-DD4 — already recomputed at training time from current TD-error). Plan: docs/superpowers/plans/2026-05-08-sp18-reward-shape-hold-attractor.md. Bumped 474 → 483 by SP17 (2026-05-08): added 9 slots [474..483) for the dueling-Q identifiability diagnostic chain. With mean-zero advantage projection `A_centered[a, z] = A[a, z] − mean_a A[a, z]` (Phase 1-3 of SP17), V and A become uniquely identifiable: V_share ≈ 0.5 + Var(A_centered) > 0 indicates a healthy dueling decomposition; V_share → 1.0 + Var(A_centered) → 0 signals regression to the pre-SP17 V-dominated regime that the optimizer was free to learn under naive `Q = V + A`. Slots: A_VAR_EMA × 4 branches at [474..478) — per-branch EMA of the over-actions variance of the centered advantage; V_SHARE × 4 branches at [478..482) — per-branch |V| / (|V| + |A_centered|) magnitude share; ADVANTAGE_CLIP_BOUND at [482] — adaptive |A_centered| upper bound producer-tracked from p99(|A_centered|) × safety_factor=1.5 (mirrors SP14 P0-A REWARD_POS_CAP producer pattern; α=0.01 slow per-fold cadence). All Pearl-A first-observation bootstraps with sentinels SENTINEL_A_VAR_EMA=0.0, SENTINEL_V_SHARE=0.5 (cold-start 50/50 split), SENTINEL_ADVANTAGE_CLIP_BOUND=1.0 (cold-start "no effective clipping"). Bounds [ADVANTAGE_CLIP_BOUND_MIN=0.1, ADVANTAGE_CLIP_BOUND_MAX=100.0] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds`: floor 0.1 below which centered advantage is essentially noise (clipping would erase the action signal); ceiling 100.0 above which a single adversarial outlier dominates per-batch EMAs. This commit is additive infrastructure — no consumer change yet; the Phase 1 mean-centering producer (atomic across `compute_expected_q` + `c51_loss` + `c51_grad` + `mag_concat_qdir`) lands in the next 4 commits per `feedback_no_partial_refactor`. Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md. Bumped 462 → 474 by SP16 T3 (2026-05-08): added 12 slots [462..474) — 6 Welford accumulators per producer kernel (T1 MIN_HOLD_TEMPERATURE + T2 HOLD_COST_SCALE). Replaces the hardcoded `alpha = 0.05f` EMA constant in both producer kernels (`min_hold_temperature_update_kernel.cu` + `hold_cost_scale_update_kernel.cu`) with a signal-driven Wiener-optimal α per `pearl_wiener_optimal_adaptive_alpha`: α = diff_var / (diff_var + sample_var + ε). Per train-multi-seed-hjzss validation, the SP16 T1+T2 chain was structurally landed but BEHAVIORALLY INERT in 5-epoch smoke (bit-identical to pfh9n baseline through epoch 3) because α=0.05 needs ~60 epochs from cold start to converge. Wiener-optimal α naturally falls in [0, 1] (clamped to [0.01, 0.95] for numerical safety): cold-start (signal jumps 1.0 → 6.4 → 7.0) → diff_var dominates → α ≈ 0.6+ → near-bootstrap responsiveness in epochs 1-3; steady-state → diff_var drops → α decays naturally → smoothing emerges without hardcoded constant. Each producer maintains 6 Welford accumulators contiguously: [HCS_TARGET_MEAN, HCS_TARGET_M2, HCS_DIFF_MEAN, HCS_DIFF_M2, HCS_PREV_TARGET, HCS_SAMPLE_COUNT] at [462..468) and [MHT_TARGET_MEAN, MHT_TARGET_M2, MHT_DIFF_MEAN, MHT_DIFF_M2, MHT_PREV_TARGET, MHT_SAMPLE_COUNT] at [468..474). FoldReset sentinel 0.0 — combined with the cold-start fallback (sample_count < 3 → α = 1.0 REPLACE), the first observation of each fold replaces blended directly, the second seeds diff statistics, the third stabilises Wiener α. Per `feedback_isv_for_adaptive_bounds.md` — α was an adaptive bound being hardcoded; lifting to ISV-driven Wiener-optimal closes that violation. Bumped 461 → 462 by SP16 Phase 2 (2026-05-08): added 1 slot [461..462) — HOLD_COST_SCALE_INDEX (461). Replaces the static `ISV[HOLD_COST_INDEX=380]` per-bar Hold-cost magnitude (~0.006) when over-holding is observed. Producer kernel `hold_cost_scale_update_kernel` reads `ISV[HOLD_RATE_OBSERVED_EMA_INDEX=382]` and `ISV[HOLD_RATE_TARGET_INDEX=381]` (same SP13 hold-pricing chain feeding SP16-P1's MIN_HOLD_TEMPERATURE — same cadence, same input pattern), maps overrun into a [1, 25] multiplier via `target_scale = SCALE_MIN + (SCALE_MAX − SCALE_MIN) × clamp(max(0, observed - target) / max(target, 0.01), 0, 1)`, and Pearl-A-bootstrap + α=0.05 mid-cadence EMA blends into slot 461. Consumer: `experience_kernels.cu` per-bar Hold cost subtractions (3 sites: segment_complete branch line 3089, per-bar positioned-Hold branch line 3553, per-bar flat-Hold branch line 3617) multiply `isv[ISV_HOLD_COST_IDX=380]` by `isv[HOLD_COST_SCALE_INDEX=461]`. Cold-start fallback: when slot 461 is at sentinel 0.0 OR outside [1, 25], the consumer uses scale=1.0 (bit-identical to pre-Phase-2 v3-P0a Hold-cost magnitude). At observed=2× target → scale=25× → effective cost ~0.15/bar competitive with per-bar reward magnitudes ~0.01-0.1 but well below SP12 asymmetric cap [-10, +5]. Bounds [1, 25] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds.md`. Bumped 459 → 461 by Class A audit-fix Batch 4-B (2026-05-08): added 2 slots [459..461) — PLAN_THRESHOLD_FLOOR_ADAPTIVE (459) + MIN_HOLD_TEMPERATURE_ADAPTIVE (460). PLAN_THRESHOLD_FLOOR_ADAPTIVE replaces the hardcoded `0.1f` floor in `plan_threshold_update_kernel.cu:44` (`fmaxf(0.1f, 0.5f * ema)`) — Design Y inline producer (no new kernel; same kernel writes both slot 49 and the slow-EMA-shadow floor at slot 459 in the same launch, then uses the freshly-written floor as the lower bound). Pearl-A bootstrap (sentinel 0.1 matches pre-fix hardcoded value for bit-identical cold-start) + α=0.005 slow EMA (per-fold cadence — the floor is a long-term shadow of `0.5 × readiness_ema` and must move much slower than the readiness EMA itself). Bounds [0.05, 0.50] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds` (probability units). MIN_HOLD_TEMPERATURE_ADAPTIVE replaces the deleted epoch-driven anneal schedule `T_end + (T_start − T_end) × exp(-epoch / decay)` (formerly state_layout.cuh::MIN_HOLD_TEMPERATURE_{START=50, END=5, DECAY=20} + training_loop.rs::min_hold_temperature_for_epoch). Producer kernel `min_hold_temperature_update_kernel` reads `ISV[AUX_DIR_ACC_SHORT_EMA_INDEX=373]` (sp13 fast EMA of binary directional accuracy), maps to a [5, 50] temperature via `temp = TEMP_MIN + (TEMP_MAX − TEMP_MIN) × confusion` where `confusion = 1 − clamp((short_ema − 0.5)/0.5, 0, 1)` (so dir_acc ≈ 0.5 plateau → temp HIGH/permissive exit ramp; dir_acc → 1.0 → temp LOW/strict informed-commitment), and slow-EMA-blends into slot 460. HIGH T = forgiving (`deficit/(deficit+T)` saturation factor → 0); LOW T = sharp (→ 1) per `compute_min_hold_penalty` at trade_physics.cuh:575. Pearl-A bootstrap (sentinel 50.0 matches the deleted MIN_HOLD_TEMPERATURE_START=50 anchor for bit-identical cold-start under both regimes) + α=0.05 mid-cadence EMA. Bounds [5, 50] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds` (matches the original schedule range). Decouples temperature from epoch number — when WR plateaus, the old schedule pinned LOW temp (sharp) at end of training (max punishment exactly when most needed forgiveness); the signal-driven version reacts to actual realized skill. Per `feedback_no_legacy_aliases.md` the legacy schedule constants + helper function are deleted atomically. Bumped 458 → 459 by Class A audit-fix Batch 4-A (2026-05-08): added 1 adaptive-DD-saturation-floor slot [458..459) — DD_SATURATION_FLOOR_ADAPTIVE (458). Replaces the hardcoded `0.25f` saturation floor in `trade_physics.cuh::apply_margin_cap` (the upper end of the linear position-size scaling ramp dd_scale = max(0.05, 1.0 − dd_frac / saturation_floor)). Distinct semantic role from `SP15_DD_THRESHOLD_INDEX=421` (the SP15 quadratic DD-penalty *trigger* threshold, a *lower* bound). Producer kernel `dd_saturation_floor_update_kernel` aggregates per-env DD_MAX from the existing `dd_state_per_env[env*6 + 1]` tile (same source as the SP15 dd_state_kernel writes to), computes p75 estimator via Welford rule (mean + Z_75 × sigma) × safety_factor=1.5, and slow-EMA blends. Pearl-A bootstrap (sentinel 0.25 matches pre-fix hardcoded value for bit-identical cold-start) + α=0.01 slow EMA (DD distribution is a slow-moving fold-volatility property, shouldn't track per-epoch noise). Bounds [0.10, 0.50] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds`. Atomically deleted alongside this slot addition: legacy `compute_drawdown_penalty` device function in trade_physics.cuh + its single caller at experience_kernels.cu:3822 + the `w_dd` and `dd_threshold` kernel args (Item 2 Case A — SP15's quadratic asymmetric DD penalty in `compute_sp15_final_reward_kernel.cu:154` is the production-grade replacement; layering both creates double-counting per `feedback_no_partial_refactor`). Bumped 454 → 458 by Class A P1-Producer (2026-05-08): added 4 adaptive-Bayesian-prior slots [454..458) — KELLY_PRIOR_WINS (454) + KELLY_PRIOR_LOSSES (455) + KELLY_PRIOR_SUM_WINS (456) + KELLY_PRIOR_SUM_LOSSES (457). Per Class A audit, hardcoded `prior_wins=2.0f / prior_losses=2.0f / prior_sum_wins=0.01f / prior_sum_losses=0.01f` (kelly_cap_update_kernel.cu:39-42 + trade_physics.cuh::kelly_position_cap:304-307) was a static "I don't know" Bayesian prior frozen at sprint-1 defaults — the cold-start of every new fold ignored what prior folds had learned about the realized Kelly distribution. Producer kernel `kelly_bayesian_priors_update_kernel` aggregates realized Kelly stats from the existing `portfolio_state[n_envs, PS_STRIDE]` buffer (same source as kelly_cap_update consumes) and slow-EMA blends into the four slots. Pearl-A bootstrap (sentinels 2.0/2.0/0.01/0.01 match pre-P1-Producer hardcoded values for bit-identical cold-start) + α=0.005 slow EMA (per-fold cadence; the Bayesian prior is a *prior belief* and should change slowly across folds). Bounds counts∈[0.5, 100], sums∈[0.001, 1.0] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds`. Bumped 452 → 454 by Class A P0-A (2026-05-08): added 2 adaptive-reward-cap slots [452..454) — REWARD_POS_CAP_ADAPTIVE (452) + REWARD_NEG_CAP_ADAPTIVE (453). Per Class A audit, hardcoded `REWARD_POS_CAP=+5.0f` / `REWARD_NEG_CAP=-10.0f` (state_layout.cuh:266-267) was structurally clipping the upper tail of realized alpha — controller signals (sharpe EMA, var_q, q_gap) all derive from the CAPPED buffer, so the controller cannot select for trades it cannot see. The state_layout comment explicitly deferred Phase 2 (ISV-driven) "IF Phase 1 validation reveals adaptive need"; 11 superprojects of WR-stuck-at-46-48% revealed adaptive need. Producer kernel `reward_cap_update_kernel` computes p99(winning_returns) × safety_factor=1.5 → POS cap and NEG = -2 × POS (preserves Kahneman 2:1 loss-aversion asymmetry per `pearl_audit_unboundedness_for_implicit_asymmetry` — moved from hardcoded scalar to producer-time multiplier, single source of truth). Pearl-A bootstrap (sentinels 5.0/-10.0) + Wiener-α=0.01 slow EMA. Bounds POS∈[1, 50], NEG∈[-100, -2] are Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds`, NOT tuning. Bumped 450 → 452 by SP14 Layer C Phase C.4b (2026-05-08): added 2 aux-prediction-horizon slots [450..452) — AUX_PRED_HORIZON_BARS (450) + AVG_WIN_HOLD_TIME_BARS (451). Aux's original label was (p_{t+1} > p_t), pure HFT-scale microstructure noise — pivoted to (p_{t+H} > p_t) with H read from ISV[450], adaptively driven from observed avg winning hold time via Pearl-A bootstrap + Wiener-α EMA. Case B: existing `hold_at_exit_per_sample` + `trade_profitable_per_sample` per-sample buffers, no aggregate slot — added new aggregate slot 451 to expose the EMA target. Bumped 444 → 450 by SP14 Layer C Phase C.1 (2026-05-08): atomically deleted 9 α-machinery slots in [385..396) AND added 6 aux-trunk control-plane slots [444..450) — AUX_TRUNK_LR (444) + AUX_TRUNK_BETA1 (445) + AUX_TRUNK_BETA2 (446) + AUX_TRUNK_EPS (447) + AUX_TRUNK_GRAD_CLIP (448) + H_S2_AUX_RMS_EMA (449). Per `feedback_no_legacy_aliases` and `feedback_no_partial_refactor`. Deleted slots left as RESERVED gap (NOT compacted) to preserve checkpoint layout fingerprint compatibility — the surviving SP14 q_disagreement_* diagnostic slots (383, 384, 389) keep their original indices. Pre-C.1 base: 173 + 210 (164 SP5 slots @ 174..278/280..340, with 2-slot gap before cross-fold-persistent Kelly block; Layer D D1 PnL outputs at [286..290); Layer D D2 health composition outputs at [290..294); Layer D D3 training metrics EMA at [294..297); SP7 T1 loss-balance Wiener stats at [297..313); SP7 activation-flag fix per-(head,branch) flags at [313..321); SP8 Fix 36 train_active_frac canary @ [321..322) + LB_MAX_BUDGET per-(head,branch) at [322..330); SP9 Fix 37 Kelly warmup floor at [330..331) + Q_VAR_MAG_EMA at [331..332) + INTENT_EVAL_DIVERGENCE at [332..333) + 3 EMA targets at [333..336) + 3 eval_dist mag bins at [336..339); SP10 Fix 38 EVAL_THOMPSON_TEMP at [339..340); SP11 Fix 39 reward-subsystem controller at [340..367) — 6 component weights [340..346) + 4 controller scalar outputs [346..350) + 2 val-sharpe canaries [350..352) + 6 mag-ratio canaries [352..358) + saboteur engagement [358] + PnL magnitude EMA [359] + popart-component magnitude EMA [360, B1b fix-up] + per-component variance EMAs [361..367, B1b smoke-recovery z-score normalization]; SP13 directional-skill instrumentation at [372..380) — TARGET_DIR_ACC [372] + AUX_DIR_ACC_SHORT/LONG_EMA [373..375) + AUX_DIR_PREDICTION [375] + DIR_SKILL_BONUS_ALPHA/BETA [376..378) + LUCK_WIN_DISCOUNT [378] + SKILL_BONUS_CAP_RATIO [379]; SP13 v3 P0a.T3 Hold-pricing controller at [380..383) — HOLD_COST [380] + HOLD_RATE_TARGET [381] + HOLD_RATE_OBSERVED_EMA [382]; SP14 q_disagreement diagnostic at [383..385), [389] — Q_DISAGREEMENT_SHORT/LONG_EMA + Q_DISAGREEMENT_VARIANCE_EMA (slots [385..389) and [390..396) RESERVED gap from C.1 deletion); SP15 [397..444) per Phase 1; SP14-C [444..450) per Phase C.1. Intentional 5-slot boundary gap at [367..372)) /// Legacy alias preserved for call sites that haven't been audited for the /// network-vs-total split. New code should pick `ISV_NETWORK_DIM` (for weight /// tensor sizing) or `ISV_TOTAL_DIM` (for the broadcast bus buffer). @@ -3659,7 +3659,29 @@ const fn layout_fingerprint_seed() -> &'static [u8] { SLOT_480_V_SHARE_ORD=480;\ SLOT_481_V_SHARE_URG=481;\ SLOT_482_ADVANTAGE_CLIP_BOUND=482;\ - ISV_TOTAL_DIM=483;\ + SLOT_483_HOLD_REWARD_POS_CAP=483;\ + SLOT_484_HOLD_REWARD_NEG_CAP=484;\ + SLOT_485_HOLD_REWARD_DECOMP_DIAG=485;\ + SLOT_486_HOLD_OPP_COST_FIRE_RATE_EMA=486;\ + SLOT_487_HRC_TARGET_MEAN=487;\ + SLOT_488_HRC_TARGET_M2=488;\ + SLOT_489_HRC_DIFF_MEAN=489;\ + SLOT_490_HRC_DIFF_M2=490;\ + SLOT_491_HRC_PREV_TARGET=491;\ + SLOT_492_HRC_SAMPLE_COUNT=492;\ + SLOT_493_TD_ERROR_MAG_EMA=493;\ + SLOT_494_Q_NEXT_TARGET_P99=494;\ + SLOT_495_Q_NEXT_MINUS_REWARD_P99=495;\ + SLOT_496_V_SHARE_TREND_DIAG=496;\ + SLOT_497_POPART_RESET_FLAG=497;\ + SLOT_498_TDB_TARGET_MEAN=498;\ + SLOT_499_TDB_TARGET_M2=499;\ + SLOT_500_TDB_DIFF_MEAN=500;\ + SLOT_501_TDB_DIFF_M2=501;\ + SLOT_502_TDB_PREV_TARGET=502;\ + SLOT_503_TDB_SAMPLE_COUNT=503;\ + SLOT_504_SP18_B_LEG_RESERVED=504;\ + ISV_TOTAL_DIM=505;\ SP14_C_AUX_TRUNK_CONTROL_PLANE=sp14_c_phase_1;\ ALPHA_MACHINERY_DELETED=sp14_c_phase_1;\ TRUNK_INPUT_DD_PCT=sp15_wave_4_1a;\ diff --git a/crates/ml/src/cuda_pipeline/sp14_isv_slots.rs b/crates/ml/src/cuda_pipeline/sp14_isv_slots.rs index 9ad87f45c..32a5f5d9e 100644 --- a/crates/ml/src/cuda_pipeline/sp14_isv_slots.rs +++ b/crates/ml/src/cuda_pipeline/sp14_isv_slots.rs @@ -680,6 +680,157 @@ pub const ADVANTAGE_CLIP_EMA_ALPHA: f32 = 0.01; pub const SP17_DUELING_SLOT_BASE: usize = 474; pub const SP17_DUELING_SLOT_END: usize = 483; +// ── SP18 (2026-05-08) — combined Hold-attractor fix: D-leg + B-leg ───── +// 22 ISV slots [483..505) split across two legs per the v2 combined plan: +// +// D-leg [483..493) — 10 slots (DD6). +// B-leg [493..505) — 12 slots (B-DD13, including 1 RESERVED). +// +// D-leg ("structural Hold opportunity-cost reward") +// ────────────────────────────────────────────────── +// Replaces the SP13 P0a + SP16 P2 + SP16 T3 reactive per-bar Hold-cost +// chain (per DD7=c) with a structural reward at the 3 Hold-cost +// subtraction sites in `experience_kernels.cu`. Each Hold-action bar pays +// `compute_sp18_hold_opportunity_cost(-2 × vol_normalized_next_log_return, +// neg_cap, pos_cap, shaping_scale)` — a counterfactual cost in the +// realised next-bar log-return units, asymmetric-capped via the new +// adaptive producer kernel `hold_reward_cap_update_kernel.cu`. +// +// [483] HOLD_REWARD_POS_CAP upper bound on positive Hold reward +// (Hold during a losing bar) +// [484] HOLD_REWARD_NEG_CAP lower bound on negative Hold reward +// (Hold during a winning bar — opp cost) +// [485] HOLD_REWARD_DECOMP_DIAG HEALTH_DIAG: per-action-bin reward +// decomposition diagnostic (Phase 0) +// [486] HOLD_OPP_COST_FIRE_RATE_EMA fraction of bars with non-zero +// Hold opp-cost reward (engagement +// canary — should be ≈ Hold% itself) +// [487..493) HRC_* Welford accumulators for Wiener-α +// EMA blend (mirrors SP16 T3 HCS_* +// layout — 6 slots) +// +// Cap producer (Phase 3): reads `p99(|step_ret|)` over Long/Short trade +// closes (DD3=b), multiplies by HOLD_REWARD_CAP_SAFETY_FACTOR=1.5, writes +// POS=that and NEG=−2 × POS (DD5=b mirrored asymmetry — same 2:1 +// loss-aversion ratio as SP14 P0-A REWARD_POS/NEG_CAP). Wiener-optimal α +// from the Welford accumulators in [487..493) per +// `pearl_wiener_optimal_adaptive_alpha`. +// +// Pearl-A first-observation bootstrap sentinels POS=5.0 / NEG=−10.0 +// match the position-side SP14 P0-A REWARD_POS/NEG_CAP_ADAPTIVE pattern +// for bit-identical cold-start before the first valid producer launch +// lands a real cap. Bounds [POS_MIN=0.5, POS_MAX=50.0] are Category-1 +// dimensional safety floors per `feedback_isv_for_adaptive_bounds.md`. +// +// SP13 [380..383)/SP16 [461..474) slots remain ALLOCATED but RETIRED per +// PP.4 — sentinel 0.0, no producer launch. Slots stay in the layout +// fingerprint to preserve checkpoint compatibility (RESERVED-gap pattern +// established at SP14-C.1). +// +// Plan: docs/superpowers/plans/2026-05-08-sp18-reward-shape-hold-attractor.md +pub const HOLD_REWARD_POS_CAP_INDEX: usize = 483; +pub const HOLD_REWARD_NEG_CAP_INDEX: usize = 484; +pub const HOLD_REWARD_DECOMP_DIAG_INDEX: usize = 485; +pub const HOLD_OPP_COST_FIRE_RATE_EMA_INDEX: usize = 486; +pub const HRC_TARGET_MEAN_INDEX: usize = 487; // Welford running mean of target_cap +pub const HRC_TARGET_M2_INDEX: usize = 488; // Welford ΣΔ² (sample variance) +pub const HRC_DIFF_MEAN_INDEX: usize = 489; // Welford running mean of consecutive diffs +pub const HRC_DIFF_M2_INDEX: usize = 490; // Welford ΣΔ² for diff series +pub const HRC_PREV_TARGET_INDEX: usize = 491; // previous-epoch target_cap (for diff) +pub const HRC_SAMPLE_COUNT_INDEX: usize = 492; // Welford counter + +// Sentinels — Pearl-A first-observation bootstrap. POS=5.0 / NEG=-10.0 +// match the SP14 P0-A REWARD_POS/NEG_CAP_ADAPTIVE pattern (bit-identical +// cold-start to the position-side adaptive cap). DIAG / FIRE_RATE bootstrap +// to 0.0 — Pearl-A direct-replace on first valid observation. +pub const SENTINEL_HOLD_REWARD_POS_CAP: f32 = 5.0; +pub const SENTINEL_HOLD_REWARD_NEG_CAP: f32 = -10.0; +pub const SENTINEL_HOLD_REWARD_DIAG: f32 = 0.0; +pub const SENTINEL_HOLD_OPP_FIRE_RATE: f32 = 0.0; + +// Category-1 dimensional safety bounds per `feedback_isv_for_adaptive_bounds`. +// Floor 0.5: below this any positive Hold reward is essentially noise; the +// asymmetric ratio NEG=-2×POS would drop NEG below -1, smaller than typical +// per-bar reward magnitudes. Ceiling 50.0: above this a single adversarial +// outlier from the Long/Short close p99 can dominate the EMA. Mirrors the +// SP14 P0-A REWARD_POS_CAP_MIN/MAX pattern. +pub const HOLD_REWARD_POS_CAP_MIN_BOUND: f32 = 0.5; +pub const HOLD_REWARD_POS_CAP_MAX_BOUND: f32 = 50.0; + +// Producer constants. Safety factor 1.5× p99 mirrors the SP14 P0-A +// REWARD_POS_CAP producer pattern. EMA α is Wiener-optimal per +// `pearl_wiener_optimal_adaptive_alpha` (Welford accumulators in +// [487..493)) — no hardcoded constant. +pub const HOLD_REWARD_CAP_SAFETY_FACTOR: f32 = 1.5; + +pub const SP18_HOLD_REWARD_SLOT_BASE: usize = 483; +pub const SP18_HOLD_REWARD_SLOT_END: usize = 493; + +// B-leg ("TD(λ) Q(s') bootstrap") +// ────────────────────────────────── +// Replaces the `q_next = rewards` self-bootstrap at +// `gpu_experience_collector.rs:4143` with a real per-branch DDQN +// `Q_target(s', argmax_a Q_online(s', a))` computed by a target-net +// forward pass on `next_states` at experience-collection time. Reuses +// existing `target_params_buf` (SP14 P4.T1B-iv Polyak-tracked) — no new +// target-net infra (B-DD1). +// +// [493] TD_ERROR_MAG_EMA HEALTH_DIAG: avg(|TD-error|) post-fix +// (B-DD9 ratio gate input) +// [494] Q_NEXT_TARGET_P99 p99 of target-Q bootstrap (bound +// sanity check on the new path) +// [495] Q_NEXT_MINUS_REWARD_P99 p99 of |q_next_target − rewards| +// (sanity: should be O(1) post-fix — +// proves the fix is doing something +// AND not blowing up) +// [496] V_SHARE_TREND_DIAG slope of V_SHARE EMA across last 4 +// epochs (B-leg synergy probe — the +// bootstrap-bias hypothesis predicts +// v_share monotone HD0→HD4) +// [497] POPART_RESET_FLAG 1.0 first epoch post-deployment; +// 0.0 thereafter (B-DD11 — gates the +// per-fold PopArt slot 63 EMA reset +// at the SP18 deployment boundary). +// NOT a FoldReset — host writes 1.0 +// once at deploy, kernel zeroes it +// after consuming. +// [498..504) TDB_* Welford accumulators for Wiener-α +// EMA blend on q_next_target stats +// (mirrors HRC_* layout — 6 slots) +// [504] RESERVED reserved for B-leg follow-up +// +// Pearl-A first-observation bootstrap sentinels match the canonical +// pattern: TD/Q/V_SHARE_TREND diagnostic slots → 0.0; POPART_RESET_FLAG +// → 1.0 (one-shot signalling first SP18-deployment epoch). +// +// Spec §B-DD12 (HEALTH_DIAG observability) + §B-DD13 (slot allocation) +// + §B-DD11 (PopArt reset gate). +pub const TD_ERROR_MAG_EMA_INDEX: usize = 493; +pub const Q_NEXT_TARGET_P99_INDEX: usize = 494; +pub const Q_NEXT_MINUS_REWARD_P99_INDEX: usize = 495; +pub const V_SHARE_TREND_DIAG_INDEX: usize = 496; +pub const POPART_RESET_FLAG_INDEX: usize = 497; +pub const TDB_TARGET_MEAN_INDEX: usize = 498; +pub const TDB_TARGET_M2_INDEX: usize = 499; +pub const TDB_DIFF_MEAN_INDEX: usize = 500; +pub const TDB_DIFF_M2_INDEX: usize = 501; +pub const TDB_PREV_TARGET_INDEX: usize = 502; +pub const TDB_SAMPLE_COUNT_INDEX: usize = 503; +pub const SP18_B_LEG_RESERVED_INDEX: usize = 504; + +// Sentinels — Pearl-A first-observation bootstrap. Diagnostic slots +// (493, 494, 495, 496) bootstrap to 0.0 — Pearl-A direct-replace on +// first valid observation. POPART_RESET_FLAG=1.0 (one-shot — gates the +// per-fold PopArt slot 63 EMA reset on the first SP18 epoch per B-DD11). +pub const SENTINEL_TD_ERROR_MAG_EMA: f32 = 0.0; +pub const SENTINEL_Q_NEXT_TARGET_P99: f32 = 0.0; +pub const SENTINEL_Q_NEXT_MINUS_REWARD_P99: f32 = 0.0; +pub const SENTINEL_V_SHARE_TREND_DIAG: f32 = 0.0; +pub const SENTINEL_POPART_RESET_FLAG: f32 = 1.0; // one-shot ⇒ PopArt reset at first SP18 epoch + +pub const SP18_TD_BOOTSTRAP_SLOT_BASE: usize = 493; +pub const SP18_TD_BOOTSTRAP_SLOT_END: usize = 505; + #[cfg(test)] mod tests { use super::*; @@ -1057,4 +1208,76 @@ mod tests { SP17_DUELING_SLOT_END, ISV_TOTAL_DIM, ); } + + /// Lock SP18 (2026-05-08 v2 combined) D-leg + B-leg slot layout. + /// 22 contiguous slots [483..505): 10 D-leg + 12 B-leg. Per + /// `feedback_no_partial_refactor`, both legs share an ISV section + /// and a single layout fingerprint bump — combined SP18 lands all 22 + /// slots atomically in Pre-Phase before Phase 0+ wiring. + #[test] + fn sp18_combined_slot_layout_locked() { + // ── D-leg [483..493) ───────────────────────────────────────── + assert_eq!(SP18_HOLD_REWARD_SLOT_BASE, 483); + assert_eq!(SP18_HOLD_REWARD_SLOT_END, 493); + assert_eq!(HOLD_REWARD_POS_CAP_INDEX, 483); + assert_eq!(HOLD_REWARD_NEG_CAP_INDEX, 484); + assert_eq!(HOLD_REWARD_DECOMP_DIAG_INDEX, 485); + assert_eq!(HOLD_OPP_COST_FIRE_RATE_EMA_INDEX, 486); + assert_eq!(HRC_TARGET_MEAN_INDEX, 487); + assert_eq!(HRC_TARGET_M2_INDEX, 488); + assert_eq!(HRC_DIFF_MEAN_INDEX, 489); + assert_eq!(HRC_DIFF_M2_INDEX, 490); + assert_eq!(HRC_PREV_TARGET_INDEX, 491); + assert_eq!(HRC_SAMPLE_COUNT_INDEX, 492); + + // ── B-leg [493..505) ───────────────────────────────────────── + assert_eq!(SP18_TD_BOOTSTRAP_SLOT_BASE, 493); + assert_eq!(SP18_TD_BOOTSTRAP_SLOT_END, 505); + assert_eq!(TD_ERROR_MAG_EMA_INDEX, 493); + assert_eq!(Q_NEXT_TARGET_P99_INDEX, 494); + assert_eq!(Q_NEXT_MINUS_REWARD_P99_INDEX, 495); + assert_eq!(V_SHARE_TREND_DIAG_INDEX, 496); + assert_eq!(POPART_RESET_FLAG_INDEX, 497); + assert_eq!(TDB_TARGET_MEAN_INDEX, 498); + assert_eq!(TDB_TARGET_M2_INDEX, 499); + assert_eq!(TDB_DIFF_MEAN_INDEX, 500); + assert_eq!(TDB_DIFF_M2_INDEX, 501); + assert_eq!(TDB_PREV_TARGET_INDEX, 502); + assert_eq!(TDB_SAMPLE_COUNT_INDEX, 503); + assert_eq!(SP18_B_LEG_RESERVED_INDEX, 504); + + // ── Sentinels — Pearl-A first-observation bootstrap (D-leg) ── + assert_eq!(SENTINEL_HOLD_REWARD_POS_CAP, 5.0); + assert_eq!(SENTINEL_HOLD_REWARD_NEG_CAP, -10.0); + assert_eq!(SENTINEL_HOLD_REWARD_DIAG, 0.0); + assert_eq!(SENTINEL_HOLD_OPP_FIRE_RATE, 0.0); + + // ── Sentinels — B-leg ───────────────────────────────────────── + assert_eq!(SENTINEL_TD_ERROR_MAG_EMA, 0.0); + assert_eq!(SENTINEL_Q_NEXT_TARGET_P99, 0.0); + assert_eq!(SENTINEL_Q_NEXT_MINUS_REWARD_P99, 0.0); + assert_eq!(SENTINEL_V_SHARE_TREND_DIAG, 0.0); + assert_eq!(SENTINEL_POPART_RESET_FLAG, 1.0); // 1.0 first epoch ⇒ PopArt reset (B-DD11) + + // ── Bounds (D-leg, Category-1 floors) ──────────────────────── + assert_eq!(HOLD_REWARD_POS_CAP_MIN_BOUND, 0.5); + assert_eq!(HOLD_REWARD_POS_CAP_MAX_BOUND, 50.0); + + // ── Producer constants ─────────────────────────────────────── + assert_eq!(HOLD_REWARD_CAP_SAFETY_FACTOR, 1.5); + } + + /// Asserts SP18 combined slot range fits within ISV_TOTAL_DIM. + /// Mirrors the SP17 / SP14 / SP5 pattern. The contract test fails + /// with a clear actionable message before any consumer can drift. + #[test] + fn all_sp18_slots_fit_within_isv_total_dim() { + use crate::cuda_pipeline::gpu_dqn_trainer::ISV_TOTAL_DIM; + assert!( + SP18_TD_BOOTSTRAP_SLOT_END <= ISV_TOTAL_DIM, + "SP18_TD_BOOTSTRAP_SLOT_END={} exceeds ISV_TOTAL_DIM={}; \ + bump ISV_TOTAL_DIM in gpu_dqn_trainer.rs (and update layout_fingerprint_seed()).", + SP18_TD_BOOTSTRAP_SLOT_END, ISV_TOTAL_DIM, + ); + } } diff --git a/crates/ml/src/cuda_pipeline/state_layout.cuh b/crates/ml/src/cuda_pipeline/state_layout.cuh index eda90fe58..f0bf09e27 100644 --- a/crates/ml/src/cuda_pipeline/state_layout.cuh +++ b/crates/ml/src/cuda_pipeline/state_layout.cuh @@ -469,6 +469,92 @@ // See the SP16 T3 SLOTS section above for the slot indices. #define DIR_ACC_RANDOM_BASELINE 0.5f +// ── SP18 (2026-05-08) — combined Hold-attractor fix: D-leg + B-leg ──── +// 22 ISV slots [483..505) split across two atomic legs per the v2 +// combined plan. Mirrors `crates/ml/src/cuda_pipeline/sp14_isv_slots.rs` +// constants of the same names (locked by `sp18_combined_slot_layout_locked` +// test). Like SP16/SP17 mirrors, the C-side header tracks slot indices +// + sentinels + bounds for kernels that read the bus directly via raw +// indices; the Rust constant is the single source of truth (see +// ISV_TOTAL_DIM macro note above). +// +// D-leg ("structural Hold opportunity-cost reward") replaces the SP13 +// P0a + SP16 P2 + SP16 T3 reactive per-bar Hold-cost chain (DD7=c) with +// a structural reward at the 3 Hold-cost subtraction sites in +// experience_kernels.cu via the new device function +// `compute_sp18_hold_opportunity_cost` (added to trade_physics.cuh in +// Phase 2). HOLD_REWARD_POS_CAP / HOLD_REWARD_NEG_CAP are adaptive bounds +// producer-tracked from p99(|step_ret|) over Long/Short trade closes × +// HOLD_REWARD_CAP_SAFETY_FACTOR=1.5 (DD3=b) with NEG = -2 × POS +// (DD5=b mirrored loss-aversion). +// +// B-leg ("TD(λ) Q(s') bootstrap") replaces the q_next = rewards +// self-bootstrap at gpu_experience_collector.rs:4143 with a real +// per-branch DDQN `Q_target(s', argmax_a Q_online(s', a))` computed via +// target-net forward pass on next_states at experience-collection time +// (Phase 4-5). All B-leg slots are observability-only (HEALTH_DIAG +// diagnostics) plus the one-shot POPART_RESET_FLAG that gates the +// per-fold PopArt EMA reset at the SP18 deployment boundary (B-DD11). +// +// Plan: docs/superpowers/plans/2026-05-08-sp18-reward-shape-hold-attractor.md + +// D-leg slots [483..493) — 10 slots +#define HOLD_REWARD_POS_CAP_INDEX 483 +#define HOLD_REWARD_NEG_CAP_INDEX 484 +#define HOLD_REWARD_DECOMP_DIAG_INDEX 485 +#define HOLD_OPP_COST_FIRE_RATE_EMA_INDEX 486 +#define HRC_TARGET_MEAN_INDEX 487 +#define HRC_TARGET_M2_INDEX 488 +#define HRC_DIFF_MEAN_INDEX 489 +#define HRC_DIFF_M2_INDEX 490 +#define HRC_PREV_TARGET_INDEX 491 +#define HRC_SAMPLE_COUNT_INDEX 492 + +// D-leg sentinels — Pearl-A first-observation bootstrap. POS=5.0 / +// NEG=-10.0 match the SP14 P0-A REWARD_POS/NEG_CAP_ADAPTIVE pattern +// (bit-identical cold-start to the position-side adaptive cap). +#define SENTINEL_HOLD_REWARD_POS_CAP 5.0f +#define SENTINEL_HOLD_REWARD_NEG_CAP -10.0f +#define SENTINEL_HOLD_REWARD_DIAG 0.0f +#define SENTINEL_HOLD_OPP_FIRE_RATE 0.0f + +// D-leg defensive consumer-side bounds (mirror Rust constants +// HOLD_REWARD_POS_CAP_MIN/MAX_BOUND in sp14_isv_slots.rs). Values +// outside this window fall back to sentinels on the consumer side as a +// defense against malformed prior state. Producer kernel clamps to the +// same range, so a healthy ISV slot will always pass the guard. +#define HOLD_REWARD_POS_CAP_MIN_BOUND 0.5f +#define HOLD_REWARD_POS_CAP_MAX_BOUND 50.0f + +// D-leg producer constant. Safety factor 1.5× p99 mirrors the SP14 P0-A +// REWARD_POS_CAP producer pattern. +#define HOLD_REWARD_CAP_SAFETY_FACTOR 1.5f + +// B-leg slots [493..505) — 12 slots +#define TD_ERROR_MAG_EMA_INDEX 493 +#define Q_NEXT_TARGET_P99_INDEX 494 +#define Q_NEXT_MINUS_REWARD_P99_INDEX 495 +#define V_SHARE_TREND_DIAG_INDEX 496 +#define POPART_RESET_FLAG_INDEX 497 +#define TDB_TARGET_MEAN_INDEX 498 +#define TDB_TARGET_M2_INDEX 499 +#define TDB_DIFF_MEAN_INDEX 500 +#define TDB_DIFF_M2_INDEX 501 +#define TDB_PREV_TARGET_INDEX 502 +#define TDB_SAMPLE_COUNT_INDEX 503 +#define SP18_B_LEG_RESERVED_INDEX 504 + +// B-leg sentinels — Pearl-A first-observation bootstrap. Diagnostic +// slots bootstrap to 0.0 (direct-replace on first valid observation). +// POPART_RESET_FLAG sentinel = 1.0 (one-shot — gates the per-fold +// PopArt slot 63 EMA reset on the first SP18 epoch per B-DD11; host +// writes 1.0 at deploy, kernel zeroes after consuming). +#define SENTINEL_TD_ERROR_MAG_EMA 0.0f +#define SENTINEL_Q_NEXT_TARGET_P99 0.0f +#define SENTINEL_Q_NEXT_MINUS_REWARD_P99 0.0f +#define SENTINEL_V_SHARE_TREND_DIAG 0.0f +#define SENTINEL_POPART_RESET_FLAG 1.0f + // ── Compile-time checks ── static_assert(SL_PADDING_START + SL_PADDING_DIM == SL_STATE_DIM, "State layout dimensions must sum to SL_STATE_DIM"); diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index 2a070dea3..ea3ff6f5b 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -2,6 +2,46 @@ **Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7. +## 2026-05-08 — SP18 v2 Pre-Phase Task PP.2: 22 ISV slots [483..505) D-leg + B-leg + +Pre-Phase scaffolding for the combined SP18 v2 plan: allocates 22 ISV slots split across two atomic legs per `feedback_no_partial_refactor`. + +**D-leg [483..493) — 10 slots** ("structural Hold opportunity-cost reward"): + +| Slot | Name | Sentinel | Notes | +|---|---|---|---| +| 483 | `HOLD_REWARD_POS_CAP` | 5.0 | Pearl-A; matches SP14 P0-A REWARD_POS_CAP for bit-identical cold-start | +| 484 | `HOLD_REWARD_NEG_CAP` | -10.0 | Pearl-A; -2× POS preserves Kahneman 2:1 loss-aversion ratio | +| 485 | `HOLD_REWARD_DECOMP_DIAG` | 0.0 | HEALTH_DIAG observability — Phase 0 reward decomposition | +| 486 | `HOLD_OPP_COST_FIRE_RATE_EMA` | 0.0 | Engagement canary — should be ≈ Hold% itself | +| 487..493 | 6 `HRC_*` Welford accumulators | 0.0 | Mirrors SP16 T3 `HCS_*` Wiener-α layout | + +**B-leg [493..505) — 12 slots** ("TD(λ) Q(s') bootstrap"): + +| Slot | Name | Sentinel | Notes | +|---|---|---|---| +| 493 | `TD_ERROR_MAG_EMA` | 0.0 | HEALTH_DIAG observability — B-DD9 ratio-gate input (`avg(\|TD-error\|)`) | +| 494 | `Q_NEXT_TARGET_P99` | 0.0 | bound check on the new target-Q bootstrap | +| 495 | `Q_NEXT_MINUS_REWARD_P99` | 0.0 | sanity: should be O(1) post-fix (proves the fix is doing something AND not blowing up) | +| 496 | `V_SHARE_TREND_DIAG` | 0.0 | B-leg synergy probe (slope of V_SHARE EMA across last 4 epochs) | +| 497 | `POPART_RESET_FLAG` | **1.0** | one-shot, NOT a FoldReset; gates per-fold PopArt slot 63 EMA reset at SP18 deployment boundary per B-DD11 | +| 498..504 | 6 `TDB_*` Welford accumulators | 0.0 | Mirrors `HRC_*` Wiener-α layout | +| 504 | `RESERVED` | 0.0 | reserved for B-leg follow-up | + +**Pearl pattern**: D-leg POS/NEG cap sentinels match the position-side SP14 P0-A `REWARD_POS_CAP_ADAPTIVE` pattern (5.0 / -10.0) for bit-identical cold-start before the cap producer (Phase 3) lands a real `p99(|step_ret|) × 1.5`. The B-leg POPART_RESET_FLAG=1.0 is the only non-zero sentinel — host writes 1.0 once at the SP18 deployment boundary, kernel zeroes after consuming (one-shot signalling, NOT a FoldReset). + +**Bounds**: D-leg POS cap clamped to `[0.5, 50.0]` Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds`. B-leg slots are observability-only (no bounds enforced — diagnostic readout only). + +**ISV_TOTAL_DIM**: 483 → 505. `layout_fingerprint_seed` extended with 22 new `SLOT_*` entries. `state_layout.cuh` mirror added (continues SP14 P0-A/P1/audit-fix-4A/4B precedent — SP16/SP17 slots intentionally not mirrored, only SP18 gets fresh `#define`s for the device-side header). + +**Atomicity envelope**: Pre-Phase scaffolding only — 22 ISV slots locked, fingerprint extended, mirror constants in C header. NO producer kernel, NO consumer wire-in, NO HEALTH_DIAG emit. Phase 0 (diagnostic emit), Phases 1–7 (consumer/producer), Phases 8–10 (validation) follow per the v2 plan. + +**SP13/SP16 retirement**: Slots [380..383) (HOLD_COST + HOLD_RATE_TARGET + HOLD_RATE_OBSERVED_EMA) and [461..474) (HOLD_COST_SCALE + Welford accumulators) remain ALLOCATED but will be RETIRED in PP.4 (sentinel 0.0, no producer launch) per the SP14-C.1 RESERVED-gap pattern — preserves checkpoint compatibility while DD7=c replaces the reactive per-bar Hold-cost chain with the structural opportunity-cost reward. + +**Plan**: `docs/superpowers/plans/2026-05-08-sp18-reward-shape-hold-attractor.md` § Pre-Phase Task PP.2. + +**Spec**: `docs/superpowers/specs/2026-05-08-sp18-reward-shape-hold-attractor-design.md` § DD6 + B-DD13. + ## 2026-05-08 — SP17 Phase 3.3: Phase 3 close-out + memory pearl Phase 3.3 finalises the SP17 dueling-Q diagnostic chain: