From 84de278dfee5b1d12f480a2eaabc42e9e6213c19 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Tue, 5 May 2026 18:51:00 +0200 Subject: [PATCH] =?UTF-8?q?feat(sp14):=20B.2=20=E2=80=94=20register=2011?= =?UTF-8?q?=20SP14=20ISV=20slots=20for=20fold-boundary=20reset?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Each EGF pearl EMA / state slot resets to its Pearl-A sentinel at fold boundary, mirroring sp13_aux_dir_acc_short_ema / long_ema entries. Atomic refactor (feedback_no_partial_refactor): both halves land together — registry entry + reset_named_state dispatch arm. Reset slots (11 total, sentinel in parens): - Q_DISAGREEMENT_SHORT/LONG_EMA (slots 383, 384) → 0.5 - K_AUX_ADAPTIVE (385) → K_BASE_AUX = 20.0 - K_Q_ADAPTIVE (386) → K_BASE_Q = 15.0 - BETA_RATE_LIMITER_ADAPTIVE (387) → BETA_BASE = 0.5 - AUX_DIR_ACC_VARIANCE_EMA, Q_DISAGREEMENT_VARIANCE_EMA, ALPHA_GRAD_RAW_VARIANCE_EMA (388, 389, 390) → 0.0 (initial k = k_base, β = β_base via ISV-driven controllers) - GATE1_OPEN_STATE (391) → 0.0 (closed) - ALPHA_GRAD_SMOOTHED (393) → 0.0 - AUX_DIR_ACC_POST_OPEN_MIN (394) → 1.0 (no min observed) ALPHA_GRAD_RAW (slot 392, recomputed every step from variance EMAs) and GRADIENT_HACK_LOCKOUT_REMAINING (slot 395, decays at epoch boundary) are NOT in the fold-reset registry; both naturally re-initialise without explicit reset. Also corrects the isv-slots.md SP14 table: slots 392 and 395 were incorrectly marked FoldReset in the B.1 entry; corrected to reflect their actual reset semantics (NOT reset / epoch-boundary decay). Producer + consumer wiring lands in subsequent tasks (B.3-B.12); this commit is additive infrastructure only — no behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../src/trainers/dqn/state_reset_registry.rs | 67 ++++++++++ .../src/trainers/dqn/trainer/training_loop.rs | 126 ++++++++++++++++++ docs/isv-slots.md | 16 ++- 3 files changed, 204 insertions(+), 5 deletions(-) diff --git a/crates/ml/src/trainers/dqn/state_reset_registry.rs b/crates/ml/src/trainers/dqn/state_reset_registry.rs index 6f85c67a3..0789a5a87 100644 --- a/crates/ml/src/trainers/dqn/state_reset_registry.rs +++ b/crates/ml/src/trainers/dqn/state_reset_registry.rs @@ -938,6 +938,73 @@ impl StateResetRegistry { category: ResetCategory::FoldReset, description: "ISV[HOLD_RATE_OBSERVED_EMA_INDEX=382] — SP13 v3 P0a.T3 per-fold EMA of the per-step Hold-pick rate `count(action_dir == DIR_HOLD) / batch_size`. Produced by `apply_fixed_alpha_ema_kernel` chained off `hold_rate_observer_kernel` (added in P0a.T3). Consumed by the host-side Hold-cost controller in `training_loop.rs`: `excess = max(0, observed − target); hold_cost = HOLD_COST_BASE × (1 + 5 × excess)` clamped to `[0.5×base, 5.0×base]`, written to ISV[HOLD_COST_INDEX=380]. The reward-composition site in `experience_kernels.cu` then subtracts the cost on every Hold-action bar — pricing the action so the policy uses Hold deliberately rather than as a free CQL-bias-anchored default. FoldReset sentinel 0.0 so Pearl A bootstraps from the new fold's first observation; the controller relaxes back to baseline cost (HOLD_COST_BASE × 0.5 floor) when observed ≤ target. Spec §'Change 1: Price Hold (replaces v2's Eliminate Hold)'.", }, + // ── SP14 fold-reset slots [383..396) ───────────────────────────── + // 11 of the 13 allocated slots reset at fold boundary; the two + // exceptions are: + // - ALPHA_GRAD_RAW_INDEX=392: recomputed every step from the + // three Welford variance EMAs (388, 389, 390); no stale value + // persists past the first step of a new fold. + // - GRADIENT_HACK_LOCKOUT_REMAINING_INDEX=395: epoch-boundary + // decay; the lockout counter naturally reaches 0 at the start + // of any new epoch/fold and must NOT be reset here (doing so + // would silently abort an active lockout mid-fold). + // Both naturally re-initialise without explicit reset. + // Spec: docs/superpowers/specs/2026-05-05-sp14-aux-q-wire-earned-gradient-flow.md + RegistryEntry { + name: "sp14_q_disagreement_short_ema", + category: ResetCategory::FoldReset, + description: "ISV[Q_DISAGREEMENT_SHORT_EMA_INDEX=383] — SP14 EGF Gate 2 fast EMA (α=0.3) of per-step K=4↔K=2 mapped argmax disagreement with Hold/Flat masked. Produced by `q_disagreement_update_kernel` (B.3) every training step. Consumed by `alpha_grad_compute_kernel` (B.4) as the Gate 2 sigmoid driver. FoldReset sentinel SENTINEL_Q_DISAGREEMENT=0.5 (analytic random alignment under masking) so Pearl A bootstraps from the new fold's first observation per `pearl_first_observation_bootstrap.md` — the EMA tracks the new fold's Q-head signal without bias from the previous fold's terminal value. Spec: docs/superpowers/specs/2026-05-05-sp14-aux-q-wire-earned-gradient-flow.md.", + }, + RegistryEntry { + name: "sp14_q_disagreement_long_ema", + category: ResetCategory::FoldReset, + description: "ISV[Q_DISAGREEMENT_LONG_EMA_INDEX=384] — SP14 EGF Gate 2 slow EMA (α=0.05) of per-step K=4↔K=2 mapped argmax disagreement with Hold/Flat masked. Produced by `q_disagreement_update_kernel` (B.3) every training step. Used alongside the short EMA for stagnation detection in the Gate 2 controller (persistent high disagreement despite training ↔ Q head stuck, not just noisy). FoldReset sentinel SENTINEL_Q_DISAGREEMENT=0.5 so Pearl A bootstraps from the new fold's first observation per `pearl_first_observation_bootstrap.md`. Spec: docs/superpowers/specs/2026-05-05-sp14-aux-q-wire-earned-gradient-flow.md.", + }, + RegistryEntry { + name: "sp14_k_aux_adaptive", + category: ResetCategory::FoldReset, + description: "ISV[K_AUX_ADAPTIVE_INDEX=385] — SP14 ISV-driven adaptive sigmoid steepness for Gate 1 (aux competence). Driven by AUX_DIR_ACC_VARIANCE_EMA_INDEX=388 via `k_aux = K_BASE_AUX × exp(-variance / VARIANCE_REF_AUX)` clamped to [K_MIN, K_BASE_AUX]; high aux variance → flatter sigmoid → more cautious gate opening. FoldReset sentinel K_BASE_AUX=20.0 (Invariant-1 anchor: start each fold with maximum steepness, decay only if the new fold's signal is noisy). Produced by `alpha_grad_compute_kernel` (B.4); consumed by Gate 1 sigmoid. Spec: docs/superpowers/specs/2026-05-05-sp14-aux-q-wire-earned-gradient-flow.md.", + }, + RegistryEntry { + name: "sp14_k_q_adaptive", + category: ResetCategory::FoldReset, + description: "ISV[K_Q_ADAPTIVE_INDEX=386] — SP14 ISV-driven adaptive sigmoid steepness for Gate 2 (Q-head disagreement). Driven by Q_DISAGREEMENT_VARIANCE_EMA_INDEX=389 via `k_q = K_BASE_Q × exp(-variance / VARIANCE_REF_Q)` clamped to [K_MIN, K_BASE_Q]; high Q-disagreement variance → flatter sigmoid → more cautious gate modulation. FoldReset sentinel K_BASE_Q=15.0 (Invariant-1 anchor: start each fold with maximum steepness). Produced by `alpha_grad_compute_kernel` (B.4); consumed by Gate 2 sigmoid. Spec: docs/superpowers/specs/2026-05-05-sp14-aux-q-wire-earned-gradient-flow.md.", + }, + RegistryEntry { + name: "sp14_beta_rate_limiter_adaptive", + category: ResetCategory::FoldReset, + description: "ISV[BETA_RATE_LIMITER_ADAPTIVE_INDEX=387] — SP14 ISV-driven adaptive rate-limiter β for the α_grad_smoothed EMA. Driven by ALPHA_GRAD_RAW_VARIANCE_EMA_INDEX=390 via `β = BETA_BASE + (BETA_MAX − BETA_BASE) × (1 − exp(-variance / VARIANCE_REF_ALPHA))` so volatile α_grad_raw → heavier smoothing (β → BETA_MAX=0.95) and stable signal → light smoothing (β → BETA_BASE=0.5). FoldReset sentinel BETA_BASE=0.5 (Invariant-1 anchor: start each fold with light smoothing). Produced by `alpha_grad_compute_kernel` (B.4); consumed by the α_grad_smoothed EMA update. Spec: docs/superpowers/specs/2026-05-05-sp14-aux-q-wire-earned-gradient-flow.md.", + }, + RegistryEntry { + name: "sp14_aux_dir_acc_variance_ema", + category: ResetCategory::FoldReset, + description: "ISV[AUX_DIR_ACC_VARIANCE_EMA_INDEX=388] — SP14 Welford variance EMA of per-step aux directional accuracy (Welford one-pass, α=0.05). Drives K_AUX_ADAPTIVE_INDEX=385: low variance → high k_aux (confident gate), high variance → low k_aux (cautious gate). FoldReset sentinel SENTINEL_VARIANCE=0.0 so that k_aux initialises at K_BASE_AUX=20.0 (maximum steepness) at the start of each fold and decays only if the new fold's aux-acc signal is genuinely noisy. Produced by `aux_variance_update_kernel` (B.3). Spec: docs/superpowers/specs/2026-05-05-sp14-aux-q-wire-earned-gradient-flow.md.", + }, + RegistryEntry { + name: "sp14_q_disagreement_variance_ema", + category: ResetCategory::FoldReset, + description: "ISV[Q_DISAGREEMENT_VARIANCE_EMA_INDEX=389] — SP14 Welford variance EMA of per-step Q-disagreement signal (Welford one-pass, α=0.05). Drives K_Q_ADAPTIVE_INDEX=386: low variance → high k_q (confident Gate 2 modulation), high variance → low k_q (cautious). FoldReset sentinel SENTINEL_VARIANCE=0.0 so that k_q initialises at K_BASE_Q=15.0 at the start of each fold. Produced by `q_disagreement_update_kernel` (B.3). Spec: docs/superpowers/specs/2026-05-05-sp14-aux-q-wire-earned-gradient-flow.md.", + }, + RegistryEntry { + name: "sp14_alpha_grad_raw_variance_ema", + category: ResetCategory::FoldReset, + description: "ISV[ALPHA_GRAD_RAW_VARIANCE_EMA_INDEX=390] — SP14 Welford variance EMA of per-step α_grad_raw (Welford one-pass, α=0.05). Drives BETA_RATE_LIMITER_ADAPTIVE_INDEX=387: low α_grad_raw variance → light smoothing (β → BETA_BASE=0.5, responsive), high variance → heavy smoothing (β → BETA_MAX=0.95, stable). FoldReset sentinel SENTINEL_VARIANCE=0.0 so that β initialises at BETA_BASE=0.5 at the start of each fold. Produced by `alpha_grad_compute_kernel` (B.4). Spec: docs/superpowers/specs/2026-05-05-sp14-aux-q-wire-earned-gradient-flow.md.", + }, + RegistryEntry { + name: "sp14_gate1_open_state", + category: ResetCategory::FoldReset, + description: "ISV[GATE1_OPEN_STATE_INDEX=391] — SP14 Schmitt-trigger persistent state for Gate 1 (0.0=closed, 1.0=open). Hysteresis: open when aux_dir_acc crosses target+SCHMITT_BAND=0.03, close when it drops below target-SCHMITT_BAND. Prevents rapid open/close cycling near the threshold. FoldReset sentinel SENTINEL_GATE1_CLOSED=0.0 (gate starts closed at each fold boundary; Gate 1 must re-earn the right to open by demonstrating competence on new fold data before α_grad can be non-zero). Produced and consumed by `alpha_grad_compute_kernel` (B.4). Spec: docs/superpowers/specs/2026-05-05-sp14-aux-q-wire-earned-gradient-flow.md.", + }, + RegistryEntry { + name: "sp14_alpha_grad_smoothed", + category: ResetCategory::FoldReset, + description: "ISV[ALPHA_GRAD_SMOOTHED_INDEX=393] — SP14 rate-limited (β-smoothed) α_grad, the actual scalar consumed by the backward path to modulate aux→Q gradient flow. Updated each step as `α_smoothed = β × α_smoothed + (1−β) × α_raw` where β = BETA_RATE_LIMITER_ADAPTIVE_INDEX=387. FoldReset sentinel SENTINEL_ALPHA_GRAD=0.0 (no gradient flow at fold start; the smoothed gate must re-ramp from zero so the policy's early fold exploration is not contaminated by the previous fold's terminal gate state). Produced by `alpha_grad_compute_kernel` (B.4); consumed by backward-pass gradient scaling (B.10). Spec: docs/superpowers/specs/2026-05-05-sp14-aux-q-wire-earned-gradient-flow.md.", + }, + RegistryEntry { + name: "sp14_aux_dir_acc_post_open_min", + category: ResetCategory::FoldReset, + description: "ISV[AUX_DIR_ACC_POST_OPEN_MIN_INDEX=394] — SP14 anti-gradient-hacking circuit breaker: running minimum of aux_dir_acc since Gate 1 last opened. Initialised to 1.0 (no minimum observed) when Gate 1 opens; updated each step post-open as `min = fminf(min, current_aux_dir_acc)`. If `open_min − current_aux_dir_acc > LOCKOUT_TRIGGER_DROP=0.05` while q_disagreement also rises by >LOCKOUT_TRIGGER_DIS_RISE=0.10, the lockout circuit fires (B.5) and writes LOCKOUT_EPOCHS=2.0 to GRADIENT_HACK_LOCKOUT_REMAINING_INDEX=395. FoldReset sentinel SENTINEL_POST_OPEN_MIN=1.0 (no minimum observed at fold start; the circuit breaker must accumulate post-open observations before it can fire). Produced by `alpha_grad_compute_kernel` (B.4). Spec: docs/superpowers/specs/2026-05-05-sp14-aux-q-wire-earned-gradient-flow.md.", + }, // SP5 Task A1: Wiener-state companion reset. The wiener_state_buf // covers SP4 (SP4_PRODUCER_COUNT=71 producers × 3 = 213 floats) + // SP5 (SP5_PRODUCER_COUNT × 3 floats) = (71 + SP5_PRODUCER_COUNT) × 3 diff --git a/crates/ml/src/trainers/dqn/trainer/training_loop.rs b/crates/ml/src/trainers/dqn/trainer/training_loop.rs index 310bf2e15..cef4421b1 100644 --- a/crates/ml/src/trainers/dqn/trainer/training_loop.rs +++ b/crates/ml/src/trainers/dqn/trainer/training_loop.rs @@ -7784,6 +7784,132 @@ impl DQNTrainer { ); } } + // SP14 (2026-05-05): EGF pearl fold-reset arms — 11 slots. + // Slots 392 (ALPHA_GRAD_RAW) and 395 (GRADIENT_HACK_LOCKOUT_REMAINING) + // are intentionally absent: raw is recomputed every step from the + // three Welford variance EMAs; lockout decays at epoch boundary and + // must NOT be force-reset mid-fold. + "sp14_q_disagreement_short_ema" => { + if let Some(ref fused) = self.fused_ctx { + use crate::cuda_pipeline::sp14_isv_slots::{ + Q_DISAGREEMENT_SHORT_EMA_INDEX, SENTINEL_Q_DISAGREEMENT, + }; + fused.trainer().write_isv_signal_at( + Q_DISAGREEMENT_SHORT_EMA_INDEX, + SENTINEL_Q_DISAGREEMENT, + ); + } + } + "sp14_q_disagreement_long_ema" => { + if let Some(ref fused) = self.fused_ctx { + use crate::cuda_pipeline::sp14_isv_slots::{ + Q_DISAGREEMENT_LONG_EMA_INDEX, SENTINEL_Q_DISAGREEMENT, + }; + fused.trainer().write_isv_signal_at( + Q_DISAGREEMENT_LONG_EMA_INDEX, + SENTINEL_Q_DISAGREEMENT, + ); + } + } + "sp14_k_aux_adaptive" => { + if let Some(ref fused) = self.fused_ctx { + use crate::cuda_pipeline::sp14_isv_slots::{ + K_AUX_ADAPTIVE_INDEX, K_BASE_AUX, + }; + fused.trainer().write_isv_signal_at( + K_AUX_ADAPTIVE_INDEX, + K_BASE_AUX, + ); + } + } + "sp14_k_q_adaptive" => { + if let Some(ref fused) = self.fused_ctx { + use crate::cuda_pipeline::sp14_isv_slots::{ + K_Q_ADAPTIVE_INDEX, K_BASE_Q, + }; + fused.trainer().write_isv_signal_at( + K_Q_ADAPTIVE_INDEX, + K_BASE_Q, + ); + } + } + "sp14_beta_rate_limiter_adaptive" => { + if let Some(ref fused) = self.fused_ctx { + use crate::cuda_pipeline::sp14_isv_slots::{ + BETA_RATE_LIMITER_ADAPTIVE_INDEX, BETA_BASE, + }; + fused.trainer().write_isv_signal_at( + BETA_RATE_LIMITER_ADAPTIVE_INDEX, + BETA_BASE, + ); + } + } + "sp14_aux_dir_acc_variance_ema" => { + if let Some(ref fused) = self.fused_ctx { + use crate::cuda_pipeline::sp14_isv_slots::{ + AUX_DIR_ACC_VARIANCE_EMA_INDEX, SENTINEL_VARIANCE, + }; + fused.trainer().write_isv_signal_at( + AUX_DIR_ACC_VARIANCE_EMA_INDEX, + SENTINEL_VARIANCE, + ); + } + } + "sp14_q_disagreement_variance_ema" => { + if let Some(ref fused) = self.fused_ctx { + use crate::cuda_pipeline::sp14_isv_slots::{ + Q_DISAGREEMENT_VARIANCE_EMA_INDEX, SENTINEL_VARIANCE, + }; + fused.trainer().write_isv_signal_at( + Q_DISAGREEMENT_VARIANCE_EMA_INDEX, + SENTINEL_VARIANCE, + ); + } + } + "sp14_alpha_grad_raw_variance_ema" => { + if let Some(ref fused) = self.fused_ctx { + use crate::cuda_pipeline::sp14_isv_slots::{ + ALPHA_GRAD_RAW_VARIANCE_EMA_INDEX, SENTINEL_VARIANCE, + }; + fused.trainer().write_isv_signal_at( + ALPHA_GRAD_RAW_VARIANCE_EMA_INDEX, + SENTINEL_VARIANCE, + ); + } + } + "sp14_gate1_open_state" => { + if let Some(ref fused) = self.fused_ctx { + use crate::cuda_pipeline::sp14_isv_slots::{ + GATE1_OPEN_STATE_INDEX, SENTINEL_GATE1_CLOSED, + }; + fused.trainer().write_isv_signal_at( + GATE1_OPEN_STATE_INDEX, + SENTINEL_GATE1_CLOSED, + ); + } + } + "sp14_alpha_grad_smoothed" => { + if let Some(ref fused) = self.fused_ctx { + use crate::cuda_pipeline::sp14_isv_slots::{ + ALPHA_GRAD_SMOOTHED_INDEX, SENTINEL_ALPHA_GRAD, + }; + fused.trainer().write_isv_signal_at( + ALPHA_GRAD_SMOOTHED_INDEX, + SENTINEL_ALPHA_GRAD, + ); + } + } + "sp14_aux_dir_acc_post_open_min" => { + if let Some(ref fused) = self.fused_ctx { + use crate::cuda_pipeline::sp14_isv_slots::{ + AUX_DIR_ACC_POST_OPEN_MIN_INDEX, SENTINEL_POST_OPEN_MIN, + }; + fused.trainer().write_isv_signal_at( + AUX_DIR_ACC_POST_OPEN_MIN_INDEX, + SENTINEL_POST_OPEN_MIN, + ); + } + } // SP11 Task A2 (2026-05-04): novelty visit-count hash table // reset arm — closes the A0 deferral per the registry entry's // docstring. 1M f32 slots, zero-filled via mapped-pinned host diff --git a/docs/isv-slots.md b/docs/isv-slots.md index d21ad6013..90b7e9780 100644 --- a/docs/isv-slots.md +++ b/docs/isv-slots.md @@ -371,16 +371,22 @@ SP14 starts at 383 (shifted +2 from the original plan which documented | 389 | `Q_DISAGREEMENT_VARIANCE_EMA_INDEX` | FoldReset (0.0) | Welford variance for k_q adaptation | | 390 | `ALPHA_GRAD_RAW_VARIANCE_EMA_INDEX` | FoldReset (0.0) | Welford variance for β adaptation | | 391 | `GATE1_OPEN_STATE_INDEX` | FoldReset (0.0) | Schmitt-trigger state (0=closed, 1=open) | -| 392 | `ALPHA_GRAD_RAW_INDEX` | FoldReset (0.0) | Raw gate output (HEALTH_DIAG visibility) | +| 392 | `ALPHA_GRAD_RAW_INDEX` | NOT reset (recomputed every step) | Raw gate output (HEALTH_DIAG visibility) | | 393 | `ALPHA_GRAD_SMOOTHED_INDEX` | FoldReset (0.0) | Rate-limited gate consumed by backward path | | 394 | `AUX_DIR_ACC_POST_OPEN_MIN_INDEX` | FoldReset (1.0) | Post-open aux_dir_acc minimum (circuit breaker) | -| 395 | `GRADIENT_HACK_LOCKOUT_REMAINING_INDEX` | FoldReset (0.0) | Lockout steps remaining (anti-gradient-hacking) | +| 395 | `GRADIENT_HACK_LOCKOUT_REMAINING_INDEX` | NOT reset (epoch-boundary decay) | Lockout epochs remaining (anti-gradient-hacking) | Slots 396-398 are conceptual buffer; no constants allocated. -All 13 slots are FoldReset per `pearl_first_observation_bootstrap.md`. -Producer + consumer wiring lands in B.2-B.12; B.1 is additive -infrastructure only (no behavior change at this commit). +11 of the 13 slots are FoldReset per `pearl_first_observation_bootstrap.md`. +Slots 392 (ALPHA_GRAD_RAW) and 395 (GRADIENT_HACK_LOCKOUT_REMAINING) are +intentionally excluded from fold-reset: raw is recomputed every step from +the three Welford variance EMAs; lockout decays at epoch boundary and must +not be force-reset mid-fold. + +Producer + consumer wiring lands in B.3-B.12; B.1 is additive +infrastructure (slot constants), B.2 wires the 11 fold-reset registry +entries + dispatch arms — both are additive only (no behavior change). **Spec:** `docs/superpowers/specs/2026-05-05-sp14-aux-q-wire-earned-gradient-flow.md` **Plan:** `docs/superpowers/plans/2026-05-05-sp14-aux-q-wire-earned-gradient-flow.md`