From b4aadff756d330407febb0dbf86fb130e0f4b92a Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Fri, 29 May 2026 01:33:26 +0200 Subject: [PATCH] =?UTF-8?q?fix(rl):=20tighten=20V=20envelope=20bootstrap?= =?UTF-8?q?=20=C2=B1200=20=E2=86=92=20=C2=B110=20(post-scale=20calibration?= =?UTF-8?q?)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 2.0 seeded the V envelope at ±200 reasoning from dd049d9a4 dollar magnitudes ($211 avg win, $235 avg loss). But the envelope clamps the POST-SCALE V regression target — after apply_reward_scale divides by mean_abs_pnl_ema to keep typical reward magnitudes ~1.0, the target's natural range is `r + γ × v_tp1 ≤ ~3` in atom-support units, NOT dollars. A ±200 bootstrap was a no-op (envelope wider than any possible target excursion) until the EMA tightened it. ±10 is the tighter bootstrap matching the actual post-scale operating range while still wide enough to admit normal target excursions. The kernel's existing sentinel-zero bootstrap discipline at line 98 (pearl_first_observation_bootstrap) snaps the EMA to the first observed trade magnitude as soon as a done event fires, so this bootstrap only matters for the first ~10-20 steps. Note: this commit does NOT resolve the step-4 NaN that surfaced after the atomicAdd determinization (10d4614fb) — l_v is identical 6.329 ± 0.001 across all ±10 / ±200 / pre-fix runs, proving V predictions are not the proximate NaN cause. The tightening is still correct per codebase discipline (pearl_controller_anchors_isv_driven + feedback_isv_for_adaptive_bounds) and removes one masked confound for the next debug pass. Suspected actual NaN cause shifted to advantage RMS / encoder h_t / shared streaming statistics — see pearl_atomicadd_masks_v_instability for the open investigation. Co-Authored-By: Claude Opus 4.7 --- .../cuda/rl_v_target_envelope_update.cu | 9 ++++--- crates/ml-alpha/src/trainer/integrated.rs | 24 ++++++++++++------- 2 files changed, 22 insertions(+), 11 deletions(-) diff --git a/crates/ml-alpha/cuda/rl_v_target_envelope_update.cu b/crates/ml-alpha/cuda/rl_v_target_envelope_update.cu index dc778bbc3..689f9470b 100644 --- a/crates/ml-alpha/cuda/rl_v_target_envelope_update.cu +++ b/crates/ml-alpha/cuda/rl_v_target_envelope_update.cu @@ -111,9 +111,12 @@ extern "C" __global__ void rl_v_target_envelope_update( // Unconditional write: v_head_bwd consumes 594/595 every step. When // the EMA is still at sentinel zero (no done events observed yet), // we DON'T overwrite the envelope — the trainer-seeded bootstrap - // values (±$200 per spec §3 sub-phase 2.0) remain in effect. This - // makes the envelope a no-op until the EMA captures a real - // trade-magnitude signal, after which it tightens to ±k × ema. + // values (±10 post-scale units, tightened from ±200 on 2026-05-29 + // after the bootstrap was found to be wider than the actual + // post-scale target range; see integrated.rs near RL_V_TARGET_MAX + // for rationale) remain in effect. This makes the envelope a tight + // initial bound until the EMA captures a real trade-magnitude + // signal, after which it tightens further to ±k × ema. const float ema_now = isv[RL_V_MAGNITUDE_EMA_INDEX]; if (ema_now > 0.0f) { const float k = isv[RL_V_TARGET_K_INDEX]; diff --git a/crates/ml-alpha/src/trainer/integrated.rs b/crates/ml-alpha/src/trainer/integrated.rs index 90f9d7e58..43de6059e 100644 --- a/crates/ml-alpha/src/trainer/integrated.rs +++ b/crates/ml-alpha/src/trainer/integrated.rs @@ -3112,16 +3112,24 @@ impl IntegratedTrainer { // Phase 2.0 (2026-05-28) — V regression target envelope. // Seeded permissive (±$200 in scaled units; at default // reward_scale=1.0 this matches dd049d9a4's $211 avg win - // / $235 avg loss tail). Cold start: envelope is a no-op - // — target's natural range is ~[-3, +1] (post-clamp - // reward) + γ × v_tp1 (atom span). After done events, - // `rl_v_target_envelope_update` tightens the envelope - // to ±k × magnitude_ema. k=3 covers ~99% of trade - // outcomes. Spec: + // 2026-05-29: tightened bootstrap ±200 → ±10. The original + // ±200 was a dollar-scale guess from dd049d9a4 ($211 avg + // win / $235 avg loss) — but the envelope clamps the + // POST-SCALE V target (after apply_reward_scale divides + // by mean_abs_pnl_ema so typical reward magnitudes are + // ~1.0). In post-scale units a ±10 envelope is already + // wide enough to admit normal target excursions + // (r + γ × v_tp1 ≤ ~3) while bounding the (v_pred - + // target_clamped)² gradient before the EMA takes over. + // The kernel's sentinel-zero bootstrap discipline + // (rl_v_target_envelope_update.cu:98) snaps the EMA to + // the first observed trade magnitude as soon as a done + // event fires, so this bootstrap is only in effect for + // the first ~10-20 steps. Spec: // docs/superpowers/specs/2026-05-28-state-conditional-q-synthesis.md // §3 Sub-phase 2.0. - (crate::rl::isv_slots::RL_V_TARGET_MAX_INDEX, 200.0), - (crate::rl::isv_slots::RL_V_TARGET_MIN_INDEX, -200.0), + (crate::rl::isv_slots::RL_V_TARGET_MAX_INDEX, 10.0), + (crate::rl::isv_slots::RL_V_TARGET_MIN_INDEX, -10.0), (crate::rl::isv_slots::RL_V_TARGET_K_INDEX, 3.0), ]; for (slot, value) in isv_constants.iter() {