fix(rl): tighten V envelope bootstrap ±200 → ±10 (post-scale calibration)
Phase 2.0 seeded the V envelope at ±200 reasoning fromdd049d9a4dollar magnitudes ($211 avg win, $235 avg loss). But the envelope clamps the POST-SCALE V regression target — after apply_reward_scale divides by mean_abs_pnl_ema to keep typical reward magnitudes ~1.0, the target's natural range is `r + γ × v_tp1 ≤ ~3` in atom-support units, NOT dollars. A ±200 bootstrap was a no-op (envelope wider than any possible target excursion) until the EMA tightened it. ±10 is the tighter bootstrap matching the actual post-scale operating range while still wide enough to admit normal target excursions. The kernel's existing sentinel-zero bootstrap discipline at line 98 (pearl_first_observation_bootstrap) snaps the EMA to the first observed trade magnitude as soon as a done event fires, so this bootstrap only matters for the first ~10-20 steps. Note: this commit does NOT resolve the step-4 NaN that surfaced after the atomicAdd determinization (10d4614fb) — l_v is identical 6.329 ± 0.001 across all ±10 / ±200 / pre-fix runs, proving V predictions are not the proximate NaN cause. The tightening is still correct per codebase discipline (pearl_controller_anchors_isv_driven + feedback_isv_for_adaptive_bounds) and removes one masked confound for the next debug pass. Suspected actual NaN cause shifted to advantage RMS / encoder h_t / shared streaming statistics — see pearl_atomicadd_masks_v_instability for the open investigation. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -111,9 +111,12 @@ extern "C" __global__ void rl_v_target_envelope_update(
|
||||
// Unconditional write: v_head_bwd consumes 594/595 every step. When
|
||||
// the EMA is still at sentinel zero (no done events observed yet),
|
||||
// we DON'T overwrite the envelope — the trainer-seeded bootstrap
|
||||
// values (±$200 per spec §3 sub-phase 2.0) remain in effect. This
|
||||
// makes the envelope a no-op until the EMA captures a real
|
||||
// trade-magnitude signal, after which it tightens to ±k × ema.
|
||||
// values (±10 post-scale units, tightened from ±200 on 2026-05-29
|
||||
// after the bootstrap was found to be wider than the actual
|
||||
// post-scale target range; see integrated.rs near RL_V_TARGET_MAX
|
||||
// for rationale) remain in effect. This makes the envelope a tight
|
||||
// initial bound until the EMA captures a real trade-magnitude
|
||||
// signal, after which it tightens further to ±k × ema.
|
||||
const float ema_now = isv[RL_V_MAGNITUDE_EMA_INDEX];
|
||||
if (ema_now > 0.0f) {
|
||||
const float k = isv[RL_V_TARGET_K_INDEX];
|
||||
|
||||
@@ -3112,16 +3112,24 @@ impl IntegratedTrainer {
|
||||
// Phase 2.0 (2026-05-28) — V regression target envelope.
|
||||
// Seeded permissive (±$200 in scaled units; at default
|
||||
// reward_scale=1.0 this matches dd049d9a4's $211 avg win
|
||||
// / $235 avg loss tail). Cold start: envelope is a no-op
|
||||
// — target's natural range is ~[-3, +1] (post-clamp
|
||||
// reward) + γ × v_tp1 (atom span). After done events,
|
||||
// `rl_v_target_envelope_update` tightens the envelope
|
||||
// to ±k × magnitude_ema. k=3 covers ~99% of trade
|
||||
// outcomes. Spec:
|
||||
// 2026-05-29: tightened bootstrap ±200 → ±10. The original
|
||||
// ±200 was a dollar-scale guess from dd049d9a4 ($211 avg
|
||||
// win / $235 avg loss) — but the envelope clamps the
|
||||
// POST-SCALE V target (after apply_reward_scale divides
|
||||
// by mean_abs_pnl_ema so typical reward magnitudes are
|
||||
// ~1.0). In post-scale units a ±10 envelope is already
|
||||
// wide enough to admit normal target excursions
|
||||
// (r + γ × v_tp1 ≤ ~3) while bounding the (v_pred -
|
||||
// target_clamped)² gradient before the EMA takes over.
|
||||
// The kernel's sentinel-zero bootstrap discipline
|
||||
// (rl_v_target_envelope_update.cu:98) snaps the EMA to
|
||||
// the first observed trade magnitude as soon as a done
|
||||
// event fires, so this bootstrap is only in effect for
|
||||
// the first ~10-20 steps. Spec:
|
||||
// docs/superpowers/specs/2026-05-28-state-conditional-q-synthesis.md
|
||||
// §3 Sub-phase 2.0.
|
||||
(crate::rl::isv_slots::RL_V_TARGET_MAX_INDEX, 200.0),
|
||||
(crate::rl::isv_slots::RL_V_TARGET_MIN_INDEX, -200.0),
|
||||
(crate::rl::isv_slots::RL_V_TARGET_MAX_INDEX, 10.0),
|
||||
(crate::rl::isv_slots::RL_V_TARGET_MIN_INDEX, -10.0),
|
||||
(crate::rl::isv_slots::RL_V_TARGET_K_INDEX, 3.0),
|
||||
];
|
||||
for (slot, value) in isv_constants.iter() {
|
||||
|
||||
Reference in New Issue
Block a user