Files
foxhunt/docs/superpowers
jgrusewski c52282fb44 feat(rl): Phase 2.0 — V head stabilization (target clamp + plateau decay)
Implements §3 Sub-phase 2.0 of the state-conditional Q synthesis spec
(docs/superpowers/specs/2026-05-28-state-conditional-q-synthesis.md).

Adds a done-gated trade-magnitude envelope that clamps the V regression
target before MSE, bounding `(v_pred - target)²` regardless of
fat-tail trade-close spikes from `returns = r + γ(1-done)·v_tp1`. The
envelope decouples from the C51 atom span (slots 484/485) so it can
tighten below the atom span floor as the trade-magnitude EMA narrows.

Wiring:
  * New ISV slots 593-596: V_MAGNITUDE_EMA, V_TARGET_MAX, V_TARGET_MIN,
    V_TARGET_K. RL_SLOTS_END = 597.
  * New kernel `rl_v_target_envelope_update.cu` — done-gated Wiener-α
    EMA of `max(|reward| | done)`; derives `V_max_eff = +k × ema,
    V_min_eff = -k × ema`. Sparse-aware (skip update on no-done steps),
    first-observation bootstrap, single-thread single-block.
  * `v_head_bwd.cu` — clamp `r_target` to `[V_min_eff, V_max_eff]`
    before computing MSE. Updates `ValueHead::backward` Rust signature
    to take the ISV device pointer.
  * Trainer integration: single helper `launch_rl_v_target_envelope_update`
    is invoked from all three reward-pipeline paths (public
    `launch_apply_reward_scale`, `step_with_lobsim_reward_and_train` —
    two inline sites) — single source of truth per
    `feedback_single_source_of_truth_no_duplicates`.
  * Bootstrap ISV slots: V_max=+200, V_min=-200, k=3.0 (matches
    dd049d9a4 baseline avg trade tail per spec). Bumps ISV constant
    array from 115 → 118 entries.

V plateau LR decay (also called out in the spec) was already wired
end-to-end via `rl_lr_controller` (slots 433/434/435/438) — no
duplication needed.

Smoke validation:
  * Real-data 1000-step run (`alpha_rl_train --n-backtests 128`):
    max l_v = 0.6024 throughout 1000 steps (well under the spec's
    5.0 ceiling).
  * Synthetic Rust integration test
    (`tests/v_head_stabilization_smoke.rs`): 1000 steps batch=1,
    cold-start window 100 steps, post-warmup max l_v = 0.0035.
2026-05-28 23:39:42 +02:00
..