fix(rl): ISV-driven output clamp on streaming var/kurtosis kernels

gxhr8 confirmed the streaming kernels work — both formerly-dead
controllers (rl_rollout_steps, rl_per_alpha) now adapt instead of
pegging at MIN. But the unclamped streaming outputs reached
advantage_var_ratio = 3e5 (when streaming-mean passed through zero
and `var/|mean|` blew up under the 1e-6 denominator floor) and
td_kurtosis = 50.6, pegging both downstream controllers at MAX
instead. Per_α at MAX over-concentrates PER sampling on outliers,
which hurts distributional Q learning (best l_q window regressed
from 2.41 → 2.69 between pdgxn and gxhr8).

## Fix: ISV-resident output clamp ceilings

Two new ISV slots hold the streaming-kernel output ceilings:

  RL_ADV_VAR_RATIO_CLAMP_INDEX = 447  (default 100.0)
  RL_TD_KURTOSIS_CLAMP_INDEX   = 448  (default  30.0)

  * 100.0 for var_ratio = 1000× ADV_VAR_RATIO_TARGET (= 0.1) — wide
    enough that healthy signal (typical 1-10) passes through, tight
    enough that 3e5 outliers don't peg rollout_steps.
  * 30.0 for kurtosis = 3× (KURT_GAUSSIAN + KURT_LIFT_SCALE) — lets
    the full per_α response range engage on heavy-tailed signal
    (≤ 10), bounds runaway above that.

Per `feedback_isv_for_adaptive_bounds`: the clamps live in ISV
(visible in diag, modifiable at runtime via re-launching the init
kernel or a future adaptive controller) rather than as kernel-side
`#define`s.

## Seeding (no HtoD per feedback_no_htod_htoh_only_mapped_pinned)

New device kernel `rl_streaming_clamp_init.cu` — single thread,
writes both clamp ceilings directly to ISV. Launched once at the
end of `with_controllers_bootstrapped` alongside the 8 existing
controller-bootstrap launches. Zero host→device transfer.

## Diag bake-in (per user request "ensure to bake in diags")

JSONL gains a new `streaming` block exposing:
  * `streaming.adv_var.{mean, m2, clamp}`
  * `streaming.td_kurt.{mean, m2, m4, clamp}`

Cross-check: when consumer-input slot (RL_ADVANTAGE_VAR_RATIO_EMA_INDEX
or RL_TD_KURTOSIS_EMA_INDEX) reads exactly the same value as
`streaming.*.clamp`, the clamp fired this step.

## Test updates

G1 (isv_bootstrap) + G3 (r5_controllers) blanket-assert that
ISV[417..END] is sentinel-zero at bootstrap. Both new slots are
seeded to non-zero values by rl_streaming_clamp_init during
bootstrap, so both tests skip these slots in the loop and assert
the seeded values separately.

## Verified gates (local sm_86)

  G1 isv_bootstrap   
  G3 controllers     
  G4 target_update   
  integrated_smoke   

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-23 22:26:12 +02:00
parent 39f90f3723
commit 66115007ab
9 changed files with 220 additions and 24 deletions

View File

@@ -64,6 +64,7 @@ const KERNELS: &[&str] = &[
// (entropy_observed_ema, ISV[420], feeds rl_entropy_coef directly via ema_update_per_step's internal mean reduce on entropy_d — no separate kernel needed.)
"rl_ppo_ratio_clamp_controller", // RL R9: PPO ratio clamp ceiling at ISV[440], anchored on ε at ISV[402] — bounds catastrophic unclipped-branch surrogate
"ppo_log_ratio_abs_max_b", // RL R9 diag: per-batch max|log π_new log π_old| → ISV[441]; surfaces ratio-clamp activity in diag JSONL
"rl_streaming_clamp_init", // RL R9: device-side seeder for streaming-kernel output clamp ceilings (ISV[447], ISV[448]) — no HtoD
];
// Cache bust v31 — five new reduce / derive kernels populate the input