# Adaptive Controller Signal Floors — Design

**Goal**: every adaptive RL controller's noise floor / target / threshold derives from observed signal statistics, not a hardcoded constant calibrated against a prior signal regime.

**Why now**: Phase 4.5 advantage normalization (commit `669578566`) shifted the operating distribution of every magnitude-dependent controller input — KL, advantage variance, TD kurtosis, Q divergence. Fold 0 walk-forward at b=1024 20k on 1/3 the ksll2 data confirmed **all 7 adaptive controllers saturate or crash**, even though release training (ksll2, full 9 files) appeared healthy. The system architecture has 7 independent controllers, each with hardcoded thresholds. Phase 4.5 invalidated all of them simultaneously.

**Scope** (revised 2026-05-30 to apply the principle consistently — including clamp bounds, not just noise floors): every adaptive RL controller with **any hardcoded threshold or clamp-bound constant** is in scope. This includes:
- Noise floors and saturation thresholds (the original Phase 4.5 saturation cause)
- Clamp bounds (MIN/MAX on controller output range)
- Step / ramp / decay rates
- EMA / Wiener-α coefficients
- Bootstrap values

This is **12 controllers** in one atomic commit per `feedback_no_partial_refactor`:

**Group A — 8 controllers that visibly saturated in fold 0:**
1. `rl_ppo_clip_controller` — stuck at MAX 0.50 (confirmed cause)
2. `rl_target_tau_controller` — never moved from bootstrap
3. `rl_rollout_steps_controller` — never moved (advantage_var_ratio definitionally near-zero under Phase 4.5)
4. `rl_entropy_coef_controller` — stuck at MIN 0.01
5. `rl_per_alpha_controller` — stuck at MIN 0.40
6. `rl_gamma_controller` — stuck at MIN 0.995 (hardcoded `GAMMA_MIN`)
7. `rl_reward_scale_controller` — crashed 250× in 100 steps
8. `rl_q_distill_lambda_controller` — stuck at MIN 0.05 (hardcoded `MIN_LAMBDA`)

**Group B — 4 controllers with hardcoded thresholds, didn't visibly saturate yet but vulnerable to the same architectural failure**:
9. `rl_v_blend_alpha_controller` — 5 hardcoded constants (`DEAD_SIGNAL_FLOOR`, `TARGET_TRACK_RATIO`, `SCHULMAN_STEP`, `EMA_ALPHA`, `BOOTSTRAP_ALPHA`)
10. `rl_ppo_ratio_clamp_controller` — `PPO_RATIO_CLAMP_MIN_OUT = 2.0f` (calibrated for pre-normalization ratios — Phase 4.5 may already have invalidated)
11. `rl_reward_clamp_controller` — `V_BOUND_FLOOR = 1.0f`, `V_BOUND_EWMA_ALPHA = 0.001f`, `MIN_WIN`, `MIN_RATIO`, `MAX_RATIO`
12. `rl_gate_threshold_controller` — hardcoded `alpha = 0.01f` for EMA smoothing (one constant in an otherwise ISV-driven controller)

**Genuinely out of scope** (audited, confirmed adaptive already, or different bug class):
- `rl_lr_controller` — all configs are ISV slots; uses only `1.0f`/`0.0f` math identities. Confirmed already adaptive.
- Phase 4.5 logic (`rl_advantage_normalize.cu`) — only adds an extra ISV write to emit `var_pre_norm`; the ε² floor itself is a math constant of the normalization, not a controller threshold.
- Controller objectives (what each controller is *trying* to achieve) — out of scope.
- Schulman bounded-step math — out of scope.

This spec replaces hardcoded constants with observed-signal-driven values. **One atomic commit. No phase split.**

---

## Diagnosis (verified, not theoretical)

Per fold 0 diag.jsonl (`/feature-cache/alpha-rl-runs/669578566/fold0/`):

| Controller | Bootstrap | Step 100 | Step 19999 | Mode |
|---|---|---|---|---|
| `γ` (gamma) | 0.995 | 0.995 | 0.995 | Stuck at MIN |
| `τ` (target_tau) | 0.005 | 0.005 | 0.005 | Never moved |
| `ε` (ppo_clip) | 0.20 | **0.50** | **0.50** | Stuck at MAX |
| entropy_coef | 0.5 | 0.01 | 0.01 | Stuck at MIN |
| per_α | 0.4 | 0.41 | 0.40 | Stuck at MIN |
| reward_scale | 1.0 | **0.0046** | **0.0042** | Crashed 250× in 100 steps |
| n_rollout_steps | 2048 | 2048 | 2048 | Never moved |
| `λ` (q_distill_lambda) | 0.01 | **0.05** | **0.05** | Stuck at MIN_LAMBDA |

Trade pnl (eval phase, files 3-5 held out): **-$1.56M, win_rate 0.235, profit_factor 0.64**. Policy did not generalize.

**Note**: `sac_alpha` trends downward over training (0.01 → 0.004) but is NOT saturated — appears to be SAC's natural alpha-annealing. Not addressed in this spec.

### Mechanism for the canonical case (PPO clip)

```c
// rl_ppo_clip_controller.cu
#define KL_NOISE_FLOOR_FRAC               0.01f
const float kl_noise_floor = kl_target * KL_NOISE_FLOOR_FRAC;  // 0.0001
if (kl_ema < kl_noise_floor) return;  // hold

if (kl_ema < kl_target / tolerance) {  // tolerance=1.5 → triggers when kl < 0.0067
    ratio = adjust_rate;               // 1.5× → widen ε
}
```

Trajectory (verified):

| Step | ε | kl_pi |
|---|---|---|
| 1 | 0.20 (bootstrap) | 0.0 |
| 50 | **0.50 (MAX)** | 0.002 |
| 100 | 0.50 | 1.9e-11 |
| 500 | 0.50 | 1.8e-3 |
| 1000+ | 0.50 | 0 or numerical noise |

At step 50 the controller saw `kl_ema = 0.002` — above the noise floor 0.0001 but below the band lower bound 0.0067. WIDEN path fired three times (1.5× per call). Bootstrap-replace-directly path on the first call wrote 0.30, Wiener blends on the next two pushed to 0.5 ceiling. Then `kl_ema` collapsed below the noise floor for the rest of training; ε never came back.

### Why ksll2 didn't show this

ksll2 had 4× the data (9 files vs 3). KL distribution was wider, and at some point `kl_ema` exceeded `kl_target * tolerance = 0.015`, triggering the TIGHTEN path that pulled ε back. With 3 files, KL never gets that high — controller never sees a reason to tighten.

This pattern repeats across all 7 controllers, each with its own variant of the same bug.

### Why this is one bug, not seven

All 7 controllers share three architectural assumptions, each invalidated by Phase 4.5:

1. **Hardcoded noise floor as fraction of target**: 4 controllers use `_NOISE_FLOOR_FRAC = 0.01f` (`ppo_clip`, `target_tau`, `rollout_steps`, `fused`). The 1% floor was calibrated against pre-normalization signal magnitudes; under Phase 4.5 it's ~100× too low.
2. **Fixed target ISV slots**: `kl_target=0.01`, `q_div_target=0.01`, `adv_var_ratio_target=5.0` etc. are bootstrapped once and never updated. They assume a signal scale that no longer holds.
3. **Aggressive bootstrap-to-update**: most controllers have a "bootstrap-replace-directly" path that lets a single observation push them off bootstrap. Combined with (1) + (2), one noisy step can drive the controller to an extremum.

---

## Approaches considered

### Approach A — Targeted patch (~12 LOC, lowest cost)

Replace `_NOISE_FLOOR_FRAC = 0.01f` with `_NOISE_FLOOR_FRAC = 0.5f` (50% of target instead of 1%). Apply to the 4 controllers that have this constant. No new kernels, no ISV changes.

**Pros**: One commit, one test, mechanical change. Validates the diagnosis quickly.

**Cons**: Still a hardcoded constant — violates `feedback_adaptive_not_tuned`. Doesn't fix `reward_scale` crash (different mechanism). Doesn't fix `gamma`/`entropy_coef`/`per_α` (no `_NOISE_FLOOR_FRAC`, different saturation modes). Might break ksll2 regime where 1% worked (50% would hold the controller too aggressively when there *is* meaningful signal). Brittle to the next normalization change.

**Verdict**: Useful as a same-day regression test for the diagnosis, but not the actual fix.

### Approach B — Signal-driven floors (~200 LOC, the architectural fix)

For each controller's input signal, track Welford-online variance in a dedicated ISV slot. Replace the hardcoded `floor = target × const` with `floor = MAX(target × 0.5, sqrt(observed_var) × 2)`. The floor naturally rescales to whatever signal distribution Phase 4.5 (or any future normalization) produces.

**Pros**: Truly adaptive — `feedback_adaptive_not_tuned` compliant. Composes with future architectural changes (Phase 4.6 normalization, new heads, etc.) without re-tuning. Same kernel pattern works for all 7 controllers. Welford is well-understood, CUDA-friendly. The added Welford-variance ISV slots are also valuable diagnostics.

**Cons**: ~200 LOC: new Welford-variance kernel (~80 LOC) + ISV slot allocation (8 new slots, one per controller input) + wiring (~80 LOC) + controller updates (~30 LOC across 7 files). Larger change, larger blast radius.

**Verdict**: This is the architectural fix the foxhunt design philosophy promised. Implementation cost is real but bounded.

### Approach C — Conservative bootstrap + asymmetric Schulman (~50 LOC)

Don't touch the floors. Instead: (1) bootstrap each controller at its TARGET, not at a safety bound, (2) require N consecutive below-band observations before widening fires (so single noisy observations don't drive drift). Hold bootstrap until cumulative evidence accumulates.

**Pros**: Smaller change than B. Preserves controller logic. Holds controllers at sensible defaults when signal is sparse.

**Cons**: Still uses fixed thresholds — same brittleness as A. Doesn't address `reward_scale` crash. Doesn't address `n_rollout_steps` which is structurally broken (its input signal `advantage_var_ratio` is *defined* to be near zero under Phase 4.5).

**Verdict**: Orthogonal to A/B. Worth folding INTO B as the "asymmetric Schulman" piece.

### Recommendation: B + the Schulman asymmetry from C, applied to ALL 12 controllers with hardcoded thresholds

The architectural fix is B. Fold C's asymmetric-Schulman idea into B's controller updates for Group A. Skip A as a separate change — it's a strict subset of B.

`reward_scale`, `n_rollout_steps`, `gamma`, `q_distill_lambda` need separate treatment within this spec — they have *structural* problems (their signals are not just noisy, they're definitionally affected by Phase 4.5 or use ramp/decay rather than Schulman). Spec covers them in dedicated Special case sections (G, Q, R).

Group B controllers (4 of them) follow the same Welford-variance pattern from B but without the asymmetric Schulman — they're not Schulman-based controllers, just have hardcoded floors/ceilings. Same kernel infrastructure, same ISV slot pattern.

**One atomic commit. 12 controllers. 52 new ISV slots. One new Welford kernel.**

---

## Detailed design (Approach B)

### Core pattern (applies 7 places)

```c
// 1. Welford variance kernel (new, runs per-step alongside EMA producers).
//    Maintains running mean + M2 in dedicated ISV slots per signal.
//    See: crates/ml-alpha/cuda/rl_signal_variance_update.cu (NEW)

extern "C" __global__ void rl_signal_variance_update(
    float* __restrict__ isv,
    int input_slot,       // e.g. RL_KL_PI_EMA_INDEX
    int count_slot,       // ISV[input_slot + COUNT_OFFSET]
    int mean_slot,        // ISV[input_slot + MEAN_OFFSET]
    int m2_slot,          // ISV[input_slot + M2_OFFSET]
    int dones_present     // 1 if this step contributed a real observation
) {
    if (threadIdx.x != 0 || !dones_present) return;
    const float x = isv[input_slot];
    if (x == 0.0f) return;                // skip sentinel
    float n = isv[count_slot] + 1.0f;
    float delta = x - isv[mean_slot];
    float mean_new = isv[mean_slot] + delta / n;
    float delta2 = x - mean_new;
    float m2_new = isv[m2_slot] + delta * delta2;
    isv[count_slot] = n;
    isv[mean_slot]  = mean_new;
    isv[m2_slot]    = m2_new;
}
```

### Controller integration (canonical example — PPO clip)

```c
// rl_ppo_clip_controller.cu (MODIFIED)
// BEFORE:
const float kl_noise_floor = kl_target * KL_NOISE_FLOOR_FRAC;  // 1% — broken

// AFTER:
const float kl_var = (isv[KL_PI_COUNT_SLOT] > 1.0f)
    ? isv[KL_PI_M2_SLOT] / (isv[KL_PI_COUNT_SLOT] - 1.0f)
    : 0.0f;
const float kl_std = sqrtf(kl_var);
const float kl_noise_floor = fmaxf(kl_target * 0.5f, kl_std * 2.0f);
//                            ↑ floor never below 50% of target
//                            ↑ AND scales with observed signal std

if (kl_ema < kl_noise_floor) return;
```

The `0.5f` constant is the only remaining tuned number, and it's safe across all signal regimes (50% of target = "less than half the desired operating distance from steady state").

### Asymmetric Schulman (folded in from Approach C)

```c
// Tighten on single observation (real signal above target = act fast).
// Widen only on N consecutive observations below band (filter noise).

if (kl_ema > kl_target * tolerance) {
    ratio = 1.0f / adjust_rate;
    isv[KL_BELOW_COUNT_SLOT] = 0.0f;    // reset
} else if (kl_ema < kl_target / tolerance) {
    isv[KL_BELOW_COUNT_SLOT] += 1.0f;
    ratio = (isv[KL_BELOW_COUNT_SLOT] >= 3.0f) ? adjust_rate : 1.0f;
} else {
    isv[KL_BELOW_COUNT_SLOT] = 0.0f;    // reset on in-band
    ratio = 1.0f;
}
```

The `3.0f` is the "N consecutive" parameter. Tunable — default 3 (~3 minibatch updates of confirmation). Could itself be ISV-driven (`SCHULMAN_WIDEN_PATIENCE_SLOT`) — TBD if needed.

### Per-controller specifics

ISV slots below allocated starting at 588 (current `RL_SLOTS_END`). Final `RL_SLOTS_END = 640` (52 new slots).

**Group A — 8 saturated controllers (Welford variance + below-band counter for asymmetric Schulman):**

| Controller | Input signal | Input slot | Variance triple (count, mean, M²) | Asym counter | Asymmetric Schulman? |
|---|---|---|---|---|---|
| ppo_clip | kl_pi_ema | 419 | 588, 589, 590 | 591 | yes |
| target_tau | q_divergence_ema | 420 | 592, 593, 594 | 595 | yes |
| rollout_steps | adv_var_pre_norm (NEW 612) | 421 → 612 | 596, 597, 598 | 599 | yes |
| entropy_coef | entropy_observed_ema | 422 | 600, 601, 602 | 603 | yes |
| per_α | td_kurtosis_ema | 423 | 604, 605, 606 | 607 | yes |
| gamma | trade_duration_ema | 417 | 608, 609, 610 | — | no (Special G) |
| reward_scale | pnl_per_batch_magnitude (NEW 614) | 614 | 615, 616, 617 | — | no (Special R) |
| q_distill_lambda | q_distill_kl_ema | 488 | 618, 619, 620 | — | no (Special Q) |

**Group B — 4 not-yet-saturated controllers with hardcoded thresholds:**

| Controller | Hardcoded → ISV slots | Notes |
|---|---|---|
| v_blend_alpha (Phase 4.4) | `DEAD_SIGNAL_FLOOR` → 621, `TARGET_TRACK_RATIO` → 622, `SCHULMAN_STEP` → 623, `EMA_ALPHA` → 624, `BOOTSTRAP_ALPHA` → 625 | 5 new slots; controller already takes one Welford variance for V_scalar magnitude — slots 626, 627, 628 |
| ppo_ratio_clamp | `MIN_OUT` (2.0f) → adaptive via ratio_variance; `MAX_OUT` (1000.0f) → 629 | Welford on ratio magnitude — slots 630, 631, 632 |
| reward_clamp | `V_BOUND_FLOOR` → 633, `V_BOUND_EWMA_ALPHA` → 634, `MIN_WIN` → 635, `MIN_RATIO` → 636, `MAX_RATIO` → 637 | 5 slots; already partially signal-driven, just removing remaining hardcoded constants |
| gate_threshold | hardcoded `alpha = 0.01f` → `RL_GATE_EMA_ALPHA_INDEX` 638 | Single slot |

**New input signals (separate from variance slots):**
- 612: `RL_ADV_VAR_PRE_NORM_INDEX` (emitted by `rl_advantage_normalize.cu` before in-place normalize)
- 613: `RL_GAMMA_MIN_ADAPTIVE_INDEX` (computed from trade_duration_ema)
- 614: `RL_REWARD_MAGNITUDE_EMA_INDEX` (per-batch pnl magnitude for reward_scale floor)
- 639: `RL_Q_DISTILL_LAMBDA_MIN_ADAPTIVE_INDEX` (computed from q_distill_kl_ema variance)

**Total: 52 new slots. `RL_SLOTS_END` grows 588 → 640.** Memory impact: +208 bytes.

### Special cases (not covered by core pattern)

#### `n_rollout_steps` — structural conflict with Phase 4.5

Its input signal `advantage_var_ratio_ema` is *definitionally* near zero under Phase 4.5 normalization (advantage var ~= ε² floor). The Welford-variance fix won't help because the signal itself is structurally diluted.

**Proposed fix**: track *pre-normalization* advantage variance in a separate ISV slot. Phase 4.5's `rl_advantage_normalize.cu` computes the mean and variance internally — emit `var_pre_norm` to a new ISV slot (`RL_ADV_VAR_PRE_NORM_INDEX`). `rl_rollout_steps_controller` reads from that instead.

~20 LOC change to `rl_advantage_normalize.cu` to emit the ISV value before in-place normalize.

#### `reward_scale` — crash, not saturation

Crashed from 1.0 → 0.004 in 100 steps. Mechanism: when `pnl_per_batch_magnitude` is small (which it is on small datasets with the agent not yet trading much), the scale controller aggressively shrinks `reward_scale` to compensate. By step 100 the reward signal is muted to 0.4% of bootstrap, which then makes everything downstream tiny → controllers see no signal → death spiral.

**Proposed fix**: rate-limit the `reward_scale` DECREASE more aggressively than INCREASE (asymmetric like Schulman above). Also: floor `reward_scale` at a fraction of bootstrap (e.g., `0.1 × REWARD_SCALE_BOOTSTRAP = 0.1`) until N steps have passed with meaningful trades. This protects against the early-training spiral.

~40 LOC in `rl_reward_scale_controller.cu`.

#### Special case G — `gamma` deadlock from hardcoded `GAMMA_MIN`

Per the trajectory, `gamma` was at MIN (0.995) at step 1 (bootstrap from sentinel) and never moved. Mechanism:

```c
// rl_gamma_controller.cu
#define GAMMA_MIN  0.995f   // hardcoded — violates feedback_adaptive_not_tuned
#define GAMMA_MAX  0.999f
// target(d) = 1 - 1/d (standard discount-from-horizon mapping)
// At d_ema = 15.6: target = 1 - 1/15.6 = 0.936
// clamp(0.936, 0.995, 0.999) = 0.995 ← always pinned at MIN
```

`target(d)` only exceeds `GAMMA_MIN = 0.995` when `d_ema > 200` — i.e., trade durations over 200 minibatch ticks (~10s+ in the surfer regime). Below that, gamma is structurally pinned at 0.995. With fold 0's `d_ema = 15.6`, target = 0.936, controller wants to lower gamma but can't (clamp), so it sticks.

**This is the same architectural bug** as the noise floors: a *hardcoded* threshold that may have been right for one regime but is wrong for another. With small data, agent makes shorter trades; the gamma floor traps the agent in a discount factor that's too long for its trade horizon, which feeds back into the value targets, which biases learning.

**Deadlock**: the agent needs higher gamma to learn longer trades, but the controller needs longer trades to allow higher gamma. With the hardcoded MIN at 0.995, the controller has no room to lower gamma to match short trades, and no signal to raise it because trades stay short.

**Proposed fix** (consistent with the rest of the spec):

Replace hardcoded `GAMMA_MIN = 0.995f` with adaptive `RL_GAMMA_MIN_ADAPTIVE_INDEX` slot, derived from observed `trade_duration_ema`:

```c
// Derived adaptive floor: allow gamma to drop low enough for current
// regime's trade horizon, with a safety floor of 0.9 to prevent the
// agent from going to bandit mode.
const float d_ema = isv[RL_MEAN_TRADE_DURATION_EMA_INDEX];
const float horizon_multiplier = 2.0f;   // gamma allows 2× the trade horizon as effective lookahead
const float adaptive_min = fmaxf(0.9f, 1.0f - 1.0f / fmaxf(d_ema * horizon_multiplier, 10.0f));
isv[RL_GAMMA_MIN_ADAPTIVE_INDEX] = adaptive_min;
```

With `d_ema=15.6`, adaptive_min = `1 - 1/31.2 = 0.968`. Now `target(15.6) = 0.936` still clamps, but to `0.968` instead of `0.995` — and the controller has room to track real trade durations. As the agent learns longer trades and `d_ema` grows, `adaptive_min` rises naturally.

The `0.9` absolute floor and `2.0` horizon multiplier are the only tuned constants — both architecturally motivated:
- `0.9` floor: per `pearl_edge_lives_at_wave_timescale_not_tick`, edge lives at wave timescale (γ≥0.99). The 0.9 floor is a hard "don't degenerate to bandit" guard, not a typical operating point.
- `2.0` horizon multiplier: standard rule-of-thumb that effective horizon should be 2× the natural transaction horizon to capture exit dynamics.

Both could be ISV-driven in a follow-up spec, but the current values are conservative defaults that don't bias the controller's adaptation.

#### Special case Q — `q_distill_lambda` ramp/decay with hardcoded MIN

Per the trajectory: `λ` bootstraps at 0.01, decays via `LAMBDA_DECAY_RATE = 0.998f` for ~100 steps because `q_distill_kl_ema ≈ 0.001` is far below `q_distill_kl_target = 0.10`. Reaches `MIN_LAMBDA = 0.05f` floor and sticks for the next 19900 steps.

```c
// rl_q_distill_lambda_controller.cu
#define MIN_LAMBDA  0.05f  // hardcoded floor
#define MAX_LAMBDA  1.0f   // hardcoded ceiling
// When kl_ema < kl_target / tolerance → λ *= LAMBDA_DECAY_RATE = 0.998
// When kl_ema > kl_target * tolerance → λ *= LAMBDA_RAMP_RATE = 1.2
// Clamp to [MIN_LAMBDA, MAX_LAMBDA]
```

Same bug class as the noise floors and gamma: a hardcoded floor (`MIN_LAMBDA`) that doesn't adapt to the actual signal regime. Phase 4.5 reduces KL signal magnitudes (which this controller also consumes via `q_distill_kl_ema`), so the "below target" path fires continuously, decaying λ to MIN.

**Proposed fix** (mirroring the gamma fix pattern): make `MIN_LAMBDA` and `MAX_LAMBDA` ISV-driven adaptive bounds derived from observed `q_distill_kl_ema` distribution:

```c
// adaptive_lambda_min = MAX(0.001, kl_ema_std * SCALE_FACTOR)
//   where SCALE_FACTOR ≈ 0.05 (provides headroom proportional to signal noise)
const float kl_std = sqrtf(isv[Q_DISTILL_KL_VAR_SLOT]);
const float adaptive_min = fmaxf(0.001f, kl_std * 0.05f);
isv[RL_Q_DISTILL_LAMBDA_MIN_ADAPTIVE_INDEX] = adaptive_min;
```

This reuses the same Welford-variance kernel infrastructure proposed in the core pattern (one variance triple for `q_distill_kl_ema`). Lambda can now decay to a level proportional to the actual signal noise rather than a hardcoded 0.05 floor.

The MAX is less critical (1.0 ceiling is rarely hit in practice) but consistency: derive MAX as `MIN(1.0, MAX(0.5, kl_std * 50.0))`. Provides upward headroom proportional to signal range.

Adds 1 ISV slot (`RL_Q_DISTILL_LAMBDA_MIN_ADAPTIVE_INDEX`) — total grows from 27 → 28.

---

## Files

### New

- `crates/ml-alpha/cuda/rl_signal_variance_update.cu` — Welford variance kernel (~80 LOC)

### Modified

- `crates/ml-alpha/src/rl/isv_slots.rs` — add 27 new slots (3 variance triples × 7 controllers + 3 input slots + 3 asymmetric Schulman counters); grow `RL_SLOTS_END` 588 → 615
- `crates/ml-alpha/cuda/rl_ppo_clip_controller.cu` — adaptive noise floor + asymmetric Schulman
- `crates/ml-alpha/cuda/rl_target_tau_controller.cu` — adaptive noise floor + asymmetric Schulman
- `crates/ml-alpha/cuda/rl_rollout_steps_controller.cu` — adaptive noise floor + asymmetric Schulman + consume `RL_ADV_VAR_PRE_NORM_INDEX` (new) instead of post-norm advantage_var_ratio
- `crates/ml-alpha/cuda/rl_entropy_coef_controller.cu` — adaptive noise floor + asymmetric Schulman
- `crates/ml-alpha/cuda/rl_per_alpha_controller.cu` — adaptive noise floor + asymmetric Schulman
- `crates/ml-alpha/cuda/rl_reward_scale_controller.cu` — asymmetric rate-limit (decrease ≤ 1.05× per step) + bootstrap-fraction floor (`reward_scale ≥ 0.1 × bootstrap` until N trades accumulated) + emit `RL_REWARD_MAGNITUDE_EMA_INDEX`
- `crates/ml-alpha/cuda/rl_gamma_controller.cu` — replace hardcoded `GAMMA_MIN = 0.995f` with adaptive `RL_GAMMA_MIN_ADAPTIVE_INDEX` derived from `trade_duration_ema` (Special case G)
- `crates/ml-alpha/cuda/rl_q_distill_lambda_controller.cu` — replace hardcoded `MIN_LAMBDA = 0.05f` and `MAX_LAMBDA = 1.0f` with adaptive ISV-driven bounds (Special case Q, below)
- `crates/ml-alpha/cuda/rl_v_blend_alpha_controller.cu` — replace 5 hardcoded constants with ISV slots (`DEAD_SIGNAL_FLOOR` → signal-driven, `TARGET_TRACK_RATIO`/`SCHULMAN_STEP`/`EMA_ALPHA`/`BOOTSTRAP_ALPHA` → ISV slots)
- `crates/ml-alpha/cuda/rl_ppo_ratio_clamp_controller.cu` — replace `PPO_RATIO_CLAMP_MIN_OUT = 2.0f` with adaptive floor derived from observed ratio variance; `MAX_OUT = 1000.0f` becomes ISV-driven ceiling
- `crates/ml-alpha/cuda/rl_reward_clamp_controller.cu` — replace `V_BOUND_FLOOR`, `V_BOUND_EWMA_ALPHA`, `MIN_WIN`, `MIN_RATIO`, `MAX_RATIO` with adaptive ISV slots (`V_BOUND_EWMA_ALPHA` becomes signal-variance-driven; ratio bounds derive from observed win/loss distribution)
- `crates/ml-alpha/cuda/rl_gate_threshold_controller.cu` — replace hardcoded `alpha = 0.01f` with ISV slot `RL_GATE_EMA_ALPHA_INDEX`
- `crates/ml-alpha/cuda/rl_advantage_normalize.cu` — emit raw `var_pre_norm` to `RL_ADV_VAR_PRE_NORM_INDEX` before in-place normalize
- `crates/ml-alpha/cuda/rl_fused_controllers.cu` — all of the above unified (this is what the trainer actually launches per-step)
- `crates/ml-alpha/src/trainer/integrated.rs` — launch `rl_signal_variance_update` per step + wire new ISV slots + populate `RL_GAMMA_MIN_ADAPTIVE_INDEX` via fused kernel

### Not touched

- Phase 4.5 logic itself (`rl_advantage_normalize.cu` body) — the only addition is one ISV write to emit `var_pre_norm` for `rl_rollout_steps_controller` consumption. The ε² floor in the normalization math stays — it's a numerical-stability constant of the kernel, not a controller threshold.
- `rl_lr_controller` — **AUDITED 2026-05-30**: all configs are ISV slots (`RL_LR_BOOTSTRAP_INDEX`, `RL_LR_MIN_INDEX`, `RL_LR_MAX_INDEX`, `RL_LR_DECAY_FACTOR_INDEX`, `RL_LR_LOSS_EMA_ALPHA_INDEX`, `RL_IMPROVEMENT_THRESHOLD_INDEX`, `RL_PLATEAU_PATIENCE_INDEX`, `RL_LR_WARMUP_STEPS_INDEX`). Kernel body uses only `1.0f`/`0.0f` math identities (not thresholds). Already complies with `feedback_adaptive_not_tuned`. Per `pearl_plateau_decay_needs_warmup_for_sparse_heads`, the plateau-decay pattern is intentionally different from Schulman; keeping it.
- All other RL kernels not listed in Modified (e.g., `rl_fused_targets.cu`, kernel utilities) — not controllers, not subject to this spec.

---

## Validation gates

### Unit / smoke gates (local, RTX 3050 Ti)

1. **G1 — Welford convergence**: feed constant signal `k=0.5`, 50 steps, assert `isv[mean_slot] ≈ k` AND `isv[m2_slot] / (n-1) ≈ 0` (no variance).
2. **G2 — Welford variance**: feed signal `[1, 2, 3, 4, 5]`, assert `isv[m2_slot] / 4 ≈ 2.5` (sample variance).
3. **G3 — PPO clip holds on small signal**: bootstrap controller, feed `kl_ema = 0.001` for 100 steps with observed_var=1e-7, assert `ε` does NOT drift past `target × tolerance = 0.015`.
4. **G4 — PPO clip tightens on big signal**: feed `kl_ema = 0.05` once (single overshoot), assert `ε` decreases by `1/adjust_rate` immediately (asymmetric tightening).
5. **G5 — PPO clip widens slowly**: feed `kl_ema = 0.001` for 10 steps, assert widen fires ONLY after 3 consecutive below-band observations (asymmetric).
6. **G6 — integrated_trainer_smoke**: existing GPU smoke must still pass.

### Cluster gates

7. **G7 — fold 0 re-run with fix**: same params (b=1024, 20k, n_folds=3, fold_idx=0). At step 19999:
   - PPO clip `ε ≤ 0.20` (held in bootstrap region, was 0.50)
   - `reward_scale ≥ 0.5` (bootstrap-fraction floor active, was 0.004)
   - `n_rollout_steps ≠ 2048` (moved off bootstrap, signals real adaptation via pre-norm advantage variance)
   - `gamma > 0.97` (adaptive_min derived from `d_ema ≈ 15` allows movement above hardcoded 0.995)
   - `q_distill_lambda > 0.05` (above old MIN floor, adaptive bound from signal variance)
   - 5+ of the 8 Group A controllers no longer pinned at extrema
8. **G8 — fold 0 eval pnl improvement**: eval pnl > -$0.5M (was -$1.56M). Profit factor > 0.85 (was 0.64). Win rate > 0.40 (was 0.235). These are improvement gates, not absolute targets — we expect *some* improvement, not full positive pnl on 3-file training.
9. **G9 — ksll2 regression test**: same params as `alpha-rl-ksll2` (n_folds=1, b=1024, 20k). Final pnl ≥ +$15M (within 20% of original +$18.3M). Confirms fix doesn't regress the large-data regime where the prior hardcoded floors happened to work. **All 12 controllers** should reach values consistent with the +$18M ksll2 trajectory (e.g., ε around target 0.01, not at 0.50; reward_scale around 0.5-1.0; gamma in [0.97, 0.999]).
10. **G10 — Group B non-regression**: the 4 Group B controllers (`v_blend_alpha`, `ppo_ratio_clamp`, `reward_clamp`, `gate_threshold`) didn't visibly saturate before this fix — verify they still operate in their prior healthy ranges:
    - `v_blend_alpha` settles in [0.0, 1.0] with non-trivial variation (was bootstrapping to 1.0 then dropping toward 0)
    - `ppo_ratio_clamp_max` settles at a meaningful value (was 10.0 bootstrap → adaptive)
    - `reward_clamp_win` / `reward_clamp_loss` track observed reward distribution (no drift toward MIN/MAX of new adaptive bounds)
    - `gate_threshold` EMA tracks dones_target as before (the only change is `alpha = 0.01f` → ISV-driven, semantically equivalent at the new default)

---

## Open questions resolved

1. ~~**ISV slot count budget**~~ → **resolved**: `RL_SLOTS_END = 588` currently, highest used 587. No upper cap. 52 new slots → `RL_SLOTS_END = 640`. Memory impact +208 bytes (negligible).
2. **Welford reset policy**: variance EMA carries over the train→eval boundary. *Decision*: accept this — eval-phase ISV reads should still see the train-phase variance estimate (controllers don't update during eval anyway).
3. **N-consecutive-below counter scope**: spec uses per-controller counters (5 ISV slots, one per Group A controller that needs asymmetric Schulman). Group B controllers don't need them (different mechanism).
4. **`reward_scale` bootstrap-fraction floor**: 0.1 (10% of bootstrap) — accepted as a tuned constant guarded by the rationale that early-training reward distributions are inherently noisy and shrinking past 10% of bootstrap is almost never beneficial. Could be ISV-driven in a follow-up.
5. **Order of implementation**: **all-in-one commit per `feedback_no_partial_refactor`** (Option B selected). Partial migration would create ambiguous validation signal — "did ppo_clip fix work but other controllers ruined it?" vs "did the architectural fix work?" — and the contract change (new ISV slots, modified controller semantics) needs atomic application. The plan decomposes into ordered TASKS that flow into ONE commit at the end, not one commit per controller. No "Phase 1/Phase 2" — 12 controllers in one commit.

6. ~~**Gamma adaptive_min coupling**~~ → **resolved** (2026-05-30): use the Welford mean of trade duration directly (already computed for `rl_per_alpha_controller`'s noise floor), with a 100-observation warmup gate. The Welford mean's natural lag (denominator-of-N smoothing) breaks the feedback loop without any explicit step-period gating or exponential cool-down. Implementation: while `count < 100` use the existing EMA; once `count >= 100` switch to the Welford mean as the input to `target(d)`. No new ISV slots, no new constants beyond the 100-observation gate (which is a confidence threshold, not a magnitude calibration). See plan Task 7 for the code.

---

## Risks & mitigations

- **Risk**: Welford with float32 + small variance values → catastrophic cancellation. **Mitigation**: standard Welford formulation already handles this; CUDA float32 is sufficient at our signal magnitudes (1e-5 to 1.0).
- **Risk**: New ISV slots break existing tests that snapshot the full ISV layout. **Mitigation**: tests need to be updated alongside; this was just done in the test-migration commit `a3dfcd63f` for similar reasons.
- **Risk**: Asymmetric Schulman makes controllers TOO conservative, blocking real adaptation. **Mitigation**: the N-consecutive parameter is small (3), tightening still fires on single observation (fast safety response), and the in-band hold preserves the steady state.
- **Risk**: The fix works on fold 0 but ksll2 (large data) actually NEEDS the loose noise floor to adapt fast enough. **Mitigation**: G9 cluster regression catches this. Fallback: ISV-driven `NOISE_FLOOR_FRAC` slot, dynamically tunable per-run.

---

## Next step after approval

Write the plan: `docs/superpowers/plans/2026-05-30-adaptive-controller-floor-plan.md` with task-level decomposition (foundation → ppo_clip → sweep → validation → cluster runs).
