Replaces hardcoded thresholds AND clamp bounds across 12 RL controllers
with observed-signal-driven ISV-slot bounds. Eliminates the architectural
failure mode that surfaced in walk-forward fold 0 (alpha-rl-m9cx5) and
fold 1 (alpha-rl-jgdh6): under Phase 4.5 advantage normalization, eight
controllers (PPO clip, target_tau, rollout_steps, entropy_coef, per_α,
gamma, reward_scale, q_distill_lambda) saturated at extrema within ~50
steps and stayed pinned for the rest of training — same pattern across
two different data slices, confirming the bug is structural rather than
data-size-dependent.
Mechanism: each controller's noise floor was hardcoded as a small
fraction of its target (`_NOISE_FLOOR_FRAC = 0.01f`), calibrated against
a pre-Phase-4.5 signal regime. Phase 4.5 normalization reduces operating
KL by ~50× — observed signal stays below the 1%-of-target floor, the
Schulman widen path fires continuously (asymmetric in the wrong
direction), and ε hits MAX 0.50 within ~50 training steps. ksll2's full-
data run (n_folds=1) happened to escape via a single above-band KL
observation that triggered tighten; both walk-forward folds (3 and 6
files) did not.
Per `feedback_adaptive_not_tuned`, `feedback_isv_for_adaptive_bounds`,
`pearl_controller_anchors_isv_driven`: every threshold now derives from
observed signal statistics (Welford online variance) rather than
constants calibrated against a prior signal regime. User explicitly
expanded scope mid-implementation: "if all clamps ISV bound should be
added to this spec and tasks" — applying the principle consistently
means hardcoded MIN/MAX clamp bounds count too, not just the saturating
noise floors. 12 controllers, single atomic commit per
`feedback_no_partial_refactor`.
Spec: docs/superpowers/specs/2026-05-30-adaptive-controller-floor-design.md
Plan: docs/superpowers/plans/2026-05-30-adaptive-controller-floor-plan.md
# New kernel
- rl_signal_variance_update.cu (80 LOC) — Welford online variance.
Single-thread single-block. Sentinel-zero skip per pearl. Per-controller
Welford triple (count, mean, M²) drives every adaptive noise floor.
# Per-controller refactors (12 .cu files)
- ppo_clip + target_tau + rollout_steps + entropy_coef + per_α —
adaptive noise floor = max(target × 0.5, sqrt(observed_var) × 2);
asymmetric Schulman (tighten on single observation, widen requires 3
consecutive below-band).
- gamma (Special G) — hardcoded GAMMA_MIN = 0.995 → adaptive via
Welford MEAN of trade duration. Per spec Q6 resolution: the Welford
mean's natural N-smoothing lag breaks the gamma↔trade_duration
feedback loop without explicit step-period gating. 100-observation
warmup falls back to EMA before Welford has enough samples.
- reward_scale (Special R) — asymmetric DECREASE rate cap (5% per
step) on both bootstrap-replace and Wiener-blend paths; bootstrap-
fraction floor (10% of bootstrap) until 100 trades close.
- q_distill_lambda (Special Q) — hardcoded MIN_LAMBDA = 0.05 → adaptive
via Welford on q_distill_kl_ema: max(0.001, std × 0.05).
- v_blend_alpha (Phase 4.4) — 5 hardcoded constants → 5 ISV slots
(DEAD_SIGNAL_FLOOR, TARGET_TRACK_RATIO, SCHULMAN_STEP, EMA_ALPHA,
BOOTSTRAP_ALPHA) + adaptive dead-signal floor from V_scalar magnitude
variance.
- ppo_ratio_clamp — adaptive MIN/MAX from observed log-ratio variance.
Architectural 2.0 absolute floor preserved (don't degenerate to
vanilla policy gradient).
- reward_clamp — V_BOUND_FLOOR, V_BOUND_EWMA_ALPHA, MIN_WIN, MIN_RATIO,
MAX_RATIO, MIN/MAX_MARGIN, MARGIN_TOLERANCE/ADJUST_RATE,
CLIP_RATE_EMA_ALPHA → 10 ISV slots.
- gate_threshold — hardcoded alpha = 0.01 → ISV slot 638.
- Clamp-bound expansion: EPS_MIN/MAX, TAU_MIN, ROLLOUT_MIN/MAX,
COEF_MIN/MAX, PER_ALPHA_MIN/MAX, GAMMA_MAX, MAX_LAMBDA, KL_TOLERANCE,
LAMBDA_RAMP_RATE, LAMBDA_DECAY_RATE → all ISV.
- WIENER_ALPHA_FLOOR shared across 9 controllers → single ISV slot 659.
# Trainer integration (integrated.rs)
- rl_signal_variance_update kernel loaded + helper method
`launch_rl_signal_variance_update`.
- Per-step Welford launches for 9 controller inputs, placed between
EMA producers and rl_fused_controllers in the per-step pipeline.
- Input-slot lookup array for rl_fused_controllers updated: rollout_steps
now consumes RL_ADV_VAR_PRE_NORM_INDEX (emitted by
rl_advantage_normalize before in-place normalize) instead of the
post-norm advantage_var_ratio (definitionally ~0 under Phase 4.5).
- ~30 new ISV bootstrap entries in with_controllers_bootstrapped.
# ISV slot allocation (isv_slots.rs)
- 72 new slots, RL_SLOTS_END 588 → 660. +288 bytes mapped-pinned.
- 9 Welford variance triples + 5 asymmetric Schulman counters +
3 new input signals (var_pre_norm, gamma_min_adaptive, reward_mag) +
v_blend (5 + 3 Welford) + ppo_ratio_clamp (1 + 3 Welford) +
reward_clamp (5) + gate_threshold (1) + q_distill (1) + 20 clamp bounds +
shared wiener floor.
# Bootstrap-clamp consistency fix (caught by Task 16 testing)
The original draft bootstrapped RL_EPS_BOOTSTRAP at 0.01 (= KL target),
but EPS_MIN was 0.05 — bootstrap value below clamp range. First post-
bootstrap step always snapped ε to MIN regardless of signal direction
("snap to MIN" behavior the unit tests surfaced). Fixed by bootstrapping
ε at MIN (0.05) so asymmetric Schulman operates from a valid state.
# Validation
- cargo build --release: clean
- cargo build --tests: clean
- 3 GPU Welford kernel tests (G1 constant→0var, G2 sequence→known var,
G3 sentinel skip): all pass
- 5 GPU adaptive-floor invariant tests (G3 PPO clip holds at bootstrap
when signal below floor, G4 tightens on single above-band, G5 widens
only after 3 consecutive below-band, G6 post-warmup uses Welford mean,
G6 pre-warmup uses EMA): all pass
- integrated_trainer_smoke (full GPU pipeline): passes
- 1k local smoke at b=128: 1000/1000 steps, completed_clean, no NaN,
controllers genuinely adapting (γ 0.90→0.974, per_α 0.40→0.52,
q_distill_λ 0.05→0.21, ε held at 0.05 = MIN per Phase 4.5 small-KL
regime — was previously stuck at MAX 0.50 throughout fold 0/1)
- compute-sanitizer memcheck b=128 5 steps: 0 errors
Cluster validation (G7-G9) submitted as follow-up runs.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
186 lines
6.0 KiB
Rust
186 lines
6.0 KiB
Rust
//! GPU-oracle tests for `rl_signal_variance_update` — Welford online
|
||
//! variance kernel for adaptive controller floors (spec
|
||
//! `2026-05-30-adaptive-controller-floor-design`).
|
||
//!
|
||
//! Three analytical invariants, zero CPU reference implementations:
|
||
//!
|
||
//! 1. `welford_constant_input_zero_variance` — feeding a constant `k`
|
||
//! for N steps must produce `mean == k` and `m2 == 0` (sample
|
||
//! variance is zero by definition).
|
||
//! 2. `welford_known_sequence_variance` — sequence [1, 2, 3, 4, 5]
|
||
//! has known sample variance 2.5; Welford output must match.
|
||
//! 3. `welford_sentinel_zero_is_skipped` — input == 0.0 sentinel
|
||
//! must produce no update (per
|
||
//! `pearl_first_observation_bootstrap`).
|
||
//!
|
||
//! Per `feedback_no_cpu_test_fallbacks`: oracles are analytical
|
||
//! identities of Welford's algorithm, not CPU reimplementations.
|
||
//! Per `feedback_no_htod_htoh_only_mapped_pinned`: ISV bus uses
|
||
//! mapped-pinned `MappedF32Buffer` (same pattern as production trainer).
|
||
//!
|
||
//! Run with:
|
||
//! `cargo test -p ml-alpha --test signal_variance_kernel -- --ignored --nocapture`
|
||
|
||
use cudarc::driver::{LaunchConfig, PushKernelArg};
|
||
use ml_alpha::pinned_mem::MappedF32Buffer;
|
||
use ml_core::device::MlDevice;
|
||
|
||
// ISV slot layout for this isolated test: input at 0, Welford triple at 1,2,3.
|
||
const INPUT_SLOT: i32 = 0;
|
||
const COUNT_SLOT: i32 = 1;
|
||
const MEAN_SLOT: i32 = 2;
|
||
const M2_SLOT: i32 = 3;
|
||
const N_SLOTS: usize = 4;
|
||
|
||
const SIGNAL_VARIANCE_UPDATE_CUBIN: &[u8] =
|
||
include_bytes!(concat!(env!("OUT_DIR"), "/rl_signal_variance_update.cubin"));
|
||
|
||
fn launch_once(
|
||
dev: &MlDevice,
|
||
isv: &MappedF32Buffer,
|
||
) {
|
||
let stream = dev.cuda_stream().expect("cuda_stream").clone();
|
||
let ctx = dev.cuda_context().expect("cuda_context");
|
||
let module = ctx
|
||
.load_cubin(SIGNAL_VARIANCE_UPDATE_CUBIN.to_vec())
|
||
.expect("load signal_variance cubin");
|
||
let func = module
|
||
.load_function("rl_signal_variance_update")
|
||
.expect("load rl_signal_variance_update fn");
|
||
let cfg = LaunchConfig {
|
||
grid_dim: (1, 1, 1),
|
||
block_dim: (1, 1, 1),
|
||
shared_mem_bytes: 0,
|
||
};
|
||
let isv_dev_ptr = isv.dev_ptr;
|
||
unsafe {
|
||
stream
|
||
.launch_builder(&func)
|
||
.arg(&isv_dev_ptr)
|
||
.arg(&INPUT_SLOT)
|
||
.arg(&COUNT_SLOT)
|
||
.arg(&MEAN_SLOT)
|
||
.arg(&M2_SLOT)
|
||
.launch(cfg)
|
||
.expect("rl_signal_variance_update launch");
|
||
}
|
||
stream.synchronize().expect("sync");
|
||
}
|
||
|
||
#[test]
|
||
#[ignore = "requires CUDA (MlDevice::cuda(0))"]
|
||
fn welford_constant_input_zero_variance() {
|
||
let dev = match MlDevice::cuda(0) {
|
||
Ok(d) => d,
|
||
Err(e) => {
|
||
eprintln!("CUDA 0 not available — skipping ({e})");
|
||
return;
|
||
}
|
||
};
|
||
let isv = unsafe { MappedF32Buffer::new(N_SLOTS) }.expect("isv alloc");
|
||
|
||
// Welford of constant k for N steps: mean = k, m2 = 0, count = N.
|
||
let k = 0.5_f32;
|
||
let n = 50_usize;
|
||
|
||
for _ in 0..n {
|
||
isv.write_record(INPUT_SLOT as usize, k);
|
||
launch_once(&dev, &isv);
|
||
}
|
||
|
||
let count = isv.read_record(COUNT_SLOT as usize);
|
||
let mean = isv.read_record(MEAN_SLOT as usize);
|
||
let m2 = isv.read_record(M2_SLOT as usize);
|
||
|
||
assert_eq!(count, n as f32, "count must be {n}; got {count}");
|
||
assert!(
|
||
(mean - k).abs() < 1e-6,
|
||
"mean of constant {k} must equal {k}; got {mean}"
|
||
);
|
||
assert!(
|
||
m2.abs() < 1e-5,
|
||
"m2 of constant input must be exactly 0 (or fp-noise); got {m2}"
|
||
);
|
||
|
||
eprintln!("G1 OK — constant k={k} for N={n}: mean={mean} m2={m2}");
|
||
}
|
||
|
||
#[test]
|
||
#[ignore = "requires CUDA (MlDevice::cuda(0))"]
|
||
fn welford_known_sequence_variance() {
|
||
let dev = match MlDevice::cuda(0) {
|
||
Ok(d) => d,
|
||
Err(e) => {
|
||
eprintln!("CUDA 0 not available — skipping ({e})");
|
||
return;
|
||
}
|
||
};
|
||
let isv = unsafe { MappedF32Buffer::new(N_SLOTS) }.expect("isv alloc");
|
||
|
||
// Sequence [1, 2, 3, 4, 5]: arithmetic mean = 3.
|
||
// Sample variance = Σ(x_i − μ)² / (n − 1) = (4+1+0+1+4)/4 = 2.5.
|
||
let seq = [1.0_f32, 2.0, 3.0, 4.0, 5.0];
|
||
|
||
for &x in seq.iter() {
|
||
isv.write_record(INPUT_SLOT as usize, x);
|
||
launch_once(&dev, &isv);
|
||
}
|
||
|
||
let count = isv.read_record(COUNT_SLOT as usize);
|
||
let mean = isv.read_record(MEAN_SLOT as usize);
|
||
let m2 = isv.read_record(M2_SLOT as usize);
|
||
let sample_var = m2 / (count - 1.0);
|
||
|
||
assert_eq!(count, seq.len() as f32, "count must be {}", seq.len());
|
||
assert!(
|
||
(mean - 3.0).abs() < 1e-5,
|
||
"mean of [1..5] must be 3.0; got {mean}"
|
||
);
|
||
assert!(
|
||
(sample_var - 2.5).abs() < 1e-5,
|
||
"sample variance of [1..5] must be 2.5; got {sample_var} (m2={m2})"
|
||
);
|
||
|
||
eprintln!("G2 OK — sequence {seq:?}: mean={mean} sample_var={sample_var}");
|
||
}
|
||
|
||
#[test]
|
||
#[ignore = "requires CUDA (MlDevice::cuda(0))"]
|
||
fn welford_sentinel_zero_is_skipped() {
|
||
let dev = match MlDevice::cuda(0) {
|
||
Ok(d) => d,
|
||
Err(e) => {
|
||
eprintln!("CUDA 0 not available — skipping ({e})");
|
||
return;
|
||
}
|
||
};
|
||
let isv = unsafe { MappedF32Buffer::new(N_SLOTS) }.expect("isv alloc");
|
||
|
||
// INPUT_SLOT remains at sentinel 0.0 (MappedF32Buffer::new zero-inits).
|
||
// Kernel must NOT count this as an observation per
|
||
// pearl_first_observation_bootstrap.
|
||
for _ in 0..10 {
|
||
launch_once(&dev, &isv);
|
||
}
|
||
assert_eq!(
|
||
isv.read_record(COUNT_SLOT as usize),
|
||
0.0,
|
||
"sentinel zero must not be counted as a real observation"
|
||
);
|
||
|
||
// Now feed a real non-zero observation; counter must advance.
|
||
isv.write_record(INPUT_SLOT as usize, 7.0);
|
||
launch_once(&dev, &isv);
|
||
assert_eq!(
|
||
isv.read_record(COUNT_SLOT as usize),
|
||
1.0,
|
||
"first real observation must be counted"
|
||
);
|
||
assert!(
|
||
(isv.read_record(MEAN_SLOT as usize) - 7.0).abs() < 1e-6,
|
||
"first observation must initialize mean directly per Welford recurrence"
|
||
);
|
||
|
||
eprintln!("G3 OK — sentinel skip honoured; first real obs initialized mean");
|
||
}
|