Files
foxhunt/crates/ml-alpha/tests/signal_variance_kernel.rs
jgrusewski 083a88f7c3 feat(rl): adaptive controller floors — 12 controllers, all signal-driven
Replaces hardcoded thresholds AND clamp bounds across 12 RL controllers
with observed-signal-driven ISV-slot bounds. Eliminates the architectural
failure mode that surfaced in walk-forward fold 0 (alpha-rl-m9cx5) and
fold 1 (alpha-rl-jgdh6): under Phase 4.5 advantage normalization, eight
controllers (PPO clip, target_tau, rollout_steps, entropy_coef, per_α,
gamma, reward_scale, q_distill_lambda) saturated at extrema within ~50
steps and stayed pinned for the rest of training — same pattern across
two different data slices, confirming the bug is structural rather than
data-size-dependent.

Mechanism: each controller's noise floor was hardcoded as a small
fraction of its target (`_NOISE_FLOOR_FRAC = 0.01f`), calibrated against
a pre-Phase-4.5 signal regime. Phase 4.5 normalization reduces operating
KL by ~50× — observed signal stays below the 1%-of-target floor, the
Schulman widen path fires continuously (asymmetric in the wrong
direction), and ε hits MAX 0.50 within ~50 training steps. ksll2's full-
data run (n_folds=1) happened to escape via a single above-band KL
observation that triggered tighten; both walk-forward folds (3 and 6
files) did not.

Per `feedback_adaptive_not_tuned`, `feedback_isv_for_adaptive_bounds`,
`pearl_controller_anchors_isv_driven`: every threshold now derives from
observed signal statistics (Welford online variance) rather than
constants calibrated against a prior signal regime. User explicitly
expanded scope mid-implementation: "if all clamps ISV bound should be
added to this spec and tasks" — applying the principle consistently
means hardcoded MIN/MAX clamp bounds count too, not just the saturating
noise floors. 12 controllers, single atomic commit per
`feedback_no_partial_refactor`.

Spec: docs/superpowers/specs/2026-05-30-adaptive-controller-floor-design.md
Plan: docs/superpowers/plans/2026-05-30-adaptive-controller-floor-plan.md

# New kernel
- rl_signal_variance_update.cu (80 LOC) — Welford online variance.
  Single-thread single-block. Sentinel-zero skip per pearl. Per-controller
  Welford triple (count, mean, M²) drives every adaptive noise floor.

# Per-controller refactors (12 .cu files)
- ppo_clip + target_tau + rollout_steps + entropy_coef + per_α —
  adaptive noise floor = max(target × 0.5, sqrt(observed_var) × 2);
  asymmetric Schulman (tighten on single observation, widen requires 3
  consecutive below-band).
- gamma (Special G) — hardcoded GAMMA_MIN = 0.995 → adaptive via
  Welford MEAN of trade duration. Per spec Q6 resolution: the Welford
  mean's natural N-smoothing lag breaks the gamma↔trade_duration
  feedback loop without explicit step-period gating. 100-observation
  warmup falls back to EMA before Welford has enough samples.
- reward_scale (Special R) — asymmetric DECREASE rate cap (5% per
  step) on both bootstrap-replace and Wiener-blend paths; bootstrap-
  fraction floor (10% of bootstrap) until 100 trades close.
- q_distill_lambda (Special Q) — hardcoded MIN_LAMBDA = 0.05 → adaptive
  via Welford on q_distill_kl_ema: max(0.001, std × 0.05).
- v_blend_alpha (Phase 4.4) — 5 hardcoded constants → 5 ISV slots
  (DEAD_SIGNAL_FLOOR, TARGET_TRACK_RATIO, SCHULMAN_STEP, EMA_ALPHA,
  BOOTSTRAP_ALPHA) + adaptive dead-signal floor from V_scalar magnitude
  variance.
- ppo_ratio_clamp — adaptive MIN/MAX from observed log-ratio variance.
  Architectural 2.0 absolute floor preserved (don't degenerate to
  vanilla policy gradient).
- reward_clamp — V_BOUND_FLOOR, V_BOUND_EWMA_ALPHA, MIN_WIN, MIN_RATIO,
  MAX_RATIO, MIN/MAX_MARGIN, MARGIN_TOLERANCE/ADJUST_RATE,
  CLIP_RATE_EMA_ALPHA → 10 ISV slots.
- gate_threshold — hardcoded alpha = 0.01 → ISV slot 638.
- Clamp-bound expansion: EPS_MIN/MAX, TAU_MIN, ROLLOUT_MIN/MAX,
  COEF_MIN/MAX, PER_ALPHA_MIN/MAX, GAMMA_MAX, MAX_LAMBDA, KL_TOLERANCE,
  LAMBDA_RAMP_RATE, LAMBDA_DECAY_RATE → all ISV.
- WIENER_ALPHA_FLOOR shared across 9 controllers → single ISV slot 659.

# Trainer integration (integrated.rs)
- rl_signal_variance_update kernel loaded + helper method
  `launch_rl_signal_variance_update`.
- Per-step Welford launches for 9 controller inputs, placed between
  EMA producers and rl_fused_controllers in the per-step pipeline.
- Input-slot lookup array for rl_fused_controllers updated: rollout_steps
  now consumes RL_ADV_VAR_PRE_NORM_INDEX (emitted by
  rl_advantage_normalize before in-place normalize) instead of the
  post-norm advantage_var_ratio (definitionally ~0 under Phase 4.5).
- ~30 new ISV bootstrap entries in with_controllers_bootstrapped.

# ISV slot allocation (isv_slots.rs)
- 72 new slots, RL_SLOTS_END 588 → 660. +288 bytes mapped-pinned.
- 9 Welford variance triples + 5 asymmetric Schulman counters +
  3 new input signals (var_pre_norm, gamma_min_adaptive, reward_mag) +
  v_blend (5 + 3 Welford) + ppo_ratio_clamp (1 + 3 Welford) +
  reward_clamp (5) + gate_threshold (1) + q_distill (1) + 20 clamp bounds +
  shared wiener floor.

# Bootstrap-clamp consistency fix (caught by Task 16 testing)
The original draft bootstrapped RL_EPS_BOOTSTRAP at 0.01 (= KL target),
but EPS_MIN was 0.05 — bootstrap value below clamp range. First post-
bootstrap step always snapped ε to MIN regardless of signal direction
("snap to MIN" behavior the unit tests surfaced). Fixed by bootstrapping
ε at MIN (0.05) so asymmetric Schulman operates from a valid state.

# Validation
- cargo build --release: clean
- cargo build --tests: clean
- 3 GPU Welford kernel tests (G1 constant→0var, G2 sequence→known var,
  G3 sentinel skip): all pass
- 5 GPU adaptive-floor invariant tests (G3 PPO clip holds at bootstrap
  when signal below floor, G4 tightens on single above-band, G5 widens
  only after 3 consecutive below-band, G6 post-warmup uses Welford mean,
  G6 pre-warmup uses EMA): all pass
- integrated_trainer_smoke (full GPU pipeline): passes
- 1k local smoke at b=128: 1000/1000 steps, completed_clean, no NaN,
  controllers genuinely adapting (γ 0.90→0.974, per_α 0.40→0.52,
  q_distill_λ 0.05→0.21, ε held at 0.05 = MIN per Phase 4.5 small-KL
  regime — was previously stuck at MAX 0.50 throughout fold 0/1)
- compute-sanitizer memcheck b=128 5 steps: 0 errors

Cluster validation (G7-G9) submitted as follow-up runs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-30 15:41:51 +02:00

186 lines
6.0 KiB
Rust
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
//! GPU-oracle tests for `rl_signal_variance_update` — Welford online
//! variance kernel for adaptive controller floors (spec
//! `2026-05-30-adaptive-controller-floor-design`).
//!
//! Three analytical invariants, zero CPU reference implementations:
//!
//! 1. `welford_constant_input_zero_variance` — feeding a constant `k`
//! for N steps must produce `mean == k` and `m2 == 0` (sample
//! variance is zero by definition).
//! 2. `welford_known_sequence_variance` — sequence [1, 2, 3, 4, 5]
//! has known sample variance 2.5; Welford output must match.
//! 3. `welford_sentinel_zero_is_skipped` — input == 0.0 sentinel
//! must produce no update (per
//! `pearl_first_observation_bootstrap`).
//!
//! Per `feedback_no_cpu_test_fallbacks`: oracles are analytical
//! identities of Welford's algorithm, not CPU reimplementations.
//! Per `feedback_no_htod_htoh_only_mapped_pinned`: ISV bus uses
//! mapped-pinned `MappedF32Buffer` (same pattern as production trainer).
//!
//! Run with:
//! `cargo test -p ml-alpha --test signal_variance_kernel -- --ignored --nocapture`
use cudarc::driver::{LaunchConfig, PushKernelArg};
use ml_alpha::pinned_mem::MappedF32Buffer;
use ml_core::device::MlDevice;
// ISV slot layout for this isolated test: input at 0, Welford triple at 1,2,3.
const INPUT_SLOT: i32 = 0;
const COUNT_SLOT: i32 = 1;
const MEAN_SLOT: i32 = 2;
const M2_SLOT: i32 = 3;
const N_SLOTS: usize = 4;
const SIGNAL_VARIANCE_UPDATE_CUBIN: &[u8] =
include_bytes!(concat!(env!("OUT_DIR"), "/rl_signal_variance_update.cubin"));
fn launch_once(
dev: &MlDevice,
isv: &MappedF32Buffer,
) {
let stream = dev.cuda_stream().expect("cuda_stream").clone();
let ctx = dev.cuda_context().expect("cuda_context");
let module = ctx
.load_cubin(SIGNAL_VARIANCE_UPDATE_CUBIN.to_vec())
.expect("load signal_variance cubin");
let func = module
.load_function("rl_signal_variance_update")
.expect("load rl_signal_variance_update fn");
let cfg = LaunchConfig {
grid_dim: (1, 1, 1),
block_dim: (1, 1, 1),
shared_mem_bytes: 0,
};
let isv_dev_ptr = isv.dev_ptr;
unsafe {
stream
.launch_builder(&func)
.arg(&isv_dev_ptr)
.arg(&INPUT_SLOT)
.arg(&COUNT_SLOT)
.arg(&MEAN_SLOT)
.arg(&M2_SLOT)
.launch(cfg)
.expect("rl_signal_variance_update launch");
}
stream.synchronize().expect("sync");
}
#[test]
#[ignore = "requires CUDA (MlDevice::cuda(0))"]
fn welford_constant_input_zero_variance() {
let dev = match MlDevice::cuda(0) {
Ok(d) => d,
Err(e) => {
eprintln!("CUDA 0 not available — skipping ({e})");
return;
}
};
let isv = unsafe { MappedF32Buffer::new(N_SLOTS) }.expect("isv alloc");
// Welford of constant k for N steps: mean = k, m2 = 0, count = N.
let k = 0.5_f32;
let n = 50_usize;
for _ in 0..n {
isv.write_record(INPUT_SLOT as usize, k);
launch_once(&dev, &isv);
}
let count = isv.read_record(COUNT_SLOT as usize);
let mean = isv.read_record(MEAN_SLOT as usize);
let m2 = isv.read_record(M2_SLOT as usize);
assert_eq!(count, n as f32, "count must be {n}; got {count}");
assert!(
(mean - k).abs() < 1e-6,
"mean of constant {k} must equal {k}; got {mean}"
);
assert!(
m2.abs() < 1e-5,
"m2 of constant input must be exactly 0 (or fp-noise); got {m2}"
);
eprintln!("G1 OK — constant k={k} for N={n}: mean={mean} m2={m2}");
}
#[test]
#[ignore = "requires CUDA (MlDevice::cuda(0))"]
fn welford_known_sequence_variance() {
let dev = match MlDevice::cuda(0) {
Ok(d) => d,
Err(e) => {
eprintln!("CUDA 0 not available — skipping ({e})");
return;
}
};
let isv = unsafe { MappedF32Buffer::new(N_SLOTS) }.expect("isv alloc");
// Sequence [1, 2, 3, 4, 5]: arithmetic mean = 3.
// Sample variance = Σ(x_i μ)² / (n 1) = (4+1+0+1+4)/4 = 2.5.
let seq = [1.0_f32, 2.0, 3.0, 4.0, 5.0];
for &x in seq.iter() {
isv.write_record(INPUT_SLOT as usize, x);
launch_once(&dev, &isv);
}
let count = isv.read_record(COUNT_SLOT as usize);
let mean = isv.read_record(MEAN_SLOT as usize);
let m2 = isv.read_record(M2_SLOT as usize);
let sample_var = m2 / (count - 1.0);
assert_eq!(count, seq.len() as f32, "count must be {}", seq.len());
assert!(
(mean - 3.0).abs() < 1e-5,
"mean of [1..5] must be 3.0; got {mean}"
);
assert!(
(sample_var - 2.5).abs() < 1e-5,
"sample variance of [1..5] must be 2.5; got {sample_var} (m2={m2})"
);
eprintln!("G2 OK — sequence {seq:?}: mean={mean} sample_var={sample_var}");
}
#[test]
#[ignore = "requires CUDA (MlDevice::cuda(0))"]
fn welford_sentinel_zero_is_skipped() {
let dev = match MlDevice::cuda(0) {
Ok(d) => d,
Err(e) => {
eprintln!("CUDA 0 not available — skipping ({e})");
return;
}
};
let isv = unsafe { MappedF32Buffer::new(N_SLOTS) }.expect("isv alloc");
// INPUT_SLOT remains at sentinel 0.0 (MappedF32Buffer::new zero-inits).
// Kernel must NOT count this as an observation per
// pearl_first_observation_bootstrap.
for _ in 0..10 {
launch_once(&dev, &isv);
}
assert_eq!(
isv.read_record(COUNT_SLOT as usize),
0.0,
"sentinel zero must not be counted as a real observation"
);
// Now feed a real non-zero observation; counter must advance.
isv.write_record(INPUT_SLOT as usize, 7.0);
launch_once(&dev, &isv);
assert_eq!(
isv.read_record(COUNT_SLOT as usize),
1.0,
"first real observation must be counted"
);
assert!(
(isv.read_record(MEAN_SLOT as usize) - 7.0).abs() < 1e-6,
"first observation must initialize mean directly per Welford recurrence"
);
eprintln!("G3 OK — sentinel skip honoured; first real obs initialized mean");
}