Replaces hardcoded thresholds AND clamp bounds across 12 RL controllers
with observed-signal-driven ISV-slot bounds. Eliminates the architectural
failure mode that surfaced in walk-forward fold 0 (alpha-rl-m9cx5) and
fold 1 (alpha-rl-jgdh6): under Phase 4.5 advantage normalization, eight
controllers (PPO clip, target_tau, rollout_steps, entropy_coef, per_α,
gamma, reward_scale, q_distill_lambda) saturated at extrema within ~50
steps and stayed pinned for the rest of training — same pattern across
two different data slices, confirming the bug is structural rather than
data-size-dependent.
Mechanism: each controller's noise floor was hardcoded as a small
fraction of its target (`_NOISE_FLOOR_FRAC = 0.01f`), calibrated against
a pre-Phase-4.5 signal regime. Phase 4.5 normalization reduces operating
KL by ~50× — observed signal stays below the 1%-of-target floor, the
Schulman widen path fires continuously (asymmetric in the wrong
direction), and ε hits MAX 0.50 within ~50 training steps. ksll2's full-
data run (n_folds=1) happened to escape via a single above-band KL
observation that triggered tighten; both walk-forward folds (3 and 6
files) did not.
Per `feedback_adaptive_not_tuned`, `feedback_isv_for_adaptive_bounds`,
`pearl_controller_anchors_isv_driven`: every threshold now derives from
observed signal statistics (Welford online variance) rather than
constants calibrated against a prior signal regime. User explicitly
expanded scope mid-implementation: "if all clamps ISV bound should be
added to this spec and tasks" — applying the principle consistently
means hardcoded MIN/MAX clamp bounds count too, not just the saturating
noise floors. 12 controllers, single atomic commit per
`feedback_no_partial_refactor`.
Spec: docs/superpowers/specs/2026-05-30-adaptive-controller-floor-design.md
Plan: docs/superpowers/plans/2026-05-30-adaptive-controller-floor-plan.md
# New kernel
- rl_signal_variance_update.cu (80 LOC) — Welford online variance.
Single-thread single-block. Sentinel-zero skip per pearl. Per-controller
Welford triple (count, mean, M²) drives every adaptive noise floor.
# Per-controller refactors (12 .cu files)
- ppo_clip + target_tau + rollout_steps + entropy_coef + per_α —
adaptive noise floor = max(target × 0.5, sqrt(observed_var) × 2);
asymmetric Schulman (tighten on single observation, widen requires 3
consecutive below-band).
- gamma (Special G) — hardcoded GAMMA_MIN = 0.995 → adaptive via
Welford MEAN of trade duration. Per spec Q6 resolution: the Welford
mean's natural N-smoothing lag breaks the gamma↔trade_duration
feedback loop without explicit step-period gating. 100-observation
warmup falls back to EMA before Welford has enough samples.
- reward_scale (Special R) — asymmetric DECREASE rate cap (5% per
step) on both bootstrap-replace and Wiener-blend paths; bootstrap-
fraction floor (10% of bootstrap) until 100 trades close.
- q_distill_lambda (Special Q) — hardcoded MIN_LAMBDA = 0.05 → adaptive
via Welford on q_distill_kl_ema: max(0.001, std × 0.05).
- v_blend_alpha (Phase 4.4) — 5 hardcoded constants → 5 ISV slots
(DEAD_SIGNAL_FLOOR, TARGET_TRACK_RATIO, SCHULMAN_STEP, EMA_ALPHA,
BOOTSTRAP_ALPHA) + adaptive dead-signal floor from V_scalar magnitude
variance.
- ppo_ratio_clamp — adaptive MIN/MAX from observed log-ratio variance.
Architectural 2.0 absolute floor preserved (don't degenerate to
vanilla policy gradient).
- reward_clamp — V_BOUND_FLOOR, V_BOUND_EWMA_ALPHA, MIN_WIN, MIN_RATIO,
MAX_RATIO, MIN/MAX_MARGIN, MARGIN_TOLERANCE/ADJUST_RATE,
CLIP_RATE_EMA_ALPHA → 10 ISV slots.
- gate_threshold — hardcoded alpha = 0.01 → ISV slot 638.
- Clamp-bound expansion: EPS_MIN/MAX, TAU_MIN, ROLLOUT_MIN/MAX,
COEF_MIN/MAX, PER_ALPHA_MIN/MAX, GAMMA_MAX, MAX_LAMBDA, KL_TOLERANCE,
LAMBDA_RAMP_RATE, LAMBDA_DECAY_RATE → all ISV.
- WIENER_ALPHA_FLOOR shared across 9 controllers → single ISV slot 659.
# Trainer integration (integrated.rs)
- rl_signal_variance_update kernel loaded + helper method
`launch_rl_signal_variance_update`.
- Per-step Welford launches for 9 controller inputs, placed between
EMA producers and rl_fused_controllers in the per-step pipeline.
- Input-slot lookup array for rl_fused_controllers updated: rollout_steps
now consumes RL_ADV_VAR_PRE_NORM_INDEX (emitted by
rl_advantage_normalize before in-place normalize) instead of the
post-norm advantage_var_ratio (definitionally ~0 under Phase 4.5).
- ~30 new ISV bootstrap entries in with_controllers_bootstrapped.
# ISV slot allocation (isv_slots.rs)
- 72 new slots, RL_SLOTS_END 588 → 660. +288 bytes mapped-pinned.
- 9 Welford variance triples + 5 asymmetric Schulman counters +
3 new input signals (var_pre_norm, gamma_min_adaptive, reward_mag) +
v_blend (5 + 3 Welford) + ppo_ratio_clamp (1 + 3 Welford) +
reward_clamp (5) + gate_threshold (1) + q_distill (1) + 20 clamp bounds +
shared wiener floor.
# Bootstrap-clamp consistency fix (caught by Task 16 testing)
The original draft bootstrapped RL_EPS_BOOTSTRAP at 0.01 (= KL target),
but EPS_MIN was 0.05 — bootstrap value below clamp range. First post-
bootstrap step always snapped ε to MIN regardless of signal direction
("snap to MIN" behavior the unit tests surfaced). Fixed by bootstrapping
ε at MIN (0.05) so asymmetric Schulman operates from a valid state.
# Validation
- cargo build --release: clean
- cargo build --tests: clean
- 3 GPU Welford kernel tests (G1 constant→0var, G2 sequence→known var,
G3 sentinel skip): all pass
- 5 GPU adaptive-floor invariant tests (G3 PPO clip holds at bootstrap
when signal below floor, G4 tightens on single above-band, G5 widens
only after 3 consecutive below-band, G6 post-warmup uses Welford mean,
G6 pre-warmup uses EMA): all pass
- integrated_trainer_smoke (full GPU pipeline): passes
- 1k local smoke at b=128: 1000/1000 steps, completed_clean, no NaN,
controllers genuinely adapting (γ 0.90→0.974, per_α 0.40→0.52,
q_distill_λ 0.05→0.21, ε held at 0.05 = MIN per Phase 4.5 small-KL
regime — was previously stuck at MAX 0.50 throughout fold 0/1)
- compute-sanitizer memcheck b=128 5 steps: 0 errors
Cluster validation (G7-G9) submitted as follow-up runs.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
525 lines
23 KiB
Rust
525 lines
23 KiB
Rust
//! Task 16 — Controller-level adaptive-floor invariant tests for the
|
||
//! 2026-05-30 adaptive-controller-floor refactor (spec
|
||
//! `docs/superpowers/specs/2026-05-30-adaptive-controller-floor-design.md`).
|
||
//!
|
||
//! Validates Validation Gates G3-G6 against the fused production launch
|
||
//! path (`launch_rl_fused_controllers`). Per
|
||
//! `feedback_no_cpu_test_fallbacks`: every oracle here is an analytical
|
||
//! identity of the controller's documented adaptive math — adaptive
|
||
//! noise floor (`max(target × 0.5, 2σ)`), asymmetric Schulman
|
||
//! (tighten on single above-band observation, widen requires N ≥ 3
|
||
//! consecutive below-band) and gamma's `1 − 1/(d × 2)` adaptive-min
|
||
//! formula. No CPU reference reimplementation; the kernel's spec is the
|
||
//! oracle.
|
||
//!
|
||
//! Per `pearl_first_observation_bootstrap`: the bootstrap-replace path
|
||
//! (`eps_prev == RL_EPS_BOOTSTRAP_INDEX`) skips the Wiener blend so the
|
||
//! first real observation lands directly at the computed `eps_target`.
|
||
//! For G4/G5 we deliberately PRE-WRITE `RL_PPO_CLIP_INDEX` to a value
|
||
//! away from the bootstrap so the Wiener-blend branch is exercised; this
|
||
//! isolates the asymmetric-Schulman invariant from the bootstrap-replace
|
||
//! quirk. G3 stays on the natural post-bootstrap state (eps_prev =
|
||
//! bootstrap = 0.05 = EPS_MIN) because the noise-floor gate returns
|
||
//! before either branch fires — the invariant under test is "controller
|
||
//! holds".
|
||
//!
|
||
//! Run with:
|
||
//! `cargo test -p ml-alpha --test controller_adaptive_floors -- --ignored --nocapture`
|
||
|
||
use ml_alpha::rl::isv_slots::{
|
||
RL_GAMMA_INDEX, RL_GAMMA_MIN_ADAPTIVE_INDEX, RL_KL_PI_BELOW_COUNT_INDEX,
|
||
RL_KL_PI_EMA_INDEX, RL_KL_PI_VAR_COUNT_INDEX, RL_KL_PI_VAR_M2_INDEX,
|
||
RL_KL_PI_VAR_MEAN_INDEX, RL_KL_TARGET_INDEX, RL_MEAN_TRADE_DURATION_EMA_INDEX,
|
||
RL_PPO_CLIP_EPS_MAX_INDEX, RL_PPO_CLIP_INDEX, RL_SCHULMAN_ADJUST_RATE_INDEX,
|
||
RL_SCHULMAN_TOLERANCE_INDEX, RL_TRADE_DUR_VAR_COUNT_INDEX,
|
||
RL_TRADE_DUR_VAR_M2_INDEX, RL_TRADE_DUR_VAR_MEAN_INDEX,
|
||
};
|
||
use ml_alpha::trainer::integrated::{IntegratedTrainer, IntegratedTrainerConfig};
|
||
use ml_alpha::trainer::perception::PerceptionTrainerConfig;
|
||
use ml_core::device::MlDevice;
|
||
|
||
/// Bootstrap value `rl_ppo_clip_controller` writes into
|
||
/// `ISV[RL_PPO_CLIP_INDEX]` at construction. Per Task 17 fix:
|
||
/// bootstrap matches `EPS_MIN` (= 0.05) so the post-bootstrap step
|
||
/// doesn't snap to MIN on first signal regardless of direction
|
||
/// (bootstrap-clamp inconsistency that surfaced in the Task 16 testing).
|
||
const EPS_BOOTSTRAP: f32 = 0.05;
|
||
|
||
/// PPO ε MIN clamp seeded by `with_controllers_bootstrapped`.
|
||
const EPS_MIN: f32 = 0.05;
|
||
|
||
/// KL target seeded by `with_controllers_bootstrapped`.
|
||
const KL_TARGET: f32 = 0.01;
|
||
|
||
/// Schulman tolerance — kl above `target × tolerance` triggers tighten,
|
||
/// below `target / tolerance` triggers (delayed) widen.
|
||
const SCHULMAN_TOLERANCE: f32 = 1.5;
|
||
|
||
/// Schulman adjust rate — tighten ratio is `1 / adjust_rate`, widen
|
||
/// ratio is `adjust_rate`.
|
||
const SCHULMAN_ADJUST_RATE: f32 = 1.5;
|
||
|
||
/// N-consecutive-below-band threshold for the widen branch (matches
|
||
/// `WIDEN_PATIENCE_CONSECUTIVE` in `rl_ppo_clip_controller.cu`).
|
||
const WIDEN_PATIENCE: f32 = 3.0;
|
||
|
||
/// Hard absolute floor for γ adaptive-min (matches `GAMMA_MIN_ABSOLUTE`
|
||
/// in `rl_gamma_controller.cu`).
|
||
const GAMMA_MIN_ABSOLUTE: f32 = 0.9;
|
||
|
||
/// Welford warmup observations before the running mean replaces the EMA
|
||
/// in the γ adaptive-min derivation (matches `WELFORD_WARMUP_OBS`).
|
||
const WELFORD_WARMUP_OBS: f32 = 100.0;
|
||
|
||
fn build_trainer() -> Option<(MlDevice, IntegratedTrainer)> {
|
||
let dev = match MlDevice::cuda(0) {
|
||
Ok(d) => d,
|
||
Err(e) => {
|
||
eprintln!("CUDA 0 not available — skipping ({e})");
|
||
return None;
|
||
}
|
||
};
|
||
let cfg = IntegratedTrainerConfig {
|
||
perception: PerceptionTrainerConfig {
|
||
seq_len: 4,
|
||
n_batch: 1,
|
||
..PerceptionTrainerConfig::default()
|
||
},
|
||
dqn_seed: 0xC1,
|
||
ppo_seed: 0xC2,
|
||
..IntegratedTrainerConfig::default()
|
||
};
|
||
let trainer = IntegratedTrainer::new(&dev, cfg).expect("IntegratedTrainer::new");
|
||
Some((dev, trainer))
|
||
}
|
||
|
||
/// Sync the trainer's production stream so subsequent mapped-pinned ISV
|
||
/// reads through `read_isv_host` observe device-side writes.
|
||
fn sync(trainer: &IntegratedTrainer) {
|
||
trainer.stream.synchronize().expect("sync trainer stream");
|
||
}
|
||
|
||
// ─────────────────────────────────────────────────────────────────────
|
||
// G3 — PPO clip holds when signal is below the adaptive noise floor.
|
||
//
|
||
// Invariant: after the bootstrap path has fired (eps_prev = bootstrap =
|
||
// 0.01), feeding `kl_pi_ema = 0.001` (< noise_floor = target × 0.5 =
|
||
// 0.005) for 50 fused launches must NOT drift ε. The noise-floor gate
|
||
// is the load-bearing fix — without it, the prior multiplicative-ratio
|
||
// design produced 1e4 ratios from sub-noise KL values and pegged ε at
|
||
// MAX 0.50 within ~50 steps (alpha-rl-mjzfk fold 0 saturation).
|
||
//
|
||
// Welford triple is pre-populated so the std-based half of the floor
|
||
// (`2σ`) is dominated by the target-fraction half (`target × 0.5`),
|
||
// keeping the floor at the steady-state value of 0.005.
|
||
// ─────────────────────────────────────────────────────────────────────
|
||
#[test]
|
||
#[ignore = "requires CUDA (MlDevice::cuda(0))"]
|
||
fn g3_ppo_clip_holds_when_signal_below_adaptive_floor() {
|
||
let Some((_dev, trainer)) = build_trainer() else { return };
|
||
|
||
// Verify pre-conditions seeded by `with_controllers_bootstrapped`.
|
||
sync(&trainer);
|
||
let eps_at_bootstrap = trainer.read_isv_host(RL_PPO_CLIP_INDEX);
|
||
assert!(
|
||
(eps_at_bootstrap - EPS_BOOTSTRAP).abs() < 1e-6,
|
||
"pre-condition: ε must be bootstrapped to {EPS_BOOTSTRAP}; got {eps_at_bootstrap}"
|
||
);
|
||
let kl_target_seeded = trainer.read_isv_host(RL_KL_TARGET_INDEX);
|
||
assert!(
|
||
(kl_target_seeded - KL_TARGET).abs() < 1e-6,
|
||
"pre-condition: KL target must be seeded to {KL_TARGET}; got {kl_target_seeded}"
|
||
);
|
||
|
||
// Pre-populate Welford triple at the kl_pi_ema slot with low M²
|
||
// so std ≈ 0 and the floor reduces to `target × 0.5 = 0.005`. count
|
||
// > 1 is required so the kernel evaluates `M²/(count − 1)` rather
|
||
// than the count ≤ 1 fallback (which would also produce std = 0).
|
||
trainer.isv_mapped.write_record(RL_KL_PI_VAR_COUNT_INDEX, 100.0);
|
||
trainer.isv_mapped.write_record(RL_KL_PI_VAR_MEAN_INDEX, 0.001);
|
||
trainer.isv_mapped.write_record(RL_KL_PI_VAR_M2_INDEX, 0.0);
|
||
|
||
// Feed below-floor signal for 50 fused launches. Re-write each step
|
||
// because a real-world EMA producer would refresh it; here we're
|
||
// testing the controller in isolation.
|
||
for _ in 0..50 {
|
||
trainer.isv_mapped.write_record(RL_KL_PI_EMA_INDEX, 0.001);
|
||
trainer
|
||
.launch_rl_fused_controllers(1)
|
||
.expect("launch_rl_fused_controllers");
|
||
}
|
||
sync(&trainer);
|
||
|
||
let eps_after = trainer.read_isv_host(RL_PPO_CLIP_INDEX);
|
||
|
||
// Primary invariant: ε held — no drift past bootstrap, MUST not have
|
||
// climbed toward MAX 0.5 (the canonical saturation pattern).
|
||
assert!(
|
||
(eps_after - EPS_BOOTSTRAP).abs() < 1e-4,
|
||
"ε must hold at bootstrap when signal is below adaptive floor; \
|
||
expected ≈ {EPS_BOOTSTRAP}, got {eps_after}"
|
||
);
|
||
assert!(
|
||
eps_after < 0.4,
|
||
"ε must NOT drift toward MAX 0.5 — this was the fold 0/1 \
|
||
saturation bug; got {eps_after}"
|
||
);
|
||
|
||
// Negative invariant: the below-band counter must have stayed at 0
|
||
// because the noise-floor gate returns BEFORE the asymmetric Schulman
|
||
// branch fires.
|
||
let below_count = trainer.read_isv_host(RL_KL_PI_BELOW_COUNT_INDEX);
|
||
assert_eq!(
|
||
below_count, 0.0,
|
||
"below-band counter must stay 0 when noise-floor gate held; got {below_count}"
|
||
);
|
||
|
||
eprintln!(
|
||
"G3 OK — ε held at {eps_after} (bootstrap {EPS_BOOTSTRAP}) over 50 below-floor steps; \
|
||
below_count = {below_count} (noise-floor gate fired)"
|
||
);
|
||
}
|
||
|
||
// ─────────────────────────────────────────────────────────────────────
|
||
// G4 — PPO clip tightens on a single above-band observation
|
||
// (asymmetric Schulman — fast tighten).
|
||
//
|
||
// Invariant: a single `kl_pi_ema > target × tolerance` observation must
|
||
// decrease ε within the same call. Tighten ratio = `1 / adjust_rate ≈
|
||
// 0.667`; the assertion is strict decrease (ε_after < ε_before).
|
||
//
|
||
// Test discipline: we pre-write ε to 0.2 (well above MIN 0.05 and away
|
||
// from bootstrap 0.01) so:
|
||
// 1. The bootstrap-replace branch DOESN'T fire (eps_prev ≠ bootstrap).
|
||
// 2. `eps_target = 0.2 × 0.667 ≈ 0.133` stays above MIN, so the clamp
|
||
// doesn't bind and obscure the tighten signal.
|
||
// The Wiener-α blend (alpha = 0.4 = Wiener floor) then produces
|
||
// `eps_new = 0.6 × 0.2 + 0.4 × 0.133 ≈ 0.173`. Below 0.2 → strict
|
||
// decrease.
|
||
// ─────────────────────────────────────────────────────────────────────
|
||
#[test]
|
||
#[ignore = "requires CUDA (MlDevice::cuda(0))"]
|
||
fn g4_ppo_clip_tightens_on_single_above_band_observation() {
|
||
let Some((_dev, trainer)) = build_trainer() else { return };
|
||
|
||
// Pre-write ε to 0.2 — away from bootstrap, above MIN clamp.
|
||
let eps_initial: f32 = 0.2;
|
||
trainer.isv_mapped.write_record(RL_PPO_CLIP_INDEX, eps_initial);
|
||
|
||
// Welford triple — std ≈ 0 so the noise floor is `target × 0.5 = 0.005`.
|
||
trainer.isv_mapped.write_record(RL_KL_PI_VAR_COUNT_INDEX, 100.0);
|
||
trainer.isv_mapped.write_record(RL_KL_PI_VAR_MEAN_INDEX, 0.01);
|
||
trainer.isv_mapped.write_record(RL_KL_PI_VAR_M2_INDEX, 0.0);
|
||
|
||
// Above-band signal: kl = 0.05 = 5× target = 3.33× tolerance bound.
|
||
trainer.isv_mapped.write_record(RL_KL_PI_EMA_INDEX, 0.05);
|
||
|
||
// Verify Schulman config matches what the controller will read.
|
||
let tolerance = trainer.read_isv_host(RL_SCHULMAN_TOLERANCE_INDEX);
|
||
let adjust_rate = trainer.read_isv_host(RL_SCHULMAN_ADJUST_RATE_INDEX);
|
||
assert!(
|
||
(tolerance - SCHULMAN_TOLERANCE).abs() < 1e-6,
|
||
"pre-condition: Schulman tolerance seeded to {SCHULMAN_TOLERANCE}; got {tolerance}"
|
||
);
|
||
assert!(
|
||
(adjust_rate - SCHULMAN_ADJUST_RATE).abs() < 1e-6,
|
||
"pre-condition: Schulman adjust_rate seeded to {SCHULMAN_ADJUST_RATE}; got {adjust_rate}"
|
||
);
|
||
|
||
trainer
|
||
.launch_rl_fused_controllers(1)
|
||
.expect("launch_rl_fused_controllers");
|
||
sync(&trainer);
|
||
|
||
let eps_after = trainer.read_isv_host(RL_PPO_CLIP_INDEX);
|
||
|
||
// Primary invariant: ε strictly decreased on a single above-band fire.
|
||
assert!(
|
||
eps_after < eps_initial,
|
||
"ε must tighten on single above-band observation; \
|
||
before = {eps_initial}, after = {eps_after}"
|
||
);
|
||
// Above the MIN clamp — the decrease is genuine adaptation, not a
|
||
// saturation pin.
|
||
assert!(
|
||
eps_after > EPS_MIN,
|
||
"ε must stay above MIN {EPS_MIN} after a single fire (clamp \
|
||
not binding); got {eps_after}"
|
||
);
|
||
|
||
// The below-band counter must reset on a tighten event (the kernel's
|
||
// `else if (kl_ema > target × tolerance) { … below_count = 0 }`).
|
||
let below_count = trainer.read_isv_host(RL_KL_PI_BELOW_COUNT_INDEX);
|
||
assert_eq!(
|
||
below_count, 0.0,
|
||
"below-band counter must reset on tighten; got {below_count}"
|
||
);
|
||
|
||
eprintln!(
|
||
"G4 OK — ε tightened {eps_initial} → {eps_after} (single above-band fire); \
|
||
below_count reset to 0"
|
||
);
|
||
}
|
||
|
||
// ─────────────────────────────────────────────────────────────────────
|
||
// G5 — PPO clip widens only after N consecutive below-band observations
|
||
// (asymmetric Schulman — slow widen).
|
||
//
|
||
// Invariant: a kl_pi_ema in the widen-band `[noise_floor, target /
|
||
// tolerance) = [0.005, 0.00667)` must increment `below_count` each step
|
||
// but NOT widen ε until `below_count ≥ WIDEN_PATIENCE_CONSECUTIVE = 3`.
|
||
//
|
||
// We choose kl = 0.006 because:
|
||
// - 0.006 ≥ 0.005 (noise floor — passes the gate)
|
||
// - 0.006 < 0.00667 (target / tolerance — triggers widen branch, not
|
||
// hold)
|
||
// To keep the noise floor low (so kl=0.006 passes it), Welford std must
|
||
// be small: count=100, M²=0 → std=0 → floor = target × 0.5 = 0.005.
|
||
//
|
||
// Test discipline: ε pre-written to 0.2 so the bootstrap-replace branch
|
||
// doesn't fire. We expect:
|
||
// step 1: below_count 0 → 1, ratio = 1 (hold), ε unchanged
|
||
// step 2: below_count 1 → 2, ratio = 1 (hold), ε unchanged
|
||
// step 3: below_count 2 → 3, ratio = adjust_rate = 1.5 (widen), ε
|
||
// increases via Wiener blend
|
||
// step 4: below_count 3 → 4, ratio = 1.5 (widen continues)
|
||
// ─────────────────────────────────────────────────────────────────────
|
||
#[test]
|
||
#[ignore = "requires CUDA (MlDevice::cuda(0))"]
|
||
fn g5_ppo_clip_widens_only_after_n_consecutive_below_band() {
|
||
let Some((_dev, trainer)) = build_trainer() else { return };
|
||
|
||
let eps_initial: f32 = 0.2;
|
||
trainer.isv_mapped.write_record(RL_PPO_CLIP_INDEX, eps_initial);
|
||
|
||
// Welford triple — low std so noise floor = target × 0.5 = 0.005.
|
||
trainer.isv_mapped.write_record(RL_KL_PI_VAR_COUNT_INDEX, 100.0);
|
||
trainer.isv_mapped.write_record(RL_KL_PI_VAR_MEAN_INDEX, 0.006);
|
||
trainer.isv_mapped.write_record(RL_KL_PI_VAR_M2_INDEX, 0.0);
|
||
|
||
// Counter must start at 0.
|
||
trainer
|
||
.isv_mapped
|
||
.write_record(RL_KL_PI_BELOW_COUNT_INDEX, 0.0);
|
||
|
||
// Widen-band signal: 0.005 ≤ 0.006 < 0.00667.
|
||
let kl_widen: f32 = 0.006;
|
||
|
||
// Compute the widen-band edges from the canonical formulas so the
|
||
// assertion that 0.006 falls inside is auditable.
|
||
let noise_floor = KL_TARGET * 0.5;
|
||
let widen_upper = KL_TARGET / SCHULMAN_TOLERANCE;
|
||
assert!(
|
||
kl_widen >= noise_floor,
|
||
"kl_widen {kl_widen} must be ≥ noise floor {noise_floor}"
|
||
);
|
||
assert!(
|
||
kl_widen < widen_upper,
|
||
"kl_widen {kl_widen} must be < widen upper edge {widen_upper}"
|
||
);
|
||
|
||
let mut below_counts = Vec::with_capacity(4);
|
||
let mut eps_values = Vec::with_capacity(4);
|
||
|
||
for _ in 0..4 {
|
||
trainer.isv_mapped.write_record(RL_KL_PI_EMA_INDEX, kl_widen);
|
||
trainer
|
||
.launch_rl_fused_controllers(1)
|
||
.expect("launch_rl_fused_controllers");
|
||
sync(&trainer);
|
||
below_counts.push(trainer.read_isv_host(RL_KL_PI_BELOW_COUNT_INDEX));
|
||
eps_values.push(trainer.read_isv_host(RL_PPO_CLIP_INDEX));
|
||
}
|
||
|
||
for (i, (bc, eps)) in below_counts.iter().zip(eps_values.iter()).enumerate() {
|
||
eprintln!(" step {}: below_count = {bc}, ε = {eps}", i + 1);
|
||
}
|
||
|
||
// Below-counter invariant: increments monotonically per step.
|
||
assert_eq!(below_counts[0], 1.0, "step 1: below_count must be 1");
|
||
assert_eq!(below_counts[1], 2.0, "step 2: below_count must be 2");
|
||
assert_eq!(
|
||
below_counts[2], WIDEN_PATIENCE,
|
||
"step 3: below_count must reach WIDEN_PATIENCE {WIDEN_PATIENCE}"
|
||
);
|
||
assert_eq!(below_counts[3], 4.0, "step 4: below_count must be 4");
|
||
|
||
// ε hold invariant: steps 1 and 2 (below_count < 3) MUST NOT widen.
|
||
// Allow only float-rounding-level drift (the controller's
|
||
// `ratio = 1.0f` branch produces eps_target = eps_prev, then the
|
||
// Wiener blend with eps_target == eps_prev is the identity).
|
||
let drift_1 = (eps_values[0] - eps_initial).abs();
|
||
let drift_2 = (eps_values[1] - eps_initial).abs();
|
||
assert!(
|
||
drift_1 < 1e-5,
|
||
"step 1: ε must hold (count < WIDEN_PATIENCE); drift {drift_1}, value {}",
|
||
eps_values[0]
|
||
);
|
||
assert!(
|
||
drift_2 < 1e-5,
|
||
"step 2: ε must hold (count < WIDEN_PATIENCE); drift {drift_2}, value {}",
|
||
eps_values[1]
|
||
);
|
||
|
||
// ε widen invariant: step 3 (below_count reaches WIDEN_PATIENCE) MUST
|
||
// strictly increase ε. Step 4 continues widening.
|
||
assert!(
|
||
eps_values[2] > eps_initial + 1e-5,
|
||
"step 3: ε must widen once below_count reaches {WIDEN_PATIENCE}; \
|
||
was {eps_initial}, got {}",
|
||
eps_values[2]
|
||
);
|
||
assert!(
|
||
eps_values[3] > eps_values[2],
|
||
"step 4: ε must continue widening; step 3 = {}, step 4 = {}",
|
||
eps_values[2],
|
||
eps_values[3]
|
||
);
|
||
|
||
// ε stays under MAX clamp — widening is real adaptation, not a
|
||
// saturation pin.
|
||
let eps_max = trainer.read_isv_host(RL_PPO_CLIP_EPS_MAX_INDEX);
|
||
assert!(
|
||
eps_values[3] < eps_max,
|
||
"ε must stay under MAX {eps_max} (no saturation pin); got {}",
|
||
eps_values[3]
|
||
);
|
||
|
||
eprintln!(
|
||
"G5 OK — ε held for 2 steps, widened on step 3 (count reached {WIDEN_PATIENCE}): \
|
||
{eps_initial} → {} → {} → {} → {}",
|
||
eps_values[0], eps_values[1], eps_values[2], eps_values[3]
|
||
);
|
||
}
|
||
|
||
// ─────────────────────────────────────────────────────────────────────
|
||
// G6 — γ adaptive_min tracks trade duration via Welford mean (post-100-
|
||
// observation warmup) and the EMA before warmup.
|
||
//
|
||
// Invariant (post-warmup): with Welford count ≥ 100 and Welford mean d,
|
||
// the kernel must write
|
||
//
|
||
// RL_GAMMA_MIN_ADAPTIVE = max(0.9, 1 − 1/max(d × 2, 10))
|
||
//
|
||
// to ISV[613]. At d = 30: `1 − 1/60 ≈ 0.98333`.
|
||
//
|
||
// Invariant (pre-warmup): with count < 100, the formula uses the EMA at
|
||
// `RL_MEAN_TRADE_DURATION_EMA_INDEX` instead of the Welford mean. At
|
||
// EMA = 15: `1 − 1/30 ≈ 0.96667`.
|
||
//
|
||
// Per spec Section "Special case G — γ_min coupling": the Welford
|
||
// mean's N-smoothing breaks the γ ↔ trade_duration feedback loop without
|
||
// any explicit cool-down. Welford count provides the confidence
|
||
// threshold for the switchover.
|
||
// ─────────────────────────────────────────────────────────────────────
|
||
#[test]
|
||
#[ignore = "requires CUDA (MlDevice::cuda(0))"]
|
||
fn g6_gamma_adaptive_min_post_warmup_uses_welford_mean() {
|
||
let Some((_dev, trainer)) = build_trainer() else { return };
|
||
|
||
// Post-warmup state: count = 100 = WELFORD_WARMUP_OBS, mean = 30.
|
||
let d_welford: f32 = 30.0;
|
||
trainer
|
||
.isv_mapped
|
||
.write_record(RL_TRADE_DUR_VAR_COUNT_INDEX, WELFORD_WARMUP_OBS);
|
||
trainer
|
||
.isv_mapped
|
||
.write_record(RL_TRADE_DUR_VAR_MEAN_INDEX, d_welford);
|
||
trainer.isv_mapped.write_record(RL_TRADE_DUR_VAR_M2_INDEX, 0.0);
|
||
// Also write the EMA — even though the kernel ignores it after
|
||
// warmup, mismatching it from the Welford mean tests that the
|
||
// controller actually selects the Welford path (not the EMA).
|
||
trainer
|
||
.isv_mapped
|
||
.write_record(RL_MEAN_TRADE_DURATION_EMA_INDEX, d_welford);
|
||
|
||
trainer
|
||
.launch_rl_fused_controllers(1)
|
||
.expect("launch_rl_fused_controllers");
|
||
sync(&trainer);
|
||
|
||
let adaptive_min = trainer.read_isv_host(RL_GAMMA_MIN_ADAPTIVE_INDEX);
|
||
|
||
// Analytical oracle: `max(0.9, 1 - 1/max(d × 2, 10))`. At d = 30,
|
||
// d × 2 = 60 > 10 = HORIZON_FLOOR, so the inner `max` resolves to 60
|
||
// and the formula reduces to `1 - 1/60`.
|
||
let horizon = (d_welford * 2.0_f32).max(10.0);
|
||
let expected = (1.0_f32 - 1.0 / horizon).max(GAMMA_MIN_ABSOLUTE);
|
||
assert!(
|
||
(adaptive_min - expected).abs() < 1e-4,
|
||
"post-warmup γ_min must be max(0.9, 1 - 1/{horizon}) = {expected}; got {adaptive_min}"
|
||
);
|
||
|
||
// γ itself must respect the new floor — bootstrap value 0.9 was
|
||
// computed against the cold-start adaptive_min 0.9; after the
|
||
// controller re-fires with the new welford mean, γ should be
|
||
// clamped to the new floor.
|
||
let gamma = trainer.read_isv_host(RL_GAMMA_INDEX);
|
||
assert!(
|
||
gamma >= adaptive_min - 1e-4,
|
||
"γ must respect adaptive_min {adaptive_min}; got {gamma}"
|
||
);
|
||
|
||
eprintln!(
|
||
"G6 OK (post-warmup) — adaptive_min = {adaptive_min} (expected {expected}) \
|
||
at Welford count = {WELFORD_WARMUP_OBS}, mean = {d_welford}; γ = {gamma}"
|
||
);
|
||
}
|
||
|
||
#[test]
|
||
#[ignore = "requires CUDA (MlDevice::cuda(0))"]
|
||
fn g6_gamma_adaptive_min_pre_warmup_uses_ema() {
|
||
let Some((_dev, trainer)) = build_trainer() else { return };
|
||
|
||
// Pre-warmup state: count = 99 < WELFORD_WARMUP_OBS = 100. The
|
||
// controller MUST fall back to the EMA slot, not the Welford mean.
|
||
// Deliberately set EMA ≠ Welford mean so the test confirms the
|
||
// controller's selection logic.
|
||
let d_ema: f32 = 15.0;
|
||
let d_welford: f32 = 30.0;
|
||
trainer
|
||
.isv_mapped
|
||
.write_record(RL_TRADE_DUR_VAR_COUNT_INDEX, WELFORD_WARMUP_OBS - 1.0);
|
||
trainer
|
||
.isv_mapped
|
||
.write_record(RL_TRADE_DUR_VAR_MEAN_INDEX, d_welford);
|
||
trainer.isv_mapped.write_record(RL_TRADE_DUR_VAR_M2_INDEX, 0.0);
|
||
trainer
|
||
.isv_mapped
|
||
.write_record(RL_MEAN_TRADE_DURATION_EMA_INDEX, d_ema);
|
||
|
||
trainer
|
||
.launch_rl_fused_controllers(1)
|
||
.expect("launch_rl_fused_controllers");
|
||
sync(&trainer);
|
||
|
||
let adaptive_min = trainer.read_isv_host(RL_GAMMA_MIN_ADAPTIVE_INDEX);
|
||
|
||
// Oracle uses the EMA (15), not the Welford mean (30).
|
||
let horizon = (d_ema * 2.0_f32).max(10.0);
|
||
let expected = (1.0_f32 - 1.0 / horizon).max(GAMMA_MIN_ABSOLUTE);
|
||
assert!(
|
||
(adaptive_min - expected).abs() < 1e-4,
|
||
"pre-warmup γ_min must use EMA, not Welford mean: \
|
||
expected max(0.9, 1 - 1/{horizon}) = {expected}; got {adaptive_min}"
|
||
);
|
||
|
||
// Counter-invariant: had the controller used the Welford mean
|
||
// instead (count = 99 bug), adaptive_min would be ≈ 0.98333. The
|
||
// test fails if that magic number appears.
|
||
let welford_horizon = (d_welford * 2.0_f32).max(10.0);
|
||
let wrong_min = (1.0_f32 - 1.0 / welford_horizon).max(GAMMA_MIN_ABSOLUTE);
|
||
assert!(
|
||
(adaptive_min - wrong_min).abs() > 1e-3,
|
||
"pre-warmup γ_min must NOT match the Welford-mean value {wrong_min} \
|
||
(would indicate the count < WARMUP guard is broken); got {adaptive_min}"
|
||
);
|
||
|
||
eprintln!(
|
||
"G6 OK (pre-warmup) — adaptive_min = {adaptive_min} (expected {expected}, \
|
||
distinct from welford-path {wrong_min}) at Welford count = {}, EMA = {d_ema}",
|
||
WELFORD_WARMUP_OBS - 1.0
|
||
);
|
||
}
|
||
|