Files
foxhunt/crates/ml-alpha/tests/controller_adaptive_floors.rs
jgrusewski 083a88f7c3 feat(rl): adaptive controller floors — 12 controllers, all signal-driven
Replaces hardcoded thresholds AND clamp bounds across 12 RL controllers
with observed-signal-driven ISV-slot bounds. Eliminates the architectural
failure mode that surfaced in walk-forward fold 0 (alpha-rl-m9cx5) and
fold 1 (alpha-rl-jgdh6): under Phase 4.5 advantage normalization, eight
controllers (PPO clip, target_tau, rollout_steps, entropy_coef, per_α,
gamma, reward_scale, q_distill_lambda) saturated at extrema within ~50
steps and stayed pinned for the rest of training — same pattern across
two different data slices, confirming the bug is structural rather than
data-size-dependent.

Mechanism: each controller's noise floor was hardcoded as a small
fraction of its target (`_NOISE_FLOOR_FRAC = 0.01f`), calibrated against
a pre-Phase-4.5 signal regime. Phase 4.5 normalization reduces operating
KL by ~50× — observed signal stays below the 1%-of-target floor, the
Schulman widen path fires continuously (asymmetric in the wrong
direction), and ε hits MAX 0.50 within ~50 training steps. ksll2's full-
data run (n_folds=1) happened to escape via a single above-band KL
observation that triggered tighten; both walk-forward folds (3 and 6
files) did not.

Per `feedback_adaptive_not_tuned`, `feedback_isv_for_adaptive_bounds`,
`pearl_controller_anchors_isv_driven`: every threshold now derives from
observed signal statistics (Welford online variance) rather than
constants calibrated against a prior signal regime. User explicitly
expanded scope mid-implementation: "if all clamps ISV bound should be
added to this spec and tasks" — applying the principle consistently
means hardcoded MIN/MAX clamp bounds count too, not just the saturating
noise floors. 12 controllers, single atomic commit per
`feedback_no_partial_refactor`.

Spec: docs/superpowers/specs/2026-05-30-adaptive-controller-floor-design.md
Plan: docs/superpowers/plans/2026-05-30-adaptive-controller-floor-plan.md

# New kernel
- rl_signal_variance_update.cu (80 LOC) — Welford online variance.
  Single-thread single-block. Sentinel-zero skip per pearl. Per-controller
  Welford triple (count, mean, M²) drives every adaptive noise floor.

# Per-controller refactors (12 .cu files)
- ppo_clip + target_tau + rollout_steps + entropy_coef + per_α —
  adaptive noise floor = max(target × 0.5, sqrt(observed_var) × 2);
  asymmetric Schulman (tighten on single observation, widen requires 3
  consecutive below-band).
- gamma (Special G) — hardcoded GAMMA_MIN = 0.995 → adaptive via
  Welford MEAN of trade duration. Per spec Q6 resolution: the Welford
  mean's natural N-smoothing lag breaks the gamma↔trade_duration
  feedback loop without explicit step-period gating. 100-observation
  warmup falls back to EMA before Welford has enough samples.
- reward_scale (Special R) — asymmetric DECREASE rate cap (5% per
  step) on both bootstrap-replace and Wiener-blend paths; bootstrap-
  fraction floor (10% of bootstrap) until 100 trades close.
- q_distill_lambda (Special Q) — hardcoded MIN_LAMBDA = 0.05 → adaptive
  via Welford on q_distill_kl_ema: max(0.001, std × 0.05).
- v_blend_alpha (Phase 4.4) — 5 hardcoded constants → 5 ISV slots
  (DEAD_SIGNAL_FLOOR, TARGET_TRACK_RATIO, SCHULMAN_STEP, EMA_ALPHA,
  BOOTSTRAP_ALPHA) + adaptive dead-signal floor from V_scalar magnitude
  variance.
- ppo_ratio_clamp — adaptive MIN/MAX from observed log-ratio variance.
  Architectural 2.0 absolute floor preserved (don't degenerate to
  vanilla policy gradient).
- reward_clamp — V_BOUND_FLOOR, V_BOUND_EWMA_ALPHA, MIN_WIN, MIN_RATIO,
  MAX_RATIO, MIN/MAX_MARGIN, MARGIN_TOLERANCE/ADJUST_RATE,
  CLIP_RATE_EMA_ALPHA → 10 ISV slots.
- gate_threshold — hardcoded alpha = 0.01 → ISV slot 638.
- Clamp-bound expansion: EPS_MIN/MAX, TAU_MIN, ROLLOUT_MIN/MAX,
  COEF_MIN/MAX, PER_ALPHA_MIN/MAX, GAMMA_MAX, MAX_LAMBDA, KL_TOLERANCE,
  LAMBDA_RAMP_RATE, LAMBDA_DECAY_RATE → all ISV.
- WIENER_ALPHA_FLOOR shared across 9 controllers → single ISV slot 659.

# Trainer integration (integrated.rs)
- rl_signal_variance_update kernel loaded + helper method
  `launch_rl_signal_variance_update`.
- Per-step Welford launches for 9 controller inputs, placed between
  EMA producers and rl_fused_controllers in the per-step pipeline.
- Input-slot lookup array for rl_fused_controllers updated: rollout_steps
  now consumes RL_ADV_VAR_PRE_NORM_INDEX (emitted by
  rl_advantage_normalize before in-place normalize) instead of the
  post-norm advantage_var_ratio (definitionally ~0 under Phase 4.5).
- ~30 new ISV bootstrap entries in with_controllers_bootstrapped.

# ISV slot allocation (isv_slots.rs)
- 72 new slots, RL_SLOTS_END 588 → 660. +288 bytes mapped-pinned.
- 9 Welford variance triples + 5 asymmetric Schulman counters +
  3 new input signals (var_pre_norm, gamma_min_adaptive, reward_mag) +
  v_blend (5 + 3 Welford) + ppo_ratio_clamp (1 + 3 Welford) +
  reward_clamp (5) + gate_threshold (1) + q_distill (1) + 20 clamp bounds +
  shared wiener floor.

# Bootstrap-clamp consistency fix (caught by Task 16 testing)
The original draft bootstrapped RL_EPS_BOOTSTRAP at 0.01 (= KL target),
but EPS_MIN was 0.05 — bootstrap value below clamp range. First post-
bootstrap step always snapped ε to MIN regardless of signal direction
("snap to MIN" behavior the unit tests surfaced). Fixed by bootstrapping
ε at MIN (0.05) so asymmetric Schulman operates from a valid state.

# Validation
- cargo build --release: clean
- cargo build --tests: clean
- 3 GPU Welford kernel tests (G1 constant→0var, G2 sequence→known var,
  G3 sentinel skip): all pass
- 5 GPU adaptive-floor invariant tests (G3 PPO clip holds at bootstrap
  when signal below floor, G4 tightens on single above-band, G5 widens
  only after 3 consecutive below-band, G6 post-warmup uses Welford mean,
  G6 pre-warmup uses EMA): all pass
- integrated_trainer_smoke (full GPU pipeline): passes
- 1k local smoke at b=128: 1000/1000 steps, completed_clean, no NaN,
  controllers genuinely adapting (γ 0.90→0.974, per_α 0.40→0.52,
  q_distill_λ 0.05→0.21, ε held at 0.05 = MIN per Phase 4.5 small-KL
  regime — was previously stuck at MAX 0.50 throughout fold 0/1)
- compute-sanitizer memcheck b=128 5 steps: 0 errors

Cluster validation (G7-G9) submitted as follow-up runs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-30 15:41:51 +02:00

525 lines
23 KiB
Rust
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
//! Task 16 — Controller-level adaptive-floor invariant tests for the
//! 2026-05-30 adaptive-controller-floor refactor (spec
//! `docs/superpowers/specs/2026-05-30-adaptive-controller-floor-design.md`).
//!
//! Validates Validation Gates G3-G6 against the fused production launch
//! path (`launch_rl_fused_controllers`). Per
//! `feedback_no_cpu_test_fallbacks`: every oracle here is an analytical
//! identity of the controller's documented adaptive math — adaptive
//! noise floor (`max(target × 0.5, 2σ)`), asymmetric Schulman
//! (tighten on single above-band observation, widen requires N ≥ 3
//! consecutive below-band) and gamma's `1 1/(d × 2)` adaptive-min
//! formula. No CPU reference reimplementation; the kernel's spec is the
//! oracle.
//!
//! Per `pearl_first_observation_bootstrap`: the bootstrap-replace path
//! (`eps_prev == RL_EPS_BOOTSTRAP_INDEX`) skips the Wiener blend so the
//! first real observation lands directly at the computed `eps_target`.
//! For G4/G5 we deliberately PRE-WRITE `RL_PPO_CLIP_INDEX` to a value
//! away from the bootstrap so the Wiener-blend branch is exercised; this
//! isolates the asymmetric-Schulman invariant from the bootstrap-replace
//! quirk. G3 stays on the natural post-bootstrap state (eps_prev =
//! bootstrap = 0.05 = EPS_MIN) because the noise-floor gate returns
//! before either branch fires — the invariant under test is "controller
//! holds".
//!
//! Run with:
//! `cargo test -p ml-alpha --test controller_adaptive_floors -- --ignored --nocapture`
use ml_alpha::rl::isv_slots::{
RL_GAMMA_INDEX, RL_GAMMA_MIN_ADAPTIVE_INDEX, RL_KL_PI_BELOW_COUNT_INDEX,
RL_KL_PI_EMA_INDEX, RL_KL_PI_VAR_COUNT_INDEX, RL_KL_PI_VAR_M2_INDEX,
RL_KL_PI_VAR_MEAN_INDEX, RL_KL_TARGET_INDEX, RL_MEAN_TRADE_DURATION_EMA_INDEX,
RL_PPO_CLIP_EPS_MAX_INDEX, RL_PPO_CLIP_INDEX, RL_SCHULMAN_ADJUST_RATE_INDEX,
RL_SCHULMAN_TOLERANCE_INDEX, RL_TRADE_DUR_VAR_COUNT_INDEX,
RL_TRADE_DUR_VAR_M2_INDEX, RL_TRADE_DUR_VAR_MEAN_INDEX,
};
use ml_alpha::trainer::integrated::{IntegratedTrainer, IntegratedTrainerConfig};
use ml_alpha::trainer::perception::PerceptionTrainerConfig;
use ml_core::device::MlDevice;
/// Bootstrap value `rl_ppo_clip_controller` writes into
/// `ISV[RL_PPO_CLIP_INDEX]` at construction. Per Task 17 fix:
/// bootstrap matches `EPS_MIN` (= 0.05) so the post-bootstrap step
/// doesn't snap to MIN on first signal regardless of direction
/// (bootstrap-clamp inconsistency that surfaced in the Task 16 testing).
const EPS_BOOTSTRAP: f32 = 0.05;
/// PPO ε MIN clamp seeded by `with_controllers_bootstrapped`.
const EPS_MIN: f32 = 0.05;
/// KL target seeded by `with_controllers_bootstrapped`.
const KL_TARGET: f32 = 0.01;
/// Schulman tolerance — kl above `target × tolerance` triggers tighten,
/// below `target / tolerance` triggers (delayed) widen.
const SCHULMAN_TOLERANCE: f32 = 1.5;
/// Schulman adjust rate — tighten ratio is `1 / adjust_rate`, widen
/// ratio is `adjust_rate`.
const SCHULMAN_ADJUST_RATE: f32 = 1.5;
/// N-consecutive-below-band threshold for the widen branch (matches
/// `WIDEN_PATIENCE_CONSECUTIVE` in `rl_ppo_clip_controller.cu`).
const WIDEN_PATIENCE: f32 = 3.0;
/// Hard absolute floor for γ adaptive-min (matches `GAMMA_MIN_ABSOLUTE`
/// in `rl_gamma_controller.cu`).
const GAMMA_MIN_ABSOLUTE: f32 = 0.9;
/// Welford warmup observations before the running mean replaces the EMA
/// in the γ adaptive-min derivation (matches `WELFORD_WARMUP_OBS`).
const WELFORD_WARMUP_OBS: f32 = 100.0;
fn build_trainer() -> Option<(MlDevice, IntegratedTrainer)> {
let dev = match MlDevice::cuda(0) {
Ok(d) => d,
Err(e) => {
eprintln!("CUDA 0 not available — skipping ({e})");
return None;
}
};
let cfg = IntegratedTrainerConfig {
perception: PerceptionTrainerConfig {
seq_len: 4,
n_batch: 1,
..PerceptionTrainerConfig::default()
},
dqn_seed: 0xC1,
ppo_seed: 0xC2,
..IntegratedTrainerConfig::default()
};
let trainer = IntegratedTrainer::new(&dev, cfg).expect("IntegratedTrainer::new");
Some((dev, trainer))
}
/// Sync the trainer's production stream so subsequent mapped-pinned ISV
/// reads through `read_isv_host` observe device-side writes.
fn sync(trainer: &IntegratedTrainer) {
trainer.stream.synchronize().expect("sync trainer stream");
}
// ─────────────────────────────────────────────────────────────────────
// G3 — PPO clip holds when signal is below the adaptive noise floor.
//
// Invariant: after the bootstrap path has fired (eps_prev = bootstrap =
// 0.01), feeding `kl_pi_ema = 0.001` (< noise_floor = target × 0.5 =
// 0.005) for 50 fused launches must NOT drift ε. The noise-floor gate
// is the load-bearing fix — without it, the prior multiplicative-ratio
// design produced 1e4 ratios from sub-noise KL values and pegged ε at
// MAX 0.50 within ~50 steps (alpha-rl-mjzfk fold 0 saturation).
//
// Welford triple is pre-populated so the std-based half of the floor
// (`2σ`) is dominated by the target-fraction half (`target × 0.5`),
// keeping the floor at the steady-state value of 0.005.
// ─────────────────────────────────────────────────────────────────────
#[test]
#[ignore = "requires CUDA (MlDevice::cuda(0))"]
fn g3_ppo_clip_holds_when_signal_below_adaptive_floor() {
let Some((_dev, trainer)) = build_trainer() else { return };
// Verify pre-conditions seeded by `with_controllers_bootstrapped`.
sync(&trainer);
let eps_at_bootstrap = trainer.read_isv_host(RL_PPO_CLIP_INDEX);
assert!(
(eps_at_bootstrap - EPS_BOOTSTRAP).abs() < 1e-6,
"pre-condition: ε must be bootstrapped to {EPS_BOOTSTRAP}; got {eps_at_bootstrap}"
);
let kl_target_seeded = trainer.read_isv_host(RL_KL_TARGET_INDEX);
assert!(
(kl_target_seeded - KL_TARGET).abs() < 1e-6,
"pre-condition: KL target must be seeded to {KL_TARGET}; got {kl_target_seeded}"
);
// Pre-populate Welford triple at the kl_pi_ema slot with low M²
// so std ≈ 0 and the floor reduces to `target × 0.5 = 0.005`. count
// > 1 is required so the kernel evaluates `M²/(count 1)` rather
// than the count ≤ 1 fallback (which would also produce std = 0).
trainer.isv_mapped.write_record(RL_KL_PI_VAR_COUNT_INDEX, 100.0);
trainer.isv_mapped.write_record(RL_KL_PI_VAR_MEAN_INDEX, 0.001);
trainer.isv_mapped.write_record(RL_KL_PI_VAR_M2_INDEX, 0.0);
// Feed below-floor signal for 50 fused launches. Re-write each step
// because a real-world EMA producer would refresh it; here we're
// testing the controller in isolation.
for _ in 0..50 {
trainer.isv_mapped.write_record(RL_KL_PI_EMA_INDEX, 0.001);
trainer
.launch_rl_fused_controllers(1)
.expect("launch_rl_fused_controllers");
}
sync(&trainer);
let eps_after = trainer.read_isv_host(RL_PPO_CLIP_INDEX);
// Primary invariant: ε held — no drift past bootstrap, MUST not have
// climbed toward MAX 0.5 (the canonical saturation pattern).
assert!(
(eps_after - EPS_BOOTSTRAP).abs() < 1e-4,
"ε must hold at bootstrap when signal is below adaptive floor; \
expected ≈ {EPS_BOOTSTRAP}, got {eps_after}"
);
assert!(
eps_after < 0.4,
"ε must NOT drift toward MAX 0.5 — this was the fold 0/1 \
saturation bug; got {eps_after}"
);
// Negative invariant: the below-band counter must have stayed at 0
// because the noise-floor gate returns BEFORE the asymmetric Schulman
// branch fires.
let below_count = trainer.read_isv_host(RL_KL_PI_BELOW_COUNT_INDEX);
assert_eq!(
below_count, 0.0,
"below-band counter must stay 0 when noise-floor gate held; got {below_count}"
);
eprintln!(
"G3 OK — ε held at {eps_after} (bootstrap {EPS_BOOTSTRAP}) over 50 below-floor steps; \
below_count = {below_count} (noise-floor gate fired)"
);
}
// ─────────────────────────────────────────────────────────────────────
// G4 — PPO clip tightens on a single above-band observation
// (asymmetric Schulman — fast tighten).
//
// Invariant: a single `kl_pi_ema > target × tolerance` observation must
// decrease ε within the same call. Tighten ratio = `1 / adjust_rate ≈
// 0.667`; the assertion is strict decrease (ε_after < ε_before).
//
// Test discipline: we pre-write ε to 0.2 (well above MIN 0.05 and away
// from bootstrap 0.01) so:
// 1. The bootstrap-replace branch DOESN'T fire (eps_prev ≠ bootstrap).
// 2. `eps_target = 0.2 × 0.667 ≈ 0.133` stays above MIN, so the clamp
// doesn't bind and obscure the tighten signal.
// The Wiener-α blend (alpha = 0.4 = Wiener floor) then produces
// `eps_new = 0.6 × 0.2 + 0.4 × 0.133 ≈ 0.173`. Below 0.2 → strict
// decrease.
// ─────────────────────────────────────────────────────────────────────
#[test]
#[ignore = "requires CUDA (MlDevice::cuda(0))"]
fn g4_ppo_clip_tightens_on_single_above_band_observation() {
let Some((_dev, trainer)) = build_trainer() else { return };
// Pre-write ε to 0.2 — away from bootstrap, above MIN clamp.
let eps_initial: f32 = 0.2;
trainer.isv_mapped.write_record(RL_PPO_CLIP_INDEX, eps_initial);
// Welford triple — std ≈ 0 so the noise floor is `target × 0.5 = 0.005`.
trainer.isv_mapped.write_record(RL_KL_PI_VAR_COUNT_INDEX, 100.0);
trainer.isv_mapped.write_record(RL_KL_PI_VAR_MEAN_INDEX, 0.01);
trainer.isv_mapped.write_record(RL_KL_PI_VAR_M2_INDEX, 0.0);
// Above-band signal: kl = 0.05 = 5× target = 3.33× tolerance bound.
trainer.isv_mapped.write_record(RL_KL_PI_EMA_INDEX, 0.05);
// Verify Schulman config matches what the controller will read.
let tolerance = trainer.read_isv_host(RL_SCHULMAN_TOLERANCE_INDEX);
let adjust_rate = trainer.read_isv_host(RL_SCHULMAN_ADJUST_RATE_INDEX);
assert!(
(tolerance - SCHULMAN_TOLERANCE).abs() < 1e-6,
"pre-condition: Schulman tolerance seeded to {SCHULMAN_TOLERANCE}; got {tolerance}"
);
assert!(
(adjust_rate - SCHULMAN_ADJUST_RATE).abs() < 1e-6,
"pre-condition: Schulman adjust_rate seeded to {SCHULMAN_ADJUST_RATE}; got {adjust_rate}"
);
trainer
.launch_rl_fused_controllers(1)
.expect("launch_rl_fused_controllers");
sync(&trainer);
let eps_after = trainer.read_isv_host(RL_PPO_CLIP_INDEX);
// Primary invariant: ε strictly decreased on a single above-band fire.
assert!(
eps_after < eps_initial,
"ε must tighten on single above-band observation; \
before = {eps_initial}, after = {eps_after}"
);
// Above the MIN clamp — the decrease is genuine adaptation, not a
// saturation pin.
assert!(
eps_after > EPS_MIN,
"ε must stay above MIN {EPS_MIN} after a single fire (clamp \
not binding); got {eps_after}"
);
// The below-band counter must reset on a tighten event (the kernel's
// `else if (kl_ema > target × tolerance) { … below_count = 0 }`).
let below_count = trainer.read_isv_host(RL_KL_PI_BELOW_COUNT_INDEX);
assert_eq!(
below_count, 0.0,
"below-band counter must reset on tighten; got {below_count}"
);
eprintln!(
"G4 OK — ε tightened {eps_initial}{eps_after} (single above-band fire); \
below_count reset to 0"
);
}
// ─────────────────────────────────────────────────────────────────────
// G5 — PPO clip widens only after N consecutive below-band observations
// (asymmetric Schulman — slow widen).
//
// Invariant: a kl_pi_ema in the widen-band `[noise_floor, target /
// tolerance) = [0.005, 0.00667)` must increment `below_count` each step
// but NOT widen ε until `below_count ≥ WIDEN_PATIENCE_CONSECUTIVE = 3`.
//
// We choose kl = 0.006 because:
// - 0.006 ≥ 0.005 (noise floor — passes the gate)
// - 0.006 < 0.00667 (target / tolerance — triggers widen branch, not
// hold)
// To keep the noise floor low (so kl=0.006 passes it), Welford std must
// be small: count=100, M²=0 → std=0 → floor = target × 0.5 = 0.005.
//
// Test discipline: ε pre-written to 0.2 so the bootstrap-replace branch
// doesn't fire. We expect:
// step 1: below_count 0 → 1, ratio = 1 (hold), ε unchanged
// step 2: below_count 1 → 2, ratio = 1 (hold), ε unchanged
// step 3: below_count 2 → 3, ratio = adjust_rate = 1.5 (widen), ε
// increases via Wiener blend
// step 4: below_count 3 → 4, ratio = 1.5 (widen continues)
// ─────────────────────────────────────────────────────────────────────
#[test]
#[ignore = "requires CUDA (MlDevice::cuda(0))"]
fn g5_ppo_clip_widens_only_after_n_consecutive_below_band() {
let Some((_dev, trainer)) = build_trainer() else { return };
let eps_initial: f32 = 0.2;
trainer.isv_mapped.write_record(RL_PPO_CLIP_INDEX, eps_initial);
// Welford triple — low std so noise floor = target × 0.5 = 0.005.
trainer.isv_mapped.write_record(RL_KL_PI_VAR_COUNT_INDEX, 100.0);
trainer.isv_mapped.write_record(RL_KL_PI_VAR_MEAN_INDEX, 0.006);
trainer.isv_mapped.write_record(RL_KL_PI_VAR_M2_INDEX, 0.0);
// Counter must start at 0.
trainer
.isv_mapped
.write_record(RL_KL_PI_BELOW_COUNT_INDEX, 0.0);
// Widen-band signal: 0.005 ≤ 0.006 < 0.00667.
let kl_widen: f32 = 0.006;
// Compute the widen-band edges from the canonical formulas so the
// assertion that 0.006 falls inside is auditable.
let noise_floor = KL_TARGET * 0.5;
let widen_upper = KL_TARGET / SCHULMAN_TOLERANCE;
assert!(
kl_widen >= noise_floor,
"kl_widen {kl_widen} must be ≥ noise floor {noise_floor}"
);
assert!(
kl_widen < widen_upper,
"kl_widen {kl_widen} must be < widen upper edge {widen_upper}"
);
let mut below_counts = Vec::with_capacity(4);
let mut eps_values = Vec::with_capacity(4);
for _ in 0..4 {
trainer.isv_mapped.write_record(RL_KL_PI_EMA_INDEX, kl_widen);
trainer
.launch_rl_fused_controllers(1)
.expect("launch_rl_fused_controllers");
sync(&trainer);
below_counts.push(trainer.read_isv_host(RL_KL_PI_BELOW_COUNT_INDEX));
eps_values.push(trainer.read_isv_host(RL_PPO_CLIP_INDEX));
}
for (i, (bc, eps)) in below_counts.iter().zip(eps_values.iter()).enumerate() {
eprintln!(" step {}: below_count = {bc}, ε = {eps}", i + 1);
}
// Below-counter invariant: increments monotonically per step.
assert_eq!(below_counts[0], 1.0, "step 1: below_count must be 1");
assert_eq!(below_counts[1], 2.0, "step 2: below_count must be 2");
assert_eq!(
below_counts[2], WIDEN_PATIENCE,
"step 3: below_count must reach WIDEN_PATIENCE {WIDEN_PATIENCE}"
);
assert_eq!(below_counts[3], 4.0, "step 4: below_count must be 4");
// ε hold invariant: steps 1 and 2 (below_count < 3) MUST NOT widen.
// Allow only float-rounding-level drift (the controller's
// `ratio = 1.0f` branch produces eps_target = eps_prev, then the
// Wiener blend with eps_target == eps_prev is the identity).
let drift_1 = (eps_values[0] - eps_initial).abs();
let drift_2 = (eps_values[1] - eps_initial).abs();
assert!(
drift_1 < 1e-5,
"step 1: ε must hold (count < WIDEN_PATIENCE); drift {drift_1}, value {}",
eps_values[0]
);
assert!(
drift_2 < 1e-5,
"step 2: ε must hold (count < WIDEN_PATIENCE); drift {drift_2}, value {}",
eps_values[1]
);
// ε widen invariant: step 3 (below_count reaches WIDEN_PATIENCE) MUST
// strictly increase ε. Step 4 continues widening.
assert!(
eps_values[2] > eps_initial + 1e-5,
"step 3: ε must widen once below_count reaches {WIDEN_PATIENCE}; \
was {eps_initial}, got {}",
eps_values[2]
);
assert!(
eps_values[3] > eps_values[2],
"step 4: ε must continue widening; step 3 = {}, step 4 = {}",
eps_values[2],
eps_values[3]
);
// ε stays under MAX clamp — widening is real adaptation, not a
// saturation pin.
let eps_max = trainer.read_isv_host(RL_PPO_CLIP_EPS_MAX_INDEX);
assert!(
eps_values[3] < eps_max,
"ε must stay under MAX {eps_max} (no saturation pin); got {}",
eps_values[3]
);
eprintln!(
"G5 OK — ε held for 2 steps, widened on step 3 (count reached {WIDEN_PATIENCE}): \
{eps_initial}{}{}{}{}",
eps_values[0], eps_values[1], eps_values[2], eps_values[3]
);
}
// ─────────────────────────────────────────────────────────────────────
// G6 — γ adaptive_min tracks trade duration via Welford mean (post-100-
// observation warmup) and the EMA before warmup.
//
// Invariant (post-warmup): with Welford count ≥ 100 and Welford mean d,
// the kernel must write
//
// RL_GAMMA_MIN_ADAPTIVE = max(0.9, 1 1/max(d × 2, 10))
//
// to ISV[613]. At d = 30: `1 1/60 ≈ 0.98333`.
//
// Invariant (pre-warmup): with count < 100, the formula uses the EMA at
// `RL_MEAN_TRADE_DURATION_EMA_INDEX` instead of the Welford mean. At
// EMA = 15: `1 1/30 ≈ 0.96667`.
//
// Per spec Section "Special case G — γ_min coupling": the Welford
// mean's N-smoothing breaks the γ ↔ trade_duration feedback loop without
// any explicit cool-down. Welford count provides the confidence
// threshold for the switchover.
// ─────────────────────────────────────────────────────────────────────
#[test]
#[ignore = "requires CUDA (MlDevice::cuda(0))"]
fn g6_gamma_adaptive_min_post_warmup_uses_welford_mean() {
let Some((_dev, trainer)) = build_trainer() else { return };
// Post-warmup state: count = 100 = WELFORD_WARMUP_OBS, mean = 30.
let d_welford: f32 = 30.0;
trainer
.isv_mapped
.write_record(RL_TRADE_DUR_VAR_COUNT_INDEX, WELFORD_WARMUP_OBS);
trainer
.isv_mapped
.write_record(RL_TRADE_DUR_VAR_MEAN_INDEX, d_welford);
trainer.isv_mapped.write_record(RL_TRADE_DUR_VAR_M2_INDEX, 0.0);
// Also write the EMA — even though the kernel ignores it after
// warmup, mismatching it from the Welford mean tests that the
// controller actually selects the Welford path (not the EMA).
trainer
.isv_mapped
.write_record(RL_MEAN_TRADE_DURATION_EMA_INDEX, d_welford);
trainer
.launch_rl_fused_controllers(1)
.expect("launch_rl_fused_controllers");
sync(&trainer);
let adaptive_min = trainer.read_isv_host(RL_GAMMA_MIN_ADAPTIVE_INDEX);
// Analytical oracle: `max(0.9, 1 - 1/max(d × 2, 10))`. At d = 30,
// d × 2 = 60 > 10 = HORIZON_FLOOR, so the inner `max` resolves to 60
// and the formula reduces to `1 - 1/60`.
let horizon = (d_welford * 2.0_f32).max(10.0);
let expected = (1.0_f32 - 1.0 / horizon).max(GAMMA_MIN_ABSOLUTE);
assert!(
(adaptive_min - expected).abs() < 1e-4,
"post-warmup γ_min must be max(0.9, 1 - 1/{horizon}) = {expected}; got {adaptive_min}"
);
// γ itself must respect the new floor — bootstrap value 0.9 was
// computed against the cold-start adaptive_min 0.9; after the
// controller re-fires with the new welford mean, γ should be
// clamped to the new floor.
let gamma = trainer.read_isv_host(RL_GAMMA_INDEX);
assert!(
gamma >= adaptive_min - 1e-4,
"γ must respect adaptive_min {adaptive_min}; got {gamma}"
);
eprintln!(
"G6 OK (post-warmup) — adaptive_min = {adaptive_min} (expected {expected}) \
at Welford count = {WELFORD_WARMUP_OBS}, mean = {d_welford}; γ = {gamma}"
);
}
#[test]
#[ignore = "requires CUDA (MlDevice::cuda(0))"]
fn g6_gamma_adaptive_min_pre_warmup_uses_ema() {
let Some((_dev, trainer)) = build_trainer() else { return };
// Pre-warmup state: count = 99 < WELFORD_WARMUP_OBS = 100. The
// controller MUST fall back to the EMA slot, not the Welford mean.
// Deliberately set EMA ≠ Welford mean so the test confirms the
// controller's selection logic.
let d_ema: f32 = 15.0;
let d_welford: f32 = 30.0;
trainer
.isv_mapped
.write_record(RL_TRADE_DUR_VAR_COUNT_INDEX, WELFORD_WARMUP_OBS - 1.0);
trainer
.isv_mapped
.write_record(RL_TRADE_DUR_VAR_MEAN_INDEX, d_welford);
trainer.isv_mapped.write_record(RL_TRADE_DUR_VAR_M2_INDEX, 0.0);
trainer
.isv_mapped
.write_record(RL_MEAN_TRADE_DURATION_EMA_INDEX, d_ema);
trainer
.launch_rl_fused_controllers(1)
.expect("launch_rl_fused_controllers");
sync(&trainer);
let adaptive_min = trainer.read_isv_host(RL_GAMMA_MIN_ADAPTIVE_INDEX);
// Oracle uses the EMA (15), not the Welford mean (30).
let horizon = (d_ema * 2.0_f32).max(10.0);
let expected = (1.0_f32 - 1.0 / horizon).max(GAMMA_MIN_ABSOLUTE);
assert!(
(adaptive_min - expected).abs() < 1e-4,
"pre-warmup γ_min must use EMA, not Welford mean: \
expected max(0.9, 1 - 1/{horizon}) = {expected}; got {adaptive_min}"
);
// Counter-invariant: had the controller used the Welford mean
// instead (count = 99 bug), adaptive_min would be ≈ 0.98333. The
// test fails if that magic number appears.
let welford_horizon = (d_welford * 2.0_f32).max(10.0);
let wrong_min = (1.0_f32 - 1.0 / welford_horizon).max(GAMMA_MIN_ABSOLUTE);
assert!(
(adaptive_min - wrong_min).abs() > 1e-3,
"pre-warmup γ_min must NOT match the Welford-mean value {wrong_min} \
(would indicate the count < WARMUP guard is broken); got {adaptive_min}"
);
eprintln!(
"G6 OK (pre-warmup) — adaptive_min = {adaptive_min} (expected {expected}, \
distinct from welford-path {wrong_min}) at Welford count = {}, EMA = {d_ema}",
WELFORD_WARMUP_OBS - 1.0
);
}