Files
foxhunt/crates/ml-alpha/tests/trunk_forward.rs
jgrusewski 248d8fe510 feat(ml-alpha): full architectural pass — recurrent CfC, GPU BCE, regime features, training discipline
Comprehensive fix for the issues identified after the BPTT-unroll cluster
run plateaued at val_loss ~0.692 with oscillating AUCs:

ARCHITECTURE
  - CfC h_old is now RECURRENT across positions. Previously reset to
    zero every step → CfC degenerated to a per-cell tanh-FC layer.
    New: h_old at step k IS h_new at step k-1. Heads still operate
    on h_new_k, but now the CfC actually carries state. Reverse-order
    backward through the K positions accumulates grad_h_old → grad_h_new
    via the new optional `grad_h_carry` arg on multi_horizon_heads_backward.
  - tau is TRAINED. cfc_step_backward now writes grad_tau (per-cell decay
    constant derivative), trainer gets a 7th AdamW group at 0.1× cfc lr.
  - 6 NEW regime features (EMA cascade computed loader-side per file)
    fill slots out[20..26] of snap_features. Gives the model multi-minute
    trend / volatility / liquidity context that is structurally unreachable
    inside the K-snapshot BPTT window. Slots: mid-z (med/slow), trend
    signal, log-vol slow, log-spread med, log-trade-rate med. All bounded
    via log1p / signed-log so no tuned constants leak in.

PERFORMANCE (NVIDIA-style)
  - GPU-fused multi-horizon BCE for the entire [K, N_HORIZONS] grid in
    ONE launch (was K host roundtrips). Native NaN-label masking.
  - K-loop is fully GPU-resident: pre-allocated per-K scratch
    (h_new_per_k, probs_per_k, labels_per_k, grad_probs_per_k), zero
    device allocs inside step(). Only TWO syncs per sequence (after
    forward, after backward) vs previously 2K+1.
  - Stream-ordered kernel launches with pointer-offset addressing into
    per-K buffers — host doesn't wait between K iterations.
  - cfc_step_backward / multi_horizon_heads_backward both use += grad
    semantics; trainer pre-zeroes accumulators once per step().
  - MAMBA2_ALPHA_MAX_K capped at 96 (was temporarily at 256). 96 covers
    h=30/100/300 with room; regime features handle h=1000/h=6000.

TRAINING DISCIPLINE
  - LR schedule: linear warmup (default 200 steps) + cosine decay to
    lr * lr_min_factor (default 0.1). Applied per training step to both
    CfC and Mamba2 AdamW groups via new set_lr_cfc/set_lr_mamba2.
  - Best-checkpoint tracking by val_loss; recorded in summary
    (best_epoch, best_val_loss, best_val_auc).
  - Early stopping on val_loss plateau (default patience = 3).
  - CRITICAL BUG FIX: validation now uses new `evaluate()` method
    (forward-only) instead of `step()`. Previous CLI called step()
    on val data, which ran the full backward + AdamW update on the
    validation set. With per-step BPTT that's ~K× more pressure than
    the old comment ("statistically negligible") assumed.

Synthetic overfit: 0.6442 → 0.1233 in 250 steps (81% drop, sharper
than the previous 70%). 77 ml-alpha tests pass.

Local 2Q smoke (seq_len=64, 600 train seqs/epoch, 4 epochs):
  val_loss 0.7011 → 0.6990, best epoch=1, h300 AUC 0.565 in epoch 0.

Phase E.3 callers (ml/examples/alpha_baseline.rs,
alpha_dqn_h600_smoke.rs) use the LEGACY Mamba2 forward_train +
backward_from_h_enriched path — unaffected by these changes (their
kernels are pre-zeroed via alloc_zeros, so the += grad semantics
remain correct in single-call mode).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-17 09:59:37 +02:00

108 lines
3.7 KiB
Rust

//! CfcTrunk forward — invariants on the sequentially-dispatched perception path.
use approx::assert_relative_eq;
use ml_alpha::cfc::snap_features::Mbp10RawInput;
use ml_alpha::cfc::{CfcConfig, CfcTrunk};
use ml_alpha::heads::{N_HORIZONS, PROJ_DIM};
use ml_core::device::MlDevice;
fn test_device() -> MlDevice {
MlDevice::cuda(0).expect("CUDA 0 required for ml-alpha tests")
}
fn synthetic_input(dt_ns: u64) -> Mbp10RawInput {
let mut bid_px = [0.0f32; 10];
let mut bid_sz = [0.0f32; 10];
let mut ask_px = [0.0f32; 10];
let mut ask_sz = [0.0f32; 10];
for i in 0..10 {
bid_px[i] = 5500.00 - 0.25 * i as f32;
ask_px[i] = 5500.25 + 0.25 * i as f32;
bid_sz[i] = 12.0 + i as f32;
ask_sz[i] = 10.0 + i as f32 * 1.5;
}
Mbp10RawInput {
bid_px,
bid_sz,
ask_px,
ask_sz,
prev_mid: 5499.875,
trade_signed_vol: 4.0,
trade_count: 7,
ts_ns: dt_ns,
prev_ts_ns: 0,
regime: [0.0; 6],
}
}
#[test]
fn forward_returns_probs_in_unit_interval() {
let dev = test_device();
let mut trunk = CfcTrunk::new_random(&dev, &CfcConfig::default(), 0xABCD).expect("init");
let (probs, proj) = trunk.forward_snapshot(&synthetic_input(1_000_000)).expect("forward");
assert_eq!(probs.len(), N_HORIZONS);
assert_eq!(proj.len(), PROJ_DIM);
for &p in &probs {
assert!((0.0..=1.0).contains(&p), "head prob {p} out of [0,1]");
}
}
#[test]
fn forward_updates_hidden_state() {
let dev = test_device();
let mut trunk = CfcTrunk::new_random(&dev, &CfcConfig::default(), 0xCAFE).expect("init");
let h0 = trunk.snapshot_hidden().expect("h0");
trunk
.forward_snapshot(&synthetic_input(20_000_000))
.expect("forward");
let h1 = trunk.snapshot_hidden().expect("h1");
let diff: f32 = h0.iter().zip(&h1).map(|(a, b)| (a - b).abs()).sum();
assert!(
diff > 1e-6,
"hidden state must change after a forward step (got total |Δh| = {diff})"
);
}
#[test]
fn forward_outputs_are_finite() {
let dev = test_device();
let mut trunk = CfcTrunk::new_random(&dev, &CfcConfig::default(), 0x1234).expect("init");
let (probs, proj) = trunk.forward_snapshot(&synthetic_input(50_000_000)).expect("forward");
for &p in &probs {
assert!(p.is_finite(), "prob {p} not finite");
}
for &v in &proj {
assert!(v.is_finite(), "proj {v} not finite");
}
}
#[test]
fn forward_layer_norm_proj_has_near_zero_mean() {
let dev = test_device();
let mut trunk = CfcTrunk::new_random(&dev, &CfcConfig::default(), 0x5EED).expect("init");
let (_, proj) = trunk.forward_snapshot(&synthetic_input(20_000_000)).expect("forward");
let mean: f32 = proj.iter().sum::<f32>() / PROJ_DIM as f32;
assert_relative_eq!(mean, 0.0, epsilon = 1e-3);
}
#[test]
fn forward_multi_step_advances_state_consistently() {
// After many forward steps, the hidden state should still be finite and
// within the bounded range expected by the CfC formula (tanh of pre).
let dev = test_device();
let mut trunk = CfcTrunk::new_random(&dev, &CfcConfig::default(), 0xBEEF).expect("init");
let mut last_ts = 0u64;
for k in 0..50 {
last_ts += 20_000_000;
let mut input = synthetic_input(last_ts);
input.prev_ts_ns = last_ts - 20_000_000;
// Slowly mutate trade_signed_vol so the input changes each step.
input.trade_signed_vol = k as f32 * 0.1;
trunk.forward_snapshot(&input).expect("forward");
}
let h = trunk.snapshot_hidden().expect("snapshot");
for (i, v) in h.iter().enumerate() {
assert!(v.is_finite(), "hidden[{i}] = {v} not finite after 50 steps");
}
}