Comprehensive fix for the issues identified after the BPTT-unroll cluster
run plateaued at val_loss ~0.692 with oscillating AUCs:
ARCHITECTURE
- CfC h_old is now RECURRENT across positions. Previously reset to
zero every step → CfC degenerated to a per-cell tanh-FC layer.
New: h_old at step k IS h_new at step k-1. Heads still operate
on h_new_k, but now the CfC actually carries state. Reverse-order
backward through the K positions accumulates grad_h_old → grad_h_new
via the new optional `grad_h_carry` arg on multi_horizon_heads_backward.
- tau is TRAINED. cfc_step_backward now writes grad_tau (per-cell decay
constant derivative), trainer gets a 7th AdamW group at 0.1× cfc lr.
- 6 NEW regime features (EMA cascade computed loader-side per file)
fill slots out[20..26] of snap_features. Gives the model multi-minute
trend / volatility / liquidity context that is structurally unreachable
inside the K-snapshot BPTT window. Slots: mid-z (med/slow), trend
signal, log-vol slow, log-spread med, log-trade-rate med. All bounded
via log1p / signed-log so no tuned constants leak in.
PERFORMANCE (NVIDIA-style)
- GPU-fused multi-horizon BCE for the entire [K, N_HORIZONS] grid in
ONE launch (was K host roundtrips). Native NaN-label masking.
- K-loop is fully GPU-resident: pre-allocated per-K scratch
(h_new_per_k, probs_per_k, labels_per_k, grad_probs_per_k), zero
device allocs inside step(). Only TWO syncs per sequence (after
forward, after backward) vs previously 2K+1.
- Stream-ordered kernel launches with pointer-offset addressing into
per-K buffers — host doesn't wait between K iterations.
- cfc_step_backward / multi_horizon_heads_backward both use += grad
semantics; trainer pre-zeroes accumulators once per step().
- MAMBA2_ALPHA_MAX_K capped at 96 (was temporarily at 256). 96 covers
h=30/100/300 with room; regime features handle h=1000/h=6000.
TRAINING DISCIPLINE
- LR schedule: linear warmup (default 200 steps) + cosine decay to
lr * lr_min_factor (default 0.1). Applied per training step to both
CfC and Mamba2 AdamW groups via new set_lr_cfc/set_lr_mamba2.
- Best-checkpoint tracking by val_loss; recorded in summary
(best_epoch, best_val_loss, best_val_auc).
- Early stopping on val_loss plateau (default patience = 3).
- CRITICAL BUG FIX: validation now uses new `evaluate()` method
(forward-only) instead of `step()`. Previous CLI called step()
on val data, which ran the full backward + AdamW update on the
validation set. With per-step BPTT that's ~K× more pressure than
the old comment ("statistically negligible") assumed.
Synthetic overfit: 0.6442 → 0.1233 in 250 steps (81% drop, sharper
than the previous 70%). 77 ml-alpha tests pass.
Local 2Q smoke (seq_len=64, 600 train seqs/epoch, 4 epochs):
val_loss 0.7011 → 0.6990, best epoch=1, h300 AUC 0.565 in epoch 0.
Phase E.3 callers (ml/examples/alpha_baseline.rs,
alpha_dqn_h600_smoke.rs) use the LEGACY Mamba2 forward_train +
backward_from_h_enriched path — unaffected by these changes (their
kernels are pre-zeroed via alloc_zeros, so the += grad semantics
remain correct in single-call mode).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
108 lines
3.7 KiB
Rust
108 lines
3.7 KiB
Rust
//! CfcTrunk forward — invariants on the sequentially-dispatched perception path.
|
|
|
|
use approx::assert_relative_eq;
|
|
use ml_alpha::cfc::snap_features::Mbp10RawInput;
|
|
use ml_alpha::cfc::{CfcConfig, CfcTrunk};
|
|
use ml_alpha::heads::{N_HORIZONS, PROJ_DIM};
|
|
use ml_core::device::MlDevice;
|
|
|
|
fn test_device() -> MlDevice {
|
|
MlDevice::cuda(0).expect("CUDA 0 required for ml-alpha tests")
|
|
}
|
|
|
|
fn synthetic_input(dt_ns: u64) -> Mbp10RawInput {
|
|
let mut bid_px = [0.0f32; 10];
|
|
let mut bid_sz = [0.0f32; 10];
|
|
let mut ask_px = [0.0f32; 10];
|
|
let mut ask_sz = [0.0f32; 10];
|
|
for i in 0..10 {
|
|
bid_px[i] = 5500.00 - 0.25 * i as f32;
|
|
ask_px[i] = 5500.25 + 0.25 * i as f32;
|
|
bid_sz[i] = 12.0 + i as f32;
|
|
ask_sz[i] = 10.0 + i as f32 * 1.5;
|
|
}
|
|
Mbp10RawInput {
|
|
bid_px,
|
|
bid_sz,
|
|
ask_px,
|
|
ask_sz,
|
|
prev_mid: 5499.875,
|
|
trade_signed_vol: 4.0,
|
|
trade_count: 7,
|
|
ts_ns: dt_ns,
|
|
prev_ts_ns: 0,
|
|
regime: [0.0; 6],
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn forward_returns_probs_in_unit_interval() {
|
|
let dev = test_device();
|
|
let mut trunk = CfcTrunk::new_random(&dev, &CfcConfig::default(), 0xABCD).expect("init");
|
|
let (probs, proj) = trunk.forward_snapshot(&synthetic_input(1_000_000)).expect("forward");
|
|
assert_eq!(probs.len(), N_HORIZONS);
|
|
assert_eq!(proj.len(), PROJ_DIM);
|
|
for &p in &probs {
|
|
assert!((0.0..=1.0).contains(&p), "head prob {p} out of [0,1]");
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn forward_updates_hidden_state() {
|
|
let dev = test_device();
|
|
let mut trunk = CfcTrunk::new_random(&dev, &CfcConfig::default(), 0xCAFE).expect("init");
|
|
let h0 = trunk.snapshot_hidden().expect("h0");
|
|
trunk
|
|
.forward_snapshot(&synthetic_input(20_000_000))
|
|
.expect("forward");
|
|
let h1 = trunk.snapshot_hidden().expect("h1");
|
|
let diff: f32 = h0.iter().zip(&h1).map(|(a, b)| (a - b).abs()).sum();
|
|
assert!(
|
|
diff > 1e-6,
|
|
"hidden state must change after a forward step (got total |Δh| = {diff})"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn forward_outputs_are_finite() {
|
|
let dev = test_device();
|
|
let mut trunk = CfcTrunk::new_random(&dev, &CfcConfig::default(), 0x1234).expect("init");
|
|
let (probs, proj) = trunk.forward_snapshot(&synthetic_input(50_000_000)).expect("forward");
|
|
for &p in &probs {
|
|
assert!(p.is_finite(), "prob {p} not finite");
|
|
}
|
|
for &v in &proj {
|
|
assert!(v.is_finite(), "proj {v} not finite");
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn forward_layer_norm_proj_has_near_zero_mean() {
|
|
let dev = test_device();
|
|
let mut trunk = CfcTrunk::new_random(&dev, &CfcConfig::default(), 0x5EED).expect("init");
|
|
let (_, proj) = trunk.forward_snapshot(&synthetic_input(20_000_000)).expect("forward");
|
|
let mean: f32 = proj.iter().sum::<f32>() / PROJ_DIM as f32;
|
|
assert_relative_eq!(mean, 0.0, epsilon = 1e-3);
|
|
}
|
|
|
|
#[test]
|
|
fn forward_multi_step_advances_state_consistently() {
|
|
// After many forward steps, the hidden state should still be finite and
|
|
// within the bounded range expected by the CfC formula (tanh of pre).
|
|
let dev = test_device();
|
|
let mut trunk = CfcTrunk::new_random(&dev, &CfcConfig::default(), 0xBEEF).expect("init");
|
|
let mut last_ts = 0u64;
|
|
for k in 0..50 {
|
|
last_ts += 20_000_000;
|
|
let mut input = synthetic_input(last_ts);
|
|
input.prev_ts_ns = last_ts - 20_000_000;
|
|
// Slowly mutate trade_signed_vol so the input changes each step.
|
|
input.trade_signed_vol = k as f32 * 0.1;
|
|
trunk.forward_snapshot(&input).expect("forward");
|
|
}
|
|
let h = trunk.snapshot_hidden().expect("snapshot");
|
|
for (i, v) in h.iter().enumerate() {
|
|
assert!(v.is_finite(), "hidden[{i}] = {v} not finite after 50 steps");
|
|
}
|
|
}
|