Wires the aux supervision path parallel to BCE:
- AuxTrunk (64-hidden single-bucket CfC) consumes the same encoder output
as the main BCE trunk
- AuxHeads (linear regression long/short) maps aux_trunk output to
per-(direction, horizon) predicted outcomes
- AuxHuberLoss supervises against D-style labels from MultiHorizonLoader
Backward path with asymmetric stop-grad at encoder boundary:
- aux_trunk gets gradient signal into its OWN params at all times
- aux_trunk's encoder-boundary gradient is INITIALLY blocked
(stop_grad_aux_to_encoder = true)
- Conditional lift per E3 design: if aux_huber_ema < 0.4 AND
aux_dir_acc_ema > 0.85 within 200 steps, lift the stop-grad
- When lifted, aux_vec_add kernel folds aux's grad_x into the main
grad_h_enriched_seq slot (element-wise += per feedback_no_atomicadd)
ISV signals added: aux_huber_per_h, aux_dir_acc_per_h (per pearl).
Per-trunk scratch + reduced grad buffers (no Adam state sharing per
pearl_adam_normalizes_loss_weights — opt_aux is its own Adam group).
New helper kernel cuda/aux_vec_add.cu: position-local dst += src for
the asymmetric stop-grad lift accumulation.
New synthetic test stacked_trainer_aux_supervision_converges_on_constant_signal
validates end-to-end:
aux_huber_ema_per_h = [0.087, 0.087, 0.087] (converged)
aux_dir_acc_ema_per_h = [1.0, 1.0, 1.0] (perfect on constant)
stop_grad_aux_to_encoder = false (lift fired)
All 5 stacked_trainer tests pass on RTX 3050 (lib still converges, no
regression from parallel aux wiring).
Not yet consumed by decision policy (B7) — aux output flows through
training only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends MultiHorizonLoaderConfig with `outcome_label_cost` (default
DEFAULT_OUTCOME_LABEL_COST_ES = 0.5 = 2 ticks for ES round-trip).
LabeledSequence + LoadedFile gain outcome_long + outcome_short arrays
of [Vec<f32>; N_HORIZONS] populated by generate_outcome_labels_d during
LoadedFile construction (same !inference_only branch as BCE labels).
CLI exposure: alpha_train.rs gains --outcome-label-cost flag with the
ES default. All 6 MultiHorizonLoaderConfig consumers updated atomically
per feedback_no_partial_refactor: alpha_train (train + val loaders),
in-file test fixtures (×2), multi_horizon_loader.rs (×2), harness.rs,
trainer_parity.rs, ring3_replay.rs.
cargo check --workspace clean; ml-alpha lib tests 41 passed.
B3+ tasks not yet touched: outcome labels are computed and propagated to
LabeledSequence but no consumer reads them yet (aux trunk Tasks B3-B5
will wire that).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per spec §2.5 and feedback_no_partial_refactor. Removes:
- LoaderMode enum (single-variant after Sequential removal → deleted entirely)
- next_sequence_sequential + sequential_file_idx + sequential_anchor_idx + last_call_was_file_boundary + is_file_boundary
- last_seen_file_boundary, train_graph_boundary_state fields
- notify_file_boundary method + boundary-state graph recapture logic
- need_attn_pool_bootstrap match — always true now (random mode always bootstraps)
- --loader-mode CLI flag + notify_file_boundary() call
- loader-mode Argo template param
Additional consumers migrated atomically (beyond the 4 files listed in
the plan): ml-alpha/tests/perception_overfit.rs,
ml-alpha/tests/multi_horizon_loader.rs, ml-backtesting/src/harness.rs,
ml-backtesting/tests/trainer_parity.rs,
ml-backtesting/tests/ring3_replay.rs — all referenced LoaderMode or the
removed config fields.
Workspace builds clean at this commit (pre-existing cudarc-cupti example
and ml-crate test errors are unrelated to MTER removal — they fail at
HEAD too). New per-horizon CfC arch lands in subsequent tasks of the
same plan.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Loader can now serve sequences in temporal order (Sequential mode);
when active, the trainer skips attn_pool reset at non-file-boundary
sequence boundaries (next commit will use this for stateful CfC + MTER).
Random mode preserved as default; ZERO behavioral change for existing
runs. Phased commit 1/8 of intervention B per
docs/superpowers/specs/2026-05-21-crt-train-intervention-b-multi-timescale-readout.md
The compile-time feature kernel-step-trace gates inclusion of the ring
code. NEW: --kernel-step-trace <PATH> CLI flag on alpha_train gates
RUNTIME activation:
- Path provided -> ring allocated, drain spawns, JSONL records written
to <PATH> (truncated). One record per kernel emission.
- Path omitted -> ring not allocated, drain not spawned, zero overhead
even with the feature compiled in.
JSONL schema: {"step": u32, "kid": u8, "kname": str, "rt": u8,
"rt_name": str, "payload": {field: f32, ...}}. The payload field names
come from the existing per-(kid, rt) decoders in gpu_log.rs (ported to
serde_json::json! in decode_to_json).
Trainer ring fields are now Option<_>; populated only when the runtime
trace path is Some. Tick kernel, step-counter shadow, smoothness
controller pointer-passing, and Drop all become path-conditional.
The tracing::info!-based drain variant is removed (file-writer is
strictly more useful; tests migrated to assert JSONL file contents).
Argo template:
- New parameter kernel-step-trace-enable (build-time feature opt-in)
- New parameter kernel-step-trace-path (runtime CLI value)
- ensure-binary cache-busts when feature toggles
- training step passes --kernel-step-trace flag conditionally
Per feedback_no_feature_flags: compile-time gate retained because the
ring carries real memory cost (~2 MiB pinned) and per-step write overhead;
the specific name kernel-step-trace narrows scope to this mechanism.
Runtime gate is Option<PathBuf>, not a boolean enable_*.
Task 5 added smoothness_base_lambda to PerceptionTrainerConfig but only
updated the struct definition + Default impl. Eight test sites in
perception_overfit.rs and one site in alpha_train.rs construct the
config via explicit field list (no ..Default::default() spread), so
they broke with E0063 missing-field errors under cargo check
--all-targets.
Per feedback_no_partial_refactor.md: when a contract changes, every
consumer migrates atomically. This commit adds smoothness_base_lambda:
0.0 to all 9 sites so the workspace compiles cleanly. The
alpha_train.rs value of 0.0 is a placeholder — Task 7 will replace it
with cli.smoothness_base_lambda once the CLI flag is added.
Per spec 2026-05-20-continuous-reasoning-trader-design.md §3.3 and §8:
decision_stride is REMOVED, not deprecated. No backwards-compat shim,
no fallback. Every consumer migrates in this commit per
feedback_no_partial_refactor.
Removed from:
- bin/fxt-backtest: RunArgs CLI flag, SweepBase field, SweepCell
override field, default_decision_stride() function, all three
BacktestHarnessConfig and RunArgs construction sites
- crates/ml-backtesting/src/harness.rs: BacktestHarnessConfig field,
MultiHorizonLoaderConfig decision_stride initializer, `let stride`
local, `if event_count % stride == 0` gate around
step_decision_with_latency; forward_step_into + step_decision now
share a single window-full guard (merged into one `if` block)
- crates/ml-alpha/src/data/loader.rs: MultiHorizonLoaderConfig field,
next_sequence stride logic simplified to stride=1 (consecutive
snapshots only)
- crates/ml-alpha/src/trainer/perception.rs: PerceptionTrainerConfig
field and Default impl; all four dt_s locals replaced with 1.0_f32
(training K-loop, graph-capture K-loop, forward_step_into CfC step,
eval K-loop)
- crates/ml-alpha/examples/alpha_train.rs: CLI flag, trainer_cfg and
both loader configs
- crates/ml/examples/alpha_baseline.rs: CLI flag, train + eval stride
gates replaced with unconditional read_all()
- config/ml/*.yaml: decision_stride: lines removed from
sweep_smoke, sweep_threshold_tuning, sweep_deployability,
sweep_decision_stride_example (file repurposed as generic example)
- tests: forward_step_golden, perception_overfit (×7 structs including
the stride=4 smoke repurposed as a second convergence check),
multi_horizon_loader (stride=4 spacing test repurposed as
ts_ns monotonicity check), ring3_replay, trainer_parity
Harness loop now invokes BOTH forward_step_into AND
step_decision_with_latency on every event whenever the snapshot window
is full. forward_step_into advances SSM state and writes alpha_probs_d;
step_decision_with_latency reads alpha_probs_d immediately after —
no CPU roundtrip, no stride gate.
n_decisions ≈ events_processed - seq_len + 1 after this commit
(vs ~9999 at stride=200 in the S2 baseline).
cargo check --workspace: clean
cargo test -p ml-backtesting --lib: 33 passed
cargo test -p ml-alpha --lib: 33 passed (6 ignored)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per project_ml_alpha_starting_capital greenfield posture: there is no V1
to differentiate from (the V1 trunk forward was dead code, no V1
checkpoint files exist in the wild). The 'v2' prefix on every identifier
was historical baggage from the migration period.
Renames:
- CheckpointV2 -> Checkpoint (also drops the version: u32 field —
bincode either deserialises a current envelope or errors; no migration
path needed)
- CheckpointVersionProbe removed (was only for V1 rejection)
- LAYER_NORM_CUBIN_V2 / VARIABLE_SELECTION_CUBIN_V2 / ATTENTION_POOL_CUBIN_V2
-> LAYER_NORM_CUBIN / VARIABLE_SELECTION_CUBIN / ATTENTION_POOL_CUBIN
- _ln_module_v2 / _vsn_module_v2 / _attn_module_v2 -> drop _v2 suffix
- smoke_load_v2_checkpoint test -> smoke_load_checkpoint
- config/ml/sweep_v2_*.yaml -> config/ml/sweep_*.yaml
- migration-era 'V2 weight skeleton' / 'V2 fields' / etc. comments
cleaned to remove the v2 prefix
Pre-existing 'v2' references in ml-backtesting CUDA files
(decision_policy.cu, pnl_track.cu) are NOT touched — those refer to
future planned 'v2' refinements (Portfolio mode, multi-fill averaging)
from the C1-C19 commits and reflect aspirational features unrelated to
this session's trunk-grows work.
Verification: ml-alpha + ml-backtesting + fxt-backtest all build clean.
perception_forward_golden bit-exact (max_diff = 0.000000).
X13: Adds PerceptionTrainer::save_checkpoint as a thin delegate to
self.trunk.save_checkpoint. Inference-only serialization — grads + AdamW
state aren't included.
X14: Inside the existing auc_h6000_improved block in alpha_train.rs,
calls trainer.save_checkpoint(out_dir / 'trunk_best_h6000.bin') so the
trained trunk lands alongside alpha_train_summary.json. Extends
AlphaTrainSummary with best_h6000_ckpt_path (Option<String>) so
downstream tooling (fxt-backtest --checkpoint) can locate the file
without re-deriving the path.
After this commit, every alpha-perception Argo workflow run produces
a CheckpointV2 file at every new-best-h6000 epoch, ready for backtest
consumption.
Verification:
- ml-alpha lib tests: 34 pass
- alpha_train example builds clean (release)
Per spec §1.1 (X13+X14).
Adds a single tracing::info!("step") emitted every 500 training steps:
if epoch_train_steps % 500 == 0 {
tracing::info!(epoch, step = epoch_train_steps, loss = loss, "step");
}
Consumed by /tmp/alpha_monitor.py (v2: per-epoch trajectories + ISV +
liveness) to render intra-epoch loss trajectory and detect stalls
faster than the once-per-epoch granularity allowed.
At 8000 steps/epoch and ~36s/epoch on L40S → ~16 step lines per epoch
per fold, well below the cluster log volume budget.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add inference_only flag to MultiHorizonLoaderConfig that skips per-file
forward-label precomputation (~half the file-load cost), plus
peek_first() and next_inference_input() chronological-streaming methods
for the ml-backtesting LOB harness.
- min_size relaxed to cfg.seq_len when inference_only=true (training
still requires seq_len + max_horizon + 1 for label generation)
- New cursor fields (inference_file_idx, inference_snap_idx) walk every
loaded snapshot in chronological order; reset() zeros both
- peek_first() seeds CfcTrunk::capture_graph_a with cur==prev semantics
(prev_ts_ns==ts_ns, trade_signed_vol=0) — natural stream-start
- next_inference_input() errors if cfg.inference_only=false (guard
against accidental mixing of training/inference paths)
- All trainer call-sites (alpha_train example + multi_horizon_loader
tests) updated with inference_only: false (zero behaviour change)
- Inline test module exercises both modes; tests skip gracefully when
fixture data isn't populated rather than panicking
See docs/superpowers/specs/2026-05-18-real-lob-integration-design.md
§1 (trainer parity) + §7 (orchestrator).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Decision-stride S lets a length-K sequence span ((K-1)*S + 1) raw
snapshots instead of K consecutive ones — expands the effective
time-window covered by each sequence at the same K-positions compute
cost. With K=64 and S=4, the window covers 256 ticks (~5s on ES MBP-10
at 20ms-tick) instead of 64 ticks (~1.3s).
Loader (crates/ml-alpha/src/data/loader.rs):
- `MultiHorizonLoaderConfig.decision_stride: usize` (default 1, must
pre-existing call sites add the new field).
- `next_sequence` reads snapshot at `anchor + k * stride`; labels at the
same indices (labels stay in absolute-snapshot horizons regardless of
stride, e.g. h=6000 always means "predict 6000 raw snapshots forward").
- `prev` snapshot for microstructure features (prev_mid, prev_ts_ns)
now points to the prior K-position (`anchor + (k-1)*stride`), NOT the
consecutive-snapshot prior, so `Δt = ts_ns - prev_ts_ns` carries the
actual elapsed time between K-positions (consumed by Mamba2's dt_s and
the planned Phase 2C TGN Fourier features).
- New `#[ignore]` real-data test: `loader_stride_4_yields_correct_spacing`
asserts Δt monotonicity at stride=4.
Mamba2 dt_s (crates/ml-alpha/src/trainer/perception.rs):
- `PerceptionTrainerConfig.decision_stride: usize` plumbs the stride
through. dispatch_train_step + evaluate_batched now use
`dt_s = decision_stride as f32` so Mamba2's selective scan
`exp(-dt * sigmoid(a))` reflects the real elapsed time. With stride=1
the behaviour is identical to before.
CLI (crates/ml-alpha/examples/alpha_train.rs):
- `--decision-stride <S>` flag (default 1) wired into both train and val
loaders + PerceptionTrainerConfig.
Argo workflow:
- `decision-stride` parameter on the template (default "1") +
`--decision-stride` script flag + propagation into the train pod's
alpha_train invocation.
Synthetic smoke (tests/perception_overfit.rs):
- `stacked_trainer_loss_shrinks_with_stride_4` proves the trainer-level
dt_s=4.0 keeps the Mamba2+LN+CfC+GRN chain numerically stable.
Converges 0.32 → 0.0000 (matches stride=1 smoke trajectory — dt_s
scaling didn't break the SSM dynamics).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three correlated fixes addressing the architectural inconsistency
surfaced by the 3-fold ISV CV: we built a horizon-aware gradient
controller (ISV) but suppressed its target horizon (h6000) to 0.36%
of the loss via auto-horizon-weights, then used a ratio formula
that never approached its own clamp ceiling. ISV's lambda was
operating on rounding error.
(1) Uniform BCE weights as auto-default
trainer/perception.rs: `auto_horizon_weights` now returns
[1.0; 5] regardless of seq_len. Prior schedule `min(1, K/h)`
gave h6000 weight 0.0053 at K=32 — combined with lambda ~1.04,
h6000's effective loss contribution was ~0.37%, indistinguishable
from zero. With uniform weights, each horizon contributes 20% and
ISV's lambda actually has something to scale.
(2) Z-score lambda derivation
cuda/horizon_lambda.cu: replace `ratio = ema_h / mean(ema)` with
`z_h = (ema_h - mean) / std(ema); lambda = clamp(1.0 + 0.5*z, 1.0, 2.0)`.
Per `pearl_zscore_normalization_for_magnitude_asymmetric_signals.md`
z-score makes lambda spread scale-invariant of the absolute EMA
level. The ratio formula gave lambdas ≤ 1.04 in our data because
per-horizon BCE clusters tightly (range ~0.04) while mean is
~0.65. Z-score fills the [1.0, 2.0] envelope: 1σ → 1.5, 2σ →
ceiling. Boost-only asymmetric clamp preserved.
Test verification on the existing smoke (after 5 steps):
ema = [0.526, 0.522, 0.608, 0.641, 0.553]
lambda = [1.00, 1.00, 1.40, 1.76, 1.00]
Previously with ratio formula, max lambda on the same data
would have been ~1.05. h1000 (1.5σ above mean BCE here) now
gets a 76% trunk-gradient boost vs uniform.
(3) Default --seq-len 32 → 64
examples/alpha_train.rs: K=32 gives the model 0.5% of the
h6000 prediction window as in-window context. K=64 doubles
that, giving Mamba2's SSM state more material to build
long-horizon predictions. Within the kernel's MAMBA2_KERNEL_SEQ_MAX
cap of 96. Per-epoch wall scales ~K (more K-loop launches in
the captured graph, ~2× wall at K=64 vs K=32 for the K-loop
portion of dispatch).
Cache-bust v5 in build.rs to force nvcc recompile against the new
horizon_lambda.cu formula on the cluster's /cargo-target PVC. Old
cubins compute a numerically different lambda; running them against
the new Rust loop would silently apply the wrong gradient scaler.
Validation: 7 perception_overfit tests + 26 lib + 23 integration
ml-alpha tests pass. Synthetic overfit still converges to 0.0006.
horizon_ema_and_lambda_track_after_training observes the new
lambda spread (1.0-1.76) and asserts the asymmetric clamp envelope.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three correlated changes for the next CV round:
1. Mamba2 state_dim cap: 16 → 32
cuda/mamba2_alpha_kernel.cu: MAMBA2_ALPHA_MAX_STATE_D 16 → 32.
Per-thread state register `float x[32]` (128 B/thread) and
per-thread x_hist replay cache `float x_hist[K*32]` (up to
12 KiB/thread of local memory at K=96). L40S/H100 register file
(256 KiB/SM) absorbs this without occupancy collapse for our
block dims (32-128 threads). Update Rust-side
MAMBA2_KERNEL_STATE_MAX + validation message + test name. Kernel
header doc updated.
2. New early-stop option: auc_h6000
examples/alpha_train.rs: add the long-horizon AUC as a third
early-stop metric. The ISV CV (3a196382f, 5d42ab0e9, 0171c8c0e)
showed mean_auc-best-epoch and h6000-best-epoch can differ by
1-2 epochs and the h6000 gap can be 5-6pt within a single run
(fblb2 fold-1: saved E10 h6000=0.681, but E11 h6000=0.739 — we
threw away the deployment-better checkpoint). For multi-minute
trading deployment we want the h6000-best checkpoint directly.
3. Summary JSON: best_auc_h6000_epoch / best_auc_h6000 /
best_auc_h6000_per_horizon
So the analysis tooling can see the h6000-best checkpoint
independently of mean_auc / val_loss bests.
Test rename: test_mamba2_config_rejects_state_over_16 →
test_mamba2_config_rejects_state_over_32 (tests now reject state_dim=33).
build.rs cache-bust v4 forces cluster nodes to recompile the kernel
against the new MAMBA2_ALPHA_MAX_STATE_D — old cubins from previous
SHAs were sized for state_d=16 and would silently truncate state_d=32
state arrays.
Validation: 26 lib + 23 integration ml-alpha tests pass. Mamba2 block
tests use state_dim=8 or 16 (well below the new cap), exercise both
the forward + backward + AdamW paths.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fold-2 of the asymmetric-ISV 3-fold CV regressed -2.0pt mean_auc
vs no-ISV (0.696 vs 0.716) while folds 0 and 1 gained +1.3pt and
+2.5pt respectively. To understand why the controller hurts that
specific regime, log the per-horizon BCE EMA and lambda multiplier
each epoch via the trainer's `loss_ema_snapshot()` /
`lambda_snapshot()` accessors (mapped-pinned reads; per-epoch
budget, not hot-path).
Single fold-2 re-run on `--cv-fold 2 --cv-n-folds 3
--cv-train-window 4` will surface:
- which horizon's BCE the controller flagged as hardest each epoch
- whether lambda saturated at the [1.0, 2.0] ceiling for one horizon
- whether the lambda trajectory has more variance vs the
stable-fold runs (suggests the regime drifts during training)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
mhzs7 reported val mean_auc=0.726 on 3a196382f — but the trainer
constructed both train and val MultiHorizonLoader with the SAME
`mbp10_root: cli.mbp10_data_dir`. The two loaders only differed by
seed. So val sequences were held-out-by-anchor from the same files
train sampled from; not temporally OOS. Per
`pearl_single_window_oos_is_not_oos.md` a single-window result that
doesn't enforce time-ordered separation can collapse across true
walk-forward folds.
Refactor: drop `mbp10_root` from `MultiHorizonLoaderConfig` (which
forced caller to share the dir between train and val). New API takes
an explicit `files: Vec<PathBuf>` — the loader preserves the order
given and does no internal shuffle, so callers control temporal
ordering. Added `discover_mbp10_files_sorted(root)` helper that
enumerates a dir and sorts by filename (chronological under the
`ES.FUT_<YEAR>-Q<n>.dbn.zst` convention).
alpha_train.rs splits the discovered files by 3 new CLI flags:
--cv-fold <k> (default 0)
--cv-n-folds <N> (default 1 — single fold)
--cv-train-window <W> (default 0 — auto)
Single-fold default (cv_n_folds=1): train on all files except the
last, val on the last file. This replaces the old "same files for
both" bug; even runs that don't think about CV now get a temporal
split by default.
Sliding-window CV (cv_n_folds > 1): fold k trains on files
[k..k+W] and validates on file [k+W]. With 9 quarterly files
(2024-Q1..2026-Q1) and `--cv-n-folds 3`, the natural layout is:
fold 0: train 2024-Q1..2024-Q4 (W=4) → val 2025-Q1
fold 1: train 2024-Q2..2025-Q1 → val 2025-Q2
fold 2: train 2024-Q3..2025-Q2 → val 2025-Q3
blind holdout: 2025-Q4, 2026-Q1
Threaded the flags through scripts/argo-alpha-perception.sh and
infra/k8s/argo/alpha-perception-template.yaml so each fold submits
as an independent workflow.
Updated tests/multi_horizon_loader.rs to the new API:
loader_yields_seq_with_valid_labels — exercises discover + load.
loader_errors_on_empty_files — replaces missing-root test.
discover_errors_on_missing_root — pinpoints the discover step.
Honors:
- feedback_no_partial_refactor.md — every consumer migrated atomically.
- feedback_no_legacy_aliases.md — no `mbp10_root` shim left behind.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
cnjfl wall-time analysis: 80% of training time was disk IO. The
MultiHorizonLoader was constructed fresh per epoch in the CLI loop,
forcing 9 file deserializations from bincode (~50s/file × 9 = 7-8
min) per epoch. For a 6-epoch run that was ~45 min of pure IO out
of ~50 min total.
Now loaders preload ALL files into RAM at construction (one-time
~7 min startup) and expose a `reset(seed: u64)` method that
re-seeds anchor sampling per epoch — no disk IO between epochs.
Memory cost: ~13-15 GB for the 9-quarter ES.FUT dataset (45M
snapshots × ~280 bytes). Well under the training pod's 64 GB
limit; current 16 GB request remains sufficient since the resident
set fits.
Expected wall-time impact at K=96, B=8, 16K seqs:
6 epochs: ~50 min → ~13 min (~3.7×)
15 epochs: ~120 min → ~23 min (~5×)
The internal LoadedFile-cache-with-cycling logic is gone — the
loader now holds Vec<LoadedFile> with all files resident. Per-call
`next_sequence` picks a uniformly random file + uniformly random
anchor inside it (instead of cycling files with a per-file budget).
Distribution is equivalent: each file contributes ~n_max_sequences /
n_files samples per epoch in expectation.
77 ml-alpha tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
cnjfl evidence: val_loss and mean_auc disagree.
val_loss best at e3 (0.5592)
mean_auc best at e4 (0.7670 — new h300 + h6000 peaks)
For downstream trading, ranking quality (AUC) matters more than
probability calibration (BCE loss). New default is mean_auc-based
early stopping, but val_loss/none remain selectable.
AUC is noisier than loss epoch-to-epoch (1-2pt bounces are common
even when long-horizon AUCs are still drifting up under
auto-horizon-weights), so patience defaults bump from 3 → 5.
CLI: --early-stop-metric {val_loss|mean_auc|none} default mean_auc
--early-stop-patience N default 5
Argo template parameters added with matching defaults.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
cnjfl run showed val_loss and mean-AUC peak at different epochs:
epoch 3: val_loss=0.5592 (best) mean_auc=0.7608
epoch 4: val_loss=0.5609 (worse) mean_auc=0.7670 (best — new h300 + h6000 peaks)
val_loss tracks probability calibration; AUC tracks ranking quality.
For downstream trading the ranking profile matters more — so we now
publish both bests in alpha_train_summary.json and log a "new best
mean_auc" line whenever a new mean-AUC peak lands.
Early stopping still gates on val_loss (the two policies stay
decoupled — mean-AUC is reported-only).
New summary fields:
best_mean_auc_epoch
best_mean_auc
best_mean_auc_per_horizon
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Plumbs the batched cfc + heads kernels added in 829ddfa62 through
PerceptionTrainer, evaluator, and the alpha_train CLI:
PerceptionTrainerConfig.n_batch — batch size, default 1
PerceptionTrainer::step_batched — process B sequences per
optimizer step using
cfc_step_batched / heads_batched
PerceptionTrainer::evaluate_batched — forward-only batched eval
PerceptionTrainer::step / evaluate — thin B=1 wrappers preserving
existing single-sequence
test/inference APIs (assert
cfg.n_batch == 1)
alpha_train CLI: --batch-size N — accumulates B sequences per
optimizer step in train loop;
val loop also batches and uses
evaluate_batched
Per-K scratch buffers all grow to [K, B, dim] layout (K-major, slot-k
contiguous). Mamba2's [B, K, H] output is transposed once after
forward via the new transpose_3d_swap_01 kernel, and grad_h_enriched_seq_t
is transposed back to [B, K, H] before Mamba2 backward. Two transposes
per training step; negligible (1.5MB at B=32).
Dead unbatched kernel handles removed from the trainer (step_fn,
step_bwd_fn, heads_fn, heads_bwd_fn, grad_x_d) — all training and
inference now go through the batched variants for B ≥ 1. The
single-sample kernels remain in CUDA for the standalone test helpers
in cfc/step.rs and heads.rs.
Local 2Q smoke (seq_len=32, B=4, --auto-horizon-weights, 800 train
seqs × 2 epochs):
epoch 0: val_loss=0.7138 AUC h30/h100/h300/h1000/h6000 = .55/.55/.57/.61/.51
epoch 1: val_loss=0.6558 AUC h30/h100/h300/h1000/h6000 = .72/.68/.75/.68/.65
vs the in-flight qf5mj baseline (B=1, K=96, no horizon weighting) which
had val_loss=0.6933 best and AUCs oscillating at ~0.50 — this batched
run hits AUC 0.75 (h300) and 0.72 (h30) in just 2 epochs of 200
optimizer updates. Batching + horizon-weighting unblocks the model.
77 ml-alpha tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
For seq_len K and horizon h with h ≫ K, the K position-supervised
labels in a single sequence are near-identical (sequential positions'
forward windows overlap by ~(h-1)/h). Per-position BCE therefore
treats ~K highly-correlated labels as independent samples, inflating
gradient pressure on long horizons by a factor of K.
Concretely at K=96:
h=30 → ~3 effective samples per seq (forward windows overlap ~97%)
h=100 → ~1 (~99%)
h=6000 → ~1 (~99.98%)
Per-position supervision was paying 96× the natural signal density on
h=6000, pulling the model toward fitting noise at long horizons.
Fix: the fused BCE kernel now accepts an optional
`loss_weights[N_HORIZONS]` (nullptr → uniform = no-op). Each (k, h)
loss + grad contribution is multiplied by w_h; the normaliser is the
sum of weighted valid entries instead of the raw valid count.
`auto_horizon_weights(K, horizons)` computes `w_h = min(1.0, K/h)` so
short horizons stay at full weight and long horizons collapse to
their independent-sample density. Exposed via CLI:
--auto-horizon-weights # K/h auto-derived
--horizon-weights "1,1,0.5,0.1,0.02" # explicit floats
Default behaviour is uniform (1.0) — apples-to-apples with the
in-flight qf5mj baseline. Synthetic overfit still 0.6268 → 0.1144 in
250 steps (82% drop). 77 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Comprehensive fix for the issues identified after the BPTT-unroll cluster
run plateaued at val_loss ~0.692 with oscillating AUCs:
ARCHITECTURE
- CfC h_old is now RECURRENT across positions. Previously reset to
zero every step → CfC degenerated to a per-cell tanh-FC layer.
New: h_old at step k IS h_new at step k-1. Heads still operate
on h_new_k, but now the CfC actually carries state. Reverse-order
backward through the K positions accumulates grad_h_old → grad_h_new
via the new optional `grad_h_carry` arg on multi_horizon_heads_backward.
- tau is TRAINED. cfc_step_backward now writes grad_tau (per-cell decay
constant derivative), trainer gets a 7th AdamW group at 0.1× cfc lr.
- 6 NEW regime features (EMA cascade computed loader-side per file)
fill slots out[20..26] of snap_features. Gives the model multi-minute
trend / volatility / liquidity context that is structurally unreachable
inside the K-snapshot BPTT window. Slots: mid-z (med/slow), trend
signal, log-vol slow, log-spread med, log-trade-rate med. All bounded
via log1p / signed-log so no tuned constants leak in.
PERFORMANCE (NVIDIA-style)
- GPU-fused multi-horizon BCE for the entire [K, N_HORIZONS] grid in
ONE launch (was K host roundtrips). Native NaN-label masking.
- K-loop is fully GPU-resident: pre-allocated per-K scratch
(h_new_per_k, probs_per_k, labels_per_k, grad_probs_per_k), zero
device allocs inside step(). Only TWO syncs per sequence (after
forward, after backward) vs previously 2K+1.
- Stream-ordered kernel launches with pointer-offset addressing into
per-K buffers — host doesn't wait between K iterations.
- cfc_step_backward / multi_horizon_heads_backward both use += grad
semantics; trainer pre-zeroes accumulators once per step().
- MAMBA2_ALPHA_MAX_K capped at 96 (was temporarily at 256). 96 covers
h=30/100/300 with room; regime features handle h=1000/h=6000.
TRAINING DISCIPLINE
- LR schedule: linear warmup (default 200 steps) + cosine decay to
lr * lr_min_factor (default 0.1). Applied per training step to both
CfC and Mamba2 AdamW groups via new set_lr_cfc/set_lr_mamba2.
- Best-checkpoint tracking by val_loss; recorded in summary
(best_epoch, best_val_loss, best_val_auc).
- Early stopping on val_loss plateau (default patience = 3).
- CRITICAL BUG FIX: validation now uses new `evaluate()` method
(forward-only) instead of `step()`. Previous CLI called step()
on val data, which ran the full backward + AdamW update on the
validation set. With per-step BPTT that's ~K× more pressure than
the old comment ("statistically negligible") assumed.
Synthetic overfit: 0.6442 → 0.1233 in 250 steps (81% drop, sharper
than the previous 70%). 77 ml-alpha tests pass.
Local 2Q smoke (seq_len=64, 600 train seqs/epoch, 4 epochs):
val_loss 0.7011 → 0.6990, best epoch=1, h300 AUC 0.565 in epoch 0.
Phase E.3 callers (ml/examples/alpha_baseline.rs,
alpha_dqn_h600_smoke.rs) use the LEGACY Mamba2 forward_train +
backward_from_h_enriched path — unaffected by these changes (their
kernels are pre-zeroed via alloc_zeros, so the += grad semantics
remain correct in single-call mode).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The final-step-only trainer (one BCE prediction per 32-snapshot
window) trained flat at chance on real ES data despite working on
synthetic overfit: train_loss=0.6953, val_loss=0.6943 across 40k
gradient steps. Gradient density was the bottleneck — one supervised
position per sequence × ~8K seqs/epoch isn't enough signal for the
SSM to find the alpha.
This commit supervises the model at EVERY position in the sequence:
mamba2_alpha_scan_fwd_seq — emits h_enriched at every t step
([N, K, sh2] instead of [N, sh2])
mamba2_alpha_scan_bwd_seq — accepts d_h_enriched_seq, injects
gradient at each t before propagating
d_state through the gate chain.
d_w_c and d_h_s2 accumulate across t.
PerceptionTrainer.step() — loop k=0..K; cfc + heads + BCE at
each valid label; cfc/heads grads
accumulate via += in kernel writes.
One Mamba2 backward call consumes the
full grad_h_enriched_seq.
cfc_step_backward — grad_w_in/w_rec/b writes changed
to += (callers MUST pre-zero).
multi_horizon_heads_backward — grad_w/grad_b writes changed to +=.
alpha_train.rs — passes per-position label rows to
step(); AUC still scored from
last-position predictions.
Phase E.3 callers (alpha_baseline.rs, alpha_dqn_h600_smoke.rs) use
the LEGACY Mamba2 forward_train + backward path with `alloc_zeros`
grad buffers — unaffected.
Synthetic overfit still converges 0.6664 → 0.1976 in 250 steps.
Local 2-quarter ES.FUT smoke shows the val AUC at h300 climbing
0.513 → 0.566 over 3 epochs (was flat-at-chance before). First
gradient signal we've gotten through the new architecture.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per user direction "no gating, this is the new default": the stacked
Mamba2 -> CfC -> heads design is THE production architecture. There's
no competing-baseline comparison to run. Validation reduces to normal
training metrics (per-horizon val AUC, train loss curve, sanity floor
of >0.5 AUC).
Deletions:
- crates/ml-alpha/src/gate/cfc_vs_mamba2.rs (gate verdict logic)
- crates/ml-alpha/src/gate/mod.rs
- crates/ml-alpha/examples/alpha_gate.rs (gate runner binary)
Renames:
- crates/ml-alpha/src/gate/auc.rs -> crates/ml-alpha/src/eval/auc.rs
- lib.rs: pub mod gate -> pub mod eval (gate implied comparison;
eval doesn't)
Spec amendments:
- Drop the "Gate baseline strategy" amendment (committed earlier
this session)
- Reframe the stacked-architecture amendment as a "decision" not a
"gate"; production path is unambiguous
- Reframe Section 4 "Validation gate: CfC must meet Mamba2" -> just
"Validation: per-horizon val AUC" with the >0.5 sanity floor
Doc cleanups: stale "Mamba2 gate baseline" mentions in build.rs and
pinned_mem.rs replaced with neutral wording. The Argo template
comment about "downstream gate consumption" becomes "for monitoring".
Test status: all 26+ ml-alpha tests pass. AUC tests (6/6) still pass
under the eval:: namespace.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The merged Mamba2 -> CfC -> heads design from the 2026-05-16 spec
amendment supersedes the CfC-alone path. Removing the old CfC-only
PerceptionTrainer and renaming the stacked Mamba2CfcTrainer to
PerceptionTrainer (one trainer, clean naming).
Deletions:
- src/trainer/perception.rs (the OLD CfC-only trainer)
- tests/perception_overfit.rs (CfC-only smoke)
- tests/perception_debug_dump.rs (CfC-only trajectory print)
- tests/stacked_overfit.rs (replaced by perception_overfit pointing
at the renamed module)
Renames:
- src/trainer/stacked.rs -> src/trainer/perception.rs
- Mamba2CfcTrainer -> PerceptionTrainer
- Mamba2CfcTrainerConfig -> PerceptionTrainerConfig
- tests/stacked_overfit.rs content -> tests/perception_overfit.rs
CLI rewrite:
examples/alpha_train.rs now drives the stacked PerceptionTrainer.
Per-step inputs are sequences (Vec<Mbp10RawInput>) of length
seq_len; labels come from the LAST position of the window
(per-horizon). Flags: --seq-len, --mamba2-state-dim, --lr-cfc,
--lr-mamba2 (no more --n-hid since hidden_dim is fixed at 128 to
match Mamba2 and CfC by design).
Test status:
- 64 GPU tests pass on local sm_86 (31 lib unit + 33 integration)
- synthetic-overfit (rebranded perception_overfit): 250 steps,
initial=0.5951 -> final=0.1917 (68% drop, well above 40% gate)
- All bit-equiv / finite-diff / invariant tests still PASS
- One ignored test: the gate_artifact integration (waiting for
cluster-trained summary inputs)
The cluster gate (Task 18) now compares stacked-trained AUC vs a
Mamba2-baseline AUC (TBD: stacked vs a simpler "Mamba2 only" config
or an external reference baseline).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLI wraps PerceptionTrainer + MultiHorizonLoader for end-to-end
training. Per-epoch loop:
- train: stream sequences from MultiHorizonLoader, step per
position with reset_hidden_state (K=1 BPTT), accumulate train
loss
- val: separate loader on disjoint seed, accumulate (probs,
labels) per horizon, compute Mann-Whitney U AUC
Emits alpha_train_summary.json with final train loss + per-horizon
val AUC for downstream gate consumption.
Adds PerceptionTrainer::last_probs() — slow-path readback of the
most recent forward's probs. Used by the eval loop to capture
per-position predictions for AUC.
Args: --mbp10-data-dir --predecoded-dir --out --epochs --n-hid
--seq-len --lr --n-train-seqs --n-val-seqs --seed (all with sane
defaults from spec Section 4 Phase A).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
