mhzs7 reported val mean_auc=0.726 on 3a196382f — but the trainer
constructed both train and val MultiHorizonLoader with the SAME
`mbp10_root: cli.mbp10_data_dir`. The two loaders only differed by
seed. So val sequences were held-out-by-anchor from the same files
train sampled from; not temporally OOS. Per
`pearl_single_window_oos_is_not_oos.md` a single-window result that
doesn't enforce time-ordered separation can collapse across true
walk-forward folds.
Refactor: drop `mbp10_root` from `MultiHorizonLoaderConfig` (which
forced caller to share the dir between train and val). New API takes
an explicit `files: Vec<PathBuf>` — the loader preserves the order
given and does no internal shuffle, so callers control temporal
ordering. Added `discover_mbp10_files_sorted(root)` helper that
enumerates a dir and sorts by filename (chronological under the
`ES.FUT_<YEAR>-Q<n>.dbn.zst` convention).
alpha_train.rs splits the discovered files by 3 new CLI flags:
--cv-fold <k> (default 0)
--cv-n-folds <N> (default 1 — single fold)
--cv-train-window <W> (default 0 — auto)
Single-fold default (cv_n_folds=1): train on all files except the
last, val on the last file. This replaces the old "same files for
both" bug; even runs that don't think about CV now get a temporal
split by default.
Sliding-window CV (cv_n_folds > 1): fold k trains on files
[k..k+W] and validates on file [k+W]. With 9 quarterly files
(2024-Q1..2026-Q1) and `--cv-n-folds 3`, the natural layout is:
fold 0: train 2024-Q1..2024-Q4 (W=4) → val 2025-Q1
fold 1: train 2024-Q2..2025-Q1 → val 2025-Q2
fold 2: train 2024-Q3..2025-Q2 → val 2025-Q3
blind holdout: 2025-Q4, 2026-Q1
Threaded the flags through scripts/argo-alpha-perception.sh and
infra/k8s/argo/alpha-perception-template.yaml so each fold submits
as an independent workflow.
Updated tests/multi_horizon_loader.rs to the new API:
loader_yields_seq_with_valid_labels — exercises discover + load.
loader_errors_on_empty_files — replaces missing-root test.
discover_errors_on_missing_root — pinpoints the discover step.
Honors:
- feedback_no_partial_refactor.md — every consumer migrated atomically.
- feedback_no_legacy_aliases.md — no `mbp10_root` shim left behind.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
cnjfl evidence: val_loss and mean_auc disagree.
val_loss best at e3 (0.5592)
mean_auc best at e4 (0.7670 — new h300 + h6000 peaks)
For downstream trading, ranking quality (AUC) matters more than
probability calibration (BCE loss). New default is mean_auc-based
early stopping, but val_loss/none remain selectable.
AUC is noisier than loss epoch-to-epoch (1-2pt bounces are common
even when long-horizon AUCs are still drifting up under
auto-horizon-weights), so patience defaults bump from 3 → 5.
CLI: --early-stop-metric {val_loss|mean_auc|none} default mean_auc
--early-stop-patience N default 5
Argo template parameters added with matching defaults.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds two new workflow parameters with backward-compatible defaults
(batch-size=1, auto-horizon-weights=false) so existing submissions
behave identically. Both flags are forwarded to alpha_train CLI.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
alpha_train --seed is clap-typed as u64, and clap's default u64
parser only accepts decimal digits. Previous default "0x4242"
hit "invalid digit found in string" at startup.
Replace with decimal equivalent 16962 (= 0x4242) in both the
submission script default and the workflow template default.
Long-term: could add a custom clap value_parser that accepts
hex/dec/oct prefixes, but for now decimal-only matches the
foxhunt convention in other CLIs (alpha_baseline, etc.).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a tiny alpine pod (check-cache) that runs first on the platform
pool (no autoscaler delay, ~3 sec end-to-end) and probes the
training-data PVC for /data/bin/$SHA/alpha_train. Outputs:
- sha: short SHA used for binary cache keying
- cache: "hit" or "miss"
ensure-binary now has `when: cache == miss` — when the binary is
already cached for the current SHA, the entire ~4.8GB ci-builder
image pull + sccache compile cycle is skipped. Re-runs on the same
SHA now go straight from submission to training in ~30 seconds
instead of ~3 minutes.
train depends on check-cache + ensure-binary; sources the SHA from
check-cache's output (works whether ensure-binary ran or was
skipped — Argo treats `when:` skip as a satisfied dependency).
Submission script (scripts/argo-alpha-perception.sh) now pre-resolves
commit-sha=HEAD to an actual git SHA via `git rev-parse origin/<branch>`
before submission. This lets the alpine check-cache pod work without
installing git in the container.
Also removed the now-stale `ci-training-h100x2|ci-training-h100-sxm`
case branch from the SM-arch detection — those pools no longer exist
post-pool-cleanup commit a252119fd.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Argo WorkflowTemplate at infra/k8s/argo/alpha-perception-template.yaml
runs the stacked Mamba2 -> CfC -> heads PerceptionTrainer on a single
L40S in fr-par-2. Two-stage DAG:
ensure-binary (ci-compile-cpu pool, sccache-backed cargo build of
alpha_train example, SHA-keyed binary cache under
/data/bin/$SHORT_SHA/)
train (ci-training-l40s pool, runs the cached binary against
/data/futures-baseline/mbp10 with predecoded sidecar
cache at /feature-cache/predecoded, writes
alpha_train_summary.json to
/feature-cache/alpha-perception-runs/$SHA/)
Defaults mirror the validated synthetic-overfit smoke config:
epochs=5, seq_len=32, mamba2_state_dim=16, lr_cfc=3e-3,
lr_mamba2=1e-3, n_train_seqs=8000, n_val_seqs=1000, seed=0x4242
Submission script scripts/argo-alpha-perception.sh wraps argo submit
with the standard L40S/H100 cuda-compute-cap mapping. --watch
follows logs.
Workflow nodeSelector pinned to fr-par-2 (consistent with the cluster
topology constraint). ttlStrategy 1h after completion;
activeDeadlineSeconds 4h cap (well above expected ~30-90 min wall).
This is the cluster entrypoint for the stacked perception design.
Once it lands a summary on MinIO, the gate runner (alpha_gate, Task
17) can consume it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
