feat(ml-alpha): walk-forward CV via file-list-driven loader

mhzs7 reported val mean_auc=0.726 on 3a196382f — but the trainer
constructed both train and val MultiHorizonLoader with the SAME
`mbp10_root: cli.mbp10_data_dir`. The two loaders only differed by
seed. So val sequences were held-out-by-anchor from the same files
train sampled from; not temporally OOS. Per
`pearl_single_window_oos_is_not_oos.md` a single-window result that
doesn't enforce time-ordered separation can collapse across true
walk-forward folds.

Refactor: drop `mbp10_root` from `MultiHorizonLoaderConfig` (which
forced caller to share the dir between train and val). New API takes
an explicit `files: Vec<PathBuf>` — the loader preserves the order
given and does no internal shuffle, so callers control temporal
ordering. Added `discover_mbp10_files_sorted(root)` helper that
enumerates a dir and sorts by filename (chronological under the
`ES.FUT_<YEAR>-Q<n>.dbn.zst` convention).

alpha_train.rs splits the discovered files by 3 new CLI flags:
  --cv-fold <k>            (default 0)
  --cv-n-folds <N>         (default 1 — single fold)
  --cv-train-window <W>    (default 0 — auto)

Single-fold default (cv_n_folds=1): train on all files except the
last, val on the last file. This replaces the old "same files for
both" bug; even runs that don't think about CV now get a temporal
split by default.

Sliding-window CV (cv_n_folds > 1): fold k trains on files
[k..k+W] and validates on file [k+W]. With 9 quarterly files
(2024-Q1..2026-Q1) and `--cv-n-folds 3`, the natural layout is:

  fold 0: train 2024-Q1..2024-Q4 (W=4) → val 2025-Q1
  fold 1: train 2024-Q2..2025-Q1       → val 2025-Q2
  fold 2: train 2024-Q3..2025-Q2       → val 2025-Q3
  blind holdout: 2025-Q4, 2026-Q1

Threaded the flags through scripts/argo-alpha-perception.sh and
infra/k8s/argo/alpha-perception-template.yaml so each fold submits
as an independent workflow.

Updated tests/multi_horizon_loader.rs to the new API:
  loader_yields_seq_with_valid_labels — exercises discover + load.
  loader_errors_on_empty_files       — replaces missing-root test.
  discover_errors_on_missing_root    — pinpoints the discover step.

Honors:
  - feedback_no_partial_refactor.md — every consumer migrated atomically.
  - feedback_no_legacy_aliases.md — no `mbp10_root` shim left behind.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-17 16:36:48 +02:00
parent 3a196382f0
commit eb51c0f9cd
5 changed files with 167 additions and 31 deletions

View File

@@ -31,6 +31,9 @@ BATCH_SIZE=1
AUTO_HORIZON_WEIGHTS=false
EARLY_STOP_METRIC=mean_auc
EARLY_STOP_PATIENCE=5
CV_FOLD=0
CV_N_FOLDS=1
CV_TRAIN_WINDOW=0
WATCH=false
usage() {
@@ -47,6 +50,9 @@ Usage: $0 [OPTIONS]
--n-train-seqs <n> Train sequences per epoch (default: $N_TRAIN_SEQS)
--n-val-seqs <n> Val sequences per epoch (default: $N_VAL_SEQS)
--seed <n> Random seed (default: $SEED)
--cv-fold <k> CV fold index (default: $CV_FOLD)
--cv-n-folds <N> Total CV folds (default: $CV_N_FOLDS)
--cv-train-window <W> Files per train window (default: $CV_TRAIN_WINDOW = auto)
--watch Follow logs via argo watch
EOF
}
@@ -68,6 +74,9 @@ while [[ $# -gt 0 ]]; do
--auto-horizon-weights) AUTO_HORIZON_WEIGHTS=true; shift ;;
--early-stop-metric) EARLY_STOP_METRIC="$2"; shift 2 ;;
--early-stop-patience) EARLY_STOP_PATIENCE="$2"; shift 2 ;;
--cv-fold) CV_FOLD="$2"; shift 2 ;;
--cv-n-folds) CV_N_FOLDS="$2"; shift 2 ;;
--cv-train-window) CV_TRAIN_WINDOW="$2"; shift 2 ;;
--watch) WATCH=true; shift ;;
-h|--help) usage; exit 0 ;;
*) echo "Unknown option: $1"; usage; exit 1 ;;
@@ -135,4 +144,7 @@ argo submit -n foxhunt --from=wftmpl/alpha-perception \
-p auto-horizon-weights="$AUTO_HORIZON_WEIGHTS" \
-p early-stop-metric="$EARLY_STOP_METRIC" \
-p early-stop-patience="$EARLY_STOP_PATIENCE" \
-p cv-fold="$CV_FOLD" \
-p cv-n-folds="$CV_N_FOLDS" \
-p cv-train-window="$CV_TRAIN_WINDOW" \
$WATCH_FLAG