alpha_rl_train reads MBP-10 data with its own predecoded cache —
it never uses the fxcache pipeline. Skip the 15-minute feature
extraction step entirely for model=alpha-rl by depending only on
ensure-binary + gpu-warmup.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
SP-chain training has been standardising on L40S since 2026-05-09, but
every invocation required an explicit `--gpu-pool ci-training-l40s`
override. The 2026-05-04 train-mnpf7 incident (sm_90 cubins deployed
to an L40S device, then resubmitted with the explicit override) was
the last incident in a long line of "forgot the pool flag" friction.
`feedback_default_to_l40s_pool.md` codified the user preference; this
commit lands the default in the actual invocation paths.
Changes:
- infra/k8s/argo/train-template.yaml: gpu-pool default H100 → L40S
- infra/k8s/argo/train-multi-seed-template.yaml: same + cuda-compute
-cap default 90 → 89
- scripts/argo-train.sh: docstring / --help / compute-cap fallback
case all flip to L40S as the bare default; H100 becomes opt-in via
`--gpu-pool ci-training-h100` for 80 GB / sm_90 workloads
- scripts/argo-test.sh: --help text aligned
Other architectural defaults (data-source=mbp10 per
feedback_mbp10_mandatory; imbalance-bar-threshold=20.0 per the 2026-
05-10 OOM-prevention fix) are already correct in the template.
Verified via `argo-train.sh dqn --branch sp20-aux-h-fixed --sha HEAD
--baseline --dry-run` — rendered workflow shows cuda-compute-cap=89,
no explicit gpu-pool override (template default L40S in effect).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The multi-seed path already does `kubectl apply` before `argo submit`,
so cluster template stays in sync with source. The single-job path used
`argo submit --from=wftmpl/train` directly, expecting the cluster's
template to already match — which silently drifts when defaults change.
Caused workflow train-jpxvn (2026-05-10) to dispatch with stale
imbalance-bar-threshold=0.5 default (the cluster's old value) when the
source had been bumped to 20.0. Triggered near-OOM in feature extraction.
One-line fix: apply the template before submission. Mirrors what
multi-seed already does. No behavior change for users who pass explicit
flags; just makes implicit defaults track source.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Two architectural cleanups, both surfaced by the wgdc8 experiment
### Part 1: volume_bar_size in cache key
Mirrors the imbalance_bar_threshold/ewma_alpha fix from `f7718b376`. The
volume bar size constant (100 contracts/bar) was previously hardcoded and
not in the fxcache key. Tuning it would have hit the same fossilization
bug as imbalance_bar_threshold did pre-fix.
Changes:
- `Hyperparams.volume_bar_size: u64` field added (default 100, matches
`DEFAULT_VOLUME_BAR_SIZE` for backwards compat).
- TrainingProfile loader reads `volume_bar_size` TOML key.
- `calculate_dbn_cache_key_full` signature 7 → 8 args. Hashed via
`to_le_bytes()`. Test `test_cache_key_includes_volume_bar_size`
added; passes alongside the 5 existing tests.
- 4 callers updated atomically (per `feedback_no_partial_refactor`):
`discover_and_load`, `data_loading.rs:146`, `train_baseline_rl.rs:599`,
`precompute_features.rs:259,720`.
- `data_loading.rs:279` now passes `self.hyperparams.volume_bar_size`
to `build_volume_bars` instead of the hardcoded `DEFAULT_VOLUME_BAR_SIZE`.
- New `--volume-bar-size` CLI arg on both binaries (default 100).
- New Argo workflow params `volume-bar-size` (default "100") and
`data-source` (default "mbp10") on both `train-template.yaml` and
`train-multi-seed-template.yaml`. Threaded into precompute + trainer
invocations.
- `scripts/argo-train.sh` exposes `--volume-bar-size <n>` and
`--data-source <s>` for ad-hoc overrides.
### Part 2: OFI front-month filter (latent bug fix)
`crates/ml/examples/precompute_features.rs:539-557` (the OFI/VPIN/Kyle's
Lambda computation branch when MBP-10 + trades data is available) was
loading trades unfiltered for per-bar microstructure feature computation.
The volume bar formation path filters front-month per-file (line 354), but
the OFI path did not.
Effect pre-fix: during contract rollover windows (e.g., ESZ24 → ESH25),
OFI per-bar microstructure features included trades from BOTH contracts
simultaneously, distorting VPIN, Kyle's Lambda, and trade imbalance
signals. Severity in production: small (front-month dominates ES.FUT
volume by 10-100×) but real and present in every prior MBP-10+trades
production run.
Fix: mirror the per-file `filter_front_month` call from the volume bar
path. Volume bar formation and OFI computation now both see the same
in-month trade tape. Added log line shows raw vs filtered count per file
for transparency.
## Why bundled
Both fixes touch trade-data plumbing in `precompute_features.rs` and the
fxcache key contract. Per `feedback_no_partial_refactor`, related
architectural cleanups land atomically. Both surfaced from the same
wgdc8 audit; bundling avoids two cache-key-invalidating commits in
sequence (each would force full fxcache regen).
## Compatibility
- `volume_bar_size` defaults to 100 → existing wgdc7-equivalent runs
reproduce, but with a *new* fxcache key (the f7718b376-era cache file
is unreachable; harmless, can GC manually).
- OFI fix is strictly more correct; no opt-out needed. Existing models
trained on contaminated OFI features may show slight feature
distribution drift on first cache regen — expected, not a regression.
- `data_source = "ohlcv"` Argo param now possible; routes precompute
through volume bar branch directly. wgdc8 experiment uses this to test
bar resolution sensitivity at volume_bar_size=500 (5× DEFAULT).
Tests: 6/6 feature_cache tests pass. Workspace + examples compile clean.
Audit-doc: `docs/dqn-wire-up-audit.md` updated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## The bug (audit 2026-05-09)
`crates/ml/src/feature_cache.rs:calculate_dbn_cache_key_full` hashed only
`(symbol, data_source, dbn_filenames+sizes)` — NOT `imbalance_bar_threshold`
or `imbalance_bar_ewma_alpha`. Combined with `precompute_features.rs:346`
unconditionally calling `build_volume_bars` regardless of `data_source`,
this meant:
1. 14 audited production runs (Apr 11–May 8) all collided on the same
fxcache key (`a3f933aa...` / `c07c960a...`) regardless of TOML
`imbalance_bar_threshold` value
2. The imbalance-bar code path was reachable only via fxcache MISS, which
never happens in production because `ensure-fxcache` always populates
first
3. Every "tuning" of `imbalance_bar_threshold` across 16+ SP runs was a
silent no-op — the system was actually running volume bars at
DEFAULT_VOLUME_BAR_SIZE (100 contracts/bar)
## The fix (this commit)
**Part A — cache key includes bar formation params:**
- `calculate_dbn_cache_key_full` signature: 5 args → 7 args. Two new f64
params hashed via `to_le_bytes()`.
- 4 callers updated atomically (per `feedback_no_partial_refactor`).
- 2 new unit tests (`test_cache_key_includes_bar_threshold`,
`test_cache_key_includes_bar_alpha`) pin the contract.
**Part B — precompute_features actually USES data_source:**
- New CLI args `--imbalance-bar-threshold` (default 0.5) and
`--imbalance-bar-ewma-alpha` (default 0.1) on both train_baseline_rl
and precompute_features.
- `precompute_features.rs:346` now branches: when
`data_source == "mbp10"` AND `mbp10_data_dir.is_some()`, calls
`mbp10_to_imbalance_bars` instead of `build_volume_bars`.
**Argo plumbing:**
- `train-template.yaml` + `train-multi-seed-template.yaml`: new workflow
parameters threaded into BOTH precompute and trainer invocations so
both compute the same fxcache key.
- `scripts/argo-train.sh`: new CLI flags for ad-hoc overrides.
- ensure-fxcache regen path: removed `rm -f /feature-cache/*.fxcache`
(with bar-params now in key, parallel experiments coexist).
## Effects going forward
- Tuning `imbalance_bar_threshold` actually changes bar density
- Configuring `data_source = "mbp10"` actually produces imbalance bars
- Multiple parallel experiments at different thresholds coexist on PVC
- `dqn-production.toml: imbalance_bar_threshold = 0.5` no longer ignored
Default values match prior production behavior → existing wgdc7-equivalent
runs reproduce, just with a *new* fxcache key (the old volume-bar cache
file is still on disk but won't be hit; harmless, can GC manually).
Audit-doc: `docs/dqn-wire-up-audit.md` updated with full context.
Tests: 5/5 feature_cache tests pass, full workspace + examples compile clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
While P0b smoke (train-sw4ws on bdc5cb8bb) runs, two prep items:
(A) scripts/argo-train.sh — add `ci-training` to the L40S sm_89 case
arm. Bare `ci-training` is an L40S pool alias in some clusters;
previously defaulted to sm_90 (Hopper), causing train-mnpf7 to
deploy with wrong-arch cubins (terminated + resubmitted manually).
Now both `*l40s*` and bare `ci-training` resolve correctly.
(C) docs/superpowers/plans/...sp13...md — Layer B section expanded
with concrete codebase locations discovered during P0a:
- aux_heads_kernel.cu, aux_heads_loss_ema_kernel.cu locations
- aux_nb_label_buf populated as column 0 of next_states (log_return)
- F1/F2 regression history note (don't alias the label buffer)
- aux_pred_to_isv_tanh_kernel.cu is a P0a placeholder per its own
header — Layer B should rewrite to read softmax logit-diff
- dir_acc kernel + oracle tests need softmax-read updates
- Layer B + P0b combined rationale post-P0a empirics
Saves the next implementer ~30 min of re-investigation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The first L40S deploy attempt (workflow `train-multi-seed-z2llf`, terminated)
failed at startup with `error: unexpected argument '--fold' found` on every
job: `train_baseline_rl` is a multi-fold walk-forward executor that accepts
`--max-folds K`, NOT `--fold N`. The original P5T1 harness assumed the
opposite and fanned out N seeds × K folds = N*K jobs, each invoking the
binary with `--seed N --fold K`.
User chose Path B: pivot to one job per seed (each runs all K folds via the
existing `--max-folds` mechanism). Per-job runtime is K× longer, but fanout
drops from N*K=30 → N=5 (matches L40S pool capacity better) and the binary
contract becomes the one the binary actually has.
4 surface changes:
1. crates/ml/examples/train_baseline_rl.rs — add `--seed N` CLI arg
(default 42 — historic implicit value). Sets `FOXHUNT_SEED` env var at
startup BEFORE any CUDA module spins up. Logs the seed value at the
training start banner.
2. crates/ml/src/cuda_pipeline/mod.rs — add `global_seed()` (reads
`FOXHUNT_SEED`, default 42) + `mix_seed(base)` (SplitMix64 avalanche
so adjacent global seeds produce uncorrelated module seeds). Six call
sites updated to mix the global seed into their previously-hardcoded
constants:
- trainer/action.rs: GpuActionSelector seed (0xDEAD_BEEF_CAFE) + the
epsilon-greedy fallback StdRng (0xAC7_DEF0).
- cuda_pipeline/gpu_iqn_head.rs: IQN Xavier-init RNG (0x1CA_1234).
- cuda_pipeline/gpu_iql_trainer.rs: V(s) Xavier-init RNG (0x1C1_9ABC).
- cuda_pipeline/gpu_her.rs: random-donor RNG (0x4E4_5678).
- cuda_pipeline/gpu_ppo_collector.rs: rng_seeds Vec for PPO
experience-collector init + reset (0xAA0_5EED).
- trainer/training_loop.rs: per-epoch regime_dropout_seed.
3. infra/k8s/argo/train-multi-seed-template.yaml — drop `fold` parameter
from `train-single` template; binary invoked as `--seed "$SEED"
--max-folds {{workflow.parameters.folds}}` so the walk-forward sweep
happens inside the single training process. Drop `FOLD` env var. Update
the nsys-rep upload filename to drop the fold suffix. Update banners /
doc comments to reflect "one-job-per-seed" semantics.
4. scripts/argo-train.sh — matrix generator drops the inner fold loop.
Each emitted task carries only `seed=${s}` and depends on the same
ensure-fxcache + gpu-warmup. The dry-run synthetic marker switches from
`seed=${s} fold=${f}` to `seed=${s} max_folds=${FOLDS}` so test harnesses
count the new shape correctly.
5. scripts/tests/test_multi_seed_harness.sh — assertions updated:
- `--multi-seed 3 --folds 2` produces 3 tasks (was 6).
- Rendered binary command must include `--max-folds
{{workflow.parameters.folds}}` placeholder.
- Rendered template must declare `folds` workflow parameter (so
`argo submit -p folds=K` overrides the default).
- Rendered binary command must NOT contain any per-fold flag — this
catches the failure mode that broke the first L40S deploy.
- Backward-compat: `--multi-seed 1 --folds 1` preserves the existing
single-template path (no DAG matrix tasks emitted).
6. docs/dqn-wire-up-audit.md — adds 1 Wired row documenting the pivot,
the new `--seed`/`mix_seed` plumbing, all 6 RNG call sites, and the
end-to-end seed-variation verification result.
Validation:
cargo check --workspace clean at 11 warnings (workspace baseline preserved).
cargo build --release --example train_baseline_rl succeeds; --help shows
the new --seed flag with documented default 42.
Seed-variation end-to-end test on RTX 3050 Ti (1 fold × 2 epochs each):
--seed 42 → F0 best Sharpe = -9.7831, best_val_metric = 1.957244,
epoch-2 train Sharpe = -16.12, val_Sharpe = +1.11.
--seed 999 → F0 best Sharpe = +92.9341, best_val_metric = 2.161012,
epoch-2 train Sharpe = +92.93, val_Sharpe = -0.25.
Different best Sharpe / best_val_metric / epoch-2 train + val Sharpe
across seeds proves the seed actually propagates through the RNG init
paths and is not just accepted-and-ignored. The seed=42 numbers match
the prompt's "deterministic baseline" expectation (F0 = -9.7831 was
bit-identical pre-pivot because no global-seed plumbing existed).
./scripts/argo-train.sh dqn --multi-seed 5 --folds 6 --dry-run produces
exactly 5 WorkflowTask markers (train-s0..train-s4), each with
`--max-folds {{workflow.parameters.folds}}` in the binary invocation.
All 3 harness tests PASS:
- test_multi_seed_harness.sh: 5 PASS lines, exit 0.
- test_nsys_harness.sh: 4 PASS lines + ALL PASS, exit 0.
- test_tier_checks.sh: PASS overall (good-fixture passes, bad-fixture
surfaces expected check rejections), exit 0.
Backward compat: existing single-job `argo-train.sh` callers (no
`--multi-seed`, no `--folds`) route to the original `train-template.yaml`
unchanged. `--seed 42` is a no-op offset for the SplitMix64 mix at the call
sites — the trajectory shifts only when the user passes `--seed` explicitly,
matching the prompt's "default 42 (historic implicit value)" requirement.
L40S pool: argo-train.sh defaults `--gpu-pool ci-training-h100`; user passes
`--gpu-pool ci-training-l40s` at deploy time. No script default change
(per constraint 5).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- argo-train.sh: --profile flag forces multi-seed render path so the
nsys wrapper + foxhunt-training-artifacts upload step are visible in
--dry-run YAML without cluster contact (test surface).
- train-multi-seed-template.yaml: new `profile` parameter (default
"false") gates the per-(seed, fold) `nsys profile
--capture-range=cudaProfilerApi` wrapper and the `mc cp` upload to
foxhunt-training-artifacts/profiles/<sha>/. mc binary fetched
on-demand (ci-builder image lacks it). MinIO creds optional —
upload warn-skips if absent.
- Dockerfile.foxhunt-training-runtime: install nsight-systems-cli
unpinned (pinning the stale 2024.4.1.61-1 from earlier plans
breaks builds when apt index advances).
- minio.yaml: add foxhunt-training-artifacts bucket to minio-init.
- compare-nsys-profiles.py: V0 regression detector — compares
cuda_gpu_kern_sum total_ns / epoch_count between two profiles;
exits 1 on >20% slowdown. NVTX per-epoch ranges deferred to T5.
- tests/test_nsys_harness.sh: dry-run grep test — verifies both
required strings appear when --profile is set, and that the
default (no --profile) path keeps profile=false in the rendered
template.
- dqn-wire-up-audit.md: Plan 5 Task 3 row added documenting the
harness + the baseline-capture deferral to T5.
Backward compat: test_multi_seed_harness.sh from P5T1 still PASS.
Adds the orchestration surface for Plan 5 Task 1 — Multi-Seed × Multi-Fold
Validation Harness:
- scripts/argo-train.sh: new --multi-seed N, --folds K, --tag T, --dry-run
flags. Default --multi-seed 1 --folds 1 routes to the existing
train-template.yaml (backward compat — existing callers unchanged). When
N>1 or K>1 the script renders train-multi-seed-template.yaml with an
inline-generated (seed, fold) matrix and either prints the YAML
(--dry-run) or applies + submits it.
- infra/k8s/argo/train-multi-seed-template.yaml: new WorkflowTemplate with
entrypoint multi-seed-matrix → ensure-binary, gpu-warmup, ensure-fxcache,
then N*K parallel train-single instances. Each train-single receives
seed/fold via inputs.parameters and forwards them to the training binary
via --seed/--fold CLI args + SEED/FOLD env vars. The dag.tasks placeholder
`# __MATRIX_TASKS__` is substituted programmatically by argo-train.sh
(awk) — no hand-written 30-task matrix.
- scripts/tests/test_multi_seed_harness.sh: dry-run regression test.
Asserts --multi-seed 3 --folds 2 emits 6 WorkflowTask markers AND
--multi-seed 1 --folds 1 emits zero (single-template path preserved).
Validation:
- argo lint --offline passes on both the source template and the rendered
3x2 / 5x6 outputs.
- test_multi_seed_harness.sh passes locally.
- Single-job dry-run still produces the unchanged train-template YAML.
Note: plan Step 0.1 pre-plan check expects ISV_TOTAL_DIM=72 and seven
ATTN_*_FOCUS_EMA_INDEX slots — both stale (Plan 4 landed
ISV_TOTAL_DIM=117 and the VSN_MAG_EMA / VSN_DIR_EMA / MAMBA2_RETENTION_EMA
slots instead). Plan 4 validation doc never landed (T8 deferred → Plan 5
T5). T1 is pure infrastructure that builds the harness consumed by T5,
so the stale pre-plan expectations do not block this commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
