foxhunt

gitadmin/foxhunt

Fork 0

Commit Graph

Author	SHA1	Message	Date
jgrusewski	fcf76701f4	plan5(task5-B): pivot multi-seed Argo from N×K (seed,fold) to N seed-only fanout The first L40S deploy attempt (workflow `train-multi-seed-z2llf`, terminated) failed at startup with `error: unexpected argument '--fold' found` on every job: `train_baseline_rl` is a multi-fold walk-forward executor that accepts `--max-folds K`, NOT `--fold N`. The original P5T1 harness assumed the opposite and fanned out N seeds × K folds = NK jobs, each invoking the binary with `--seed N --fold K`. User chose Path B: pivot to one job per seed (each runs all K folds via the existing `--max-folds` mechanism). Per-job runtime is K× longer, but fanout drops from NK=30 → N=5 (matches L40S pool capacity better) and the binary contract becomes the one the binary actually has. 4 surface changes: 1. crates/ml/examples/train_baseline_rl.rs — add `--seed N` CLI arg (default 42 — historic implicit value). Sets `FOXHUNT_SEED` env var at startup BEFORE any CUDA module spins up. Logs the seed value at the training start banner. 2. crates/ml/src/cuda_pipeline/mod.rs — add `global_seed()` (reads `FOXHUNT_SEED`, default 42) + `mix_seed(base)` (SplitMix64 avalanche so adjacent global seeds produce uncorrelated module seeds). Six call sites updated to mix the global seed into their previously-hardcoded constants: - trainer/action.rs: GpuActionSelector seed (0xDEAD_BEEF_CAFE) + the epsilon-greedy fallback StdRng (0xAC7_DEF0). - cuda_pipeline/gpu_iqn_head.rs: IQN Xavier-init RNG (0x1CA_1234). - cuda_pipeline/gpu_iql_trainer.rs: V(s) Xavier-init RNG (0x1C1_9ABC). - cuda_pipeline/gpu_her.rs: random-donor RNG (0x4E4_5678). - cuda_pipeline/gpu_ppo_collector.rs: rng_seeds Vec for PPO experience-collector init + reset (0xAA0_5EED). - trainer/training_loop.rs: per-epoch regime_dropout_seed. 3. infra/k8s/argo/train-multi-seed-template.yaml — drop `fold` parameter from `train-single` template; binary invoked as `--seed "$SEED" --max-folds {{workflow.parameters.folds}}` so the walk-forward sweep happens inside the single training process. Drop `FOLD` env var. Update the nsys-rep upload filename to drop the fold suffix. Update banners / doc comments to reflect "one-job-per-seed" semantics. 4. scripts/argo-train.sh — matrix generator drops the inner fold loop. Each emitted task carries only `seed=${s}` and depends on the same ensure-fxcache + gpu-warmup. The dry-run synthetic marker switches from `seed=${s} fold=${f}` to `seed=${s} max_folds=${FOLDS}` so test harnesses count the new shape correctly. 5. scripts/tests/test_multi_seed_harness.sh — assertions updated: - `--multi-seed 3 --folds 2` produces 3 tasks (was 6). - Rendered binary command must include `--max-folds {{workflow.parameters.folds}}` placeholder. - Rendered template must declare `folds` workflow parameter (so `argo submit -p folds=K` overrides the default). - Rendered binary command must NOT contain any per-fold flag — this catches the failure mode that broke the first L40S deploy. - Backward-compat: `--multi-seed 1 --folds 1` preserves the existing single-template path (no DAG matrix tasks emitted). 6. docs/dqn-wire-up-audit.md — adds 1 Wired row documenting the pivot, the new `--seed`/`mix_seed` plumbing, all 6 RNG call sites, and the end-to-end seed-variation verification result. Validation: cargo check --workspace clean at 11 warnings (workspace baseline preserved). cargo build --release --example train_baseline_rl succeeds; --help shows the new --seed flag with documented default 42. Seed-variation end-to-end test on RTX 3050 Ti (1 fold × 2 epochs each): --seed 42 → F0 best Sharpe = -9.7831, best_val_metric = 1.957244, epoch-2 train Sharpe = -16.12, val_Sharpe = +1.11. --seed 999 → F0 best Sharpe = +92.9341, best_val_metric = 2.161012, epoch-2 train Sharpe = +92.93, val_Sharpe = -0.25. Different best Sharpe / best_val_metric / epoch-2 train + val Sharpe across seeds proves the seed actually propagates through the RNG init paths and is not just accepted-and-ignored. The seed=42 numbers match the prompt's "deterministic baseline" expectation (F0 = -9.7831 was bit-identical pre-pivot because no global-seed plumbing existed). ./scripts/argo-train.sh dqn --multi-seed 5 --folds 6 --dry-run produces exactly 5 WorkflowTask markers (train-s0..train-s4), each with `--max-folds {{workflow.parameters.folds}}` in the binary invocation. All 3 harness tests PASS: - test_multi_seed_harness.sh: 5 PASS lines, exit 0. - test_nsys_harness.sh: 4 PASS lines + ALL PASS, exit 0. - test_tier_checks.sh: PASS overall (good-fixture passes, bad-fixture surfaces expected check rejections), exit 0. Backward compat: existing single-job `argo-train.sh` callers (no `--multi-seed`, no `--folds`) route to the original `train-template.yaml` unchanged. `--seed 42` is a no-op offset for the SplitMix64 mix at the call sites — the trajectory shifts only when the user passes `--seed` explicitly, matching the prompt's "default 42 (historic implicit value)" requirement. L40S pool: argo-train.sh defaults `--gpu-pool ci-training-h100`; user passes `--gpu-pool ci-training-l40s` at deploy time. No script default change (per constraint 5). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-26 14:11:32 +02:00
jgrusewski	2606506cd8	plan5(task3): A.4.1 nsys profile harness with regression-comparison script - argo-train.sh: --profile flag forces multi-seed render path so the nsys wrapper + foxhunt-training-artifacts upload step are visible in --dry-run YAML without cluster contact (test surface). - train-multi-seed-template.yaml: new `profile` parameter (default "false") gates the per-(seed, fold) `nsys profile --capture-range=cudaProfilerApi` wrapper and the `mc cp` upload to foxhunt-training-artifacts/profiles/<sha>/. mc binary fetched on-demand (ci-builder image lacks it). MinIO creds optional — upload warn-skips if absent. - Dockerfile.foxhunt-training-runtime: install nsight-systems-cli unpinned (pinning the stale 2024.4.1.61-1 from earlier plans breaks builds when apt index advances). - minio.yaml: add foxhunt-training-artifacts bucket to minio-init. - compare-nsys-profiles.py: V0 regression detector — compares cuda_gpu_kern_sum total_ns / epoch_count between two profiles; exits 1 on >20% slowdown. NVTX per-epoch ranges deferred to T5. - tests/test_nsys_harness.sh: dry-run grep test — verifies both required strings appear when --profile is set, and that the default (no --profile) path keeps profile=false in the rendered template. - dqn-wire-up-audit.md: Plan 5 Task 3 row added documenting the harness + the baseline-capture deferral to T5. Backward compat: test_multi_seed_harness.sh from P5T1 still PASS.	2026-04-26 12:25:35 +02:00
jgrusewski	c6634254e4	plan5(task1A): multi-seed × multi-fold Argo DAG template Adds the orchestration surface for Plan 5 Task 1 — Multi-Seed × Multi-Fold Validation Harness: - scripts/argo-train.sh: new --multi-seed N, --folds K, --tag T, --dry-run flags. Default --multi-seed 1 --folds 1 routes to the existing train-template.yaml (backward compat — existing callers unchanged). When N>1 or K>1 the script renders train-multi-seed-template.yaml with an inline-generated (seed, fold) matrix and either prints the YAML (--dry-run) or applies + submits it. - infra/k8s/argo/train-multi-seed-template.yaml: new WorkflowTemplate with entrypoint multi-seed-matrix → ensure-binary, gpu-warmup, ensure-fxcache, then NK parallel train-single instances. Each train-single receives seed/fold via inputs.parameters and forwards them to the training binary via --seed/--fold CLI args + SEED/FOLD env vars. The dag.tasks placeholder `# __MATRIX_TASKS__` is substituted programmatically by argo-train.sh (awk) — no hand-written 30-task matrix. - scripts/tests/test_multi_seed_harness.sh: dry-run regression test. Asserts --multi-seed 3 --folds 2 emits 6 WorkflowTask markers AND --multi-seed 1 --folds 1 emits zero (single-template path preserved). Validation: - argo lint --offline passes on both the source template and the rendered 3x2 / 5x6 outputs. - test_multi_seed_harness.sh passes locally. - Single-job dry-run still produces the unchanged train-template YAML. Note: plan Step 0.1 pre-plan check expects ISV_TOTAL_DIM=72 and seven ATTN__FOCUS_EMA_INDEX slots — both stale (Plan 4 landed ISV_TOTAL_DIM=117 and the VSN_MAG_EMA / VSN_DIR_EMA / MAMBA2_RETENTION_EMA slots instead). Plan 4 validation doc never landed (T8 deferred → Plan 5 T5). T1 is pure infrastructure that builds the harness consumed by T5, so the stale pre-plan expectations do not block this commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-26 10:55:12 +02:00

Author

SHA1

Message

Date

jgrusewski

fcf76701f4

plan5(task5-B): pivot multi-seed Argo from N×K (seed,fold) to N seed-only fanout

The first L40S deploy attempt (workflow `train-multi-seed-z2llf`, terminated)
failed at startup with `error: unexpected argument '--fold' found` on every
job: `train_baseline_rl` is a multi-fold walk-forward executor that accepts
`--max-folds K`, NOT `--fold N`. The original P5T1 harness assumed the
opposite and fanned out N seeds × K folds = N*K jobs, each invoking the
binary with `--seed N --fold K`.

User chose Path B: pivot to one job per seed (each runs all K folds via the
existing `--max-folds` mechanism). Per-job runtime is K× longer, but fanout
drops from N*K=30 → N=5 (matches L40S pool capacity better) and the binary
contract becomes the one the binary actually has.

4 surface changes:

1. crates/ml/examples/train_baseline_rl.rs — add `--seed N` CLI arg
   (default 42 — historic implicit value). Sets `FOXHUNT_SEED` env var at
   startup BEFORE any CUDA module spins up. Logs the seed value at the
   training start banner.

2. crates/ml/src/cuda_pipeline/mod.rs — add `global_seed()` (reads
   `FOXHUNT_SEED`, default 42) + `mix_seed(base)` (SplitMix64 avalanche
   so adjacent global seeds produce uncorrelated module seeds). Six call
   sites updated to mix the global seed into their previously-hardcoded
   constants:
   - trainer/action.rs: GpuActionSelector seed (0xDEAD_BEEF_CAFE) + the
     epsilon-greedy fallback StdRng (0xAC7_DEF0).
   - cuda_pipeline/gpu_iqn_head.rs: IQN Xavier-init RNG (0x1CA_1234).
   - cuda_pipeline/gpu_iql_trainer.rs: V(s) Xavier-init RNG (0x1C1_9ABC).
   - cuda_pipeline/gpu_her.rs: random-donor RNG (0x4E4_5678).
   - cuda_pipeline/gpu_ppo_collector.rs: rng_seeds Vec for PPO
     experience-collector init + reset (0xAA0_5EED).
   - trainer/training_loop.rs: per-epoch regime_dropout_seed.

3. infra/k8s/argo/train-multi-seed-template.yaml — drop `fold` parameter
   from `train-single` template; binary invoked as `--seed "$SEED"
   --max-folds {{workflow.parameters.folds}}` so the walk-forward sweep
   happens inside the single training process. Drop `FOLD` env var. Update
   the nsys-rep upload filename to drop the fold suffix. Update banners /
   doc comments to reflect "one-job-per-seed" semantics.

4. scripts/argo-train.sh — matrix generator drops the inner fold loop.
   Each emitted task carries only `seed=${s}` and depends on the same
   ensure-fxcache + gpu-warmup. The dry-run synthetic marker switches from
   `seed=${s} fold=${f}` to `seed=${s} max_folds=${FOLDS}` so test harnesses
   count the new shape correctly.

5. scripts/tests/test_multi_seed_harness.sh — assertions updated:
   - `--multi-seed 3 --folds 2` produces 3 tasks (was 6).
   - Rendered binary command must include `--max-folds
     {{workflow.parameters.folds}}` placeholder.
   - Rendered template must declare `folds` workflow parameter (so
     `argo submit -p folds=K` overrides the default).
   - Rendered binary command must NOT contain any per-fold flag — this
     catches the failure mode that broke the first L40S deploy.
   - Backward-compat: `--multi-seed 1 --folds 1` preserves the existing
     single-template path (no DAG matrix tasks emitted).

6. docs/dqn-wire-up-audit.md — adds 1 Wired row documenting the pivot,
   the new `--seed`/`mix_seed` plumbing, all 6 RNG call sites, and the
   end-to-end seed-variation verification result.

Validation:

  cargo check --workspace clean at 11 warnings (workspace baseline preserved).

  cargo build --release --example train_baseline_rl succeeds; --help shows
  the new --seed flag with documented default 42.

  Seed-variation end-to-end test on RTX 3050 Ti (1 fold × 2 epochs each):
    --seed 42  → F0 best Sharpe = -9.7831, best_val_metric = 1.957244,
                 epoch-2 train Sharpe = -16.12, val_Sharpe = +1.11.
    --seed 999 → F0 best Sharpe = +92.9341, best_val_metric = 2.161012,
                 epoch-2 train Sharpe = +92.93, val_Sharpe = -0.25.
  Different best Sharpe / best_val_metric / epoch-2 train + val Sharpe
  across seeds proves the seed actually propagates through the RNG init
  paths and is not just accepted-and-ignored. The seed=42 numbers match
  the prompt's "deterministic baseline" expectation (F0 = -9.7831 was
  bit-identical pre-pivot because no global-seed plumbing existed).

  ./scripts/argo-train.sh dqn --multi-seed 5 --folds 6 --dry-run produces
  exactly 5 WorkflowTask markers (train-s0..train-s4), each with
  `--max-folds {{workflow.parameters.folds}}` in the binary invocation.

  All 3 harness tests PASS:
    - test_multi_seed_harness.sh: 5 PASS lines, exit 0.
    - test_nsys_harness.sh: 4 PASS lines + ALL PASS, exit 0.
    - test_tier_checks.sh: PASS overall (good-fixture passes, bad-fixture
      surfaces expected check rejections), exit 0.

Backward compat: existing single-job `argo-train.sh` callers (no
`--multi-seed`, no `--folds`) route to the original `train-template.yaml`
unchanged. `--seed 42` is a no-op offset for the SplitMix64 mix at the call
sites — the trajectory shifts only when the user passes `--seed` explicitly,
matching the prompt's "default 42 (historic implicit value)" requirement.

L40S pool: argo-train.sh defaults `--gpu-pool ci-training-h100`; user passes
`--gpu-pool ci-training-l40s` at deploy time. No script default change
(per constraint 5).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-26 14:11:32 +02:00

jgrusewski

2606506cd8

plan5(task3): A.4.1 nsys profile harness with regression-comparison script

- argo-train.sh: --profile flag forces multi-seed render path so the
  nsys wrapper + foxhunt-training-artifacts upload step are visible in
  --dry-run YAML without cluster contact (test surface).
- train-multi-seed-template.yaml: new `profile` parameter (default
  "false") gates the per-(seed, fold) `nsys profile
  --capture-range=cudaProfilerApi` wrapper and the `mc cp` upload to
  foxhunt-training-artifacts/profiles/<sha>/. mc binary fetched
  on-demand (ci-builder image lacks it). MinIO creds optional —
  upload warn-skips if absent.
- Dockerfile.foxhunt-training-runtime: install nsight-systems-cli
  unpinned (pinning the stale 2024.4.1.61-1 from earlier plans
  breaks builds when apt index advances).
- minio.yaml: add foxhunt-training-artifacts bucket to minio-init.
- compare-nsys-profiles.py: V0 regression detector — compares
  cuda_gpu_kern_sum total_ns / epoch_count between two profiles;
  exits 1 on >20% slowdown. NVTX per-epoch ranges deferred to T5.
- tests/test_nsys_harness.sh: dry-run grep test — verifies both
  required strings appear when --profile is set, and that the
  default (no --profile) path keeps profile=false in the rendered
  template.
- dqn-wire-up-audit.md: Plan 5 Task 3 row added documenting the
  harness + the baseline-capture deferral to T5.

Backward compat: test_multi_seed_harness.sh from P5T1 still PASS.

2026-04-26 12:25:35 +02:00

jgrusewski

c6634254e4

plan5(task1A): multi-seed × multi-fold Argo DAG template

Adds the orchestration surface for Plan 5 Task 1 — Multi-Seed × Multi-Fold
Validation Harness:

- scripts/argo-train.sh: new --multi-seed N, --folds K, --tag T, --dry-run
  flags. Default --multi-seed 1 --folds 1 routes to the existing
  train-template.yaml (backward compat — existing callers unchanged). When
  N>1 or K>1 the script renders train-multi-seed-template.yaml with an
  inline-generated (seed, fold) matrix and either prints the YAML
  (--dry-run) or applies + submits it.

- infra/k8s/argo/train-multi-seed-template.yaml: new WorkflowTemplate with
  entrypoint multi-seed-matrix → ensure-binary, gpu-warmup, ensure-fxcache,
  then N*K parallel train-single instances. Each train-single receives
  seed/fold via inputs.parameters and forwards them to the training binary
  via --seed/--fold CLI args + SEED/FOLD env vars. The dag.tasks placeholder
  `# __MATRIX_TASKS__` is substituted programmatically by argo-train.sh
  (awk) — no hand-written 30-task matrix.

- scripts/tests/test_multi_seed_harness.sh: dry-run regression test.
  Asserts --multi-seed 3 --folds 2 emits 6 WorkflowTask markers AND
  --multi-seed 1 --folds 1 emits zero (single-template path preserved).

Validation:
- argo lint --offline passes on both the source template and the rendered
  3x2 / 5x6 outputs.
- test_multi_seed_harness.sh passes locally.
- Single-job dry-run still produces the unchanged train-template YAML.

Note: plan Step 0.1 pre-plan check expects ISV_TOTAL_DIM=72 and seven
ATTN_*_FOCUS_EMA_INDEX slots — both stale (Plan 4 landed
ISV_TOTAL_DIM=117 and the VSN_MAG_EMA / VSN_DIR_EMA / MAMBA2_RETENTION_EMA
slots instead). Plan 4 validation doc never landed (T8 deferred → Plan 5
T5). T1 is pure infrastructure that builds the harness consumed by T5,
so the stale pre-plan expectations do not block this commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-26 10:55:12 +02:00

3 Commits