fcf76701f4
The first L40S deploy attempt (workflow `train-multi-seed-z2llf`, terminated)
failed at startup with `error: unexpected argument '--fold' found` on every
job: `train_baseline_rl` is a multi-fold walk-forward executor that accepts
`--max-folds K`, NOT `--fold N`. The original P5T1 harness assumed the
opposite and fanned out N seeds × K folds = N*K jobs, each invoking the
binary with `--seed N --fold K`.
User chose Path B: pivot to one job per seed (each runs all K folds via the
existing `--max-folds` mechanism). Per-job runtime is K× longer, but fanout
drops from N*K=30 → N=5 (matches L40S pool capacity better) and the binary
contract becomes the one the binary actually has.
4 surface changes:
1. crates/ml/examples/train_baseline_rl.rs — add `--seed N` CLI arg
(default 42 — historic implicit value). Sets `FOXHUNT_SEED` env var at
startup BEFORE any CUDA module spins up. Logs the seed value at the
training start banner.
2. crates/ml/src/cuda_pipeline/mod.rs — add `global_seed()` (reads
`FOXHUNT_SEED`, default 42) + `mix_seed(base)` (SplitMix64 avalanche
so adjacent global seeds produce uncorrelated module seeds). Six call
sites updated to mix the global seed into their previously-hardcoded
constants:
- trainer/action.rs: GpuActionSelector seed (0xDEAD_BEEF_CAFE) + the
epsilon-greedy fallback StdRng (0xAC7_DEF0).
- cuda_pipeline/gpu_iqn_head.rs: IQN Xavier-init RNG (0x1CA_1234).
- cuda_pipeline/gpu_iql_trainer.rs: V(s) Xavier-init RNG (0x1C1_9ABC).
- cuda_pipeline/gpu_her.rs: random-donor RNG (0x4E4_5678).
- cuda_pipeline/gpu_ppo_collector.rs: rng_seeds Vec for PPO
experience-collector init + reset (0xAA0_5EED).
- trainer/training_loop.rs: per-epoch regime_dropout_seed.
3. infra/k8s/argo/train-multi-seed-template.yaml — drop `fold` parameter
from `train-single` template; binary invoked as `--seed "$SEED"
--max-folds {{workflow.parameters.folds}}` so the walk-forward sweep
happens inside the single training process. Drop `FOLD` env var. Update
the nsys-rep upload filename to drop the fold suffix. Update banners /
doc comments to reflect "one-job-per-seed" semantics.
4. scripts/argo-train.sh — matrix generator drops the inner fold loop.
Each emitted task carries only `seed=${s}` and depends on the same
ensure-fxcache + gpu-warmup. The dry-run synthetic marker switches from
`seed=${s} fold=${f}` to `seed=${s} max_folds=${FOLDS}` so test harnesses
count the new shape correctly.
5. scripts/tests/test_multi_seed_harness.sh — assertions updated:
- `--multi-seed 3 --folds 2` produces 3 tasks (was 6).
- Rendered binary command must include `--max-folds
{{workflow.parameters.folds}}` placeholder.
- Rendered template must declare `folds` workflow parameter (so
`argo submit -p folds=K` overrides the default).
- Rendered binary command must NOT contain any per-fold flag — this
catches the failure mode that broke the first L40S deploy.
- Backward-compat: `--multi-seed 1 --folds 1` preserves the existing
single-template path (no DAG matrix tasks emitted).
6. docs/dqn-wire-up-audit.md — adds 1 Wired row documenting the pivot,
the new `--seed`/`mix_seed` plumbing, all 6 RNG call sites, and the
end-to-end seed-variation verification result.
Validation:
cargo check --workspace clean at 11 warnings (workspace baseline preserved).
cargo build --release --example train_baseline_rl succeeds; --help shows
the new --seed flag with documented default 42.
Seed-variation end-to-end test on RTX 3050 Ti (1 fold × 2 epochs each):
--seed 42 → F0 best Sharpe = -9.7831, best_val_metric = 1.957244,
epoch-2 train Sharpe = -16.12, val_Sharpe = +1.11.
--seed 999 → F0 best Sharpe = +92.9341, best_val_metric = 2.161012,
epoch-2 train Sharpe = +92.93, val_Sharpe = -0.25.
Different best Sharpe / best_val_metric / epoch-2 train + val Sharpe
across seeds proves the seed actually propagates through the RNG init
paths and is not just accepted-and-ignored. The seed=42 numbers match
the prompt's "deterministic baseline" expectation (F0 = -9.7831 was
bit-identical pre-pivot because no global-seed plumbing existed).
./scripts/argo-train.sh dqn --multi-seed 5 --folds 6 --dry-run produces
exactly 5 WorkflowTask markers (train-s0..train-s4), each with
`--max-folds {{workflow.parameters.folds}}` in the binary invocation.
All 3 harness tests PASS:
- test_multi_seed_harness.sh: 5 PASS lines, exit 0.
- test_nsys_harness.sh: 4 PASS lines + ALL PASS, exit 0.
- test_tier_checks.sh: PASS overall (good-fixture passes, bad-fixture
surfaces expected check rejections), exit 0.
Backward compat: existing single-job `argo-train.sh` callers (no
`--multi-seed`, no `--folds`) route to the original `train-template.yaml`
unchanged. `--seed 42` is a no-op offset for the SplitMix64 mix at the call
sites — the trajectory shifts only when the user passes `--seed` explicitly,
matching the prompt's "default 42 (historic implicit value)" requirement.
L40S pool: argo-train.sh defaults `--gpu-pool ci-training-h100`; user passes
`--gpu-pool ci-training-l40s` at deploy time. No script default change
(per constraint 5).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
74 lines
3.7 KiB
Bash
Executable File
74 lines
3.7 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Test: argo-train.sh --multi-seed N --folds K --dry-run emits one
|
|
# WorkflowTask per seed (N total). Each task internally runs all K folds via
|
|
# the binary's `--max-folds K` argument — train_baseline_rl is a multi-fold
|
|
# walk-forward executor, NOT a single-fold one.
|
|
#
|
|
# This is the validation surface for the Plan 5 Task 5 Phase B multi-seed
|
|
# Argo DAG (was N*K (seed, fold) jobs in Task 1A; pivoted to N seed-only jobs
|
|
# after the first deploy failed at startup with `--fold` not recognized).
|
|
# Verifies the matrix + binary command is generated correctly without
|
|
# submitting to a live cluster.
|
|
set -euo pipefail
|
|
|
|
# Resolve repo root from this script's location so the test runs from any cwd.
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
|
|
cd "${REPO_ROOT}"
|
|
|
|
# Dry-run: --multi-seed 3 --folds 2 should emit 3 training jobs (one per seed),
|
|
# each invoking the binary with `--max-folds 2` to run both folds internally.
|
|
output=$(./scripts/argo-train.sh dqn --multi-seed 3 --folds 2 --dry-run 2>&1)
|
|
actual_count=$(echo "$output" | grep -c "kind: WorkflowTask" || true)
|
|
expected=3
|
|
if [[ "$actual_count" -ne "$expected" ]]; then
|
|
echo "FAIL: expected $expected jobs (3 seeds, fold sweep inside binary), got $actual_count"
|
|
echo "--- output ---"
|
|
echo "$output"
|
|
exit 1
|
|
fi
|
|
echo "PASS: multi-seed harness produces $expected per-seed jobs for --multi-seed 3 --folds 2"
|
|
|
|
# The rendered binary command must include `--max-folds {{workflow.parameters.folds}}`
|
|
# (drives the walk-forward sweep inside the single training process) and must
|
|
# NOT include any per-fold flag — `train_baseline_rl` rejects `--fold`, which
|
|
# is what broke the first L40S deploy attempt (workflow train-multi-seed-z2llf,
|
|
# every job exited at startup with `error: unexpected argument '--fold' found`).
|
|
# The Argo `{{workflow.parameters.folds}}` placeholder is resolved at
|
|
# workflow submission time (argo-train.sh passes `-p folds=$FOLDS` to argo
|
|
# submit), not at template render time, so the dry-run shows the placeholder.
|
|
if ! echo "$output" | grep -qE -- "--max-folds[[:space:]]+\{\{workflow\.parameters\.folds\}\}"; then
|
|
echo "FAIL: rendered binary command missing '--max-folds {{workflow.parameters.folds}}'"
|
|
echo "--- output (rendered template, first 300 lines) ---"
|
|
echo "$output" | head -300
|
|
exit 1
|
|
fi
|
|
echo "PASS: rendered binary invocation includes --max-folds placeholder"
|
|
|
|
# Folds parameter must be declared at the workflow level so `argo submit -p
|
|
# folds=K` (which argo-train.sh emits) can override the default at submit time.
|
|
if ! echo "$output" | grep -qE "^[[:space:]]*-[[:space:]]*name:[[:space:]]*folds[[:space:]]*$"; then
|
|
echo "FAIL: rendered template does not declare 'folds' workflow parameter"
|
|
exit 1
|
|
fi
|
|
echo "PASS: rendered template declares folds workflow parameter"
|
|
|
|
if echo "$output" | grep -qE -- '--fold[[:space:]]+"?[0-9{]'; then
|
|
echo "FAIL: rendered binary command still contains a per-fold flag (--fold N)"
|
|
echo " train_baseline_rl rejects --fold; only --max-folds is supported."
|
|
echo "--- offending lines ---"
|
|
echo "$output" | grep -E -- '--fold[[:space:]]+"?[0-9{]'
|
|
exit 1
|
|
fi
|
|
echo "PASS: rendered binary invocation has no per-fold flag (Path B compliance)"
|
|
|
|
# Backward-compat: --multi-seed 1 --folds 1 must NOT emit WorkflowTask lines
|
|
# (single-job path uses the existing template, no DAG matrix).
|
|
output_single=$(./scripts/argo-train.sh dqn --multi-seed 1 --folds 1 --dry-run 2>&1)
|
|
single_tasks=$(echo "$output_single" | grep -c "kind: WorkflowTask" || true)
|
|
if [[ "$single_tasks" -ne 0 ]]; then
|
|
echo "FAIL: single-job dry-run should not emit multi-seed DAG tasks (got $single_tasks)"
|
|
exit 1
|
|
fi
|
|
echo "PASS: single-job dry-run preserves existing single-template path"
|