plan5(task5-B): pivot multi-seed Argo from N×K (seed,fold) to N seed-only fanout

The first L40S deploy attempt (workflow `train-multi-seed-z2llf`, terminated)
failed at startup with `error: unexpected argument '--fold' found` on every
job: `train_baseline_rl` is a multi-fold walk-forward executor that accepts
`--max-folds K`, NOT `--fold N`. The original P5T1 harness assumed the
opposite and fanned out N seeds × K folds = N*K jobs, each invoking the
binary with `--seed N --fold K`.

User chose Path B: pivot to one job per seed (each runs all K folds via the
existing `--max-folds` mechanism). Per-job runtime is K× longer, but fanout
drops from N*K=30 → N=5 (matches L40S pool capacity better) and the binary
contract becomes the one the binary actually has.

4 surface changes:

1. crates/ml/examples/train_baseline_rl.rs — add `--seed N` CLI arg
   (default 42 — historic implicit value). Sets `FOXHUNT_SEED` env var at
   startup BEFORE any CUDA module spins up. Logs the seed value at the
   training start banner.

2. crates/ml/src/cuda_pipeline/mod.rs — add `global_seed()` (reads
   `FOXHUNT_SEED`, default 42) + `mix_seed(base)` (SplitMix64 avalanche
   so adjacent global seeds produce uncorrelated module seeds). Six call
   sites updated to mix the global seed into their previously-hardcoded
   constants:
   - trainer/action.rs: GpuActionSelector seed (0xDEAD_BEEF_CAFE) + the
     epsilon-greedy fallback StdRng (0xAC7_DEF0).
   - cuda_pipeline/gpu_iqn_head.rs: IQN Xavier-init RNG (0x1CA_1234).
   - cuda_pipeline/gpu_iql_trainer.rs: V(s) Xavier-init RNG (0x1C1_9ABC).
   - cuda_pipeline/gpu_her.rs: random-donor RNG (0x4E4_5678).
   - cuda_pipeline/gpu_ppo_collector.rs: rng_seeds Vec for PPO
     experience-collector init + reset (0xAA0_5EED).
   - trainer/training_loop.rs: per-epoch regime_dropout_seed.

3. infra/k8s/argo/train-multi-seed-template.yaml — drop `fold` parameter
   from `train-single` template; binary invoked as `--seed "$SEED"
   --max-folds {{workflow.parameters.folds}}` so the walk-forward sweep
   happens inside the single training process. Drop `FOLD` env var. Update
   the nsys-rep upload filename to drop the fold suffix. Update banners /
   doc comments to reflect "one-job-per-seed" semantics.

4. scripts/argo-train.sh — matrix generator drops the inner fold loop.
   Each emitted task carries only `seed=${s}` and depends on the same
   ensure-fxcache + gpu-warmup. The dry-run synthetic marker switches from
   `seed=${s} fold=${f}` to `seed=${s} max_folds=${FOLDS}` so test harnesses
   count the new shape correctly.

5. scripts/tests/test_multi_seed_harness.sh — assertions updated:
   - `--multi-seed 3 --folds 2` produces 3 tasks (was 6).
   - Rendered binary command must include `--max-folds
     {{workflow.parameters.folds}}` placeholder.
   - Rendered template must declare `folds` workflow parameter (so
     `argo submit -p folds=K` overrides the default).
   - Rendered binary command must NOT contain any per-fold flag — this
     catches the failure mode that broke the first L40S deploy.
   - Backward-compat: `--multi-seed 1 --folds 1` preserves the existing
     single-template path (no DAG matrix tasks emitted).

6. docs/dqn-wire-up-audit.md — adds 1 Wired row documenting the pivot,
   the new `--seed`/`mix_seed` plumbing, all 6 RNG call sites, and the
   end-to-end seed-variation verification result.

Validation:

  cargo check --workspace clean at 11 warnings (workspace baseline preserved).

  cargo build --release --example train_baseline_rl succeeds; --help shows
  the new --seed flag with documented default 42.

  Seed-variation end-to-end test on RTX 3050 Ti (1 fold × 2 epochs each):
    --seed 42  → F0 best Sharpe = -9.7831, best_val_metric = 1.957244,
                 epoch-2 train Sharpe = -16.12, val_Sharpe = +1.11.
    --seed 999 → F0 best Sharpe = +92.9341, best_val_metric = 2.161012,
                 epoch-2 train Sharpe = +92.93, val_Sharpe = -0.25.
  Different best Sharpe / best_val_metric / epoch-2 train + val Sharpe
  across seeds proves the seed actually propagates through the RNG init
  paths and is not just accepted-and-ignored. The seed=42 numbers match
  the prompt's "deterministic baseline" expectation (F0 = -9.7831 was
  bit-identical pre-pivot because no global-seed plumbing existed).

  ./scripts/argo-train.sh dqn --multi-seed 5 --folds 6 --dry-run produces
  exactly 5 WorkflowTask markers (train-s0..train-s4), each with
  `--max-folds {{workflow.parameters.folds}}` in the binary invocation.

  All 3 harness tests PASS:
    - test_multi_seed_harness.sh: 5 PASS lines, exit 0.
    - test_nsys_harness.sh: 4 PASS lines + ALL PASS, exit 0.
    - test_tier_checks.sh: PASS overall (good-fixture passes, bad-fixture
      surfaces expected check rejections), exit 0.

Backward compat: existing single-job `argo-train.sh` callers (no
`--multi-seed`, no `--folds`) route to the original `train-template.yaml`
unchanged. `--seed 42` is a no-op offset for the SplitMix64 mix at the call
sites — the trajectory shifts only when the user passes `--seed` explicitly,
matching the prompt's "default 42 (historic implicit value)" requirement.

L40S pool: argo-train.sh defaults `--gpu-pool ci-training-h100`; user passes
`--gpu-pool ci-training-l40s` at deploy time. No script default change
(per constraint 5).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

scripts/argo-train.sh

@@ -170,12 +170,16 @@ if [[ "$USE_MULTI_SEED" == "false" ]]; then
   exit 0
 fi
 
-# ── Multi-seed × multi-fold path ──
-# Render the train-multi-seed-template.yaml with the (seed, fold) matrix
-# expanded inline. The base template ships with a placeholder marker
-# (`# __MATRIX_TASKS__`) which we replace with N*K generated WorkflowTask
-# stanzas. This avoids hand-writing a 30-task matrix and keeps the source
-# template human-readable.
+# ── Multi-seed path (one job per seed; folds run inside the binary) ──
+# Render the train-multi-seed-template.yaml with the per-seed matrix
+# expanded inline. Plan 5 Task 5 Phase B pivot: was N*K (seed,fold) jobs;
+# is now N (seed-only) jobs because `train_baseline_rl` is a multi-fold
+# walk-forward executor — it accepts `--max-folds K`, NOT `--fold K`. Each
+# rendered task invokes the binary with `--seed N --max-folds K` so the
+# walk-forward sweep happens inside the single training process.
+#
+# The base template ships with a placeholder marker (`# __MATRIX_TASKS__`)
+# which we replace with N generated WorkflowTask stanzas.
 TEMPLATE_SRC="infra/k8s/argo/train-multi-seed-template.yaml"
 if [[ ! -f "$TEMPLATE_SRC" ]]; then
   echo "Error: multi-seed template not found at $TEMPLATE_SRC"
@@ -183,34 +187,30 @@ if [[ ! -f "$TEMPLATE_SRC" ]]; then
 fi
 
 # Build matrix YAML. Each task is a dag.tasks[] entry that targets the
-# `train-single` template with seed/fold parameters bound from inputs.
+# `train-single` template with the `seed` parameter bound from inputs.
+# The template forwards `seed` as `--seed` and reads `--max-folds` from the
+# workflow-scoped `folds` parameter, so each task internally runs all K folds.
 build_matrix_tasks() {
   local seeds="$1"
-  local folds="$2"
   # 10 spaces — matches the sibling `- name: ensure-binary` list-item indent
   # under `dag.tasks:` (which is itself at 8 spaces). Wrong indent here
   # produces a YAML parse error in the rendered template.
   local indent="          "
   local s
-  local f
   for ((s=0; s<seeds; s++)); do
-    for ((f=0; f<folds; f++)); do
-      cat <<EOF
-${indent}- name: train-s${s}-f${f}
+    cat <<EOF
+${indent}- name: train-s${s}
 ${indent}  template: train-single
 ${indent}  dependencies: [ensure-fxcache, gpu-warmup]
 ${indent}  arguments:
 ${indent}    parameters:
 ${indent}      - name: seed
 ${indent}        value: "${s}"
-${indent}      - name: fold
-${indent}        value: "${f}"
 EOF
-    done
   done
 }
 
-MATRIX_TASKS=$(build_matrix_tasks "$MULTI_SEED" "$FOLDS")
+MATRIX_TASKS=$(build_matrix_tasks "$MULTI_SEED")
 
 # Substitute the placeholder. Match the *exact* placeholder line (leading
 # whitespace + the marker as the only content on the line) — the marker
@@ -226,12 +226,12 @@ if [[ "$DRY_RUN" == "true" ]]; then
   # Emit the rendered template + a synthetic per-task marker line so test
   # harnesses can count generated jobs without piping through `argo submit`
   # (which would require a live cluster). The `kind: WorkflowTask` marker
-  # is what the test_multi_seed_harness.sh asserts against.
+  # is what the test_multi_seed_harness.sh asserts against. Plan 5 Task 5
+  # Phase B: one marker per seed (not per (seed, fold) pair) because each
+  # job now sweeps all K folds via `--max-folds`.
   echo "$RENDERED"
   for ((s=0; s<MULTI_SEED; s++)); do
-    for ((f=0; f<FOLDS; f++)); do
-      echo "# kind: WorkflowTask  seed=${s} fold=${f}"
-    done
+    echo "# kind: WorkflowTask  seed=${s} max_folds=${FOLDS}"
   done
   exit 0
 fi
@@ -247,8 +247,8 @@ echo "  sha:        $SHA"
 echo "  branch:     $BRANCH"
 echo "  model:      $MODEL"
 echo "  multi-seed: $MULTI_SEED"
-echo "  folds:      $FOLDS"
-echo "  total jobs: $((MULTI_SEED * FOLDS))"
+echo "  folds:      $FOLDS (each job runs all folds via --max-folds)"
+echo "  total jobs: $MULTI_SEED"
 [[ "$PROFILE" == "true" ]] && echo "  profile:    nsys (per-job .nsys-rep upload)"
 [[ -n "$TAG" ]] && echo "  tag:        $TAG"
 [[ -n "$EPOCHS" ]] && echo "  epochs:     $EPOCHS"

scripts/tests/test_multi_seed_harness.sh

@@ -1,10 +1,14 @@
 #!/usr/bin/env bash
 # Test: argo-train.sh --multi-seed N --folds K --dry-run emits one
-# WorkflowTask per (seed, fold) pair (N * K total).
+# WorkflowTask per seed (N total). Each task internally runs all K folds via
+# the binary's `--max-folds K` argument — train_baseline_rl is a multi-fold
+# walk-forward executor, NOT a single-fold one.
 #
-# This is the validation surface for the Plan 5 Task 1A multi-seed Argo DAG —
-# verifies the matrix is generated correctly without submitting to a live
-# cluster.
+# This is the validation surface for the Plan 5 Task 5 Phase B multi-seed
+# Argo DAG (was N*K (seed, fold) jobs in Task 1A; pivoted to N seed-only jobs
+# after the first deploy failed at startup with `--fold` not recognized).
+# Verifies the matrix + binary command is generated correctly without
+# submitting to a live cluster.
 set -euo pipefail
 
 # Resolve repo root from this script's location so the test runs from any cwd.
@@ -12,17 +16,51 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
 cd "${REPO_ROOT}"
 
-# Dry-run: --multi-seed 3 --folds 2 should emit a workflow with 6 training jobs.
+# Dry-run: --multi-seed 3 --folds 2 should emit 3 training jobs (one per seed),
+# each invoking the binary with `--max-folds 2` to run both folds internally.
 output=$(./scripts/argo-train.sh dqn --multi-seed 3 --folds 2 --dry-run 2>&1)
 actual_count=$(echo "$output" | grep -c "kind: WorkflowTask" || true)
-expected=6
+expected=3
 if [[ "$actual_count" -ne "$expected" ]]; then
-    echo "FAIL: expected $expected jobs (3 seeds x 2 folds), got $actual_count"
+    echo "FAIL: expected $expected jobs (3 seeds, fold sweep inside binary), got $actual_count"
     echo "--- output ---"
     echo "$output"
     exit 1
 fi
-echo "PASS: multi-seed harness produces $expected jobs for 3x2 matrix"
+echo "PASS: multi-seed harness produces $expected per-seed jobs for --multi-seed 3 --folds 2"
+
+# The rendered binary command must include `--max-folds {{workflow.parameters.folds}}`
+# (drives the walk-forward sweep inside the single training process) and must
+# NOT include any per-fold flag — `train_baseline_rl` rejects `--fold`, which
+# is what broke the first L40S deploy attempt (workflow train-multi-seed-z2llf,
+# every job exited at startup with `error: unexpected argument '--fold' found`).
+# The Argo `{{workflow.parameters.folds}}` placeholder is resolved at
+# workflow submission time (argo-train.sh passes `-p folds=$FOLDS` to argo
+# submit), not at template render time, so the dry-run shows the placeholder.
+if ! echo "$output" | grep -qE -- "--max-folds[[:space:]]+\{\{workflow\.parameters\.folds\}\}"; then
+    echo "FAIL: rendered binary command missing '--max-folds {{workflow.parameters.folds}}'"
+    echo "--- output (rendered template, first 300 lines) ---"
+    echo "$output" | head -300
+    exit 1
+fi
+echo "PASS: rendered binary invocation includes --max-folds placeholder"
+
+# Folds parameter must be declared at the workflow level so `argo submit -p
+# folds=K` (which argo-train.sh emits) can override the default at submit time.
+if ! echo "$output" | grep -qE "^[[:space:]]*-[[:space:]]*name:[[:space:]]*folds[[:space:]]*$"; then
+    echo "FAIL: rendered template does not declare 'folds' workflow parameter"
+    exit 1
+fi
+echo "PASS: rendered template declares folds workflow parameter"
+
+if echo "$output" | grep -qE -- '--fold[[:space:]]+"?[0-9{]'; then
+    echo "FAIL: rendered binary command still contains a per-fold flag (--fold N)"
+    echo "      train_baseline_rl rejects --fold; only --max-folds is supported."
+    echo "--- offending lines ---"
+    echo "$output" | grep -E -- '--fold[[:space:]]+"?[0-9{]'
+    exit 1
+fi
+echo "PASS: rendered binary invocation has no per-fold flag (Path B compliance)"
 
 # Backward-compat: --multi-seed 1 --folds 1 must NOT emit WorkflowTask lines
 # (single-job path uses the existing template, no DAG matrix).