gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
7e2eb708a7fc3a7cfa2f37d47a047ce94d20123a
foxhunt/scripts/argo-train.sh
jgrusewski fcf76701f4 plan5(task5-B): pivot multi-seed Argo from N×K (seed,fold) to N seed-only fanout 
The first L40S deploy attempt (workflow `train-multi-seed-z2llf`, terminated)
failed at startup with `error: unexpected argument '--fold' found` on every
job: `train_baseline_rl` is a multi-fold walk-forward executor that accepts
`--max-folds K`, NOT `--fold N`. The original P5T1 harness assumed the
opposite and fanned out N seeds × K folds = N*K jobs, each invoking the
binary with `--seed N --fold K`.

User chose Path B: pivot to one job per seed (each runs all K folds via the
existing `--max-folds` mechanism). Per-job runtime is K× longer, but fanout
drops from N*K=30 → N=5 (matches L40S pool capacity better) and the binary
contract becomes the one the binary actually has.

4 surface changes:

1. crates/ml/examples/train_baseline_rl.rs — add `--seed N` CLI arg
   (default 42 — historic implicit value). Sets `FOXHUNT_SEED` env var at
   startup BEFORE any CUDA module spins up. Logs the seed value at the
   training start banner.

2. crates/ml/src/cuda_pipeline/mod.rs — add `global_seed()` (reads
   `FOXHUNT_SEED`, default 42) + `mix_seed(base)` (SplitMix64 avalanche
   so adjacent global seeds produce uncorrelated module seeds). Six call
   sites updated to mix the global seed into their previously-hardcoded
   constants:
   - trainer/action.rs: GpuActionSelector seed (0xDEAD_BEEF_CAFE) + the
     epsilon-greedy fallback StdRng (0xAC7_DEF0).
   - cuda_pipeline/gpu_iqn_head.rs: IQN Xavier-init RNG (0x1CA_1234).
   - cuda_pipeline/gpu_iql_trainer.rs: V(s) Xavier-init RNG (0x1C1_9ABC).
   - cuda_pipeline/gpu_her.rs: random-donor RNG (0x4E4_5678).
   - cuda_pipeline/gpu_ppo_collector.rs: rng_seeds Vec for PPO
     experience-collector init + reset (0xAA0_5EED).
   - trainer/training_loop.rs: per-epoch regime_dropout_seed.

3. infra/k8s/argo/train-multi-seed-template.yaml — drop `fold` parameter
   from `train-single` template; binary invoked as `--seed "$SEED"
   --max-folds {{workflow.parameters.folds}}` so the walk-forward sweep
   happens inside the single training process. Drop `FOLD` env var. Update
   the nsys-rep upload filename to drop the fold suffix. Update banners /
   doc comments to reflect "one-job-per-seed" semantics.

4. scripts/argo-train.sh — matrix generator drops the inner fold loop.
   Each emitted task carries only `seed=${s}` and depends on the same
   ensure-fxcache + gpu-warmup. The dry-run synthetic marker switches from
   `seed=${s} fold=${f}` to `seed=${s} max_folds=${FOLDS}` so test harnesses
   count the new shape correctly.

5. scripts/tests/test_multi_seed_harness.sh — assertions updated:
   - `--multi-seed 3 --folds 2` produces 3 tasks (was 6).
   - Rendered binary command must include `--max-folds
     {{workflow.parameters.folds}}` placeholder.
   - Rendered template must declare `folds` workflow parameter (so
     `argo submit -p folds=K` overrides the default).
   - Rendered binary command must NOT contain any per-fold flag — this
     catches the failure mode that broke the first L40S deploy.
   - Backward-compat: `--multi-seed 1 --folds 1` preserves the existing
     single-template path (no DAG matrix tasks emitted).

6. docs/dqn-wire-up-audit.md — adds 1 Wired row documenting the pivot,
   the new `--seed`/`mix_seed` plumbing, all 6 RNG call sites, and the
   end-to-end seed-variation verification result.

Validation:

  cargo check --workspace clean at 11 warnings (workspace baseline preserved).

  cargo build --release --example train_baseline_rl succeeds; --help shows
  the new --seed flag with documented default 42.

  Seed-variation end-to-end test on RTX 3050 Ti (1 fold × 2 epochs each):
    --seed 42  → F0 best Sharpe = -9.7831, best_val_metric = 1.957244,
                 epoch-2 train Sharpe = -16.12, val_Sharpe = +1.11.
    --seed 999 → F0 best Sharpe = +92.9341, best_val_metric = 2.161012,
                 epoch-2 train Sharpe = +92.93, val_Sharpe = -0.25.
  Different best Sharpe / best_val_metric / epoch-2 train + val Sharpe
  across seeds proves the seed actually propagates through the RNG init
  paths and is not just accepted-and-ignored. The seed=42 numbers match
  the prompt's "deterministic baseline" expectation (F0 = -9.7831 was
  bit-identical pre-pivot because no global-seed plumbing existed).

  ./scripts/argo-train.sh dqn --multi-seed 5 --folds 6 --dry-run produces
  exactly 5 WorkflowTask markers (train-s0..train-s4), each with
  `--max-folds {{workflow.parameters.folds}}` in the binary invocation.

  All 3 harness tests PASS:
    - test_multi_seed_harness.sh: 5 PASS lines, exit 0.
    - test_nsys_harness.sh: 4 PASS lines + ALL PASS, exit 0.
    - test_tier_checks.sh: PASS overall (good-fixture passes, bad-fixture
      surfaces expected check rejections), exit 0.

Backward compat: existing single-job `argo-train.sh` callers (no
`--multi-seed`, no `--folds`) route to the original `train-template.yaml`
unchanged. `--seed 42` is a no-op offset for the SplitMix64 mix at the call
sites — the trajectory shifts only when the user passes `--seed` explicitly,
matching the prompt's "default 42 (historic implicit value)" requirement.

L40S pool: argo-train.sh defaults `--gpu-pool ci-training-h100`; user passes
`--gpu-pool ci-training-l40s` at deploy time. No script default change
(per constraint 5).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-26 14:11:32 +02:00

283 lines
11 KiB
Bash
Executable File
Raw Blame History

#!/usr/bin/env bash
# Train a model via Argo Workflows.
#
# Usage:
# ./scripts/argo-train.sh dqn # defaults: HEAD, H100, 50 epochs
# ./scripts/argo-train.sh dqn --sha abc1234 # specific commit
# ./scripts/argo-train.sh dqn --epochs 100 --trials 40 # override training params
# ./scripts/argo-train.sh ppo --gpu-pool ci-training # L40S instead of H100
# ./scripts/argo-train.sh dqn --baseline # skip hyperopt
# ./scripts/argo-train.sh dqn --watch # follow logs
#
# Supported models:
# RL: dqn, ppo
# Supervised: tft, mamba2, tggn, tlob, liquid, kan, xlstm, diffusion
#
# Requires: argo CLI
set -euo pipefail
SHA="HEAD"
BRANCH="main"
TRIALS=""
EPOCHS=""
GPU_POOL=""
SYMBOL=""
WATCH=false
BASELINE=false
CAPITAL=""
SANITIZER="none"
MULTI_SEED=1
FOLDS=1
DRY_RUN=false
TAG=""
PROFILE=false
usage() {
cat <<EOF
Usage: $(basename "$0") <model> [OPTIONS]
Models:
dqn, ppo (RL)
tft, mamba2, tggn, tlob, liquid, kan, xlstm (supervised)
Options:
--sha <commit> Git commit SHA (default: HEAD)
--branch <branch> Git branch (default: main)
--trials <n> Hyperopt trials (default: 20)
--epochs <n> Training epochs (default: 50)
--gpu-pool <pool> GPU node pool (default: ci-training-h100)
--symbol <sym> Trading symbol (default: ES.FUT)
--capital <n> Initial capital (default: 35000)
--baseline Skip hyperopt (trials=0)
--sanitizer <tool> Run under compute-sanitizer (memcheck|racecheck|synccheck)
--watch Follow workflow logs
--multi-seed <n> Run N seeds in parallel (default: 1, fans out via DAG when >1)
--folds <k> Walk-forward fold count (default: 1, fans out via DAG when >1)
--tag <t> Label workflow with foxhunt-tag=<t> for log aggregation
--profile Wrap training under nsys (NVIDIA Nsight Systems) and
upload .nsys-rep artefacts to MinIO bucket
foxhunt-training-artifacts/profiles/<sha>/. Forces the
multi-seed render path (template rendered locally) so
the nsys wrapper is visible in --dry-run output without
cluster contact. Plan 5 Task 3 (A.4.1).
--dry-run Print rendered workflow YAML to stdout, do not submit
-h, --help Show this help
EOF
exit 0
}
[[ $# -eq 0 ]] && { echo "Error: model argument required"; usage; }
MODEL="$1"; shift
case "$MODEL" in
dqn|ppo|tft|mamba2|tggn|tlob|liquid|kan|xlstm|diffusion) ;;
*) echo "Error: unknown model '$MODEL'"; usage ;;
esac
while [[ $# -gt 0 ]]; do
case $1 in
--sha) SHA="$2"; shift 2 ;;
--branch) BRANCH="$2"; shift 2 ;;
--trials) TRIALS="$2"; shift 2 ;;
--epochs) EPOCHS="$2"; shift 2 ;;
--gpu-pool) GPU_POOL="$2"; shift 2 ;;
--symbol) SYMBOL="$2"; shift 2 ;;
--capital) CAPITAL="$2"; shift 2 ;;
--baseline) BASELINE=true; shift ;;
--sanitizer) SANITIZER="$2"; shift 2 ;;
--watch) WATCH=true; shift ;;
--multi-seed) MULTI_SEED="$2"; shift 2 ;;
--folds) FOLDS="$2"; shift 2 ;;
--tag) TAG="$2"; shift 2 ;;
--profile) PROFILE=true; shift ;;
--dry-run) DRY_RUN=true; shift ;;
-h|--help) usage ;;
*) echo "Unknown option: $1"; usage ;;
esac
done
# Validate seed/fold counts are positive integers.
if ! [[ "$MULTI_SEED" =~ ^[0-9]+$ ]] || [[ "$MULTI_SEED" -lt 1 ]]; then
echo "Error: --multi-seed must be a positive integer, got '$MULTI_SEED'"
exit 1
fi
if ! [[ "$FOLDS" =~ ^[0-9]+$ ]] || [[ "$FOLDS" -lt 1 ]]; then
echo "Error: --folds must be a positive integer, got '$FOLDS'"
exit 1
fi
# Auto-derive cuda-compute-cap from GPU pool — cubins must match device sm_XX.
# Default pool is ci-training-h100 (sm_90). Override for other architectures:
# ci-training-h100* → sm_90 (Hopper)
# ci-training-l40s → sm_89 (Ada Lovelace)
case "${GPU_POOL:-ci-training-h100}" in
*l40s*) CUDA_COMPUTE_CAP="89" ;;
*h100*|*) CUDA_COMPUTE_CAP="90" ;; # default Hopper
esac
# ── Route: single-job (existing template) vs multi-seed DAG (new template) ──
# Backward compat: --multi-seed 1 --folds 1 keeps the existing single-template
# call path verbatim. Only when N>1 OR K>1 do we render the matrix DAG.
#
# --profile forces the multi-seed render path even at N=K=1 because the
# single-job dry-run goes through `argo submit --dry-run -o yaml` which
# resolves the WorkflowTemplate against a live cluster — not available in
# local CI. The multi-seed renderer reads the template file directly so
# `--profile --dry-run` works without cluster access. Plan 5 Task 3.
USE_MULTI_SEED=false
if [[ "$MULTI_SEED" -gt 1 || "$FOLDS" -gt 1 || "$PROFILE" == "true" ]]; then
USE_MULTI_SEED=true
fi
if [[ "$USE_MULTI_SEED" == "false" ]]; then
CMD="argo submit -n foxhunt --from=wftmpl/train"
CMD="$CMD -p commit-sha=$SHA"
CMD="$CMD -p git-branch=$BRANCH"
CMD="$CMD -p model=$MODEL"
CMD="$CMD -p cuda-compute-cap=$CUDA_COMPUTE_CAP"
[[ -n "$TRIALS" ]] && CMD="$CMD -p hyperopt-trials=$TRIALS"
[[ -n "$EPOCHS" ]] && CMD="$CMD -p train-epochs=$EPOCHS"
[[ -n "$GPU_POOL" ]] && CMD="$CMD -p gpu-pool=$GPU_POOL"
[[ -n "$SYMBOL" ]] && CMD="$CMD -p symbol=$SYMBOL"
[[ -n "$CAPITAL" ]] && CMD="$CMD -p initial-capital=$CAPITAL"
[[ "$SANITIZER" != "none" ]] && CMD="$CMD -p sanitizer=$SANITIZER"
$BASELINE && CMD="$CMD -p hyperopt-trials=0"
$WATCH && CMD="$CMD --watch"
if [[ "$DRY_RUN" == "true" ]]; then
# `argo submit --dry-run -o yaml` renders the resolved Workflow without
# contacting the cluster. Single-template path emits one Workflow object
# (no `kind: WorkflowTask` lines — that string only appears in the
# multi-seed DAG matrix).
eval "$CMD --dry-run -o yaml"
exit 0
fi
echo "Submitting $MODEL training workflow..."
echo " sha: $SHA"
echo " branch: $BRANCH"
echo " model: $MODEL"
$BASELINE && echo " mode: baseline (no hyperopt)"
[[ -n "$TRIALS" ]] && echo " trials: $TRIALS"
[[ -n "$EPOCHS" ]] && echo " epochs: $EPOCHS"
[[ -n "$GPU_POOL" ]] && echo " gpu: $GPU_POOL"
echo " sm: $CUDA_COMPUTE_CAP"
echo ""
eval "$CMD"
exit 0
fi
# ── Multi-seed path (one job per seed; folds run inside the binary) ──
# Render the train-multi-seed-template.yaml with the per-seed matrix
# expanded inline. Plan 5 Task 5 Phase B pivot: was N*K (seed,fold) jobs;
# is now N (seed-only) jobs because `train_baseline_rl` is a multi-fold
# walk-forward executor — it accepts `--max-folds K`, NOT `--fold K`. Each
# rendered task invokes the binary with `--seed N --max-folds K` so the
# walk-forward sweep happens inside the single training process.
#
# The base template ships with a placeholder marker (`# __MATRIX_TASKS__`)
# which we replace with N generated WorkflowTask stanzas.
TEMPLATE_SRC="infra/k8s/argo/train-multi-seed-template.yaml"
if [[ ! -f "$TEMPLATE_SRC" ]]; then
echo "Error: multi-seed template not found at $TEMPLATE_SRC"
exit 1
fi
# Build matrix YAML. Each task is a dag.tasks[] entry that targets the
# `train-single` template with the `seed` parameter bound from inputs.
# The template forwards `seed` as `--seed` and reads `--max-folds` from the
# workflow-scoped `folds` parameter, so each task internally runs all K folds.
build_matrix_tasks() {
local seeds="$1"
# 10 spaces — matches the sibling `- name: ensure-binary` list-item indent
# under `dag.tasks:` (which is itself at 8 spaces). Wrong indent here
# produces a YAML parse error in the rendered template.
local indent=" "
local s
for ((s=0; s<seeds; s++)); do
cat <<EOF
${indent}- name: train-s${s}
${indent} template: train-single
${indent} dependencies: [ensure-fxcache, gpu-warmup]
${indent} arguments:
${indent} parameters:
${indent} - name: seed
${indent} value: "${s}"
EOF
done
}
MATRIX_TASKS=$(build_matrix_tasks "$MULTI_SEED")
# Substitute the placeholder. Match the *exact* placeholder line (leading
# whitespace + the marker as the only content on the line) — the marker
# string also appears in doc comments above and we must not replace those.
# Use awk (not sed) — multi-line replacement with sed is fragile across
# platforms.
RENDERED=$(awk -v repl="$MATRIX_TASKS" '
/^[[:space:]]*# __MATRIX_TASKS__[[:space:]]*$/ { print repl; next }
{ print }
' "$TEMPLATE_SRC")
if [[ "$DRY_RUN" == "true" ]]; then
# Emit the rendered template + a synthetic per-task marker line so test
# harnesses can count generated jobs without piping through `argo submit`
# (which would require a live cluster). The `kind: WorkflowTask` marker
# is what the test_multi_seed_harness.sh asserts against. Plan 5 Task 5
# Phase B: one marker per seed (not per (seed, fold) pair) because each
# job now sweeps all K folds via `--max-folds`.
echo "$RENDERED"
for ((s=0; s<MULTI_SEED; s++)); do
echo "# kind: WorkflowTask seed=${s} max_folds=${FOLDS}"
done
exit 0
fi
# Submit: write the rendered template to a temp file, apply it as a
# WorkflowTemplate, then submit a Workflow that references it.
TMP_TEMPLATE=$(mktemp -t train-multi-seed.XXXXXX.yaml)
trap 'rm -f "$TMP_TEMPLATE"' EXIT
echo "$RENDERED" > "$TMP_TEMPLATE"
echo "Submitting multi-seed $MODEL training workflow..."
echo " sha: $SHA"
echo " branch: $BRANCH"
echo " model: $MODEL"
echo " multi-seed: $MULTI_SEED"
echo " folds: $FOLDS (each job runs all folds via --max-folds)"
echo " total jobs: $MULTI_SEED"
[[ "$PROFILE" == "true" ]] && echo " profile: nsys (per-job .nsys-rep upload)"
[[ -n "$TAG" ]] && echo " tag: $TAG"
[[ -n "$EPOCHS" ]] && echo " epochs: $EPOCHS"
[[ -n "$GPU_POOL" ]] && echo " gpu: $GPU_POOL"
echo " sm: $CUDA_COMPUTE_CAP"
echo ""
# Apply the rendered WorkflowTemplate, then submit a workflow from it.
kubectl apply -n foxhunt -f "$TMP_TEMPLATE"
CMD="argo submit -n foxhunt --from=wftmpl/train-multi-seed"
CMD="$CMD -p commit-sha=$SHA"
CMD="$CMD -p git-branch=$BRANCH"
CMD="$CMD -p model=$MODEL"
CMD="$CMD -p cuda-compute-cap=$CUDA_COMPUTE_CAP"
CMD="$CMD -p multi-seed=$MULTI_SEED"
CMD="$CMD -p folds=$FOLDS"
CMD="$CMD -p profile=$PROFILE"
[[ -n "$TRIALS" ]] && CMD="$CMD -p hyperopt-trials=$TRIALS"
[[ -n "$EPOCHS" ]] && CMD="$CMD -p train-epochs=$EPOCHS"
[[ -n "$GPU_POOL" ]] && CMD="$CMD -p gpu-pool=$GPU_POOL"
[[ -n "$SYMBOL" ]] && CMD="$CMD -p symbol=$SYMBOL"
[[ -n "$CAPITAL" ]] && CMD="$CMD -p initial-capital=$CAPITAL"
[[ -n "$TAG" ]] && CMD="$CMD --labels foxhunt-tag=$TAG"
[[ "$SANITIZER" != "none" ]] && CMD="$CMD -p sanitizer=$SANITIZER"
$BASELINE && CMD="$CMD -p hyperopt-trials=0"
$WATCH && CMD="$CMD --watch"
eval "$CMD"
Reference in New Issue View Git Blame Copy Permalink