gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

infra(argo): default GPU pool to ci-training-l40s (sm_89) per feedback_default_to_l40s_pool

Browse Source 
SP-chain training has been standardising on L40S since 2026-05-09, but
every invocation required an explicit `--gpu-pool ci-training-l40s`
override. The 2026-05-04 train-mnpf7 incident (sm_90 cubins deployed
to an L40S device, then resubmitted with the explicit override) was
the last incident in a long line of "forgot the pool flag" friction.
`feedback_default_to_l40s_pool.md` codified the user preference; this
commit lands the default in the actual invocation paths.

Changes:

  - infra/k8s/argo/train-template.yaml: gpu-pool default H100 → L40S
  - infra/k8s/argo/train-multi-seed-template.yaml: same + cuda-compute
    -cap default 90 → 89
  - scripts/argo-train.sh: docstring / --help / compute-cap fallback
    case all flip to L40S as the bare default; H100 becomes opt-in via
    `--gpu-pool ci-training-h100` for 80 GB / sm_90 workloads
  - scripts/argo-test.sh: --help text aligned

Other architectural defaults (data-source=mbp10 per
feedback_mbp10_mandatory; imbalance-bar-threshold=20.0 per the 2026-
05-10 OOM-prevention fix) are already correct in the template.

Verified via `argo-train.sh dqn --branch sp20-aux-h-fixed --sha HEAD
--baseline --dry-run` — rendered workflow shows cuda-compute-cap=89,
no explicit gpu-pool override (template default L40S in effect).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-14 14:22:19 +02:00
parent bbd52c3aa7
commit 8b8bb1af70
4 changed files with 28 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
infra/k8s/argo/train-multi-seed-template.yaml
View File

@@ -58,12 +58,17 @@ spec:
value: HEAD
- name: git-branch
value: main
# Default sm_89 / ci-training-l40s as of 2026-05-14 per
# `feedback_default_to_l40s_pool.md`. argo-train.sh's
# CUDA_COMPUTE_CAP derivation passes "89" by default to match;
# explicit --gpu-pool ci-training-h100 overrides both back to
# sm_90 + H100.
- name: cuda-compute-cap
value: "90"
value: "89"
- name: model
value: dqn
- name: gpu-pool
value: ci-training-h100
value: ci-training-l40s
- name: hyperopt-trials
value: "0" # Multi-seed runs typically skip hyperopt (re-use baseline params)
- name: hyperopt-epochs

10
infra/k8s/argo/train-template.yaml
View File

@@ -45,8 +45,16 @@ spec:
value: "90"
- name: model
value: dqn
# gpu-pool default: ci-training-l40s (set 2026-05-14 per
# `feedback_default_to_l40s_pool.md` — SP-chain training has been
# standardising on L40S since 2026-05-09; the prior ci-training-h100
# default required every SP-run invocation to pass an explicit
# `--gpu-pool ci-training-l40s` override. H100 remains opt-in via
# `--gpu-pool ci-training-h100` for runs that genuinely need
# 80 GB VRAM or sm_90 features). Compute-cap derivation in
# `argo-train.sh` matches this default to sm_89 (Ada Lovelace).
- name: gpu-pool
value: ci-training-h100
value: ci-training-l40s
- name: hyperopt-trials
value: "20"
- name: hyperopt-epochs

3
scripts/argo-test.sh
View File

@@ -25,7 +25,8 @@ Options:
Use "all" for all 10 models
--scope <scope> lib, integration, or all (default: all)
--ref <ref> Git ref to test (default: HEAD)
--gpu-pool <pool> GPU node pool (default: ci-training-h100)
--gpu-pool <pool> GPU node pool (default: ci-training-l40s; opt into
ci-training-h100 for 80 GB / sm_90 workloads)
--watch Follow workflow logs
-h, --help Show this help
EOF

17
scripts/argo-train.sh
View File

@@ -2,10 +2,10 @@
# Train a model via Argo Workflows.
#
# Usage:
# ./scripts/argo-train.sh dqn # defaults: HEAD, H100, 50 epochs
# ./scripts/argo-train.sh dqn # defaults: HEAD, L40S, 50 epochs
# ./scripts/argo-train.sh dqn --sha abc1234 # specific commit
# ./scripts/argo-train.sh dqn --epochs 100 --trials 40 # override training params
# ./scripts/argo-train.sh ppo --gpu-pool ci-training # L40S instead of H100
# ./scripts/argo-train.sh ppo --gpu-pool ci-training-h100 # opt into H100 (80 GB)
# ./scripts/argo-train.sh dqn --baseline # skip hyperopt
# ./scripts/argo-train.sh dqn --watch # follow logs
#
@@ -49,7 +49,8 @@ Options:
--branch <branch> Git branch (default: main)
--trials <n> Hyperopt trials (default: 20)
--epochs <n> Training epochs (default: 50)
--gpu-pool <pool> GPU node pool (default: ci-training-h100)
--gpu-pool <pool> GPU node pool (default: ci-training-l40s; opt into
ci-training-h100 for 80 GB / sm_90 workloads)
--symbol <sym> Trading symbol (default: ES.FUT)
--capital <n> Initial capital (default: 35000)
--baseline Skip hyperopt (trials=0)
@@ -125,16 +126,18 @@ if ! [[ "$FOLDS" =~ ^[0-9]+$ ]] || [[ "$FOLDS" -lt 1 ]]; then
fi
# Auto-derive cuda-compute-cap from GPU pool — cubins must match device sm_XX.
# Default pool is ci-training-h100 (sm_90). Override for other architectures:
# Default pool is ci-training-l40s (sm_89) as of 2026-05-14 per
# `feedback_default_to_l40s_pool.md`. Override for other architectures:
# ci-training-h100* → sm_90 (Hopper)
# ci-training-l40s → sm_89 (Ada Lovelace)
# ci-training-l40s → sm_89 (Ada Lovelace, current default)
# ci-training → sm_89 (alias for L40S — pool is named bare in some
# clusters; fixed 2026-05-04 after train-mnpf7 deployed
# with sm_90 cubins on L40S device, requiring terminate
# + resubmit with explicit --gpu-pool ci-training-l40s).
case "${GPU_POOL:-ci-training-h100}" in
case "${GPU_POOL:-ci-training-l40s}" in
*h100*) CUDA_COMPUTE_CAP="90" ;; # Hopper (opt-in)
*l40s*|ci-training) CUDA_COMPUTE_CAP="89" ;;
*h100*|*) CUDA_COMPUTE_CAP="90" ;; # default Hopper
*) CUDA_COMPUTE_CAP="89" ;; # default Ada Lovelace
esac
# ── Route: single-job (existing template) vs multi-seed DAG (new template) ──