Files
foxhunt/infra/k8s
jgrusewski 8b8bb1af70 infra(argo): default GPU pool to ci-training-l40s (sm_89) per feedback_default_to_l40s_pool
SP-chain training has been standardising on L40S since 2026-05-09, but
every invocation required an explicit `--gpu-pool ci-training-l40s`
override. The 2026-05-04 train-mnpf7 incident (sm_90 cubins deployed
to an L40S device, then resubmitted with the explicit override) was
the last incident in a long line of "forgot the pool flag" friction.
`feedback_default_to_l40s_pool.md` codified the user preference; this
commit lands the default in the actual invocation paths.

Changes:

  - infra/k8s/argo/train-template.yaml: gpu-pool default H100 → L40S
  - infra/k8s/argo/train-multi-seed-template.yaml: same + cuda-compute
    -cap default 90 → 89
  - scripts/argo-train.sh: docstring / --help / compute-cap fallback
    case all flip to L40S as the bare default; H100 becomes opt-in via
    `--gpu-pool ci-training-h100` for 80 GB / sm_90 workloads
  - scripts/argo-test.sh: --help text aligned

Other architectural defaults (data-source=mbp10 per
feedback_mbp10_mandatory; imbalance-bar-threshold=20.0 per the 2026-
05-10 OOM-prevention fix) are already correct in the template.

Verified via `argo-train.sh dqn --branch sp20-aux-h-fixed --sha HEAD
--baseline --dry-run` — rendered workflow shows cuda-compute-cap=89,
no explicit gpu-pool override (template default L40S in effect).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 14:22:19 +02:00
..