8b8bb1af70
SP-chain training has been standardising on L40S since 2026-05-09, but
every invocation required an explicit `--gpu-pool ci-training-l40s`
override. The 2026-05-04 train-mnpf7 incident (sm_90 cubins deployed
to an L40S device, then resubmitted with the explicit override) was
the last incident in a long line of "forgot the pool flag" friction.
`feedback_default_to_l40s_pool.md` codified the user preference; this
commit lands the default in the actual invocation paths.
Changes:
- infra/k8s/argo/train-template.yaml: gpu-pool default H100 → L40S
- infra/k8s/argo/train-multi-seed-template.yaml: same + cuda-compute
-cap default 90 → 89
- scripts/argo-train.sh: docstring / --help / compute-cap fallback
case all flip to L40S as the bare default; H100 becomes opt-in via
`--gpu-pool ci-training-h100` for 80 GB / sm_90 workloads
- scripts/argo-test.sh: --help text aligned
Other architectural defaults (data-source=mbp10 per
feedback_mbp10_mandatory; imbalance-bar-threshold=20.0 per the 2026-
05-10 OOM-prevention fix) are already correct in the template.
Verified via `argo-train.sh dqn --branch sp20-aux-h-fixed --sha HEAD
--baseline --dry-run` — rendered workflow shows cuda-compute-cap=89,
no explicit gpu-pool override (template default L40S in effect).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
70 lines
1.9 KiB
Bash
Executable File
70 lines
1.9 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Run GPU/CUDA tests via Argo Workflows on H100.
|
|
#
|
|
# Usage:
|
|
# ./scripts/argo-test.sh # defaults: dqn,ppo,tft
|
|
# ./scripts/argo-test.sh --models dqn --scope lib # DQN lib tests only
|
|
# ./scripts/argo-test.sh --models all # all 10 models
|
|
# ./scripts/argo-test.sh --watch # follow logs
|
|
#
|
|
# Requires: argo CLI
|
|
set -euo pipefail
|
|
|
|
MODELS="dqn,ppo,tft"
|
|
SCOPE="all"
|
|
COMMIT_REF="HEAD"
|
|
GPU_POOL=""
|
|
WATCH=false
|
|
|
|
usage() {
|
|
cat <<EOF
|
|
Usage: $(basename "$0") [OPTIONS]
|
|
|
|
Options:
|
|
--models <list> Comma-separated model list (default: dqn,ppo,tft)
|
|
Use "all" for all 10 models
|
|
--scope <scope> lib, integration, or all (default: all)
|
|
--ref <ref> Git ref to test (default: HEAD)
|
|
--gpu-pool <pool> GPU node pool (default: ci-training-l40s; opt into
|
|
ci-training-h100 for 80 GB / sm_90 workloads)
|
|
--watch Follow workflow logs
|
|
-h, --help Show this help
|
|
EOF
|
|
exit 0
|
|
}
|
|
|
|
while [[ $# -gt 0 ]]; do
|
|
case $1 in
|
|
--models) MODELS="$2"; shift 2 ;;
|
|
--scope) SCOPE="$2"; shift 2 ;;
|
|
--ref) COMMIT_REF="$2"; shift 2 ;;
|
|
--gpu-pool) GPU_POOL="$2"; shift 2 ;;
|
|
--watch) WATCH=true; shift ;;
|
|
-h|--help) usage ;;
|
|
*) echo "Unknown option: $1"; usage ;;
|
|
esac
|
|
done
|
|
|
|
# Resolve HEAD to actual SHA if needed
|
|
if [ "$COMMIT_REF" = "HEAD" ]; then
|
|
COMMIT_REF=$(git rev-parse HEAD)
|
|
fi
|
|
|
|
CMD="argo submit -n foxhunt --from=wftmpl/gpu-test-pipeline"
|
|
CMD="$CMD -p commit-ref=$COMMIT_REF"
|
|
CMD="$CMD -p models=$MODELS"
|
|
CMD="$CMD -p test-scope=$SCOPE"
|
|
|
|
[[ -n "$GPU_POOL" ]] && CMD="$CMD -p gpu-pool=$GPU_POOL"
|
|
|
|
if $WATCH; then
|
|
CMD="$CMD --watch"
|
|
fi
|
|
|
|
echo "Submitting GPU test workflow..."
|
|
echo " commit: $(echo $COMMIT_REF | cut -c1-8)"
|
|
echo " models: $MODELS"
|
|
echo " scope: $SCOPE"
|
|
echo ""
|
|
eval "$CMD"
|