infra(argo): plain smoke-test template + scripts/argo-smoke.sh

Adds a no-wrapper L40S smoke runner alongside the nsys/sanitizer
templates. Same compile + checkout + data-mount layout, no profiler
binary, no sanitizer instrumentation, no MinIO artefact upload — just
a plain test execution against the full training-data PVC.

Use case: validate a smoke test passes on the real 27-month dataset
when local data is too short (laptop only has 1 quarter of MBP-10/
trades, fxcache truncates below the 10-month minimum).

- infra/k8s/argo/smoke-test-template.yaml: WorkflowTemplate
  smoke-test, entrypoint smoke-run, default test
  multi_fold_convergence::test_multi_fold_convergence.
- infra/k8s/argo/kustomization.yaml: register the new template.
- infra/k8s/argo/argo-workflow-netpol.yaml: extend the
  sanitizer/nsys NetworkPolicy podSelector to include smoke-test
  (identical egress requirements: git fetch, ci-builder pull,
  training-data PVC).
- scripts/argo-smoke.sh: thin wrapper mirroring argo-nsys.sh /
  argo-sanitizer.sh (--multi-fold default, --test, --ref, --watch).

Verified: kustomize dry-run clean, kubectl apply -k creates the
template + reconfigures the netpol live in foxhunt namespace.
This commit is contained in:
jgrusewski
2026-04-29 07:52:11 +02:00
parent 99487c9871
commit 27f536ba6e
4 changed files with 330 additions and 1 deletions

81
scripts/argo-smoke.sh Executable file
View File

@@ -0,0 +1,81 @@
#!/usr/bin/env bash
# Run a smoke test on L40S via Argo Workflows — no profiler, no sanitizer.
#
# Use this when you want to validate that a smoke test passes against the
# full training-data PVC (27 months of OHLCV + MBP-10 + trades) without
# nsys profile overhead or sanitizer instrumentation. Iteration cycle is
# faster than argo-nsys.sh / argo-sanitizer.sh because there's no artefact
# upload step.
#
# Why L40S not local: the laptop has only 1 quarter of MBP-10/trades data,
# which truncates the fxcache below the 10-month minimum that the
# multi_fold_convergence smoke needs.
#
# Usage:
# ./scripts/argo-smoke.sh # multi_fold_convergence (default)
# ./scripts/argo-smoke.sh --test magnitude_distribution # different test
# ./scripts/argo-smoke.sh --ref <sha> # specific commit
# ./scripts/argo-smoke.sh --watch # follow logs
#
# Requires: argo CLI
set -euo pipefail
TEST_NAME="multi_fold_convergence::test_multi_fold_convergence"
COMMIT_REF="HEAD"
GPU_POOL=""
WATCH=false
usage() {
cat <<EOF
Usage: $(basename "$0") [OPTIONS]
Plain smoke-test runner on L40S (no profiler, no sanitizer).
Options:
--test <name> Smoke test name (default: $TEST_NAME)
Common values:
multi_fold_convergence::test_multi_fold_convergence
magnitude_distribution::test_magnitude_distribution
performance::test_real_data_single_epoch
iqn_quantile_monotonicity::iqn_multi_quantile_heads_produce_monotonic_estimates
--multi-fold Shortcut: --test multi_fold_convergence::test_multi_fold_convergence
(default — exercises fold-boundary code paths)
--ref <ref> Git ref to test (default: HEAD)
--gpu-pool <pool> GPU node pool (default: ci-training-l40s)
--watch Follow workflow logs
-h, --help Show this help
EOF
exit 0
}
while [[ $# -gt 0 ]]; do
case $1 in
--test) TEST_NAME="$2"; shift 2 ;;
--multi-fold) TEST_NAME="multi_fold_convergence::test_multi_fold_convergence"; shift ;;
--ref) COMMIT_REF="$2"; shift 2 ;;
--gpu-pool) GPU_POOL="$2"; shift 2 ;;
--watch) WATCH=true; shift ;;
-h|--help) usage ;;
*) echo "Unknown option: $1"; usage ;;
esac
done
if [ "$COMMIT_REF" = "HEAD" ]; then
COMMIT_REF=$(git rev-parse HEAD)
fi
CMD="argo submit -n foxhunt --from=wftmpl/smoke-test"
CMD="$CMD -p commit-ref=$COMMIT_REF"
CMD="$CMD -p test-name=$TEST_NAME"
[[ -n "$GPU_POOL" ]] && CMD="$CMD -p gpu-pool=$GPU_POOL"
if $WATCH; then
CMD="$CMD --watch"
fi
echo "Submitting smoke workflow..."
echo " commit: $(echo "$COMMIT_REF" | cut -c1-8)"
echo " test: $TEST_NAME"
echo ""
eval "$CMD"