plan5(task3): A.4.1 nsys profile harness with regression-comparison script

- argo-train.sh: --profile flag forces multi-seed render path so the
  nsys wrapper + foxhunt-training-artifacts upload step are visible in
  --dry-run YAML without cluster contact (test surface).
- train-multi-seed-template.yaml: new `profile` parameter (default
  "false") gates the per-(seed, fold) `nsys profile
  --capture-range=cudaProfilerApi` wrapper and the `mc cp` upload to
  foxhunt-training-artifacts/profiles/<sha>/. mc binary fetched
  on-demand (ci-builder image lacks it). MinIO creds optional —
  upload warn-skips if absent.
- Dockerfile.foxhunt-training-runtime: install nsight-systems-cli
  unpinned (pinning the stale 2024.4.1.61-1 from earlier plans
  breaks builds when apt index advances).
- minio.yaml: add foxhunt-training-artifacts bucket to minio-init.
- compare-nsys-profiles.py: V0 regression detector — compares
  cuda_gpu_kern_sum total_ns / epoch_count between two profiles;
  exits 1 on >20% slowdown. NVTX per-epoch ranges deferred to T5.
- tests/test_nsys_harness.sh: dry-run grep test — verifies both
  required strings appear when --profile is set, and that the
  default (no --profile) path keeps profile=false in the rendered
  template.
- dqn-wire-up-audit.md: Plan 5 Task 3 row added documenting the
  harness + the baseline-capture deferral to T5.

Backward compat: test_multi_seed_harness.sh from P5T1 still PASS.

scripts/argo-train.sh

@@ -30,6 +30,7 @@ MULTI_SEED=1
 FOLDS=1
 DRY_RUN=false
 TAG=""
+PROFILE=false
 
 usage() {
   cat <<EOF
@@ -53,6 +54,12 @@ Options:
   --multi-seed <n>     Run N seeds in parallel (default: 1, fans out via DAG when >1)
   --folds <k>          Walk-forward fold count (default: 1, fans out via DAG when >1)
   --tag <t>            Label workflow with foxhunt-tag=<t> for log aggregation
+  --profile            Wrap training under nsys (NVIDIA Nsight Systems) and
+                       upload .nsys-rep artefacts to MinIO bucket
+                       foxhunt-training-artifacts/profiles/<sha>/. Forces the
+                       multi-seed render path (template rendered locally) so
+                       the nsys wrapper is visible in --dry-run output without
+                       cluster contact. Plan 5 Task 3 (A.4.1).
   --dry-run            Print rendered workflow YAML to stdout, do not submit
   -h, --help           Show this help
 EOF
@@ -83,6 +90,7 @@ while [[ $# -gt 0 ]]; do
     --multi-seed)  MULTI_SEED="$2"; shift 2 ;;
     --folds)       FOLDS="$2"; shift 2 ;;
     --tag)         TAG="$2"; shift 2 ;;
+    --profile)     PROFILE=true; shift ;;
     --dry-run)     DRY_RUN=true; shift ;;
     -h|--help)     usage ;;
     *)             echo "Unknown option: $1"; usage ;;
@@ -111,8 +119,14 @@ esac
 # ── Route: single-job (existing template) vs multi-seed DAG (new template) ──
 # Backward compat: --multi-seed 1 --folds 1 keeps the existing single-template
 # call path verbatim. Only when N>1 OR K>1 do we render the matrix DAG.
+#
+# --profile forces the multi-seed render path even at N=K=1 because the
+# single-job dry-run goes through `argo submit --dry-run -o yaml` which
+# resolves the WorkflowTemplate against a live cluster — not available in
+# local CI. The multi-seed renderer reads the template file directly so
+# `--profile --dry-run` works without cluster access. Plan 5 Task 3.
 USE_MULTI_SEED=false
-if [[ "$MULTI_SEED" -gt 1 || "$FOLDS" -gt 1 ]]; then
+if [[ "$MULTI_SEED" -gt 1 || "$FOLDS" -gt 1 || "$PROFILE" == "true" ]]; then
   USE_MULTI_SEED=true
 fi
 
@@ -235,6 +249,7 @@ echo "  model:      $MODEL"
 echo "  multi-seed: $MULTI_SEED"
 echo "  folds:      $FOLDS"
 echo "  total jobs: $((MULTI_SEED * FOLDS))"
+[[ "$PROFILE" == "true" ]] && echo "  profile:    nsys (per-job .nsys-rep upload)"
 [[ -n "$TAG" ]] && echo "  tag:        $TAG"
 [[ -n "$EPOCHS" ]] && echo "  epochs:     $EPOCHS"
 [[ -n "$GPU_POOL" ]] && echo "  gpu:        $GPU_POOL"
@@ -251,6 +266,7 @@ CMD="$CMD -p model=$MODEL"
 CMD="$CMD -p cuda-compute-cap=$CUDA_COMPUTE_CAP"
 CMD="$CMD -p multi-seed=$MULTI_SEED"
 CMD="$CMD -p folds=$FOLDS"
+CMD="$CMD -p profile=$PROFILE"
 
 [[ -n "$TRIALS" ]]   && CMD="$CMD -p hyperopt-trials=$TRIALS"
 [[ -n "$EPOCHS" ]]   && CMD="$CMD -p train-epochs=$EPOCHS"