diff --git a/docs/superpowers/plans/2026-03-13-gpu-test-workflow.md b/docs/superpowers/plans/2026-03-13-gpu-test-workflow.md new file mode 100644 index 000000000..b0ce53420 --- /dev/null +++ b/docs/superpowers/plans/2026-03-13-gpu-test-workflow.md @@ -0,0 +1,962 @@ +# H100 GPU Test Workflow Implementation Plan + +> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Argo WorkflowTemplate that compiles and runs the full GPU/CUDA test suite on H100 with real market data, triggered manually, on-push (gated), or nightly. + +**Architecture:** Two new PVCs (`cargo-target-cuda-test` 30Gi, `test-data-pvc` 10Gi) isolate test infrastructure from training. Single H100 pod compiles with `--features cuda` then runs tests sequentially per model family. Continue-on-failure captures per-model results. + +**Tech Stack:** Argo Workflows, Kubernetes, Rust/Cargo, CUDA 12.9, ci-builder image, bash + +**Spec:** `docs/superpowers/specs/2026-03-13-gpu-test-workflow-design.md` + +--- + +## File Structure + +### New Files (7) + +| File | Responsibility | +|------|---------------| +| `infra/k8s/argo/cargo-target-cuda-test-pvc.yaml` | PVC for test compilation cache (30Gi) | +| `infra/k8s/training/test-data-pvc.yaml` | PVC for curated test data subset (10Gi) | +| `infra/k8s/training/populate-test-data-job.yaml` | One-time job: copy data from training PVC to test PVC | +| `infra/k8s/argo/gpu-test-pipeline-template.yaml` | Main WorkflowTemplate: compile + test on H100 | +| `infra/k8s/argo/gpu-test-nightly-cron.yaml` | CronWorkflow (suspended) for nightly runs | +| `scripts/argo-test.sh` | CLI wrapper for manual `argo submit` | +| `scripts/refresh-test-data.sh` | Re-run populate job | + +### Modified Files (5) + +| File | Change | +|------|--------| +| `infra/k8s/argo/ci-pipeline-template.yaml` | Add `ml-changed` output + `gpu-test` DAG task | +| `crates/ml/tests/smoke_test_real_data.rs:99-130` | Add `TEST_DATA_DIR` to `smoke_data_dir()` and `try_data()` | +| `crates/ml/tests/dqn_training_smoke_test.rs:94-105` | Add `TEST_DATA_DIR` to `get_dbn_data_dir()` | +| `crates/ml/tests/dqn_training_pipeline_test.rs:100-117` | Add `TEST_DATA_DIR` to `get_es_fut_data_dir()` | +| `crates/ml/tests/dqn_early_stopping_termination_test.rs:93-104` | Add `TEST_DATA_DIR` to `get_dbn_data_dir()` | + +--- + +## Task 1: Create PVCs + +**Files:** +- Create: `infra/k8s/argo/cargo-target-cuda-test-pvc.yaml` +- Create: `infra/k8s/training/test-data-pvc.yaml` + +- [ ] **Step 1: Create cargo-target-cuda-test PVC** + +```yaml +# infra/k8s/argo/cargo-target-cuda-test-pvc.yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: cargo-target-cuda-test + namespace: foxhunt + labels: + app.kubernetes.io/name: cargo-target + app.kubernetes.io/component: ci-cache + app.kubernetes.io/part-of: foxhunt +spec: + accessModes: [ReadWriteOnce] + storageClassName: scw-bssd-retain + resources: + requests: + storage: 30Gi +``` + +Pattern matches `infra/k8s/argo/sccache-pvcs.yaml` (same `scw-bssd-retain` storage class). + +- [ ] **Step 2: Create test-data PVC** + +```yaml +# infra/k8s/training/test-data-pvc.yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: test-data-pvc + namespace: foxhunt + labels: + app.kubernetes.io/name: test-data + app.kubernetes.io/part-of: foxhunt +spec: + accessModes: [ReadWriteOnce] + storageClassName: scw-bssd + resources: + requests: + storage: 10Gi +``` + +- [ ] **Step 3: Commit** + +```bash +git add infra/k8s/argo/cargo-target-cuda-test-pvc.yaml infra/k8s/training/test-data-pvc.yaml +git commit -m "infra: add cargo-target-cuda-test (30Gi) and test-data-pvc (10Gi)" +``` + +--- + +## Task 2: Create populate-test-data job + +**Files:** +- Create: `infra/k8s/training/populate-test-data-job.yaml` +- Create: `scripts/refresh-test-data.sh` + +- [ ] **Step 1: Create the populate job** + +This job runs on the H100 node pool (same as test-data-pvc) and copies one quarter of ES.FUT data per schema from the training PVC. + +```yaml +# infra/k8s/training/populate-test-data-job.yaml +apiVersion: batch/v1 +kind: Job +metadata: + generateName: populate-test-data- + namespace: foxhunt + labels: + app.kubernetes.io/name: populate-test-data + app.kubernetes.io/part-of: foxhunt +spec: + backoffLimit: 1 + ttlSecondsAfterFinished: 300 + template: + spec: + restartPolicy: Never + nodeSelector: + k8s.scaleway.com/pool-name: ci-training-h100 + tolerations: + - key: nvidia.com/gpu + operator: Exists + effect: NoSchedule + - key: node.cilium.io/agent-not-ready + operator: Exists + effect: NoSchedule + containers: + - name: populate + image: busybox:1.37 + command: ["/bin/sh", "-c"] + args: + - | + set -e + echo "=== Populating test data PVC ===" + + # Target directories + mkdir -p /test-data/ohlcv/ES.FUT + mkdir -p /test-data/mbp10/ES.FUT + mkdir -p /test-data/trades/ES.FUT + + # Copy one quarter per schema (2025-Q2) + echo "Copying OHLCV..." + cp /training-data/futures-baseline/ES.FUT/ES.FUT_2025-Q2.dbn.zst \ + /test-data/ohlcv/ES.FUT/ 2>/dev/null || \ + cp /training-data/futures-baseline/ES.FUT/*2025*Q2* \ + /test-data/ohlcv/ES.FUT/ 2>/dev/null || \ + { echo "WARN: No 2025-Q2 OHLCV found, copying first available"; \ + cp /training-data/futures-baseline/ES.FUT/*.dbn.zst \ + /test-data/ohlcv/ES.FUT/ 2>/dev/null | head -1; } + + echo "Copying MBP-10..." + cp /training-data/futures-baseline-mbp10/ES.FUT/ES.FUT_2025-Q2.dbn.zst \ + /test-data/mbp10/ES.FUT/ 2>/dev/null || \ + cp /training-data/futures-baseline-mbp10/ES.FUT/*2025*Q2* \ + /test-data/mbp10/ES.FUT/ 2>/dev/null || \ + echo "WARN: No MBP-10 2025-Q2 found" + + echo "Copying trades..." + cp /training-data/futures-baseline-trades/ES.FUT/ES.FUT_2025-Q2.dbn.zst \ + /test-data/trades/ES.FUT/ 2>/dev/null || \ + cp /training-data/futures-baseline-trades/ES.FUT/*2025*Q2* \ + /test-data/trades/ES.FUT/ 2>/dev/null || \ + echo "WARN: No trades 2025-Q2 found" + + echo "=== Test data contents ===" + find /test-data -type f -exec ls -lh {} \; + echo "Total: $(du -sh /test-data | cut -f1)" + echo "=== Done ===" + resources: + requests: + nvidia.com/gpu: "1" + cpu: 100m + memory: 128Mi + limits: + nvidia.com/gpu: "1" + cpu: 500m + memory: 256Mi + volumeMounts: + - name: training-data + mountPath: /training-data + readOnly: true + - name: test-data + mountPath: /test-data + volumes: + - name: training-data + persistentVolumeClaim: + claimName: training-data-pvc + readOnly: true + - name: test-data + persistentVolumeClaim: + claimName: test-data-pvc +``` + +Note: requests `nvidia.com/gpu: 1` only to schedule on the H100 node (same as test-data-pvc). GPU isn't used for the copy. + +- [ ] **Step 2: Create refresh script** + +```bash +#!/usr/bin/env bash +# Re-populate the test-data-pvc from training-data-pvc. +# Usage: ./scripts/refresh-test-data.sh +set -euo pipefail + +echo "Submitting populate-test-data job..." +kubectl -n foxhunt create -f infra/k8s/training/populate-test-data-job.yaml + +echo "Waiting for completion..." +kubectl -n foxhunt wait --for=condition=complete job \ + -l app.kubernetes.io/name=populate-test-data \ + --timeout=300s + +echo "Done. Test data PVC refreshed." +``` + +Save to `scripts/refresh-test-data.sh`, chmod +x. + +- [ ] **Step 3: Commit** + +```bash +chmod +x scripts/refresh-test-data.sh +git add infra/k8s/training/populate-test-data-job.yaml scripts/refresh-test-data.sh +git commit -m "infra: add populate-test-data job and refresh script" +``` + +--- + +## Task 3: Wire TEST_DATA_DIR into test helpers + +**Files:** +- Modify: `crates/ml/tests/smoke_test_real_data.rs:99-130` +- Modify: `crates/ml/tests/dqn_training_smoke_test.rs:94-105` +- Modify: `crates/ml/tests/dqn_training_pipeline_test.rs:100-117` +- Modify: `crates/ml/tests/dqn_early_stopping_termination_test.rs:93-104` + +- [ ] **Step 1: Update `smoke_data_dir()` and `try_data()` in smoke_test_real_data.rs** + +Replace `smoke_data_dir()`: +```rust +fn smoke_data_dir() -> PathBuf { + // CI: TEST_DATA_DIR points to test-data-pvc on H100 + if let Ok(dir) = std::env::var("TEST_DATA_DIR") { + return PathBuf::from(dir); + } + // Local dev: relative path from workspace root + workspace_root().join("data").join("smoke-test") +} +``` + +`try_data()` doesn't need changes — it already calls `smoke_data_dir()`. + +- [ ] **Step 2: Update `get_dbn_data_dir()` in dqn_training_smoke_test.rs** + +Replace lines 94-105: +```rust +fn get_dbn_data_dir() -> Result { + // CI: TEST_DATA_DIR points to test-data-pvc on H100 + if let Ok(dir) = std::env::var("TEST_DATA_DIR") { + let ohlcv = std::path::PathBuf::from(&dir).join("ohlcv"); + if ohlcv.exists() { + return Ok(ohlcv.to_string_lossy().to_string()); + } + // Flat layout fallback + return Ok(dir); + } + let workspace_root = PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .ancestors() + .find(|p| p.join("test_data").exists()) + .context("Failed to find workspace root with test_data/")? + .to_path_buf(); + let data_dir = workspace_root.join("test_data/futures-baseline"); + if !data_dir.exists() { + anyhow::bail!("DBN data not found: {}", data_dir.display()); + } + Ok(data_dir.to_string_lossy().to_string()) +} +``` + +- [ ] **Step 3: Update `get_es_fut_data_dir()` in dqn_training_pipeline_test.rs** + +Replace lines 100-117: +```rust +fn get_es_fut_data_dir() -> Result { + if let Ok(dir) = std::env::var("TEST_DATA_DIR") { + let ohlcv = std::path::PathBuf::from(&dir).join("ohlcv"); + if ohlcv.exists() { + return Ok(ohlcv.to_string_lossy().to_string()); + } + return Ok(dir); + } + let workspace_root = PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .ancestors() + .find(|p| p.join("test_data").exists()) + .context("Failed to find workspace root with test_data/")? + .to_path_buf(); + let data_dir = workspace_root.join("test_data/futures-baseline"); + if !data_dir.exists() { + anyhow::bail!( + "ES.FUT data directory not found: {}. Run data acquisition first.", + data_dir.display() + ); + } + Ok(data_dir.to_string_lossy().to_string()) +} +``` + +- [ ] **Step 4: Update `get_dbn_data_dir()` in dqn_early_stopping_termination_test.rs** + +Replace lines 93-104: +```rust +#[cfg(feature = "cuda")] +fn get_dbn_data_dir() -> Option { + if let Ok(dir) = std::env::var("TEST_DATA_DIR") { + let ohlcv = std::path::PathBuf::from(&dir).join("ohlcv"); + if ohlcv.exists() { + return Some(ohlcv.to_string_lossy().to_string()); + } + return Some(dir); + } + let workspace_root = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .ancestors() + .find(|p| p.join("test_data").exists())? + .to_path_buf(); + let data_dir = workspace_root.join("test_data/futures-baseline"); + if !data_dir.exists() { + eprintln!("SKIP: DBN data not found: {}", data_dir.display()); + return None; + } + Some(data_dir.to_string_lossy().to_string()) +} +``` + +- [ ] **Step 5: Verify compilation** + +```bash +SQLX_OFFLINE=true cargo test -p ml --features cuda --no-run 2>&1 | tail -5 +``` + +Expected: compiles with 0 errors. + +- [ ] **Step 6: Verify local tests still work (env var NOT set)** + +```bash +unset TEST_DATA_DIR +SQLX_OFFLINE=true cargo test -p ml --test smoke_test_real_data --features cuda -- smoke_ohlcv_parse_and_extract_features --nocapture 2>&1 | tail -5 +``` + +Expected: PASS (falls back to `data/smoke-test/` path). + +- [ ] **Step 7: Verify TEST_DATA_DIR override works** + +```bash +TEST_DATA_DIR=/home/jgrusewski/Work/foxhunt/data/smoke-test \ + SQLX_OFFLINE=true cargo test -p ml --test smoke_test_real_data --features cuda -- smoke_ohlcv_parse_and_extract_features --nocapture 2>&1 | tail -5 +``` + +Expected: PASS (uses env var path). + +- [ ] **Step 8: Commit** + +```bash +git add crates/ml/tests/smoke_test_real_data.rs crates/ml/tests/dqn_training_smoke_test.rs \ + crates/ml/tests/dqn_training_pipeline_test.rs crates/ml/tests/dqn_early_stopping_termination_test.rs +git commit -m "test: add TEST_DATA_DIR env var support for H100 CI test workflow" +``` + +--- + +## Task 4: Create GPU test WorkflowTemplate + +**Files:** +- Create: `infra/k8s/argo/gpu-test-pipeline-template.yaml` + +This is the core workflow. Reference `compile-and-train-template.yaml` for patterns (git checkout, env vars, tolerations, notification). + +- [ ] **Step 1: Create the WorkflowTemplate** + +Create `infra/k8s/argo/gpu-test-pipeline-template.yaml` with: + +**Header:** +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: WorkflowTemplate +metadata: + name: gpu-test-pipeline + namespace: foxhunt + labels: + app.kubernetes.io/name: gpu-test-pipeline + app.kubernetes.io/part-of: foxhunt +spec: + entrypoint: pipeline + onExit: notify-result + serviceAccountName: argo-workflow + podMetadata: + labels: + app.kubernetes.io/part-of: foxhunt + app.kubernetes.io/component: gpu-test + securityContext: + fsGroup: 0 + podGC: + strategy: OnPodCompletion + ttlStrategy: + secondsAfterCompletion: 3600 + activeDeadlineSeconds: 5400 +``` + +**Parameters:** +```yaml + arguments: + parameters: + - name: commit-ref + value: HEAD + - name: models + value: "dqn,ppo,tft" + - name: test-scope + value: all + - name: epochs + value: "3" + - name: gpu-pool + value: ci-training-h100 + - name: cuda-compute-cap + value: "90" + - name: notify + value: "true" +``` + +**Volumes:** +```yaml + volumes: + - name: git-ssh-key + secret: + secretName: argo-git-ssh-key + defaultMode: 256 + - name: cargo-target + persistentVolumeClaim: + claimName: cargo-target-cuda-test + - name: test-data + persistentVolumeClaim: + claimName: test-data-pvc + readOnly: true +``` + +**DAG:** +```yaml + templates: + - name: pipeline + dag: + tasks: + - name: gpu-warmup + template: gpu-warmup + - name: compile-and-test + template: compile-and-test + dependencies: [gpu-warmup] +``` + +**gpu-warmup template:** Copy from `compile-and-train-template.yaml:266-291`, replace hardcoded pool with `{{workflow.parameters.gpu-pool}}`. + +**compile-and-test template:** Single container on H100. The shell script: + +1. SSH + git checkout (same pattern as compile-and-train lines 170-206) +2. Set `CUDA_COMPUTE_CAP={{workflow.parameters.cuda-compute-cap}}` +3. PVC size guard (same as compile-and-train lines 214-220) +4. `cargo test -p ml -p ml-dqn -p ml-core --features cuda --no-run` (compile) +5. Parse `models` parameter, loop over each model +6. Per model: run `cargo test` commands, capture exit code, continue on failure +7. Print summary table, exit with aggregate status + +**Key env vars on the compile-and-test container:** +```yaml +env: + - name: SQLX_OFFLINE + value: "true" + - name: CARGO_TERM_COLOR + value: always + - name: CARGO_TARGET_DIR + value: /cargo-target + - name: CARGO_HOME + value: /cargo-target/cargo-home + - name: CUDA_VISIBLE_DEVICES + value: "0" + - name: CUBLAS_WORKSPACE_CONFIG + value: ":4096:8" + - name: TEST_DATA_DIR + value: /data/test-data + - name: RUST_LOG + value: info + - name: DQN_SMOKE_EPOCHS + value: "{{workflow.parameters.epochs}}" +``` + +**Key volume mounts:** +```yaml +volumeMounts: + - name: git-ssh-key + mountPath: /etc/git-ssh + readOnly: true + - name: cargo-target + mountPath: /cargo-target + - name: test-data + mountPath: /data/test-data + readOnly: true +``` + +**Resources:** +```yaml +resources: + requests: + nvidia.com/gpu: "1" + cpu: "4" + memory: 32Gi + limits: + nvidia.com/gpu: "1" + cpu: "8" + memory: 64Gi +``` + +**Shell script for compile-and-test (inline in args):** + +```bash +set -e + +REF="{{workflow.parameters.commit-ref}}" +MODELS="{{workflow.parameters.models}}" +SCOPE="{{workflow.parameters.test-scope}}" + +# --- SSH setup (same as compile-and-train) --- +mkdir -p ~/.ssh +cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519 +chmod 600 ~/.ssh/id_ed25519 +printf 'Host *\n StrictHostKeyChecking no\n UserKnownHostsFile /dev/null\n' > ~/.ssh/config +chmod 600 ~/.ssh/config + +REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git" +BUILD="/cargo-target/src" +git config --global --add safe.directory "$BUILD" + +# --- Persistent checkout on PVC (same as compile-and-train) --- +if [ -d "$BUILD/.git" ]; then + cd "$BUILD" + CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none") + TARGET=$(git ls-remote origin "$REF" 2>/dev/null | cut -f1 || echo "$REF") + if [ "$CURRENT" = "$TARGET" ] || [ "$CURRENT" = "$REF" ]; then + echo "=== Already at $REF ===" + else + echo "=== Updating checkout ===" + git fetch origin + git checkout --force "$REF" + git clean -fd + fi +else + echo "=== Initial clone ===" + git clone --filter=blob:none "$REPO" "$BUILD" + cd "$BUILD" + git checkout "$REF" +fi +echo "Checked out $(git rev-parse --short HEAD)" + +export PATH="${CARGO_HOME}/bin:${PATH}" +export CUDA_COMPUTE_CAP={{workflow.parameters.cuda-compute-cap}} + +# --- PVC size guard --- +TARGET_SIZE_MB=$(du -sm "$CARGO_TARGET_DIR" 2>/dev/null | cut -f1 || echo 0) +echo "PVC usage: ${TARGET_SIZE_MB}MB" +if [ "$TARGET_SIZE_MB" -gt 25000 ]; then + echo "PVC exceeds 25GB, pruning..." + rm -rf "$CARGO_TARGET_DIR/release" "$CARGO_TARGET_DIR/debug" +fi + +# --- Compile --- +echo "=== Compiling test binaries (--features cuda) ===" +cargo test -p ml -p ml-dqn -p ml-core --features cuda --no-run 2>&1 +echo "=== Compilation done ===" + +# --- Expand "all" --- +if [ "$MODELS" = "all" ]; then + MODELS="dqn,ppo,tft,mamba2,tggn,tlob,liquid,kan,xlstm,diffusion" +fi + +# --- Test runner --- +RESULTS="" +FAILURES=0 + +run_tests() { + local NAME="$1"; shift + echo "" + echo "========================================" + echo " TEST: $NAME" + echo "========================================" + set +e + "$@" --nocapture 2>&1 + EXIT=$? + set -e + if [ $EXIT -eq 0 ]; then + RESULTS="${RESULTS}${NAME}:PASS\n" + else + RESULTS="${RESULTS}${NAME}:FAIL\n" + FAILURES=$((FAILURES + 1)) + fi +} + +# --- Core tests (always run) --- +run_tests "core-lib" cargo test -p ml-core --features cuda --lib + +# --- Per-model tests --- +IFS=',' read -ra MODEL_LIST <<< "$MODELS" +for MODEL in "${MODEL_LIST[@]}"; do + case "$MODEL" in + dqn) + if [ "$SCOPE" = "lib" ] || [ "$SCOPE" = "all" ]; then + run_tests "dqn-lib" cargo test -p ml-dqn --features cuda --lib + run_tests "dqn-ml-lib" cargo test -p ml --features cuda --lib -- dqn + fi + if [ "$SCOPE" = "integration" ] || [ "$SCOPE" = "all" ]; then + run_tests "dqn-smoke" cargo test -p ml --features cuda --test smoke_test_real_data + run_tests "dqn-pipeline" cargo test -p ml --features cuda --test dqn_training_pipeline_test + run_tests "dqn-smoke-train" cargo test -p ml --features cuda --test dqn_training_smoke_test + run_tests "dqn-early-stop" cargo test -p ml --features cuda --test dqn_early_stopping_termination_test + run_tests "dqn-collapse" cargo test -p ml --features cuda --test dqn_action_collapse_fix_test + fi + ;; + ppo) + if [ "$SCOPE" = "lib" ] || [ "$SCOPE" = "all" ]; then + run_tests "ppo-lib" cargo test -p ml --features cuda --lib -- ppo + fi + if [ "$SCOPE" = "integration" ] || [ "$SCOPE" = "all" ]; then + run_tests "ppo-barrier" cargo test -p ml --features cuda --test barrier_optimization_test + fi + ;; + *) + # Supervised models + if [ "$SCOPE" = "lib" ] || [ "$SCOPE" = "all" ]; then + run_tests "${MODEL}-lib" cargo test -p ml --features cuda --lib -- "$MODEL" + fi + if [ "$SCOPE" = "integration" ] || [ "$SCOPE" = "all" ]; then + # Run model-specific integration tests if they exist (non-fatal if none) + run_tests "${MODEL}-integ" cargo test -p ml --features cuda --test "${MODEL}_*" 2>/dev/null || true + fi + ;; + esac +done + +# --- Summary --- +echo "" +echo "========================================" +echo " GPU TEST RESULTS" +echo "========================================" +printf "$RESULTS" | while IFS=: read -r name status; do + [ -z "$name" ] && continue + printf " %-25s %s\n" "$name" "$status" +done +echo "========================================" +echo " Total failures: $FAILURES" +echo "========================================" + +# Write results for notification step +mkdir -p /tmp/outputs +printf "$RESULTS" > /tmp/outputs/results +echo "$FAILURES" > /tmp/outputs/failures + +exit $FAILURES +``` + +**Outputs on compile-and-test:** +```yaml +outputs: + parameters: + - name: results + valueFrom: + path: /tmp/outputs/results + default: "unknown:FAIL" + - name: failures + valueFrom: + path: /tmp/outputs/failures + default: "1" +``` + +**notify-result template:** Same pattern as `compile-and-train-template.yaml:586-627` and `ci-pipeline-template.yaml:553-594`. Use `{{workflow.status}}` and `{{workflow.name}}`. Gated on `notify` parameter. + +- [ ] **Step 2: Validate YAML syntax** + +```bash +python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/gpu-test-pipeline-template.yaml'))" && echo "YAML OK" +``` + +- [ ] **Step 3: Commit** + +```bash +git add infra/k8s/argo/gpu-test-pipeline-template.yaml +git commit -m "feat(ci): add gpu-test-pipeline WorkflowTemplate for H100 CUDA testing" +``` + +--- + +## Task 5: Create nightly CronWorkflow + +**Files:** +- Create: `infra/k8s/argo/gpu-test-nightly-cron.yaml` + +- [ ] **Step 1: Create CronWorkflow (suspended)** + +```yaml +# infra/k8s/argo/gpu-test-nightly-cron.yaml +apiVersion: argoproj.io/v1alpha1 +kind: CronWorkflow +metadata: + name: gpu-test-nightly + namespace: foxhunt + labels: + app.kubernetes.io/name: gpu-test-nightly + app.kubernetes.io/part-of: foxhunt +spec: + schedule: "0 2 * * *" + timezone: "UTC" + suspend: true # Enable when ready: kubectl patch cronworkflow gpu-test-nightly -n foxhunt -p '{"spec":{"suspend":false}}' + concurrencyPolicy: Replace + successfulJobsHistoryLimit: 3 + failedJobsHistoryLimit: 5 + workflowSpec: + workflowTemplateRef: + name: gpu-test-pipeline + arguments: + parameters: + - name: models + value: "dqn,ppo,tft,mamba2,tggn,tlob,liquid,kan,xlstm,diffusion" + - name: epochs + value: "5" + - name: commit-ref + value: "main" +``` + +- [ ] **Step 2: Commit** + +```bash +git add infra/k8s/argo/gpu-test-nightly-cron.yaml +git commit -m "feat(ci): add gpu-test-nightly CronWorkflow (suspended)" +``` + +--- + +## Task 6: Create argo-test.sh CLI + +**Files:** +- Create: `scripts/argo-test.sh` + +- [ ] **Step 1: Create the script** + +Pattern matches `scripts/argo-train.sh` (same argument parsing style, same `argo submit` invocation). + +```bash +#!/usr/bin/env bash +# Run GPU/CUDA tests via Argo Workflows on H100. +# +# Usage: +# ./scripts/argo-test.sh # defaults: dqn,ppo,tft +# ./scripts/argo-test.sh --models dqn --scope lib # DQN lib tests only +# ./scripts/argo-test.sh --models all --epochs 5 # all 10 models, 5 epochs +# ./scripts/argo-test.sh --watch # follow logs +# +# Requires: argo CLI +set -euo pipefail + +MODELS="dqn,ppo,tft" +SCOPE="all" +EPOCHS="3" +COMMIT_REF="HEAD" +GPU_POOL="" +WATCH=false + +usage() { + cat < Comma-separated model list (default: dqn,ppo,tft) + Use "all" for all 10 models + --scope lib, integration, or all (default: all) + --epochs Training epochs for e2e tests (default: 3) + --ref Git ref to test (default: HEAD) + --gpu-pool GPU node pool (default: ci-training-h100) + --watch Follow workflow logs + -h, --help Show this help +EOF + exit 0 +} + +while [[ $# -gt 0 ]]; do + case $1 in + --models) MODELS="$2"; shift 2 ;; + --scope) SCOPE="$2"; shift 2 ;; + --epochs) EPOCHS="$2"; shift 2 ;; + --ref) COMMIT_REF="$2"; shift 2 ;; + --gpu-pool) GPU_POOL="$2"; shift 2 ;; + --watch) WATCH=true; shift ;; + -h|--help) usage ;; + *) echo "Unknown option: $1"; usage ;; + esac +done + +# Resolve HEAD to actual SHA if needed +if [ "$COMMIT_REF" = "HEAD" ]; then + COMMIT_REF=$(git rev-parse HEAD) +fi + +CMD="argo submit -n foxhunt --from=wftmpl/gpu-test-pipeline" +CMD="$CMD -p commit-ref=$COMMIT_REF" +CMD="$CMD -p models=$MODELS" +CMD="$CMD -p test-scope=$SCOPE" +CMD="$CMD -p epochs=$EPOCHS" + +[[ -n "$GPU_POOL" ]] && CMD="$CMD -p gpu-pool=$GPU_POOL" + +if $WATCH; then + CMD="$CMD --watch" +fi + +echo "Submitting GPU test workflow..." +echo " commit: $(echo $COMMIT_REF | cut -c1-8)" +echo " models: $MODELS" +echo " scope: $SCOPE" +echo " epochs: $EPOCHS" +echo "" +eval "$CMD" +``` + +- [ ] **Step 2: Make executable and commit** + +```bash +chmod +x scripts/argo-test.sh +git add scripts/argo-test.sh +git commit -m "feat(ci): add argo-test.sh CLI for GPU test workflow" +``` + +--- + +## Task 7: Add on-push trigger to CI pipeline + +**Files:** +- Modify: `infra/k8s/argo/ci-pipeline-template.yaml` + +- [ ] **Step 1: Add `ml-changed` to detect-changes** + +In `ci-pipeline-template.yaml`, inside the `detect-changes` template's shell script (around line 199), add: + +```bash +ML_CHANGED=$(check_paths "crates/ml crates/ml-dqn crates/ml-core crates/ml-regime crates/ml-features crates/ml-supervised crates/ml-ppo crates/ml-ensemble crates/ml-hyperopt crates/ml-labeling") +``` + +And add the output file write (around line 218): +```bash +echo -n "$ML_CHANGED" > /tmp/outputs/ml-changed +``` + +And add the log line (around line 210): +```bash +echo "ml-changed: $ML_CHANGED" +``` + +- [ ] **Step 2: Add ml-changed output parameter** + +In the `outputs.parameters` section of `detect-changes` (around line 238), add: +```yaml +- name: ml-changed + valueFrom: + path: /tmp/outputs/ml-changed +``` + +- [ ] **Step 3: Add gpu-test DAG task** + +In the `pipeline` DAG `tasks` section (around line 143), add: +```yaml +- name: gpu-test + depends: "detect-changes" + templateRef: + name: gpu-test-pipeline + template: pipeline + arguments: + parameters: + - name: commit-ref + value: "{{workflow.parameters.commit-sha}}" + - name: models + value: "dqn,ppo,tft" + - name: epochs + value: "3" + when: "{{tasks.detect-changes.outputs.parameters.ml-changed}} == true" +``` + +- [ ] **Step 4: Validate YAML syntax** + +```bash +python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))" && echo "YAML OK" +``` + +- [ ] **Step 5: Commit** + +```bash +git add infra/k8s/argo/ci-pipeline-template.yaml +git commit -m "feat(ci): add gpu-test on-push trigger gated on ML crate changes" +``` + +--- + +## Task 8: Apply to cluster and validate + +- [ ] **Step 1: Apply PVCs** + +```bash +kubectl -n foxhunt apply -f infra/k8s/argo/cargo-target-cuda-test-pvc.yaml +kubectl -n foxhunt apply -f infra/k8s/training/test-data-pvc.yaml +``` + +- [ ] **Step 2: Populate test data** + +```bash +./scripts/refresh-test-data.sh +``` + +Wait for job completion. Verify data: +```bash +kubectl -n foxhunt exec -it $(kubectl -n foxhunt get pods -l app.kubernetes.io/name=populate-test-data -o name | head -1) -- ls -lhR /test-data/ +``` + +- [ ] **Step 3: Apply WorkflowTemplate and CronWorkflow** + +```bash +kubectl -n foxhunt apply -f infra/k8s/argo/gpu-test-pipeline-template.yaml +kubectl -n foxhunt apply -f infra/k8s/argo/gpu-test-nightly-cron.yaml +``` + +- [ ] **Step 4: Run a quick manual test (DQN lib only)** + +```bash +./scripts/argo-test.sh --models dqn --scope lib --watch +``` + +Expected: compile + DQN lib tests pass on H100. + +- [ ] **Step 5: Run full default suite** + +```bash +./scripts/argo-test.sh --watch +``` + +Expected: DQN + PPO + TFT tests all pass. + +- [ ] **Step 6: Apply updated CI pipeline** + +```bash +kubectl -n foxhunt apply -f infra/k8s/argo/ci-pipeline-template.yaml +``` + +- [ ] **Step 7: Final commit (if any fixes needed)** + +```bash +git add -A +git commit -m "fix(ci): adjustments from H100 GPU test validation" +```