foxhunt/docs/superpowers/specs/2026-03-13-gpu-test-workflow-design.md

# H100 GPU Test Workflow — Design Spec

**Date**: 2026-03-13
**Status**: Approved (rev 2 — post spec-review fixes)
**Scope**: Argo WorkflowTemplate for compile + GPU/CUDA test execution on H100

## Problem

GPU smoke tests take 42+ minutes on a 4GB RTX 3050 Ti in debug mode and OOM on multi-test concurrency. Full model coverage (DQN, PPO, 8 supervised models) is impossible locally. We need a fast, reproducible GPU test path on H100 that validates CUDA hot-path correctness with real market data.

## Decision Summary

| Decision | Choice |
|----------|--------|
| Test scope | Parameterized — all model families by default, selectable subset |
| Test data | Dedicated `test-data-pvc` (10Gi) with curated subset |
| PVC strategy | Two new PVCs — zero contention with training workflows |
| Compile strategy | Compile + test on H100 (single pod, same node = no binary transfer) |
| Triggers | Manual (active) + on-push gated (active) + nightly (suspended) |

## 1. Test Data Infrastructure

### 1.1 test-data-pvc

New PersistentVolumeClaim:
- **Name**: `test-data-pvc`
- **Namespace**: `foxhunt`
- **Size**: 10Gi
- **StorageClass**: `scw-bssd`
- **AccessMode**: `ReadWriteOnce`

Separate from `training-data-pvc` — no RWO contention with training workflows.

### 1.2 Data Layout

```
/data/test-data/
├── ohlcv/ES.FUT/ES.FUT_2025-Q2.dbn.zst      # ~2MB — OHLCV bars
├── mbp10/ES.FUT/ES.FUT_2025-Q2.dbn.zst       # ~126MB — orderbook snapshots
└── trades/ES.FUT/ES.FUT_2025-Q2.dbn.zst      # ~300MB — tick trades
```

**~430MB total** — ES.FUT 2025-Q2 across all three schemas. Exercises every code path: feature extraction, OFI calculation, VPIN, regime classification, walk-forward windows.

### 1.3 Setup Job

`infra/k8s/training/populate-test-data-job.yaml` — one-time Kubernetes Job that:
1. Runs on H100 node pool (same node as test-data-pvc to avoid RWO cross-node)
2. Mounts `training-data-pvc` read-only at `/data/training`
3. Mounts `test-data-pvc` read-write at `/data/test-data`
4. Copies ES.FUT 2025-Q2 files from each schema directory
5. Validates file integrity (size check, dbn header parse)

**Node affinity**: Must schedule on the same node pool as the GPU test workflow (`ci-training-h100`) to ensure the RWO PVC stays on the correct node.

Re-runnable via `scripts/refresh-test-data.sh`.

### 1.4 Test Code Integration

Tests discover data via `TEST_DATA_DIR` environment variable:
- **H100 workflow**: `TEST_DATA_DIR=/data/test-data`
- **Local dev**: Falls back to existing `data/smoke-test/` relative path discovery

**Required code change**: Add `TEST_DATA_DIR` env var lookup as primary path in `try_data()` / `get_dbn_data_dir()` / `get_es_fut_data_dir()` test helpers. This is a prerequisite before the workflow can run.

## 2. Workflow Architecture

### 2.1 WorkflowTemplate: `gpu-test-pipeline`

File: `infra/k8s/argo/gpu-test-pipeline-template.yaml`

DAG structure:
```
gpu-warmup ──────────┐
                     ├──> compile-and-test (H100, single pod)
                     │
notify-result (onExit, always runs)
```

**Key design decision**: Compile AND test run in the **same pod** on the H100. This eliminates:
- Cross-node PVC transfer (BLOCKER in rev 1)
- Non-deterministic test binary path discovery
- GitLab packages upload/download for test binaries
- Complexity of separating compile artifacts from test harness

Cost overhead: ~5 min of H100 time for compilation (~$0.25/run). Acceptable given the simplicity gain.

### 2.2 cargo-target-cuda-test PVC

New PersistentVolumeClaim — **independent from training's `cargo-target-cuda`**:
- **Name**: `cargo-target-cuda-test`
- **Namespace**: `foxhunt`
- **Size**: 30Gi
- **StorageClass**: `scw-bssd-retain` (survives PVC deletion, matches sccache pattern)
- **AccessMode**: `ReadWriteOnce`

Contains persistent git checkout + cargo target directory (same pattern as `cargo-target-cuda` in compile-and-train). Only changed files get new mtimes → cargo skips recompiling unchanged crates.

**Zero contention**: Training workflows use `cargo-target-cuda`, test workflows use `cargo-target-cuda-test`. Neither blocks the other.

### 2.3 Compile-and-Test Step

Single pod on `ci-training-h100`:

- **Image**: `ci-builder` (CUDA 12.9 + Rust 1.89 + mold)
- **Node pool**: `ci-training-h100`
- **Resources**:
  ```yaml
  requests:
    nvidia.com/gpu: "1"
    cpu: "4"
    memory: 32Gi
  limits:
    nvidia.com/gpu: "1"
    cpu: "8"
    memory: 64Gi
  ```
- **Tolerations**:
  ```yaml
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
  - key: node.cilium.io/agent-not-ready
    operator: Exists
    effect: NoSchedule
  ```
- **Volumes**:
  - `cargo-target-cuda-test` PVC at `/cargo-target` — persistent checkout + incremental build
  - `test-data-pvc` PVC read-only at `/data/test-data`
  - `git-ssh-key` secret at `/etc/git-ssh`
- **Environment**:
  ```bash
  SQLX_OFFLINE=true
  CARGO_TARGET_DIR=/cargo-target
  CARGO_HOME=/cargo-target/cargo-home
  CARGO_TERM_COLOR=always
  CUDA_COMPUTE_CAP=90          # H100 Hopper architecture
  CUDA_VISIBLE_DEVICES=0
  CUBLAS_WORKSPACE_CONFIG=:4096:8  # Deterministic cuBLAS for reproducible tests
  TEST_DATA_DIR=/data/test-data
  RUST_LOG=info
  ```
- **Execution flow**:
  ```bash
  # 1. Checkout (persistent on PVC, incremental)
  git fetch origin && git checkout --force $SHA

  # 2. Compile test binaries (incremental, ~5-6 min warm)
  cargo test -p ml -p ml-dqn -p ml-core --features cuda --no-run

  # 3. Run tests sequentially per model (see section 4)
  # Captures per-model exit codes, continues on failure
  ```
- **Expected duration**: ~5-6 min compile (warm) + ~15-20 min tests = **~20-25 min total**

### 2.4 GPU Warmup Step

Runs in parallel with nothing (triggers H100 autoscale while the workflow is starting):

```yaml
- name: gpu-warmup
  nodeSelector:
    k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
    - key: node.cilium.io/agent-not-ready
      operator: Exists
      effect: NoSchedule
  container:
    image: busybox:1.37
    command: ["/bin/sh", "-c", "echo 'GPU warmup: node scheduled' && exit 0"]
    resources:
      requests:
        nvidia.com/gpu: "1"
        cpu: 100m
        memory: 64Mi
      limits:
        nvidia.com/gpu: "1"
        cpu: 200m
        memory: 128Mi
```

Same pattern as `compile-and-train-template.yaml:265-291`. Saves 3-5 min of autoscale wait.

### 2.5 Notify Step (onExit)

Uses `onExit` handler (NOT a DAG dependency) — **always runs** regardless of test success/failure:

```yaml
onExit: notify-result
```

Same pattern as `compile-and-train-template.yaml:22` and `ci-pipeline-template.yaml:13`. Posts per-model pass/fail summary to Mattermost webhook.

### 2.6 Workflow Metadata

```yaml
securityContext:
  fsGroup: 0                          # Match compile-and-train
podGC:
  strategy: OnPodCompletion           # Match ci-pipeline
ttlStrategy:
  secondsAfterCompletion: 3600        # 1 hour cleanup
activeDeadlineSeconds: 5400           # 90 min hard limit (generous for 25 min expected)
serviceAccountName: argo-workflow
```

## 3. Parameterization

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `models` | string | `dqn,ppo,tft` | Comma-separated model list |
| `test-scope` | enum | `all` | `lib`, `integration`, or `all` |
| `epochs` | int | `3` | Training epochs for e2e smoke tests |
| `commit-ref` | string | `HEAD` | Git ref to compile and test (SHA, branch, or tag) |
| `gpu-pool` | string | `ci-training-h100` | GPU node pool name |
| `cuda-compute-cap` | string | `90` | CUDA compute capability (90=H100) |
| `notify` | bool | `true` | Send Mattermost notification |

**Full model list**: `dqn,ppo,tft,mamba2,tggn,tlob,liquid,kan,xlstm,diffusion`

**Special value**: `models=all` expands to the full 10-model list.

## 4. Test Matrix

### 4.1 Execution Strategy

Tests run sequentially per model family in a single pod. **Continue on failure**: each model's tests run regardless of previous model failures. Exit codes are captured per-model and aggregated at the end.

```bash
RESULTS=""
run_model_tests() {
  MODEL="$1"; shift
  echo "=== Testing: $MODEL ==="
  "$@" --nocapture 2>&1 | tee "/tmp/test-${MODEL}.log"
  EXIT=${PIPESTATUS[0]}
  RESULTS="${RESULTS}${MODEL}:${EXIT}\n"
  return 0  # Always continue
}
```

### 4.2 DQN Tests

```bash
# lib tests
run_model_tests dqn-lib cargo test -p ml-dqn --features cuda --lib
run_model_tests dqn-ml-lib cargo test -p ml --features cuda --lib -- dqn

# integration tests (if scope=integration or scope=all)
run_model_tests dqn-smoke cargo test -p ml --features cuda --test smoke_test_real_data
run_model_tests dqn-pipeline cargo test -p ml --features cuda --test dqn_training_pipeline_test
run_model_tests dqn-smoke-train cargo test -p ml --features cuda --test dqn_training_smoke_test
run_model_tests dqn-early-stop cargo test -p ml --features cuda --test dqn_early_stopping_termination_test
run_model_tests dqn-collapse cargo test -p ml --features cuda --test dqn_action_collapse_fix_test
```

### 4.3 PPO Tests

```bash
run_model_tests ppo-lib cargo test -p ml --features cuda --lib -- ppo
run_model_tests ppo-barrier cargo test -p ml --features cuda --test barrier_optimization_test
```

### 4.4 Supervised Model Tests

Per model (tft, mamba2, tggn, tlob, liquid, kan, xlstm, diffusion):
```bash
run_model_tests ${MODEL}-lib cargo test -p ml --features cuda --lib -- $MODEL
run_model_tests ${MODEL}-integ cargo test -p ml --features cuda --test "${MODEL}_*" 2>/dev/null || true
```

### 4.5 Shared/Core Tests

Always run regardless of model selection:
```bash
run_model_tests core cargo test -p ml-core --features cuda --lib
run_model_tests bayesian cargo test -p ml --features cuda --test bayesian_changepoint_test
```

### 4.6 Final Aggregation

```bash
# Print summary
echo "=== GPU TEST RESULTS ==="
printf "$RESULTS" | while IFS=: read model exit; do
  [ "$exit" = "0" ] && STATUS="PASS" || STATUS="FAIL"
  printf "  %-20s %s\n" "$model" "$STATUS"
done

# Exit with failure if ANY model failed
printf "$RESULTS" | grep -v ":0$" && exit 1 || exit 0
```

## 5. Trigger Configuration

### 5.1 Manual

`scripts/argo-test.sh` — CLI wrapper for `argo submit`:

```bash
# Full suite (default)
./scripts/argo-test.sh

# Quick DQN-only
./scripts/argo-test.sh --models dqn --scope lib

# All 10 models, 5 epochs
./scripts/argo-test.sh --models all --epochs 5

# Watch logs
./scripts/argo-test.sh --models dqn,ppo --watch
```

**Status**: Active from day one.

### 5.2 On-Push (Gated)

New DAG task in `ci-pipeline-template.yaml`:

```yaml
- name: gpu-test
  depends: "detect-changes"
  templateRef:
    name: gpu-test-pipeline
    template: pipeline
  arguments:
    parameters:
      - name: commit-ref
        value: "{{workflow.parameters.commit-sha}}"
      - name: models
        value: "dqn,ppo,tft"
      - name: epochs
        value: "3"
  when: "{{tasks.detect-changes.outputs.parameters.ml-changed}} == true"
```

**Change detection**: New `ml-changed` output added to `detect-changes` template. `true` when `git diff` includes files matching:
- `crates/ml/**` (includes `src/cuda_pipeline/*.cu` and `*.cuh`)
- `crates/ml-dqn/**`
- `crates/ml-core/**`
- `crates/ml-regime/**`
- `crates/ml-features/**`
- `crates/ml-supervised/**`
- `crates/ml-ppo/**`
- `crates/ml-ensemble/**`
- `crates/ml-hyperopt/**`
- `crates/ml-labeling/**`

**Implementation**: Add to the `check_paths` section of `detect-changes`:
```bash
ML_CHANGED=$(check_paths "crates/ml crates/ml-dqn crates/ml-core crates/ml-regime crates/ml-features crates/ml-supervised crates/ml-ppo crates/ml-ensemble crates/ml-hyperopt crates/ml-labeling")
echo -n "$ML_CHANGED" > /tmp/outputs/ml-changed
```

And add the output parameter:
```yaml
- name: ml-changed
  valueFrom:
    path: /tmp/outputs/ml-changed
```

**Status**: Active from day one. Gated — zero cost when non-ML code changes.

### 5.3 Nightly

`infra/k8s/argo/gpu-test-nightly-cron.yaml`:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: gpu-test-nightly
  namespace: foxhunt
spec:
  schedule: "0 2 * * *"
  suspend: true  # Created but not active — enable when ready
  workflowSpec:
    workflowTemplateRef:
      name: gpu-test-pipeline
    arguments:
      parameters:
        - name: models
          value: "dqn,ppo,tft,mamba2,tggn,tlob,liquid,kan,xlstm,diffusion"
        - name: epochs
          value: "5"
        - name: commit-ref
          value: "main"
```

**Status**: Created but `suspend: true`. Enable with:
```bash
kubectl patch cronworkflow gpu-test-nightly -n foxhunt -p '{"spec":{"suspend":false}}'
```

## 6. File Inventory

### New files to create

| File | Purpose |
|------|---------|
| `infra/k8s/argo/gpu-test-pipeline-template.yaml` | Main WorkflowTemplate |
| `infra/k8s/argo/gpu-test-nightly-cron.yaml` | CronWorkflow (suspended) |
| `infra/k8s/argo/cargo-target-cuda-test-pvc.yaml` | 30Gi build cache PVC |
| `infra/k8s/training/test-data-pvc.yaml` | 10Gi test data PVC |
| `infra/k8s/training/populate-test-data-job.yaml` | One-time data copy job |
| `scripts/argo-test.sh` | Manual trigger CLI |
| `scripts/refresh-test-data.sh` | Re-run populate job |

### Files to modify

| File | Change |
|------|--------|
| `infra/k8s/argo/ci-pipeline-template.yaml` | Add `gpu-test` DAG task + `ml-changed` output to detect-changes |
| `crates/ml/tests/smoke_test_real_data.rs` | Add `TEST_DATA_DIR` env var to `try_data()` |
| `crates/ml/tests/dqn_training_pipeline_test.rs` | Add `TEST_DATA_DIR` env var to `get_es_fut_data_dir()` |
| `crates/ml/tests/dqn_training_smoke_test.rs` | Add `TEST_DATA_DIR` env var to `get_dbn_data_dir()` |
| `crates/ml/tests/dqn_early_stopping_termination_test.rs` | Add `TEST_DATA_DIR` env var to `get_dbn_data_dir()` |

## 7. Cost Estimate

| Trigger | H100 time/run | Frequency | Monthly cost estimate |
|---------|--------------|-----------|----------------------|
| Manual | ~25 min | ~5/week | ~$10 |
| On-push (gated) | ~25 min | ~10/week | ~$20 |
| Nightly (when enabled) | ~35 min | 30/month | ~$50 |

Based on Scaleway H100 pricing for `ci-training-h100` node pool. Includes ~5 min compile overhead on GPU node (simpler architecture vs CPU-compile + binary transfer).

## 8. Blocker Resolutions (from rev 1 review)

| Blocker | Resolution |
|---------|-----------|
| Workspace PVC can't cross nodes (RWO) | Eliminated: compile + test in same H100 pod |
| `cargo-target-cuda` contention | New `cargo-target-cuda-test` PVC — zero sharing |
| sccache not configured | Removed: using cargo incremental compilation (persistent checkout on PVC), matching `compile-and-train` pattern |
| Non-deterministic test binary paths | Eliminated: `cargo test` runs in same env that compiled — no binary discovery needed |
| No `activeDeadlineSeconds` | Added: 5400s (90 min) |
| No `onExit` handler | Added: `onExit: notify-result` |
| No `podGC` | Added: `OnPodCompletion` |
| Missing `fsGroup` | Added: `fsGroup: 0` |
| Missing `gpu-warmup` | Added: parallel warmup step |
| Incomplete change detection | Added: ml-ensemble, ml-hyperopt, ml-labeling paths |
| Missing `CUDA_COMPUTE_CAP` | Added: `cuda-compute-cap` parameter, default `90` |
| TEST_DATA_DIR not in test helpers | Marked as required prerequisite code change |
| No partial failure handling | Added: continue-on-failure with per-model exit code capture |

## 9. Success Criteria

1. `scripts/argo-test.sh` submits workflow and all default tests pass on H100
2. On-push trigger fires on ML crate changes, skips on non-ML changes
3. All 8 smoke_test_real_data tests pass with test-data-pvc data
4. DQN e2e training loop completes 3 epochs in <5 min (vs 42 min local)
5. Per-model pass/fail reported to Mattermost
6. No interference with training workflows (separate PVCs)
7. `activeDeadlineSeconds` kills stuck workflows within 90 min
8. Completed pods cleaned up via `podGC`