foxhunt/docs/superpowers/specs/2026-03-13-gpu-test-workflow-design.md

H100 GPU Test Workflow — Design Spec

Date: 2026-03-13 Status: Approved (rev 2 — post spec-review fixes) Scope: Argo WorkflowTemplate for compile + GPU/CUDA test execution on H100

Problem

GPU smoke tests take 42+ minutes on a 4GB RTX 3050 Ti in debug mode and OOM on multi-test concurrency. Full model coverage (DQN, PPO, 8 supervised models) is impossible locally. We need a fast, reproducible GPU test path on H100 that validates CUDA hot-path correctness with real market data.

Decision Summary

Decision	Choice
Test scope	Parameterized — all model families by default, selectable subset
Test data	Dedicated test-data-pvc (10Gi) with curated subset
PVC strategy	Two new PVCs — zero contention with training workflows
Compile strategy	Compile + test on H100 (single pod, same node = no binary transfer)
Triggers	Manual (active) + on-push gated (active) + nightly (suspended)

1. Test Data Infrastructure

1.1 test-data-pvc

New PersistentVolumeClaim:

• Name: test-data-pvc
• Namespace: foxhunt
• Size: 10Gi
• StorageClass: scw-bssd
• AccessMode: ReadWriteOnce

Separate from training-data-pvc — no RWO contention with training workflows.

1.2 Data Layout

/data/test-data/
├── ohlcv/ES.FUT/ES.FUT_2025-Q2.dbn.zst      # ~2MB — OHLCV bars
├── mbp10/ES.FUT/ES.FUT_2025-Q2.dbn.zst       # ~126MB — orderbook snapshots
└── trades/ES.FUT/ES.FUT_2025-Q2.dbn.zst      # ~300MB — tick trades

~430MB total — ES.FUT 2025-Q2 across all three schemas. Exercises every code path: feature extraction, OFI calculation, VPIN, regime classification, walk-forward windows.

1.3 Setup Job

infra/k8s/training/populate-test-data-job.yaml — one-time Kubernetes Job that:

1. Runs on H100 node pool (same node as test-data-pvc to avoid RWO cross-node)
2. Mounts training-data-pvc read-only at /data/training
3. Mounts test-data-pvc read-write at /data/test-data
4. Copies ES.FUT 2025-Q2 files from each schema directory
5. Validates file integrity (size check, dbn header parse)

Node affinity: Must schedule on the same node pool as the GPU test workflow (ci-training-h100) to ensure the RWO PVC stays on the correct node.

Re-runnable via scripts/refresh-test-data.sh.

1.4 Test Code Integration

Tests discover data via TEST_DATA_DIR environment variable:

• H100 workflow: TEST_DATA_DIR=/data/test-data
• Local dev: Falls back to existing data/smoke-test/ relative path discovery

Required code change: Add TEST_DATA_DIR env var lookup as primary path in try_data() / get_dbn_data_dir() / get_es_fut_data_dir() test helpers. This is a prerequisite before the workflow can run.

2. Workflow Architecture

2.1 WorkflowTemplate: gpu-test-pipeline

File: infra/k8s/argo/gpu-test-pipeline-template.yaml

DAG structure:

gpu-warmup ──────────┐
                     ├──> compile-and-test (H100, single pod)
                     │
notify-result (onExit, always runs)

Key design decision: Compile AND test run in the same pod on the H100. This eliminates:

• Cross-node PVC transfer (BLOCKER in rev 1)
• Non-deterministic test binary path discovery
• GitLab packages upload/download for test binaries
• Complexity of separating compile artifacts from test harness

Cost overhead: 5 min of H100 time for compilation ($0.25/run). Acceptable given the simplicity gain.

2.2 cargo-target-cuda-test PVC

New PersistentVolumeClaim — independent from training's cargo-target-cuda:

• Name: cargo-target-cuda-test
• Namespace: foxhunt
• Size: 30Gi
• StorageClass: scw-bssd-retain (survives PVC deletion, matches sccache pattern)
• AccessMode: ReadWriteOnce

Contains persistent git checkout + cargo target directory (same pattern as cargo-target-cuda in compile-and-train). Only changed files get new mtimes → cargo skips recompiling unchanged crates.

Zero contention: Training workflows use cargo-target-cuda, test workflows use cargo-target-cuda-test. Neither blocks the other.

2.3 Compile-and-Test Step

Single pod on ci-training-h100:

• Image: ci-builder (CUDA 12.9 + Rust 1.89 + mold)
• Node pool: ci-training-h100
• Resources:

  requests:
    nvidia.com/gpu: "1"
    cpu: "4"
    memory: 32Gi
  limits:
    nvidia.com/gpu: "1"
    cpu: "8"
    memory: 64Gi
  

• Tolerations:

  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
  - key: node.cilium.io/agent-not-ready
    operator: Exists
    effect: NoSchedule
  

• Volumes:
  • cargo-target-cuda-test PVC at /cargo-target — persistent checkout + incremental build
  • test-data-pvc PVC read-only at /data/test-data
  • git-ssh-key secret at /etc/git-ssh
• Environment:

  SQLX_OFFLINE=true
  CARGO_TARGET_DIR=/cargo-target
  CARGO_HOME=/cargo-target/cargo-home
  CARGO_TERM_COLOR=always
  CUDA_COMPUTE_CAP=90          # H100 Hopper architecture
  CUDA_VISIBLE_DEVICES=0
  CUBLAS_WORKSPACE_CONFIG=:4096:8  # Deterministic cuBLAS for reproducible tests
  TEST_DATA_DIR=/data/test-data
  RUST_LOG=info
  

• Execution flow:

  # 1. Checkout (persistent on PVC, incremental)
  git fetch origin && git checkout --force $SHA
  
  # 2. Compile test binaries (incremental, ~5-6 min warm)
  cargo test -p ml -p ml-dqn -p ml-core --features cuda --no-run
  
  # 3. Run tests sequentially per model (see section 4)
  # Captures per-model exit codes, continues on failure
  

• Expected duration: ~5-6 min compile (warm) + ~15-20 min tests = ~20-25 min total

2.4 GPU Warmup Step

Runs in parallel with nothing (triggers H100 autoscale while the workflow is starting):

- name: gpu-warmup
  nodeSelector:
    k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
    - key: node.cilium.io/agent-not-ready
      operator: Exists
      effect: NoSchedule
  container:
    image: busybox:1.37
    command: ["/bin/sh", "-c", "echo 'GPU warmup: node scheduled' && exit 0"]
    resources:
      requests:
        nvidia.com/gpu: "1"
        cpu: 100m
        memory: 64Mi
      limits:
        nvidia.com/gpu: "1"
        cpu: 200m
        memory: 128Mi

Same pattern as compile-and-train-template.yaml:265-291. Saves 3-5 min of autoscale wait.

2.5 Notify Step (onExit)

Uses onExit handler (NOT a DAG dependency) — always runs regardless of test success/failure:

onExit: notify-result

Same pattern as compile-and-train-template.yaml:22 and ci-pipeline-template.yaml:13. Posts per-model pass/fail summary to Mattermost webhook.

2.6 Workflow Metadata

securityContext:
  fsGroup: 0                          # Match compile-and-train
podGC:
  strategy: OnPodCompletion           # Match ci-pipeline
ttlStrategy:
  secondsAfterCompletion: 3600        # 1 hour cleanup
activeDeadlineSeconds: 5400           # 90 min hard limit (generous for 25 min expected)
serviceAccountName: argo-workflow

3. Parameterization

Parameter	Type	Default	Description
models	string	dqn,ppo,tft	Comma-separated model list
test-scope	enum	all	lib, integration, or all
epochs	int	3	Training epochs for e2e smoke tests
commit-ref	string	HEAD	Git ref to compile and test (SHA, branch, or tag)
gpu-pool	string	ci-training-h100	GPU node pool name
cuda-compute-cap	string	90	CUDA compute capability (90=H100)
notify	bool	true	Send Mattermost notification

Full model list: dqn,ppo,tft,mamba2,tggn,tlob,liquid,kan,xlstm,diffusion

Special value: models=all expands to the full 10-model list.

4. Test Matrix

4.1 Execution Strategy

Tests run sequentially per model family in a single pod. Continue on failure: each model's tests run regardless of previous model failures. Exit codes are captured per-model and aggregated at the end.

RESULTS=""
run_model_tests() {
  MODEL="$1"; shift
  echo "=== Testing: $MODEL ==="
  "$@" --nocapture 2>&1 | tee "/tmp/test-${MODEL}.log"
  EXIT=${PIPESTATUS[0]}
  RESULTS="${RESULTS}${MODEL}:${EXIT}\n"
  return 0  # Always continue
}

4.2 DQN Tests

# lib tests
run_model_tests dqn-lib cargo test -p ml-dqn --features cuda --lib
run_model_tests dqn-ml-lib cargo test -p ml --features cuda --lib -- dqn

# integration tests (if scope=integration or scope=all)
run_model_tests dqn-smoke cargo test -p ml --features cuda --test smoke_test_real_data
run_model_tests dqn-pipeline cargo test -p ml --features cuda --test dqn_training_pipeline_test
run_model_tests dqn-smoke-train cargo test -p ml --features cuda --test dqn_training_smoke_test
run_model_tests dqn-early-stop cargo test -p ml --features cuda --test dqn_early_stopping_termination_test
run_model_tests dqn-collapse cargo test -p ml --features cuda --test dqn_action_collapse_fix_test

4.3 PPO Tests

run_model_tests ppo-lib cargo test -p ml --features cuda --lib -- ppo
run_model_tests ppo-barrier cargo test -p ml --features cuda --test barrier_optimization_test

4.4 Supervised Model Tests

Per model (tft, mamba2, tggn, tlob, liquid, kan, xlstm, diffusion):

run_model_tests ${MODEL}-lib cargo test -p ml --features cuda --lib -- $MODEL
run_model_tests ${MODEL}-integ cargo test -p ml --features cuda --test "${MODEL}_*" 2>/dev/null || true

4.5 Shared/Core Tests

Always run regardless of model selection:

run_model_tests core cargo test -p ml-core --features cuda --lib
run_model_tests bayesian cargo test -p ml --features cuda --test bayesian_changepoint_test

4.6 Final Aggregation

# Print summary
echo "=== GPU TEST RESULTS ==="
printf "$RESULTS" | while IFS=: read model exit; do
  [ "$exit" = "0" ] && STATUS="PASS" || STATUS="FAIL"
  printf "  %-20s %s\n" "$model" "$STATUS"
done

# Exit with failure if ANY model failed
printf "$RESULTS" | grep -v ":0$" && exit 1 || exit 0

5. Trigger Configuration

5.1 Manual

scripts/argo-test.sh — CLI wrapper for argo submit:

# Full suite (default)
./scripts/argo-test.sh

# Quick DQN-only
./scripts/argo-test.sh --models dqn --scope lib

# All 10 models, 5 epochs
./scripts/argo-test.sh --models all --epochs 5

# Watch logs
./scripts/argo-test.sh --models dqn,ppo --watch

Status: Active from day one.

5.2 On-Push (Gated)

New DAG task in ci-pipeline-template.yaml:

- name: gpu-test
  depends: "detect-changes"
  templateRef:
    name: gpu-test-pipeline
    template: pipeline
  arguments:
    parameters:
      - name: commit-ref
        value: "{{workflow.parameters.commit-sha}}"
      - name: models
        value: "dqn,ppo,tft"
      - name: epochs
        value: "3"
  when: "{{tasks.detect-changes.outputs.parameters.ml-changed}} == true"

Change detection: New ml-changed output added to detect-changes template. true when git diff includes files matching:

• crates/ml/** (includes src/cuda_pipeline/*.cu and *.cuh)
• crates/ml-dqn/**
• crates/ml-core/**
• crates/ml-regime/**
• crates/ml-features/**
• crates/ml-supervised/**
• crates/ml-ppo/**
• crates/ml-ensemble/**
• crates/ml-hyperopt/**
• crates/ml-labeling/**

Implementation: Add to the check_paths section of detect-changes:

ML_CHANGED=$(check_paths "crates/ml crates/ml-dqn crates/ml-core crates/ml-regime crates/ml-features crates/ml-supervised crates/ml-ppo crates/ml-ensemble crates/ml-hyperopt crates/ml-labeling")
echo -n "$ML_CHANGED" > /tmp/outputs/ml-changed

And add the output parameter:

- name: ml-changed
  valueFrom:
    path: /tmp/outputs/ml-changed

Status: Active from day one. Gated — zero cost when non-ML code changes.

5.3 Nightly

infra/k8s/argo/gpu-test-nightly-cron.yaml:

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: gpu-test-nightly
  namespace: foxhunt
spec:
  schedule: "0 2 * * *"
  suspend: true  # Created but not active — enable when ready
  workflowSpec:
    workflowTemplateRef:
      name: gpu-test-pipeline
    arguments:
      parameters:
        - name: models
          value: "dqn,ppo,tft,mamba2,tggn,tlob,liquid,kan,xlstm,diffusion"
        - name: epochs
          value: "5"
        - name: commit-ref
          value: "main"

Status: Created but suspend: true. Enable with:

kubectl patch cronworkflow gpu-test-nightly -n foxhunt -p '{"spec":{"suspend":false}}'

6. File Inventory

New files to create

File	Purpose
infra/k8s/argo/gpu-test-pipeline-template.yaml	Main WorkflowTemplate
infra/k8s/argo/gpu-test-nightly-cron.yaml	CronWorkflow (suspended)
infra/k8s/argo/cargo-target-cuda-test-pvc.yaml	30Gi build cache PVC
infra/k8s/training/test-data-pvc.yaml	10Gi test data PVC
infra/k8s/training/populate-test-data-job.yaml	One-time data copy job
scripts/argo-test.sh	Manual trigger CLI
scripts/refresh-test-data.sh	Re-run populate job

Files to modify

File	Change
infra/k8s/argo/ci-pipeline-template.yaml	Add gpu-test DAG task + ml-changed output to detect-changes
crates/ml/tests/smoke_test_real_data.rs	Add TEST_DATA_DIR env var to try_data()
crates/ml/tests/dqn_training_pipeline_test.rs	Add TEST_DATA_DIR env var to get_es_fut_data_dir()
crates/ml/tests/dqn_training_smoke_test.rs	Add TEST_DATA_DIR env var to get_dbn_data_dir()
crates/ml/tests/dqn_early_stopping_termination_test.rs	Add TEST_DATA_DIR env var to get_dbn_data_dir()

7. Cost Estimate

Trigger	H100 time/run	Frequency	Monthly cost estimate
Manual	~25 min	~5/week	~$10
On-push (gated)	~25 min	~10/week	~$20
Nightly (when enabled)	~35 min	30/month	~$50

Based on Scaleway H100 pricing for ci-training-h100 node pool. Includes ~5 min compile overhead on GPU node (simpler architecture vs CPU-compile + binary transfer).

8. Blocker Resolutions (from rev 1 review)

Blocker	Resolution
Workspace PVC can't cross nodes (RWO)	Eliminated: compile + test in same H100 pod
cargo-target-cuda contention	New cargo-target-cuda-test PVC — zero sharing
sccache not configured	Removed: using cargo incremental compilation (persistent checkout on PVC), matching compile-and-train pattern
Non-deterministic test binary paths	Eliminated: cargo test runs in same env that compiled — no binary discovery needed
No activeDeadlineSeconds	Added: 5400s (90 min)
No onExit handler	Added: onExit: notify-result
No podGC	Added: OnPodCompletion
Missing fsGroup	Added: fsGroup: 0
Missing gpu-warmup	Added: parallel warmup step
Incomplete change detection	Added: ml-ensemble, ml-hyperopt, ml-labeling paths
Missing CUDA_COMPUTE_CAP	Added: cuda-compute-cap parameter, default 90
TEST_DATA_DIR not in test helpers	Marked as required prerequisite code change
No partial failure handling	Added: continue-on-failure with per-model exit code capture

9. Success Criteria

1. scripts/argo-test.sh submits workflow and all default tests pass on H100
2. On-push trigger fires on ML crate changes, skips on non-ML changes
3. All 8 smoke_test_real_data tests pass with test-data-pvc data
4. DQN e2e training loop completes 3 epochs in <5 min (vs 42 min local)
5. Per-model pass/fail reported to Mattermost
6. No interference with training workflows (separate PVCs)
7. activeDeadlineSeconds kills stuck workflows within 90 min
8. Completed pods cleaned up via podGC