docs: add GPU test workflow implementation plan (8 tasks)
Covers PVC creation, test data population, TEST_DATA_DIR wiring, WorkflowTemplate, CronWorkflow, argo-test.sh CLI, and CI integration. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
962
docs/superpowers/plans/2026-03-13-gpu-test-workflow.md
Normal file
962
docs/superpowers/plans/2026-03-13-gpu-test-workflow.md
Normal file
@@ -0,0 +1,962 @@
|
||||
# H100 GPU Test Workflow Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Argo WorkflowTemplate that compiles and runs the full GPU/CUDA test suite on H100 with real market data, triggered manually, on-push (gated), or nightly.
|
||||
|
||||
**Architecture:** Two new PVCs (`cargo-target-cuda-test` 30Gi, `test-data-pvc` 10Gi) isolate test infrastructure from training. Single H100 pod compiles with `--features cuda` then runs tests sequentially per model family. Continue-on-failure captures per-model results.
|
||||
|
||||
**Tech Stack:** Argo Workflows, Kubernetes, Rust/Cargo, CUDA 12.9, ci-builder image, bash
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-03-13-gpu-test-workflow-design.md`
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
### New Files (7)
|
||||
|
||||
| File | Responsibility |
|
||||
|------|---------------|
|
||||
| `infra/k8s/argo/cargo-target-cuda-test-pvc.yaml` | PVC for test compilation cache (30Gi) |
|
||||
| `infra/k8s/training/test-data-pvc.yaml` | PVC for curated test data subset (10Gi) |
|
||||
| `infra/k8s/training/populate-test-data-job.yaml` | One-time job: copy data from training PVC to test PVC |
|
||||
| `infra/k8s/argo/gpu-test-pipeline-template.yaml` | Main WorkflowTemplate: compile + test on H100 |
|
||||
| `infra/k8s/argo/gpu-test-nightly-cron.yaml` | CronWorkflow (suspended) for nightly runs |
|
||||
| `scripts/argo-test.sh` | CLI wrapper for manual `argo submit` |
|
||||
| `scripts/refresh-test-data.sh` | Re-run populate job |
|
||||
|
||||
### Modified Files (5)
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `infra/k8s/argo/ci-pipeline-template.yaml` | Add `ml-changed` output + `gpu-test` DAG task |
|
||||
| `crates/ml/tests/smoke_test_real_data.rs:99-130` | Add `TEST_DATA_DIR` to `smoke_data_dir()` and `try_data()` |
|
||||
| `crates/ml/tests/dqn_training_smoke_test.rs:94-105` | Add `TEST_DATA_DIR` to `get_dbn_data_dir()` |
|
||||
| `crates/ml/tests/dqn_training_pipeline_test.rs:100-117` | Add `TEST_DATA_DIR` to `get_es_fut_data_dir()` |
|
||||
| `crates/ml/tests/dqn_early_stopping_termination_test.rs:93-104` | Add `TEST_DATA_DIR` to `get_dbn_data_dir()` |
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Create PVCs
|
||||
|
||||
**Files:**
|
||||
- Create: `infra/k8s/argo/cargo-target-cuda-test-pvc.yaml`
|
||||
- Create: `infra/k8s/training/test-data-pvc.yaml`
|
||||
|
||||
- [ ] **Step 1: Create cargo-target-cuda-test PVC**
|
||||
|
||||
```yaml
|
||||
# infra/k8s/argo/cargo-target-cuda-test-pvc.yaml
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: cargo-target-cuda-test
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: cargo-target
|
||||
app.kubernetes.io/component: ci-cache
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: scw-bssd-retain
|
||||
resources:
|
||||
requests:
|
||||
storage: 30Gi
|
||||
```
|
||||
|
||||
Pattern matches `infra/k8s/argo/sccache-pvcs.yaml` (same `scw-bssd-retain` storage class).
|
||||
|
||||
- [ ] **Step 2: Create test-data PVC**
|
||||
|
||||
```yaml
|
||||
# infra/k8s/training/test-data-pvc.yaml
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: test-data-pvc
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: test-data
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: scw-bssd
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/k8s/argo/cargo-target-cuda-test-pvc.yaml infra/k8s/training/test-data-pvc.yaml
|
||||
git commit -m "infra: add cargo-target-cuda-test (30Gi) and test-data-pvc (10Gi)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Create populate-test-data job
|
||||
|
||||
**Files:**
|
||||
- Create: `infra/k8s/training/populate-test-data-job.yaml`
|
||||
- Create: `scripts/refresh-test-data.sh`
|
||||
|
||||
- [ ] **Step 1: Create the populate job**
|
||||
|
||||
This job runs on the H100 node pool (same as test-data-pvc) and copies one quarter of ES.FUT data per schema from the training PVC.
|
||||
|
||||
```yaml
|
||||
# infra/k8s/training/populate-test-data-job.yaml
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
generateName: populate-test-data-
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: populate-test-data
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
backoffLimit: 1
|
||||
ttlSecondsAfterFinished: 300
|
||||
template:
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: ci-training-h100
|
||||
tolerations:
|
||||
- key: nvidia.com/gpu
|
||||
operator: Exists
|
||||
effect: NoSchedule
|
||||
- key: node.cilium.io/agent-not-ready
|
||||
operator: Exists
|
||||
effect: NoSchedule
|
||||
containers:
|
||||
- name: populate
|
||||
image: busybox:1.37
|
||||
command: ["/bin/sh", "-c"]
|
||||
args:
|
||||
- |
|
||||
set -e
|
||||
echo "=== Populating test data PVC ==="
|
||||
|
||||
# Target directories
|
||||
mkdir -p /test-data/ohlcv/ES.FUT
|
||||
mkdir -p /test-data/mbp10/ES.FUT
|
||||
mkdir -p /test-data/trades/ES.FUT
|
||||
|
||||
# Copy one quarter per schema (2025-Q2)
|
||||
echo "Copying OHLCV..."
|
||||
cp /training-data/futures-baseline/ES.FUT/ES.FUT_2025-Q2.dbn.zst \
|
||||
/test-data/ohlcv/ES.FUT/ 2>/dev/null || \
|
||||
cp /training-data/futures-baseline/ES.FUT/*2025*Q2* \
|
||||
/test-data/ohlcv/ES.FUT/ 2>/dev/null || \
|
||||
{ echo "WARN: No 2025-Q2 OHLCV found, copying first available"; \
|
||||
cp /training-data/futures-baseline/ES.FUT/*.dbn.zst \
|
||||
/test-data/ohlcv/ES.FUT/ 2>/dev/null | head -1; }
|
||||
|
||||
echo "Copying MBP-10..."
|
||||
cp /training-data/futures-baseline-mbp10/ES.FUT/ES.FUT_2025-Q2.dbn.zst \
|
||||
/test-data/mbp10/ES.FUT/ 2>/dev/null || \
|
||||
cp /training-data/futures-baseline-mbp10/ES.FUT/*2025*Q2* \
|
||||
/test-data/mbp10/ES.FUT/ 2>/dev/null || \
|
||||
echo "WARN: No MBP-10 2025-Q2 found"
|
||||
|
||||
echo "Copying trades..."
|
||||
cp /training-data/futures-baseline-trades/ES.FUT/ES.FUT_2025-Q2.dbn.zst \
|
||||
/test-data/trades/ES.FUT/ 2>/dev/null || \
|
||||
cp /training-data/futures-baseline-trades/ES.FUT/*2025*Q2* \
|
||||
/test-data/trades/ES.FUT/ 2>/dev/null || \
|
||||
echo "WARN: No trades 2025-Q2 found"
|
||||
|
||||
echo "=== Test data contents ==="
|
||||
find /test-data -type f -exec ls -lh {} \;
|
||||
echo "Total: $(du -sh /test-data | cut -f1)"
|
||||
echo "=== Done ==="
|
||||
resources:
|
||||
requests:
|
||||
nvidia.com/gpu: "1"
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
nvidia.com/gpu: "1"
|
||||
cpu: 500m
|
||||
memory: 256Mi
|
||||
volumeMounts:
|
||||
- name: training-data
|
||||
mountPath: /training-data
|
||||
readOnly: true
|
||||
- name: test-data
|
||||
mountPath: /test-data
|
||||
volumes:
|
||||
- name: training-data
|
||||
persistentVolumeClaim:
|
||||
claimName: training-data-pvc
|
||||
readOnly: true
|
||||
- name: test-data
|
||||
persistentVolumeClaim:
|
||||
claimName: test-data-pvc
|
||||
```
|
||||
|
||||
Note: requests `nvidia.com/gpu: 1` only to schedule on the H100 node (same as test-data-pvc). GPU isn't used for the copy.
|
||||
|
||||
- [ ] **Step 2: Create refresh script**
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
# Re-populate the test-data-pvc from training-data-pvc.
|
||||
# Usage: ./scripts/refresh-test-data.sh
|
||||
set -euo pipefail
|
||||
|
||||
echo "Submitting populate-test-data job..."
|
||||
kubectl -n foxhunt create -f infra/k8s/training/populate-test-data-job.yaml
|
||||
|
||||
echo "Waiting for completion..."
|
||||
kubectl -n foxhunt wait --for=condition=complete job \
|
||||
-l app.kubernetes.io/name=populate-test-data \
|
||||
--timeout=300s
|
||||
|
||||
echo "Done. Test data PVC refreshed."
|
||||
```
|
||||
|
||||
Save to `scripts/refresh-test-data.sh`, chmod +x.
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
chmod +x scripts/refresh-test-data.sh
|
||||
git add infra/k8s/training/populate-test-data-job.yaml scripts/refresh-test-data.sh
|
||||
git commit -m "infra: add populate-test-data job and refresh script"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Wire TEST_DATA_DIR into test helpers
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/tests/smoke_test_real_data.rs:99-130`
|
||||
- Modify: `crates/ml/tests/dqn_training_smoke_test.rs:94-105`
|
||||
- Modify: `crates/ml/tests/dqn_training_pipeline_test.rs:100-117`
|
||||
- Modify: `crates/ml/tests/dqn_early_stopping_termination_test.rs:93-104`
|
||||
|
||||
- [ ] **Step 1: Update `smoke_data_dir()` and `try_data()` in smoke_test_real_data.rs**
|
||||
|
||||
Replace `smoke_data_dir()`:
|
||||
```rust
|
||||
fn smoke_data_dir() -> PathBuf {
|
||||
// CI: TEST_DATA_DIR points to test-data-pvc on H100
|
||||
if let Ok(dir) = std::env::var("TEST_DATA_DIR") {
|
||||
return PathBuf::from(dir);
|
||||
}
|
||||
// Local dev: relative path from workspace root
|
||||
workspace_root().join("data").join("smoke-test")
|
||||
}
|
||||
```
|
||||
|
||||
`try_data()` doesn't need changes — it already calls `smoke_data_dir()`.
|
||||
|
||||
- [ ] **Step 2: Update `get_dbn_data_dir()` in dqn_training_smoke_test.rs**
|
||||
|
||||
Replace lines 94-105:
|
||||
```rust
|
||||
fn get_dbn_data_dir() -> Result<String> {
|
||||
// CI: TEST_DATA_DIR points to test-data-pvc on H100
|
||||
if let Ok(dir) = std::env::var("TEST_DATA_DIR") {
|
||||
let ohlcv = std::path::PathBuf::from(&dir).join("ohlcv");
|
||||
if ohlcv.exists() {
|
||||
return Ok(ohlcv.to_string_lossy().to_string());
|
||||
}
|
||||
// Flat layout fallback
|
||||
return Ok(dir);
|
||||
}
|
||||
let workspace_root = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
|
||||
.ancestors()
|
||||
.find(|p| p.join("test_data").exists())
|
||||
.context("Failed to find workspace root with test_data/")?
|
||||
.to_path_buf();
|
||||
let data_dir = workspace_root.join("test_data/futures-baseline");
|
||||
if !data_dir.exists() {
|
||||
anyhow::bail!("DBN data not found: {}", data_dir.display());
|
||||
}
|
||||
Ok(data_dir.to_string_lossy().to_string())
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Update `get_es_fut_data_dir()` in dqn_training_pipeline_test.rs**
|
||||
|
||||
Replace lines 100-117:
|
||||
```rust
|
||||
fn get_es_fut_data_dir() -> Result<String> {
|
||||
if let Ok(dir) = std::env::var("TEST_DATA_DIR") {
|
||||
let ohlcv = std::path::PathBuf::from(&dir).join("ohlcv");
|
||||
if ohlcv.exists() {
|
||||
return Ok(ohlcv.to_string_lossy().to_string());
|
||||
}
|
||||
return Ok(dir);
|
||||
}
|
||||
let workspace_root = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
|
||||
.ancestors()
|
||||
.find(|p| p.join("test_data").exists())
|
||||
.context("Failed to find workspace root with test_data/")?
|
||||
.to_path_buf();
|
||||
let data_dir = workspace_root.join("test_data/futures-baseline");
|
||||
if !data_dir.exists() {
|
||||
anyhow::bail!(
|
||||
"ES.FUT data directory not found: {}. Run data acquisition first.",
|
||||
data_dir.display()
|
||||
);
|
||||
}
|
||||
Ok(data_dir.to_string_lossy().to_string())
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Update `get_dbn_data_dir()` in dqn_early_stopping_termination_test.rs**
|
||||
|
||||
Replace lines 93-104:
|
||||
```rust
|
||||
#[cfg(feature = "cuda")]
|
||||
fn get_dbn_data_dir() -> Option<String> {
|
||||
if let Ok(dir) = std::env::var("TEST_DATA_DIR") {
|
||||
let ohlcv = std::path::PathBuf::from(&dir).join("ohlcv");
|
||||
if ohlcv.exists() {
|
||||
return Some(ohlcv.to_string_lossy().to_string());
|
||||
}
|
||||
return Some(dir);
|
||||
}
|
||||
let workspace_root = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"))
|
||||
.ancestors()
|
||||
.find(|p| p.join("test_data").exists())?
|
||||
.to_path_buf();
|
||||
let data_dir = workspace_root.join("test_data/futures-baseline");
|
||||
if !data_dir.exists() {
|
||||
eprintln!("SKIP: DBN data not found: {}", data_dir.display());
|
||||
return None;
|
||||
}
|
||||
Some(data_dir.to_string_lossy().to_string())
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Verify compilation**
|
||||
|
||||
```bash
|
||||
SQLX_OFFLINE=true cargo test -p ml --features cuda --no-run 2>&1 | tail -5
|
||||
```
|
||||
|
||||
Expected: compiles with 0 errors.
|
||||
|
||||
- [ ] **Step 6: Verify local tests still work (env var NOT set)**
|
||||
|
||||
```bash
|
||||
unset TEST_DATA_DIR
|
||||
SQLX_OFFLINE=true cargo test -p ml --test smoke_test_real_data --features cuda -- smoke_ohlcv_parse_and_extract_features --nocapture 2>&1 | tail -5
|
||||
```
|
||||
|
||||
Expected: PASS (falls back to `data/smoke-test/` path).
|
||||
|
||||
- [ ] **Step 7: Verify TEST_DATA_DIR override works**
|
||||
|
||||
```bash
|
||||
TEST_DATA_DIR=/home/jgrusewski/Work/foxhunt/data/smoke-test \
|
||||
SQLX_OFFLINE=true cargo test -p ml --test smoke_test_real_data --features cuda -- smoke_ohlcv_parse_and_extract_features --nocapture 2>&1 | tail -5
|
||||
```
|
||||
|
||||
Expected: PASS (uses env var path).
|
||||
|
||||
- [ ] **Step 8: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/ml/tests/smoke_test_real_data.rs crates/ml/tests/dqn_training_smoke_test.rs \
|
||||
crates/ml/tests/dqn_training_pipeline_test.rs crates/ml/tests/dqn_early_stopping_termination_test.rs
|
||||
git commit -m "test: add TEST_DATA_DIR env var support for H100 CI test workflow"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Create GPU test WorkflowTemplate
|
||||
|
||||
**Files:**
|
||||
- Create: `infra/k8s/argo/gpu-test-pipeline-template.yaml`
|
||||
|
||||
This is the core workflow. Reference `compile-and-train-template.yaml` for patterns (git checkout, env vars, tolerations, notification).
|
||||
|
||||
- [ ] **Step 1: Create the WorkflowTemplate**
|
||||
|
||||
Create `infra/k8s/argo/gpu-test-pipeline-template.yaml` with:
|
||||
|
||||
**Header:**
|
||||
```yaml
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: WorkflowTemplate
|
||||
metadata:
|
||||
name: gpu-test-pipeline
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: gpu-test-pipeline
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
entrypoint: pipeline
|
||||
onExit: notify-result
|
||||
serviceAccountName: argo-workflow
|
||||
podMetadata:
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
app.kubernetes.io/component: gpu-test
|
||||
securityContext:
|
||||
fsGroup: 0
|
||||
podGC:
|
||||
strategy: OnPodCompletion
|
||||
ttlStrategy:
|
||||
secondsAfterCompletion: 3600
|
||||
activeDeadlineSeconds: 5400
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
```yaml
|
||||
arguments:
|
||||
parameters:
|
||||
- name: commit-ref
|
||||
value: HEAD
|
||||
- name: models
|
||||
value: "dqn,ppo,tft"
|
||||
- name: test-scope
|
||||
value: all
|
||||
- name: epochs
|
||||
value: "3"
|
||||
- name: gpu-pool
|
||||
value: ci-training-h100
|
||||
- name: cuda-compute-cap
|
||||
value: "90"
|
||||
- name: notify
|
||||
value: "true"
|
||||
```
|
||||
|
||||
**Volumes:**
|
||||
```yaml
|
||||
volumes:
|
||||
- name: git-ssh-key
|
||||
secret:
|
||||
secretName: argo-git-ssh-key
|
||||
defaultMode: 256
|
||||
- name: cargo-target
|
||||
persistentVolumeClaim:
|
||||
claimName: cargo-target-cuda-test
|
||||
- name: test-data
|
||||
persistentVolumeClaim:
|
||||
claimName: test-data-pvc
|
||||
readOnly: true
|
||||
```
|
||||
|
||||
**DAG:**
|
||||
```yaml
|
||||
templates:
|
||||
- name: pipeline
|
||||
dag:
|
||||
tasks:
|
||||
- name: gpu-warmup
|
||||
template: gpu-warmup
|
||||
- name: compile-and-test
|
||||
template: compile-and-test
|
||||
dependencies: [gpu-warmup]
|
||||
```
|
||||
|
||||
**gpu-warmup template:** Copy from `compile-and-train-template.yaml:266-291`, replace hardcoded pool with `{{workflow.parameters.gpu-pool}}`.
|
||||
|
||||
**compile-and-test template:** Single container on H100. The shell script:
|
||||
|
||||
1. SSH + git checkout (same pattern as compile-and-train lines 170-206)
|
||||
2. Set `CUDA_COMPUTE_CAP={{workflow.parameters.cuda-compute-cap}}`
|
||||
3. PVC size guard (same as compile-and-train lines 214-220)
|
||||
4. `cargo test -p ml -p ml-dqn -p ml-core --features cuda --no-run` (compile)
|
||||
5. Parse `models` parameter, loop over each model
|
||||
6. Per model: run `cargo test` commands, capture exit code, continue on failure
|
||||
7. Print summary table, exit with aggregate status
|
||||
|
||||
**Key env vars on the compile-and-test container:**
|
||||
```yaml
|
||||
env:
|
||||
- name: SQLX_OFFLINE
|
||||
value: "true"
|
||||
- name: CARGO_TERM_COLOR
|
||||
value: always
|
||||
- name: CARGO_TARGET_DIR
|
||||
value: /cargo-target
|
||||
- name: CARGO_HOME
|
||||
value: /cargo-target/cargo-home
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: "0"
|
||||
- name: CUBLAS_WORKSPACE_CONFIG
|
||||
value: ":4096:8"
|
||||
- name: TEST_DATA_DIR
|
||||
value: /data/test-data
|
||||
- name: RUST_LOG
|
||||
value: info
|
||||
- name: DQN_SMOKE_EPOCHS
|
||||
value: "{{workflow.parameters.epochs}}"
|
||||
```
|
||||
|
||||
**Key volume mounts:**
|
||||
```yaml
|
||||
volumeMounts:
|
||||
- name: git-ssh-key
|
||||
mountPath: /etc/git-ssh
|
||||
readOnly: true
|
||||
- name: cargo-target
|
||||
mountPath: /cargo-target
|
||||
- name: test-data
|
||||
mountPath: /data/test-data
|
||||
readOnly: true
|
||||
```
|
||||
|
||||
**Resources:**
|
||||
```yaml
|
||||
resources:
|
||||
requests:
|
||||
nvidia.com/gpu: "1"
|
||||
cpu: "4"
|
||||
memory: 32Gi
|
||||
limits:
|
||||
nvidia.com/gpu: "1"
|
||||
cpu: "8"
|
||||
memory: 64Gi
|
||||
```
|
||||
|
||||
**Shell script for compile-and-test (inline in args):**
|
||||
|
||||
```bash
|
||||
set -e
|
||||
|
||||
REF="{{workflow.parameters.commit-ref}}"
|
||||
MODELS="{{workflow.parameters.models}}"
|
||||
SCOPE="{{workflow.parameters.test-scope}}"
|
||||
|
||||
# --- SSH setup (same as compile-and-train) ---
|
||||
mkdir -p ~/.ssh
|
||||
cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
|
||||
chmod 600 ~/.ssh/id_ed25519
|
||||
printf 'Host *\n StrictHostKeyChecking no\n UserKnownHostsFile /dev/null\n' > ~/.ssh/config
|
||||
chmod 600 ~/.ssh/config
|
||||
|
||||
REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
|
||||
BUILD="/cargo-target/src"
|
||||
git config --global --add safe.directory "$BUILD"
|
||||
|
||||
# --- Persistent checkout on PVC (same as compile-and-train) ---
|
||||
if [ -d "$BUILD/.git" ]; then
|
||||
cd "$BUILD"
|
||||
CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
|
||||
TARGET=$(git ls-remote origin "$REF" 2>/dev/null | cut -f1 || echo "$REF")
|
||||
if [ "$CURRENT" = "$TARGET" ] || [ "$CURRENT" = "$REF" ]; then
|
||||
echo "=== Already at $REF ==="
|
||||
else
|
||||
echo "=== Updating checkout ==="
|
||||
git fetch origin
|
||||
git checkout --force "$REF"
|
||||
git clean -fd
|
||||
fi
|
||||
else
|
||||
echo "=== Initial clone ==="
|
||||
git clone --filter=blob:none "$REPO" "$BUILD"
|
||||
cd "$BUILD"
|
||||
git checkout "$REF"
|
||||
fi
|
||||
echo "Checked out $(git rev-parse --short HEAD)"
|
||||
|
||||
export PATH="${CARGO_HOME}/bin:${PATH}"
|
||||
export CUDA_COMPUTE_CAP={{workflow.parameters.cuda-compute-cap}}
|
||||
|
||||
# --- PVC size guard ---
|
||||
TARGET_SIZE_MB=$(du -sm "$CARGO_TARGET_DIR" 2>/dev/null | cut -f1 || echo 0)
|
||||
echo "PVC usage: ${TARGET_SIZE_MB}MB"
|
||||
if [ "$TARGET_SIZE_MB" -gt 25000 ]; then
|
||||
echo "PVC exceeds 25GB, pruning..."
|
||||
rm -rf "$CARGO_TARGET_DIR/release" "$CARGO_TARGET_DIR/debug"
|
||||
fi
|
||||
|
||||
# --- Compile ---
|
||||
echo "=== Compiling test binaries (--features cuda) ==="
|
||||
cargo test -p ml -p ml-dqn -p ml-core --features cuda --no-run 2>&1
|
||||
echo "=== Compilation done ==="
|
||||
|
||||
# --- Expand "all" ---
|
||||
if [ "$MODELS" = "all" ]; then
|
||||
MODELS="dqn,ppo,tft,mamba2,tggn,tlob,liquid,kan,xlstm,diffusion"
|
||||
fi
|
||||
|
||||
# --- Test runner ---
|
||||
RESULTS=""
|
||||
FAILURES=0
|
||||
|
||||
run_tests() {
|
||||
local NAME="$1"; shift
|
||||
echo ""
|
||||
echo "========================================"
|
||||
echo " TEST: $NAME"
|
||||
echo "========================================"
|
||||
set +e
|
||||
"$@" --nocapture 2>&1
|
||||
EXIT=$?
|
||||
set -e
|
||||
if [ $EXIT -eq 0 ]; then
|
||||
RESULTS="${RESULTS}${NAME}:PASS\n"
|
||||
else
|
||||
RESULTS="${RESULTS}${NAME}:FAIL\n"
|
||||
FAILURES=$((FAILURES + 1))
|
||||
fi
|
||||
}
|
||||
|
||||
# --- Core tests (always run) ---
|
||||
run_tests "core-lib" cargo test -p ml-core --features cuda --lib
|
||||
|
||||
# --- Per-model tests ---
|
||||
IFS=',' read -ra MODEL_LIST <<< "$MODELS"
|
||||
for MODEL in "${MODEL_LIST[@]}"; do
|
||||
case "$MODEL" in
|
||||
dqn)
|
||||
if [ "$SCOPE" = "lib" ] || [ "$SCOPE" = "all" ]; then
|
||||
run_tests "dqn-lib" cargo test -p ml-dqn --features cuda --lib
|
||||
run_tests "dqn-ml-lib" cargo test -p ml --features cuda --lib -- dqn
|
||||
fi
|
||||
if [ "$SCOPE" = "integration" ] || [ "$SCOPE" = "all" ]; then
|
||||
run_tests "dqn-smoke" cargo test -p ml --features cuda --test smoke_test_real_data
|
||||
run_tests "dqn-pipeline" cargo test -p ml --features cuda --test dqn_training_pipeline_test
|
||||
run_tests "dqn-smoke-train" cargo test -p ml --features cuda --test dqn_training_smoke_test
|
||||
run_tests "dqn-early-stop" cargo test -p ml --features cuda --test dqn_early_stopping_termination_test
|
||||
run_tests "dqn-collapse" cargo test -p ml --features cuda --test dqn_action_collapse_fix_test
|
||||
fi
|
||||
;;
|
||||
ppo)
|
||||
if [ "$SCOPE" = "lib" ] || [ "$SCOPE" = "all" ]; then
|
||||
run_tests "ppo-lib" cargo test -p ml --features cuda --lib -- ppo
|
||||
fi
|
||||
if [ "$SCOPE" = "integration" ] || [ "$SCOPE" = "all" ]; then
|
||||
run_tests "ppo-barrier" cargo test -p ml --features cuda --test barrier_optimization_test
|
||||
fi
|
||||
;;
|
||||
*)
|
||||
# Supervised models
|
||||
if [ "$SCOPE" = "lib" ] || [ "$SCOPE" = "all" ]; then
|
||||
run_tests "${MODEL}-lib" cargo test -p ml --features cuda --lib -- "$MODEL"
|
||||
fi
|
||||
if [ "$SCOPE" = "integration" ] || [ "$SCOPE" = "all" ]; then
|
||||
# Run model-specific integration tests if they exist (non-fatal if none)
|
||||
run_tests "${MODEL}-integ" cargo test -p ml --features cuda --test "${MODEL}_*" 2>/dev/null || true
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# --- Summary ---
|
||||
echo ""
|
||||
echo "========================================"
|
||||
echo " GPU TEST RESULTS"
|
||||
echo "========================================"
|
||||
printf "$RESULTS" | while IFS=: read -r name status; do
|
||||
[ -z "$name" ] && continue
|
||||
printf " %-25s %s\n" "$name" "$status"
|
||||
done
|
||||
echo "========================================"
|
||||
echo " Total failures: $FAILURES"
|
||||
echo "========================================"
|
||||
|
||||
# Write results for notification step
|
||||
mkdir -p /tmp/outputs
|
||||
printf "$RESULTS" > /tmp/outputs/results
|
||||
echo "$FAILURES" > /tmp/outputs/failures
|
||||
|
||||
exit $FAILURES
|
||||
```
|
||||
|
||||
**Outputs on compile-and-test:**
|
||||
```yaml
|
||||
outputs:
|
||||
parameters:
|
||||
- name: results
|
||||
valueFrom:
|
||||
path: /tmp/outputs/results
|
||||
default: "unknown:FAIL"
|
||||
- name: failures
|
||||
valueFrom:
|
||||
path: /tmp/outputs/failures
|
||||
default: "1"
|
||||
```
|
||||
|
||||
**notify-result template:** Same pattern as `compile-and-train-template.yaml:586-627` and `ci-pipeline-template.yaml:553-594`. Use `{{workflow.status}}` and `{{workflow.name}}`. Gated on `notify` parameter.
|
||||
|
||||
- [ ] **Step 2: Validate YAML syntax**
|
||||
|
||||
```bash
|
||||
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/gpu-test-pipeline-template.yaml'))" && echo "YAML OK"
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/k8s/argo/gpu-test-pipeline-template.yaml
|
||||
git commit -m "feat(ci): add gpu-test-pipeline WorkflowTemplate for H100 CUDA testing"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Create nightly CronWorkflow
|
||||
|
||||
**Files:**
|
||||
- Create: `infra/k8s/argo/gpu-test-nightly-cron.yaml`
|
||||
|
||||
- [ ] **Step 1: Create CronWorkflow (suspended)**
|
||||
|
||||
```yaml
|
||||
# infra/k8s/argo/gpu-test-nightly-cron.yaml
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: CronWorkflow
|
||||
metadata:
|
||||
name: gpu-test-nightly
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: gpu-test-nightly
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
schedule: "0 2 * * *"
|
||||
timezone: "UTC"
|
||||
suspend: true # Enable when ready: kubectl patch cronworkflow gpu-test-nightly -n foxhunt -p '{"spec":{"suspend":false}}'
|
||||
concurrencyPolicy: Replace
|
||||
successfulJobsHistoryLimit: 3
|
||||
failedJobsHistoryLimit: 5
|
||||
workflowSpec:
|
||||
workflowTemplateRef:
|
||||
name: gpu-test-pipeline
|
||||
arguments:
|
||||
parameters:
|
||||
- name: models
|
||||
value: "dqn,ppo,tft,mamba2,tggn,tlob,liquid,kan,xlstm,diffusion"
|
||||
- name: epochs
|
||||
value: "5"
|
||||
- name: commit-ref
|
||||
value: "main"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/k8s/argo/gpu-test-nightly-cron.yaml
|
||||
git commit -m "feat(ci): add gpu-test-nightly CronWorkflow (suspended)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Create argo-test.sh CLI
|
||||
|
||||
**Files:**
|
||||
- Create: `scripts/argo-test.sh`
|
||||
|
||||
- [ ] **Step 1: Create the script**
|
||||
|
||||
Pattern matches `scripts/argo-train.sh` (same argument parsing style, same `argo submit` invocation).
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
# Run GPU/CUDA tests via Argo Workflows on H100.
|
||||
#
|
||||
# Usage:
|
||||
# ./scripts/argo-test.sh # defaults: dqn,ppo,tft
|
||||
# ./scripts/argo-test.sh --models dqn --scope lib # DQN lib tests only
|
||||
# ./scripts/argo-test.sh --models all --epochs 5 # all 10 models, 5 epochs
|
||||
# ./scripts/argo-test.sh --watch # follow logs
|
||||
#
|
||||
# Requires: argo CLI
|
||||
set -euo pipefail
|
||||
|
||||
MODELS="dqn,ppo,tft"
|
||||
SCOPE="all"
|
||||
EPOCHS="3"
|
||||
COMMIT_REF="HEAD"
|
||||
GPU_POOL=""
|
||||
WATCH=false
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: $(basename "$0") [OPTIONS]
|
||||
|
||||
Options:
|
||||
--models <list> Comma-separated model list (default: dqn,ppo,tft)
|
||||
Use "all" for all 10 models
|
||||
--scope <scope> lib, integration, or all (default: all)
|
||||
--epochs <n> Training epochs for e2e tests (default: 3)
|
||||
--ref <ref> Git ref to test (default: HEAD)
|
||||
--gpu-pool <pool> GPU node pool (default: ci-training-h100)
|
||||
--watch Follow workflow logs
|
||||
-h, --help Show this help
|
||||
EOF
|
||||
exit 0
|
||||
}
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--models) MODELS="$2"; shift 2 ;;
|
||||
--scope) SCOPE="$2"; shift 2 ;;
|
||||
--epochs) EPOCHS="$2"; shift 2 ;;
|
||||
--ref) COMMIT_REF="$2"; shift 2 ;;
|
||||
--gpu-pool) GPU_POOL="$2"; shift 2 ;;
|
||||
--watch) WATCH=true; shift ;;
|
||||
-h|--help) usage ;;
|
||||
*) echo "Unknown option: $1"; usage ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Resolve HEAD to actual SHA if needed
|
||||
if [ "$COMMIT_REF" = "HEAD" ]; then
|
||||
COMMIT_REF=$(git rev-parse HEAD)
|
||||
fi
|
||||
|
||||
CMD="argo submit -n foxhunt --from=wftmpl/gpu-test-pipeline"
|
||||
CMD="$CMD -p commit-ref=$COMMIT_REF"
|
||||
CMD="$CMD -p models=$MODELS"
|
||||
CMD="$CMD -p test-scope=$SCOPE"
|
||||
CMD="$CMD -p epochs=$EPOCHS"
|
||||
|
||||
[[ -n "$GPU_POOL" ]] && CMD="$CMD -p gpu-pool=$GPU_POOL"
|
||||
|
||||
if $WATCH; then
|
||||
CMD="$CMD --watch"
|
||||
fi
|
||||
|
||||
echo "Submitting GPU test workflow..."
|
||||
echo " commit: $(echo $COMMIT_REF | cut -c1-8)"
|
||||
echo " models: $MODELS"
|
||||
echo " scope: $SCOPE"
|
||||
echo " epochs: $EPOCHS"
|
||||
echo ""
|
||||
eval "$CMD"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Make executable and commit**
|
||||
|
||||
```bash
|
||||
chmod +x scripts/argo-test.sh
|
||||
git add scripts/argo-test.sh
|
||||
git commit -m "feat(ci): add argo-test.sh CLI for GPU test workflow"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 7: Add on-push trigger to CI pipeline
|
||||
|
||||
**Files:**
|
||||
- Modify: `infra/k8s/argo/ci-pipeline-template.yaml`
|
||||
|
||||
- [ ] **Step 1: Add `ml-changed` to detect-changes**
|
||||
|
||||
In `ci-pipeline-template.yaml`, inside the `detect-changes` template's shell script (around line 199), add:
|
||||
|
||||
```bash
|
||||
ML_CHANGED=$(check_paths "crates/ml crates/ml-dqn crates/ml-core crates/ml-regime crates/ml-features crates/ml-supervised crates/ml-ppo crates/ml-ensemble crates/ml-hyperopt crates/ml-labeling")
|
||||
```
|
||||
|
||||
And add the output file write (around line 218):
|
||||
```bash
|
||||
echo -n "$ML_CHANGED" > /tmp/outputs/ml-changed
|
||||
```
|
||||
|
||||
And add the log line (around line 210):
|
||||
```bash
|
||||
echo "ml-changed: $ML_CHANGED"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add ml-changed output parameter**
|
||||
|
||||
In the `outputs.parameters` section of `detect-changes` (around line 238), add:
|
||||
```yaml
|
||||
- name: ml-changed
|
||||
valueFrom:
|
||||
path: /tmp/outputs/ml-changed
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Add gpu-test DAG task**
|
||||
|
||||
In the `pipeline` DAG `tasks` section (around line 143), add:
|
||||
```yaml
|
||||
- name: gpu-test
|
||||
depends: "detect-changes"
|
||||
templateRef:
|
||||
name: gpu-test-pipeline
|
||||
template: pipeline
|
||||
arguments:
|
||||
parameters:
|
||||
- name: commit-ref
|
||||
value: "{{workflow.parameters.commit-sha}}"
|
||||
- name: models
|
||||
value: "dqn,ppo,tft"
|
||||
- name: epochs
|
||||
value: "3"
|
||||
when: "{{tasks.detect-changes.outputs.parameters.ml-changed}} == true"
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Validate YAML syntax**
|
||||
|
||||
```bash
|
||||
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))" && echo "YAML OK"
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/k8s/argo/ci-pipeline-template.yaml
|
||||
git commit -m "feat(ci): add gpu-test on-push trigger gated on ML crate changes"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 8: Apply to cluster and validate
|
||||
|
||||
- [ ] **Step 1: Apply PVCs**
|
||||
|
||||
```bash
|
||||
kubectl -n foxhunt apply -f infra/k8s/argo/cargo-target-cuda-test-pvc.yaml
|
||||
kubectl -n foxhunt apply -f infra/k8s/training/test-data-pvc.yaml
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Populate test data**
|
||||
|
||||
```bash
|
||||
./scripts/refresh-test-data.sh
|
||||
```
|
||||
|
||||
Wait for job completion. Verify data:
|
||||
```bash
|
||||
kubectl -n foxhunt exec -it $(kubectl -n foxhunt get pods -l app.kubernetes.io/name=populate-test-data -o name | head -1) -- ls -lhR /test-data/
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Apply WorkflowTemplate and CronWorkflow**
|
||||
|
||||
```bash
|
||||
kubectl -n foxhunt apply -f infra/k8s/argo/gpu-test-pipeline-template.yaml
|
||||
kubectl -n foxhunt apply -f infra/k8s/argo/gpu-test-nightly-cron.yaml
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run a quick manual test (DQN lib only)**
|
||||
|
||||
```bash
|
||||
./scripts/argo-test.sh --models dqn --scope lib --watch
|
||||
```
|
||||
|
||||
Expected: compile + DQN lib tests pass on H100.
|
||||
|
||||
- [ ] **Step 5: Run full default suite**
|
||||
|
||||
```bash
|
||||
./scripts/argo-test.sh --watch
|
||||
```
|
||||
|
||||
Expected: DQN + PPO + TFT tests all pass.
|
||||
|
||||
- [ ] **Step 6: Apply updated CI pipeline**
|
||||
|
||||
```bash
|
||||
kubectl -n foxhunt apply -f infra/k8s/argo/ci-pipeline-template.yaml
|
||||
```
|
||||
|
||||
- [ ] **Step 7: Final commit (if any fixes needed)**
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "fix(ci): adjustments from H100 GPU test validation"
|
||||
```
|
||||
Reference in New Issue
Block a user