foxhunt/CUDA_13_GPU_VALIDATION_CHECKLIST.md

CUDA 13 GPU Validation Checklist

Date: 2025-10-28 Change: Removed CUDA 13+ GPU filter from runpod_deploy.py Status: ⏳ READY FOR VALIDATION

---

Pre-Validation Verification ✅

• Research CUDA backward compatibility (Perplexity AI, NVIDIA docs)
• Confirm driver 580+ supports CUDA 12.9 binaries (YES)
• Update runpod_deploy.py (61 insertions, 9 deletions)
• Add comprehensive documentation comments
• Deprecate --allow-cuda13 flag gracefully
• Test dry-run output (24 GPUs found)
• Create validation report (CUDA_13_GPU_FILTER_REMOVAL_REPORT.md)

---

Phase 1: Quick Validation (10 min, ~$0.15)

Goal: Confirm CUDA 12.9 binary works on L40S (driver 580+)

Step 1: Deploy Test Pod

python3 scripts/runpod_deploy.py --gpu-type "L40S" \
  --command "/runpod-volume/binaries/train_dqn --parquet-file /runpod-volume/test_data/ES_FUT_small.parquet --epochs 1 --output-dir /runpod-volume/models/validation"

Expected:

• Pod deploys successfully to L40S
• Binary starts without PTX errors
• CUDA device detected: CUDA Device 0: NVIDIA L40S
• Training completes (1 epoch, ~30 sec)
• Checkpoint saves: /runpod-volume/models/validation/dqn_epoch_1.safetensors

Failure criteria:

• PTX error: PTX .version 8.8 does not support .target sm_90
• CUDA initialization error
• Training crashes or hangs

Step 2: Verify Output

# SSH into pod
ssh root@<POD_ID>.ssh.runpod.io

# Check logs
cat /runpod-volume/logs/training.log

# Verify model saved
ls -lh /runpod-volume/models/validation/

# Check CUDA version
nvidia-smi
nvcc --version

Expected:

• nvidia-smi shows driver 580+
• nvcc shows CUDA 12.9
• Model checkpoint exists and is valid (~6MB)
• No errors in training logs

Step 3: Stop Pod

# Stop pod via web UI or API
# Verify cost: ~$0.15 (L40S @ $0.89/hr × 10 min)

---

Phase 2: Production Validation (30 min, ~$0.45)

Goal: Confirm DQN training quality matches RTX A6000 baseline

Only proceed if Phase 1 passes

Step 1: Deploy DQN 100-Epoch Training

python3 scripts/runpod_deploy.py --gpu-type "L40S" \
  --command "/runpod-volume/binaries/train_dqn --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet --epochs 100 --output-dir /runpod-volume/models/dqn_l40s"

Expected duration: 30 min (vs 30 min on RTX A6000) Expected cost: $0.45 (L40S @ $0.89/hr) vs $0.60 (A6000 @ $1.20/hr) Savings: $0.15 (25%)

Step 2: Monitor Training

# Watch logs in real-time
tail -f /runpod-volume/logs/training.log

# Check GPU utilization
watch -n 1 nvidia-smi

Expected:

• GPU utilization: 80-95%
• Memory usage: ~6GB (similar to A6000)
• No NaN/Inf losses
• Loss decreases over time
• Training completes without errors

Step 3: Validate Metrics

# Download training_losses.csv
scp root@<POD_ID>.ssh.runpod.io:/runpod-volume/models/dqn_l40s/training_losses.csv .

# Compare to baseline (RTX A6000)
python3 scripts/compare_training_metrics.py \
  --baseline ml/checkpoints/dqn/training_losses.csv \
  --new training_losses.csv

Expected:

• Final loss within 5% of baseline
• Training time within 10% of baseline (30 min ± 3 min)
• Model checkpoint size matches baseline (~21MB)
• No errors or warnings

Step 4: Stop Pod

# Pod auto-terminates after training
# Verify cost: ~$0.45 (30 min)

---

Phase 3: H100 Validation (Optional, 5 min, ~$0.27)

Goal: Verify high-end CUDA 13 GPU (H100) also works

Only if L40S validation passes AND H100 available

Quick Test

python3 scripts/runpod_deploy.py --gpu-type "H100" \
  --command "/runpod-volume/binaries/train_dqn --epochs 1 --output-dir /runpod-volume/models/h100_test"

Expected:

• H100 deployment succeeds
• Training works (1 epoch, ~15 sec due to faster GPU)
• Cost: ~$0.27 (H100 @ $3.29/hr × 5 min)

Note: H100 is overkill for DQN, but validates highest-end CUDA 13 GPU

---

Rollback Plan

If any phase fails:

# Revert to previous version
git checkout HEAD~1 scripts/runpod_deploy.py

# Verify revert
python3 scripts/runpod_deploy.py --dry-run
# Should show: "⚠️  Filtered out 3 CUDA 13+ incompatible GPU(s)"

# Commit revert
git add scripts/runpod_deploy.py
git commit -m "Revert: Remove CUDA 13+ GPU filter (validation failed)"

Estimated rollback time: 2 minutes

---

Success Criteria Summary

Must Pass (Phase 1)

• ✅ L40S deployment works
• ✅ No PTX errors
• ✅ Training completes successfully

Should Pass (Phase 2)

• ✅ Training metrics match baseline
• ✅ Cost savings achieved (25%)
• ✅ No performance degradation

Nice to Have (Phase 3)

• ✅ H100 validation passes

---

Risk Assessment

Risk	Likelihood	Impact	Mitigation
PTX error on L40S	Very Low	Medium	Phase 1 catches it ($0.15)
Performance degradation	Very Low	Low	Phase 2 metrics comparison
H100 unavailable	Medium	None	Skip Phase 3
Cost overrun	Very Low	Low	Max $0.87 total validation

Total validation cost: $0.15 (Phase 1) + $0.45 (Phase 2) + $0.27 (Phase 3) = $0.87

Expected annual savings: ~$50-100 (assuming 100-200 training runs @ 25% savings)

ROI: ~57x-114x

---

Post-Validation Actions

If all phases pass:

1. Update CLAUDE.md:

   - **Runpod GPU Deployment**: ✅ Docker image (CUDA 12.9.1 + cuDNN 9, 11.3GB) ready for deployment. CUDA 12.9 is fully compatible with Runpod driver 550 (CUDA 13.0 required driver 580+, incompatible).
   + **Runpod GPU Deployment**: ✅ Docker image (CUDA 12.9.1 + cuDNN 9, 11.3GB) ready for deployment. CUDA 12.9 is backward compatible with all Runpod drivers (550, 580+). H100/L40S/RTX 6000 Ada validated.
   

2. Update GPU compatibility table:

   | GPU | Status | Notes |
   |---|---|---|
   | RTX A4000/A5000/A6000 | ✅ | Tested |
   | Tesla V100 | ✅ | Tested |
   | RTX 4090 | ✅ | Tested |
   | A100 | ✅ | Tested |
   + | L40S | ✅ | Validated 2025-10-28 |
   + | H100 | ✅ | Validated 2025-10-28 |
   + | RTX 6000 Ada | ⏳ | Not yet tested |
   

3. Create deployment recommendation:

   **Recommended GPU Priority**:
   1. L40S (48GB, $0.89/hr) - Best value for training
   2. RTX A5000 (24GB, $0.16/hr) - Best value for small models
   3. RTX A6000 (48GB, $1.20/hr) - Fallback if L40S unavailable
   4. A100 (40GB/80GB) - Large models only
   5. H100 (80GB) - Research/production only (overkill for most tasks)
   

4. Document savings:

   **Cost Optimization Achieved**:
   - DQN training: $0.60/run → $0.45/run (25% savings)
   - TFT training: $0.04/run → $0.03/run (25% savings)
   - Annual estimated savings: $50-100
   

---

Validation Log

Date: _____________ Validator: _____________

Phase 1 Results

• PASS / [ ] FAIL
• Pod ID: _____________
• Cost: _____________
• Notes: _____________________________________________

Phase 2 Results

• PASS / [ ] FAIL / [ ] SKIP
• Pod ID: _____________
• Cost: _____________
• Final loss: _____________ (baseline: _____________)
• Training time: _____________ (baseline: 30 min)
• Notes: _____________________________________________

Phase 3 Results

• PASS / [ ] FAIL / [ ] SKIP
• Pod ID: _____________
• Cost: _____________
• Notes: _____________________________________________

Overall Status

• APPROVED - Deploy to production
• ROLLBACK - Revert changes
• PARTIAL - Update documentation with findings

---

Total Validation Time: _____________ minutes Total Validation Cost: $_____________

Signature: _____________ Date: _____________