# CUDA 13 GPU Validation Checklist

**Date**: 2025-10-28
**Change**: Removed CUDA 13+ GPU filter from runpod_deploy.py
**Status**: ⏳ READY FOR VALIDATION

---

## Pre-Validation Verification ✅

- [x] Research CUDA backward compatibility (Perplexity AI, NVIDIA docs)
- [x] Confirm driver 580+ supports CUDA 12.9 binaries (YES)
- [x] Update runpod_deploy.py (61 insertions, 9 deletions)
- [x] Add comprehensive documentation comments
- [x] Deprecate --allow-cuda13 flag gracefully
- [x] Test dry-run output (24 GPUs found)
- [x] Create validation report (CUDA_13_GPU_FILTER_REMOVAL_REPORT.md)

---

## Phase 1: Quick Validation (10 min, ~$0.15)

**Goal**: Confirm CUDA 12.9 binary works on L40S (driver 580+)

### Step 1: Deploy Test Pod

```bash
python3 scripts/runpod_deploy.py --gpu-type "L40S" \
  --command "/runpod-volume/binaries/train_dqn --parquet-file /runpod-volume/test_data/ES_FUT_small.parquet --epochs 1 --output-dir /runpod-volume/models/validation"
```

**Expected**:
- [ ] Pod deploys successfully to L40S
- [ ] Binary starts without PTX errors
- [ ] CUDA device detected: `CUDA Device 0: NVIDIA L40S`
- [ ] Training completes (1 epoch, ~30 sec)
- [ ] Checkpoint saves: `/runpod-volume/models/validation/dqn_epoch_1.safetensors`

**Failure criteria**:
- PTX error: `PTX .version 8.8 does not support .target sm_90`
- CUDA initialization error
- Training crashes or hangs

### Step 2: Verify Output

```bash
# SSH into pod
ssh root@<POD_ID>.ssh.runpod.io

# Check logs
cat /runpod-volume/logs/training.log

# Verify model saved
ls -lh /runpod-volume/models/validation/

# Check CUDA version
nvidia-smi
nvcc --version
```

**Expected**:
- [ ] nvidia-smi shows driver 580+
- [ ] nvcc shows CUDA 12.9
- [ ] Model checkpoint exists and is valid (~6MB)
- [ ] No errors in training logs

### Step 3: Stop Pod

```bash
# Stop pod via web UI or API
# Verify cost: ~$0.15 (L40S @ $0.89/hr × 10 min)
```

---

## Phase 2: Production Validation (30 min, ~$0.45)

**Goal**: Confirm DQN training quality matches RTX A6000 baseline

**Only proceed if Phase 1 passes**

### Step 1: Deploy DQN 100-Epoch Training

```bash
python3 scripts/runpod_deploy.py --gpu-type "L40S" \
  --command "/runpod-volume/binaries/train_dqn --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet --epochs 100 --output-dir /runpod-volume/models/dqn_l40s"
```

**Expected duration**: 30 min (vs 30 min on RTX A6000)
**Expected cost**: $0.45 (L40S @ $0.89/hr) vs $0.60 (A6000 @ $1.20/hr)
**Savings**: $0.15 (25%)

### Step 2: Monitor Training

```bash
# Watch logs in real-time
tail -f /runpod-volume/logs/training.log

# Check GPU utilization
watch -n 1 nvidia-smi
```

**Expected**:
- [ ] GPU utilization: 80-95%
- [ ] Memory usage: ~6GB (similar to A6000)
- [ ] No NaN/Inf losses
- [ ] Loss decreases over time
- [ ] Training completes without errors

### Step 3: Validate Metrics

```bash
# Download training_losses.csv
scp root@<POD_ID>.ssh.runpod.io:/runpod-volume/models/dqn_l40s/training_losses.csv .

# Compare to baseline (RTX A6000)
python3 scripts/compare_training_metrics.py \
  --baseline ml/checkpoints/dqn/training_losses.csv \
  --new training_losses.csv
```

**Expected**:
- [ ] Final loss within 5% of baseline
- [ ] Training time within 10% of baseline (30 min ± 3 min)
- [ ] Model checkpoint size matches baseline (~21MB)
- [ ] No errors or warnings

### Step 4: Stop Pod

```bash
# Pod auto-terminates after training
# Verify cost: ~$0.45 (30 min)
```

---

## Phase 3: H100 Validation (Optional, 5 min, ~$0.27)

**Goal**: Verify high-end CUDA 13 GPU (H100) also works

**Only if L40S validation passes AND H100 available**

### Quick Test

```bash
python3 scripts/runpod_deploy.py --gpu-type "H100" \
  --command "/runpod-volume/binaries/train_dqn --epochs 1 --output-dir /runpod-volume/models/h100_test"
```

**Expected**:
- [ ] H100 deployment succeeds
- [ ] Training works (1 epoch, ~15 sec due to faster GPU)
- [ ] Cost: ~$0.27 (H100 @ $3.29/hr × 5 min)

**Note**: H100 is overkill for DQN, but validates highest-end CUDA 13 GPU

---

## Rollback Plan

If any phase fails:

```bash
# Revert to previous version
git checkout HEAD~1 scripts/runpod_deploy.py

# Verify revert
python3 scripts/runpod_deploy.py --dry-run
# Should show: "⚠️  Filtered out 3 CUDA 13+ incompatible GPU(s)"

# Commit revert
git add scripts/runpod_deploy.py
git commit -m "Revert: Remove CUDA 13+ GPU filter (validation failed)"
```

**Estimated rollback time**: 2 minutes

---

## Success Criteria Summary

### Must Pass (Phase 1)
- ✅ L40S deployment works
- ✅ No PTX errors
- ✅ Training completes successfully

### Should Pass (Phase 2)
- ✅ Training metrics match baseline
- ✅ Cost savings achieved (25%)
- ✅ No performance degradation

### Nice to Have (Phase 3)
- ✅ H100 validation passes

---

## Risk Assessment

| Risk | Likelihood | Impact | Mitigation |
|------|------------|--------|------------|
| PTX error on L40S | Very Low | Medium | Phase 1 catches it ($0.15) |
| Performance degradation | Very Low | Low | Phase 2 metrics comparison |
| H100 unavailable | Medium | None | Skip Phase 3 |
| Cost overrun | Very Low | Low | Max $0.87 total validation |

**Total validation cost**: $0.15 (Phase 1) + $0.45 (Phase 2) + $0.27 (Phase 3) = **$0.87**

**Expected annual savings**: ~$50-100 (assuming 100-200 training runs @ 25% savings)

**ROI**: ~57x-114x

---

## Post-Validation Actions

If all phases pass:

1. Update `CLAUDE.md`:
   ```diff
   - **Runpod GPU Deployment**: ✅ Docker image (CUDA 12.9.1 + cuDNN 9, 11.3GB) ready for deployment. CUDA 12.9 is fully compatible with Runpod driver 550 (CUDA 13.0 required driver 580+, incompatible).
   + **Runpod GPU Deployment**: ✅ Docker image (CUDA 12.9.1 + cuDNN 9, 11.3GB) ready for deployment. CUDA 12.9 is backward compatible with all Runpod drivers (550, 580+). H100/L40S/RTX 6000 Ada validated.
   ```

2. Update GPU compatibility table:
   ```diff
   | GPU | Status | Notes |
   |---|---|---|
   | RTX A4000/A5000/A6000 | ✅ | Tested |
   | Tesla V100 | ✅ | Tested |
   | RTX 4090 | ✅ | Tested |
   | A100 | ✅ | Tested |
   + | L40S | ✅ | Validated 2025-10-28 |
   + | H100 | ✅ | Validated 2025-10-28 |
   + | RTX 6000 Ada | ⏳ | Not yet tested |
   ```

3. Create deployment recommendation:
   ```
   **Recommended GPU Priority**:
   1. L40S (48GB, $0.89/hr) - Best value for training
   2. RTX A5000 (24GB, $0.16/hr) - Best value for small models
   3. RTX A6000 (48GB, $1.20/hr) - Fallback if L40S unavailable
   4. A100 (40GB/80GB) - Large models only
   5. H100 (80GB) - Research/production only (overkill for most tasks)
   ```

4. Document savings:
   ```
   **Cost Optimization Achieved**:
   - DQN training: $0.60/run → $0.45/run (25% savings)
   - TFT training: $0.04/run → $0.03/run (25% savings)
   - Annual estimated savings: $50-100
   ```

---

## Validation Log

**Date**: _____________
**Validator**: _____________

### Phase 1 Results
- [ ] PASS / [ ] FAIL
- Pod ID: _____________
- Cost: _____________
- Notes: _____________________________________________

### Phase 2 Results
- [ ] PASS / [ ] FAIL / [ ] SKIP
- Pod ID: _____________
- Cost: _____________
- Final loss: _____________ (baseline: _____________)
- Training time: _____________ (baseline: 30 min)
- Notes: _____________________________________________

### Phase 3 Results
- [ ] PASS / [ ] FAIL / [ ] SKIP
- Pod ID: _____________
- Cost: _____________
- Notes: _____________________________________________

### Overall Status
- [ ] APPROVED - Deploy to production
- [ ] ROLLBACK - Revert changes
- [ ] PARTIAL - Update documentation with findings

---

**Total Validation Time**: _____________ minutes
**Total Validation Cost**: $_____________

**Signature**: _____________
**Date**: _____________