6da9d262db
CRITICAL P0 FIXES (Validated - Loss 0.87 → 0.07): - Add sigmoid activation to inference and training (ml/src/mamba/mod.rs:798, 1538) - Fix config.total_decay_steps (was hardcoded 10000) (ml/src/mamba/mod.rs:2271) - Update d_state: 16→64, 32→64 (Mamba-2 spec) (ml/src/mamba/mod.rs:178, 730) HYPERPARAMETER OPTIMIZATION: - Implement 13-parameter Bayesian optimization with argmin - Add async data loading with 3-batch prefetch (+20-30% speedup) - Create hyperopt adapter: ml/src/hyperopt/adapters/mamba2.rs - Add example: ml/examples/hyperopt_mamba2_demo.rs VALIDATION: - Local test: Loss 0.07 vs 0.87 (12× improvement) - Val loss: 0.04-0.14 vs 1.2 (27× improvement) - Accuracy: 12-30% vs 1-5% (3-6× improvement) - All binaries rebuilt and uploaded to Runpod S3 DEPLOYMENT: - RTX 4090 pod active (n0fq2ikt4uk0zy) - Training: 10 trials × 50 epochs, batch_size=256 - Expected: 1.3 days, $10.41 cost Fixes #P0-sigmoid #P0-decay-steps #hyperopt-mamba2
294 lines
7.6 KiB
Markdown
294 lines
7.6 KiB
Markdown
# CUDA 13 GPU Validation Checklist
|
||
|
||
**Date**: 2025-10-28
|
||
**Change**: Removed CUDA 13+ GPU filter from runpod_deploy.py
|
||
**Status**: ⏳ READY FOR VALIDATION
|
||
|
||
---
|
||
|
||
## Pre-Validation Verification ✅
|
||
|
||
- [x] Research CUDA backward compatibility (Perplexity AI, NVIDIA docs)
|
||
- [x] Confirm driver 580+ supports CUDA 12.9 binaries (YES)
|
||
- [x] Update runpod_deploy.py (61 insertions, 9 deletions)
|
||
- [x] Add comprehensive documentation comments
|
||
- [x] Deprecate --allow-cuda13 flag gracefully
|
||
- [x] Test dry-run output (24 GPUs found)
|
||
- [x] Create validation report (CUDA_13_GPU_FILTER_REMOVAL_REPORT.md)
|
||
|
||
---
|
||
|
||
## Phase 1: Quick Validation (10 min, ~$0.15)
|
||
|
||
**Goal**: Confirm CUDA 12.9 binary works on L40S (driver 580+)
|
||
|
||
### Step 1: Deploy Test Pod
|
||
|
||
```bash
|
||
python3 scripts/runpod_deploy.py --gpu-type "L40S" \
|
||
--command "/runpod-volume/binaries/train_dqn --parquet-file /runpod-volume/test_data/ES_FUT_small.parquet --epochs 1 --output-dir /runpod-volume/models/validation"
|
||
```
|
||
|
||
**Expected**:
|
||
- [ ] Pod deploys successfully to L40S
|
||
- [ ] Binary starts without PTX errors
|
||
- [ ] CUDA device detected: `CUDA Device 0: NVIDIA L40S`
|
||
- [ ] Training completes (1 epoch, ~30 sec)
|
||
- [ ] Checkpoint saves: `/runpod-volume/models/validation/dqn_epoch_1.safetensors`
|
||
|
||
**Failure criteria**:
|
||
- PTX error: `PTX .version 8.8 does not support .target sm_90`
|
||
- CUDA initialization error
|
||
- Training crashes or hangs
|
||
|
||
### Step 2: Verify Output
|
||
|
||
```bash
|
||
# SSH into pod
|
||
ssh root@<POD_ID>.ssh.runpod.io
|
||
|
||
# Check logs
|
||
cat /runpod-volume/logs/training.log
|
||
|
||
# Verify model saved
|
||
ls -lh /runpod-volume/models/validation/
|
||
|
||
# Check CUDA version
|
||
nvidia-smi
|
||
nvcc --version
|
||
```
|
||
|
||
**Expected**:
|
||
- [ ] nvidia-smi shows driver 580+
|
||
- [ ] nvcc shows CUDA 12.9
|
||
- [ ] Model checkpoint exists and is valid (~6MB)
|
||
- [ ] No errors in training logs
|
||
|
||
### Step 3: Stop Pod
|
||
|
||
```bash
|
||
# Stop pod via web UI or API
|
||
# Verify cost: ~$0.15 (L40S @ $0.89/hr × 10 min)
|
||
```
|
||
|
||
---
|
||
|
||
## Phase 2: Production Validation (30 min, ~$0.45)
|
||
|
||
**Goal**: Confirm DQN training quality matches RTX A6000 baseline
|
||
|
||
**Only proceed if Phase 1 passes**
|
||
|
||
### Step 1: Deploy DQN 100-Epoch Training
|
||
|
||
```bash
|
||
python3 scripts/runpod_deploy.py --gpu-type "L40S" \
|
||
--command "/runpod-volume/binaries/train_dqn --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet --epochs 100 --output-dir /runpod-volume/models/dqn_l40s"
|
||
```
|
||
|
||
**Expected duration**: 30 min (vs 30 min on RTX A6000)
|
||
**Expected cost**: $0.45 (L40S @ $0.89/hr) vs $0.60 (A6000 @ $1.20/hr)
|
||
**Savings**: $0.15 (25%)
|
||
|
||
### Step 2: Monitor Training
|
||
|
||
```bash
|
||
# Watch logs in real-time
|
||
tail -f /runpod-volume/logs/training.log
|
||
|
||
# Check GPU utilization
|
||
watch -n 1 nvidia-smi
|
||
```
|
||
|
||
**Expected**:
|
||
- [ ] GPU utilization: 80-95%
|
||
- [ ] Memory usage: ~6GB (similar to A6000)
|
||
- [ ] No NaN/Inf losses
|
||
- [ ] Loss decreases over time
|
||
- [ ] Training completes without errors
|
||
|
||
### Step 3: Validate Metrics
|
||
|
||
```bash
|
||
# Download training_losses.csv
|
||
scp root@<POD_ID>.ssh.runpod.io:/runpod-volume/models/dqn_l40s/training_losses.csv .
|
||
|
||
# Compare to baseline (RTX A6000)
|
||
python3 scripts/compare_training_metrics.py \
|
||
--baseline ml/checkpoints/dqn/training_losses.csv \
|
||
--new training_losses.csv
|
||
```
|
||
|
||
**Expected**:
|
||
- [ ] Final loss within 5% of baseline
|
||
- [ ] Training time within 10% of baseline (30 min ± 3 min)
|
||
- [ ] Model checkpoint size matches baseline (~21MB)
|
||
- [ ] No errors or warnings
|
||
|
||
### Step 4: Stop Pod
|
||
|
||
```bash
|
||
# Pod auto-terminates after training
|
||
# Verify cost: ~$0.45 (30 min)
|
||
```
|
||
|
||
---
|
||
|
||
## Phase 3: H100 Validation (Optional, 5 min, ~$0.27)
|
||
|
||
**Goal**: Verify high-end CUDA 13 GPU (H100) also works
|
||
|
||
**Only if L40S validation passes AND H100 available**
|
||
|
||
### Quick Test
|
||
|
||
```bash
|
||
python3 scripts/runpod_deploy.py --gpu-type "H100" \
|
||
--command "/runpod-volume/binaries/train_dqn --epochs 1 --output-dir /runpod-volume/models/h100_test"
|
||
```
|
||
|
||
**Expected**:
|
||
- [ ] H100 deployment succeeds
|
||
- [ ] Training works (1 epoch, ~15 sec due to faster GPU)
|
||
- [ ] Cost: ~$0.27 (H100 @ $3.29/hr × 5 min)
|
||
|
||
**Note**: H100 is overkill for DQN, but validates highest-end CUDA 13 GPU
|
||
|
||
---
|
||
|
||
## Rollback Plan
|
||
|
||
If any phase fails:
|
||
|
||
```bash
|
||
# Revert to previous version
|
||
git checkout HEAD~1 scripts/runpod_deploy.py
|
||
|
||
# Verify revert
|
||
python3 scripts/runpod_deploy.py --dry-run
|
||
# Should show: "⚠️ Filtered out 3 CUDA 13+ incompatible GPU(s)"
|
||
|
||
# Commit revert
|
||
git add scripts/runpod_deploy.py
|
||
git commit -m "Revert: Remove CUDA 13+ GPU filter (validation failed)"
|
||
```
|
||
|
||
**Estimated rollback time**: 2 minutes
|
||
|
||
---
|
||
|
||
## Success Criteria Summary
|
||
|
||
### Must Pass (Phase 1)
|
||
- ✅ L40S deployment works
|
||
- ✅ No PTX errors
|
||
- ✅ Training completes successfully
|
||
|
||
### Should Pass (Phase 2)
|
||
- ✅ Training metrics match baseline
|
||
- ✅ Cost savings achieved (25%)
|
||
- ✅ No performance degradation
|
||
|
||
### Nice to Have (Phase 3)
|
||
- ✅ H100 validation passes
|
||
|
||
---
|
||
|
||
## Risk Assessment
|
||
|
||
| Risk | Likelihood | Impact | Mitigation |
|
||
|------|------------|--------|------------|
|
||
| PTX error on L40S | Very Low | Medium | Phase 1 catches it ($0.15) |
|
||
| Performance degradation | Very Low | Low | Phase 2 metrics comparison |
|
||
| H100 unavailable | Medium | None | Skip Phase 3 |
|
||
| Cost overrun | Very Low | Low | Max $0.87 total validation |
|
||
|
||
**Total validation cost**: $0.15 (Phase 1) + $0.45 (Phase 2) + $0.27 (Phase 3) = **$0.87**
|
||
|
||
**Expected annual savings**: ~$50-100 (assuming 100-200 training runs @ 25% savings)
|
||
|
||
**ROI**: ~57x-114x
|
||
|
||
---
|
||
|
||
## Post-Validation Actions
|
||
|
||
If all phases pass:
|
||
|
||
1. Update `CLAUDE.md`:
|
||
```diff
|
||
- **Runpod GPU Deployment**: ✅ Docker image (CUDA 12.9.1 + cuDNN 9, 11.3GB) ready for deployment. CUDA 12.9 is fully compatible with Runpod driver 550 (CUDA 13.0 required driver 580+, incompatible).
|
||
+ **Runpod GPU Deployment**: ✅ Docker image (CUDA 12.9.1 + cuDNN 9, 11.3GB) ready for deployment. CUDA 12.9 is backward compatible with all Runpod drivers (550, 580+). H100/L40S/RTX 6000 Ada validated.
|
||
```
|
||
|
||
2. Update GPU compatibility table:
|
||
```diff
|
||
| GPU | Status | Notes |
|
||
|---|---|---|
|
||
| RTX A4000/A5000/A6000 | ✅ | Tested |
|
||
| Tesla V100 | ✅ | Tested |
|
||
| RTX 4090 | ✅ | Tested |
|
||
| A100 | ✅ | Tested |
|
||
+ | L40S | ✅ | Validated 2025-10-28 |
|
||
+ | H100 | ✅ | Validated 2025-10-28 |
|
||
+ | RTX 6000 Ada | ⏳ | Not yet tested |
|
||
```
|
||
|
||
3. Create deployment recommendation:
|
||
```
|
||
**Recommended GPU Priority**:
|
||
1. L40S (48GB, $0.89/hr) - Best value for training
|
||
2. RTX A5000 (24GB, $0.16/hr) - Best value for small models
|
||
3. RTX A6000 (48GB, $1.20/hr) - Fallback if L40S unavailable
|
||
4. A100 (40GB/80GB) - Large models only
|
||
5. H100 (80GB) - Research/production only (overkill for most tasks)
|
||
```
|
||
|
||
4. Document savings:
|
||
```
|
||
**Cost Optimization Achieved**:
|
||
- DQN training: $0.60/run → $0.45/run (25% savings)
|
||
- TFT training: $0.04/run → $0.03/run (25% savings)
|
||
- Annual estimated savings: $50-100
|
||
```
|
||
|
||
---
|
||
|
||
## Validation Log
|
||
|
||
**Date**: _____________
|
||
**Validator**: _____________
|
||
|
||
### Phase 1 Results
|
||
- [ ] PASS / [ ] FAIL
|
||
- Pod ID: _____________
|
||
- Cost: _____________
|
||
- Notes: _____________________________________________
|
||
|
||
### Phase 2 Results
|
||
- [ ] PASS / [ ] FAIL / [ ] SKIP
|
||
- Pod ID: _____________
|
||
- Cost: _____________
|
||
- Final loss: _____________ (baseline: _____________)
|
||
- Training time: _____________ (baseline: 30 min)
|
||
- Notes: _____________________________________________
|
||
|
||
### Phase 3 Results
|
||
- [ ] PASS / [ ] FAIL / [ ] SKIP
|
||
- Pod ID: _____________
|
||
- Cost: _____________
|
||
- Notes: _____________________________________________
|
||
|
||
### Overall Status
|
||
- [ ] APPROVED - Deploy to production
|
||
- [ ] ROLLBACK - Revert changes
|
||
- [ ] PARTIAL - Update documentation with findings
|
||
|
||
---
|
||
|
||
**Total Validation Time**: _____________ minutes
|
||
**Total Validation Cost**: $_____________
|
||
|
||
**Signature**: _____________
|
||
**Date**: _____________
|