foxhunt/AGENT_35_CLOUD_GPU_RECOMMENDATION.md

# AGENT-35: Cloud GPU Recommendation for Production ML Training

**Date**: 2025-10-21
**Status**: ✅ COMPLETE
**Agent**: AGENT-35 (Cloud GPU Research & Recommendation)
**Context**: Wave 12 ML Production - Cloud GPU strategy for 90-180 day training datasets

---

## 🎯 Executive Summary

**Recommendation**: **RunPod Community Cloud (RTX 4090)** for immediate production training.

**Key Findings**:
- **Best Value**: RunPod RTX 4090 @ $0.34/hr (spot pricing)
- **Estimated 4-Model Training Cost**: **$3.40 - $6.80** (10-20 hours total)
- **ROI vs Local GPU Upgrade**: Cloud wins by **$1,293 - $1,596** (avoiding $1,600-$2,400 RTX 4090 purchase)
- **Memory Headroom**: RTX 4090 (24GB) provides **5.5x more VRAM** than current RTX 3050 Ti (4GB)
- **Speedup**: Expected **3-5x faster** training vs RTX 3050 Ti (based on CUDA core count: 16,384 vs 2,560)

---

## 📊 Cloud GPU Provider Comparison

### Tier 1: Budget-Friendly Options (Recommended)

| Provider | GPU Model | VRAM | $/hour (Spot) | $/hour (On-Demand) | Billing | Docker | SSH | CUDA 11.8+ | Rust Support | Notes |
|----------|-----------|------|---------------|-------------------|---------|--------|-----|-----------|--------------|-------|
| **🏆 RunPod** | **RTX 4090** | **24GB** | **$0.34** | **$0.59** | **Per-second** | ✅ | ✅ | ✅ 12.1 | ✅ | **BEST VALUE** - Pay-per-second, 31 regions |
| **Vast.ai** | RTX 4090 | 24GB | $0.29 | $0.61 | Per-hour | ✅ | ✅ | ✅ 12.x | ✅ | Marketplace (variable availability) |
| **RunPod** | RTX 3090 | 24GB | $0.27 | $0.43 | Per-second | ✅ | ✅ | ✅ 11.8 | ✅ | Lower CUDA cores (10,496 vs 16,384) |
| **Vast.ai** | A100 PCIe | 40GB | $0.67 | $1.20 | Per-hour | ✅ | ✅ | ✅ 11.8+ | ✅ | More VRAM, better for large models |
| **RunPod** | A100 80GB | 80GB | $1.64 | $1.99 | Per-second | ✅ | ✅ | ✅ 11.8+ | ✅ | Overkill for our 440MB memory budget |

### Tier 2: Mid-Range (Enterprise Support)

| Provider | GPU Model | VRAM | $/hour (On-Demand) | Billing | Support | CUDA | Notes |
|----------|-----------|------|-------------------|---------|---------|------|-------|
| **Lambda Labs** | A100 80GB (1x GPU) | 80GB | $3.29 | Per-hour | Premium | ✅ 11.8+ | ML-focused, JupyterLab pre-configured |
| **Lambda Labs** | A100 80GB (8x GPU) | 640GB | $1.79/GPU ($14.32 total) | Per-hour | Premium | ✅ 11.8+ | Multi-GPU training (not needed for us) |
| **Lambda Labs** | H100 80GB (1x GPU) | 80GB | $3.29 | Per-hour | Premium | ✅ 12.0+ | Latest generation (overkill) |
| **DigitalOcean** | H100 80GB | 80GB | $1.99 | Per-hour | Excellent | ✅ 12.0+ | New offering, great developer UX |

### Tier 3: Major Cloud Providers (AWS/GCP/Azure)

| Provider | Instance Type | GPU Model | VRAM | $/hour (On-Demand) | $/hour (Spot) | Savings Plans | Notes |
|----------|---------------|-----------|------|-------------------|---------------|---------------|-------|
| **AWS** | ml.g5.xlarge | A10G | 24GB | $1.41 | ~$0.42 (70% off) | Up to 64% off | SageMaker integration |
| **AWS** | ml.p4d.24xlarge | A100 (8x) | 320GB | $32.77 | ~$9.83 (70% off) | Up to 64% off | Multi-GPU (overkill) |
| **AWS** | p3.2xlarge | V100 | 16GB | $3.06 | ~$0.92 (70% off) | N/A | Older generation |
| **GCP** | n1-standard-8 + T4 | T4 | 16GB | $0.56 | ~$0.17 (70% off) | N/A | Budget option |
| **GCP** | a2-highgpu-1g | A100 | 40GB | $3.67 | ~$1.10 (70% off) | N/A | Enterprise support |
| **Azure** | NC A100 v4 | A100 | 80GB | ~$3.67 | Variable | Reserved | Enterprise compliance |

### Tier 4: Specialized ML Platforms

| Provider | GPU Options | $/hour Range | Strengths | Weaknesses |
|----------|-------------|--------------|-----------|------------|
| **Paperspace** | RTX 4000, A100 | $0.51 - $2.00 | Free tier, pre-configured ML | Limited GPU selection |
| **Jarvis Labs** | RTX 3090, A100 | $0.49 - $1.89 | Developer-focused | Smaller community |
| **Genesis Cloud** | RTX 3090, A100 | $0.40 - $1.80 | Reliable uptime | Less flexible than RunPod |
| **Hyperstack** | A100, H100 | $1.42 - $2.74 | Production reliability | Higher base price |

---

## 💰 Cost Analysis: 4-Model Training (90-180 Day Datasets)

### Assumptions
- **Dataset**: 90-180 days per asset (ES.FUT, NQ.FUT, 6E.FUT, ZN.FUT)
- **Training Time Estimates** (based on current RTX 3050 Ti benchmarks, scaled by CUDA cores):
  - **MAMBA-2**: 1.9 min (current) → **0.5 min** (RTX 4090, 6.4x CUDA cores)
  - **DQN**: 15 sec (current) → **5 sec** (RTX 4090)
  - **PPO**: 7 sec (current) → **2 sec** (RTX 4090)
  - **TFT-INT8**: 3.5 min (current) → **1.0 min** (RTX 4090)
  - **Total per asset**: ~5.5 min (current) → **~1.5 min** (RTX 4090)
- **4 Assets**: 4 × 1.5 min = **6 min** (optimistic)
- **50 Epochs**: 6 min × 50 = **300 min = 5 hours**
- **Safety Buffer (2x)**: 5 hours × 2 = **10 hours** (realistic estimate)
- **Hyperparameter Tuning**: 10 hours × 2 = **20 hours** (with tuning)

### Cost Breakdown by Provider

| Provider | GPU | $/hour | 10h Cost | 20h Cost | 100h Cost (Monthly) | Notes |
|----------|-----|--------|----------|----------|---------------------|-------|
| **🥇 RunPod (Spot)** | **RTX 4090** | **$0.34** | **$3.40** | **$6.80** | **$34.00** | **BEST VALUE** |
| **🥈 Vast.ai** | RTX 4090 | $0.29 | $2.90 | $5.80 | $29.00 | Slightly cheaper, less stable |
| **🥉 RunPod (Spot)** | RTX 3090 | $0.27 | $2.70 | $5.40 | $27.00 | Slower (40% fewer CUDA cores) |
| Vast.ai | A100 PCIe | $0.67 | $6.70 | $13.40 | $67.00 | More VRAM, overkill for us |
| RunPod (On-Demand) | RTX 4090 | $0.59 | $5.90 | $11.80 | $59.00 | Guaranteed availability |
| Lambda Labs | A100 80GB | $3.29 | $32.90 | $65.80 | $329.00 | Premium support |
| AWS (Spot) | ml.g5.xlarge | $0.42 | $4.20 | $8.40 | $42.00 | SageMaker integration |
| AWS (On-Demand) | ml.g5.xlarge | $1.41 | $14.10 | $28.20 | $141.00 | Enterprise support |

### Annual Cost Projection (Monthly Retraining)

| Provider | GPU | $/month (10h) | $/year | vs Local GPU ($1,600) | ROI Year 1 |
|----------|-----|---------------|--------|-----------------------|------------|
| **RunPod** | **RTX 4090** | **$34.00** | **$408** | **+$1,192 savings** | **293% ROI** |
| Vast.ai | RTX 4090 | $29.00 | $348 | +$1,252 savings | 360% ROI |
| Lambda Labs | A100 80GB | $329.00 | $3,948 | -$2,348 loss | -147% ROI |
| AWS (Spot) | ml.g5.xlarge | $42.00 | $504 | +$1,096 savings | 217% ROI |

**Verdict**: Cloud GPU (RunPod/Vast.ai) is **drastically cheaper** than buying local hardware for infrequent training.

---

## 🏆 Top 3 Recommendations

### 🥇 #1: RunPod Community Cloud (RTX 4090) - **RECOMMENDED**

**Why This is the Best Choice**:
- ✅ **Best Price-Performance**: $0.34/hr spot pricing (68% cheaper than on-demand)
- ✅ **Per-Second Billing**: Only pay for actual training time (no waste)
- ✅ **Proven Rust Support**: Docker with CUDA 12.1 pre-configured
- ✅ **24GB VRAM**: 5.5x more than RTX 3050 Ti (handles all 4 models easily)
- ✅ **31 Global Regions**: Low latency, high availability
- ✅ **SSH + Jupyter**: Full development environment access
- ✅ **Auto-Shutdown**: No accidental overcharges
- ✅ **Community Support**: Large user base, active Discord

**Cost for Our Use Case**:
- **Initial Training (50 epochs, 4 models)**: $3.40 - $6.80 (10-20 hours)
- **Monthly Retraining (10h/month)**: $34.00/month
- **Annual Cost**: $408/year (vs $1,600 local GPU purchase)

**Integration Complexity**: ⭐⭐⭐⭐☆ (4/5 - Easy)
- Upload Parquet files via SSH/SCP
- Run training commands via SSH
- Download checkpoints via SCP/rsync
- Minimal ML Training Service changes required

**Limitations**:
- ⚠️ Spot instances can be preempted (rare, but possible)
- ⚠️ Requires manual instance management (start/stop)
- ⚠️ No native integration with ML Training Service (requires custom script)

**When to Use**:
- ✅ Production training (monthly retraining cycles)
- ✅ Hyperparameter tuning experiments
- ✅ Cost-sensitive projects
- ✅ Rapid iteration workflows

---

### 🥈 #2: AWS SageMaker (ml.g5.xlarge Spot) - **ENTERPRISE CHOICE**

**Why Consider This**:
- ✅ **AWS Ecosystem Integration**: S3, IAM, CloudWatch, EventBridge
- ✅ **Spot Instances**: 70% savings ($1.41 → $0.42/hr)
- ✅ **Enterprise Support**: 24/7 support, SLAs, compliance (SOC2, HIPAA)
- ✅ **Managed ML Pipeline**: SageMaker Training Jobs API
- ✅ **Auto-Scaling**: Scale to multiple GPUs if needed
- ✅ **Built-in Monitoring**: CloudWatch metrics, logs, alarms

**Cost for Our Use Case**:
- **Initial Training (50 epochs, 4 models)**: $4.20 - $8.40 (10-20 hours)
- **Monthly Retraining (10h/month)**: $42.00/month
- **Annual Cost**: $504/year

**Integration Complexity**: ⭐⭐⭐☆☆ (3/5 - Moderate)
- Requires AWS SDK integration in ML Training Service
- Dockerfile for SageMaker container
- S3 for data storage (upload/download)
- IAM roles and policies
- EventBridge for scheduling

**Limitations**:
- ⚠️ Higher cost than RunPod ($42/month vs $34/month)
- ⚠️ Spot instances less predictable than on-demand
- ⚠️ AWS learning curve (IAM, S3, SageMaker APIs)
- ⚠️ Minimum 1-minute billing (vs RunPod's per-second)

**When to Use**:
- ✅ Already using AWS infrastructure
- ✅ Need enterprise compliance (SOC2, HIPAA)
- ✅ Want managed ML pipeline
- ✅ Scaling to multi-GPU training in future

---

### 🥉 #3: Lambda Labs (A100 80GB) - **PREMIUM CHOICE**

**Why Consider This**:
- ✅ **ML-Optimized**: Pre-configured PyTorch, TensorFlow, CUDA 11.8+
- ✅ **Zero Setup**: JupyterLab, SSH, Docker ready on boot
- ✅ **Premium Support**: Fast response times, ML expertise
- ✅ **Latest Hardware**: H100, H200 early access
- ✅ **80GB VRAM**: Future-proof for larger models
- ✅ **Persistent Storage**: NVMe SSD included (19.5TB)

**Cost for Our Use Case**:
- **Initial Training (50 epochs, 4 models)**: $32.90 - $65.80 (10-20 hours)
- **Monthly Retraining (10h/month)**: $329.00/month
- **Annual Cost**: $3,948/year

**Integration Complexity**: ⭐⭐⭐⭐⭐ (5/5 - Very Easy)
- Pre-configured CUDA environment
- One-click JupyterLab access
- SSH access for automation
- Minimal setup required

**Limitations**:
- ⚠️ **10x more expensive** than RunPod ($329/month vs $34/month)
- ⚠️ Overkill for our 440MB memory budget
- ⚠️ Per-hour billing (no per-second option)
- ⚠️ Limited global regions (US-only for some GPUs)

**When to Use**:
- ✅ Need premium support
- ✅ Training very large models (>10GB weights)
- ✅ Budget is not a constraint
- ✅ Want zero-friction ML development

---

## 🏗️ Integration Architecture

### Current Architecture (Local GPU)

```
┌─────────────────────────────────────────────────────────────┐
│                    Foxhunt System (Local)                    │
│                                                               │
│  ┌──────────────┐      ┌──────────────┐      ┌──────────┐   │
│  │ ML Training  │─────▶│  RTX 3050 Ti │─────▶│  Models  │   │
│  │   Service    │      │   (4GB VRAM) │      │ (.safetensors)│
│  │  (Port 50054)│      │              │      │              │   │
│  └──────────────┘      └──────────────┘      └──────────┘   │
│         │                                            │        │
│         │                                            │        │
│         ▼                                            ▼        │
│  ┌──────────────┐                          ┌──────────────┐  │
│  │   Parquet    │                          │ PostgreSQL   │  │
│  │   Files      │                          │  (Metadata)  │  │
│  │ (test_data/) │                          │              │  │
│  └──────────────┘                          └──────────────┘  │
└─────────────────────────────────────────────────────────────┘
```

### Hybrid Architecture (Cloud GPU Training)

```
┌─────────────────────────────────────────────────────────────────────┐
│                      Foxhunt System (Local)                          │
│                                                                      │
│  ┌──────────────┐      ┌──────────────┐      ┌──────────────┐      │
│  │ ML Training  │─────▶│ Cloud GPU    │─────▶│   Models     │      │
│  │   Service    │ gRPC │  Orchestrator│      │  (.safetensors)│     │
│  │  (Port 50054)│      │              │      │              │      │
│  └──────────────┘      └──────────────┘      └──────────────┘      │
│         │                      │                       │             │
│         │                      │                       │             │
│         │                      ▼                       ▼             │
│  ┌──────────────┐      ┌──────────────┐      ┌──────────────┐      │
│  │   Parquet    │─────▶│  S3 Bucket   │      │ PostgreSQL   │      │
│  │   Files      │ rsync│  (Training   │      │  (Metadata)  │      │
│  │ (test_data/) │      │    Data)     │      │              │      │
│  └──────────────┘      └──────────────┘      └──────────────┘      │
└─────────────────────────┼────────────────────────────────────────────┘
                          │
                          │ SSH/API
                          ▼
┌─────────────────────────────────────────────────────────────────────┐
│                        RunPod Cloud GPU                              │
│                                                                      │
│  ┌──────────────┐      ┌──────────────┐      ┌──────────────┐      │
│  │  Docker      │      │  RTX 4090    │      │  Training    │      │
│  │  Container   │─────▶│  (24GB VRAM) │─────▶│  Checkpoint  │      │
│  │  (Rust+CUDA) │      │              │      │  (.safetensors)│     │
│  └──────────────┘      └──────────────┘      └──────────────┘      │
│         │                                            │               │
│         │                                            │               │
│         ▼                                            ▼               │
│  ┌──────────────┐                          ┌──────────────┐         │
│  │  Parquet     │                          │  Auto-       │         │
│  │  Files       │                          │  Shutdown    │         │
│  │ (downloaded) │                          │  (No Waste)  │         │
│  └──────────────┘                          └──────────────┘         │
└─────────────────────────────────────────────────────────────────────┘
```

### TLI Integration (Future Enhancement)

```
┌──────────────────────────────────────────────────────────────────┐
│                      TLI (Terminal Client)                        │
│                                                                   │
│  $ tli ml train --cloud runpod --gpu rtx-4090 \                  │
│      --model mamba2 --epochs 50 --asset ES.FUT                   │
│                                                                   │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │ 1. Upload Parquet files to RunPod via rsync            │    │
│  │ 2. Spin up RTX 4090 instance (spot, $0.34/hr)         │    │
│  │ 3. Execute training command via SSH                    │    │
│  │ 4. Monitor progress (live logs streaming)              │    │
│  │ 5. Download checkpoints to ml/trained_models/          │    │
│  │ 6. Auto-shutdown instance (stop billing)               │    │
│  └─────────────────────────────────────────────────────────┘    │
│                                                                   │
│  Output:                                                          │
│  ✅ Training complete in 10.5 hours                              │
│  💾 Checkpoints downloaded to ml/trained_models/                 │
│  💰 Total cost: $3.57 (10.5h × $0.34/hr)                        │
│  🔌 Instance terminated (no ongoing charges)                     │
└──────────────────────────────────────────────────────────────────┘
```

---

## 🛣️ Migration Path (Local → Cloud Training)

### Phase 1: Manual Cloud Training (Immediate - 1 week)

**Goal**: Validate cloud GPU workflow without code changes

**Steps**:

1. **Setup RunPod Account**
   ```bash
   # Sign up at https://www.runpod.io/
   # Add payment method
   # Generate API key
   ```

2. **Create Docker Image with Rust + CUDA**
   ```dockerfile
   # Dockerfile
   FROM nvidia/cuda:12.1.0-devel-ubuntu22.04

   # Install Rust
   RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
   ENV PATH="/root/.cargo/bin:${PATH}"

   # Install dependencies
   RUN apt-get update && apt-get install -y \
       build-essential \
       libssl-dev \
       pkg-config \
       git \
       rsync

   # Copy Foxhunt codebase
   WORKDIR /workspace
   COPY . .

   # Pre-compile (cache dependencies)
   RUN cargo build --release --features cuda -p ml

   ENTRYPOINT ["/bin/bash"]
   ```

3. **Upload Parquet Files**
   ```bash
   # Local machine
   rsync -avz --progress test_data/*.parquet \
     root@runpod-instance:/workspace/test_data/
   ```

4. **Run Training via SSH**
   ```bash
   # SSH into RunPod instance
   ssh root@runpod-instance

   # Inside RunPod container
   cd /workspace

   # Train MAMBA-2
   cargo run -p ml --example train_mamba2_parquet --release --features cuda -- \
     --parquet-file test_data/ES_FUT_180d.parquet \
     --epochs 50

   # Train DQN
   cargo run -p ml --example train_dqn --release --features cuda -- \
     --parquet-file test_data/NQ_FUT_180d.parquet \
     --epochs 100

   # Train PPO
   cargo run -p ml --example train_ppo_parquet --release --features cuda -- \
     --parquet-file test_data/ZN_FUT_90d_clean.parquet \
     --epochs 50

   # Train TFT
   cargo run -p ml --example train_tft_parquet --release --features cuda -- \
     --parquet-file test_data/6E_FUT_180d.parquet \
     --epochs 50
   ```

5. **Download Checkpoints**
   ```bash
   # Local machine
   rsync -avz --progress \
     root@runpod-instance:/workspace/ml/trained_models/ \
     ml/trained_models/
   ```

6. **Terminate Instance**
   ```bash
   # RunPod web console or CLI
   runpod stop <instance-id>
   ```

**Validation**:
- ✅ All 4 models train successfully on cloud GPU
- ✅ Checkpoints download cleanly
- ✅ Cost matches estimate ($3-7 for 10-20 hours)
- ✅ Performance is 3-5x faster than local GPU

**Estimated Time**: 1 week (includes testing, validation)

---

### Phase 2: Automated Cloud Training (2-3 weeks)

**Goal**: TLI command triggers cloud training automatically

**Implementation**:

1. **Add RunPod Integration to Config**
   ```toml
   # config/config.toml
   [cloud_gpu]
   provider = "runpod"
   api_key = "${RUNPOD_API_KEY}"  # From Vault
   gpu_type = "rtx-4090"
   instance_type = "community"  # or "secure"
   auto_shutdown = true

   [cloud_gpu.storage]
   type = "s3"  # or "runpod_network"
   bucket = "foxhunt-training-data"
   checkpoint_dir = "ml/trained_models"
   ```

2. **Create Cloud GPU Orchestrator**
   ```rust
   // services/ml_training_service/src/cloud_gpu.rs

   pub struct CloudGpuOrchestrator {
       provider: CloudGpuProvider,
       config: CloudGpuConfig,
       storage: Arc<dyn CloudStorage>,
   }

   impl CloudGpuOrchestrator {
       /// Spin up cloud GPU instance
       pub async fn provision_instance(&self) -> Result<CloudInstance> {
           // 1. Call RunPod API to create instance
           // 2. Wait for instance to be ready
           // 3. Return SSH connection details
       }

       /// Upload training data to cloud
       pub async fn upload_data(&self, parquet_files: Vec<PathBuf>) -> Result<()> {
           // rsync or S3 upload
       }

       /// Execute training command via SSH
       pub async fn run_training(&self, instance: &CloudInstance, cmd: &str) -> Result<()> {
           // SSH into instance, run command, stream logs
       }

       /// Download trained models
       pub async fn download_models(&self, instance: &CloudInstance) -> Result<()> {
           // rsync or S3 download
       }

       /// Terminate instance (stop billing)
       pub async fn terminate_instance(&self, instance: CloudInstance) -> Result<()> {
           // Call RunPod API to stop instance
       }
   }
   ```

3. **Add TLI Command**
   ```rust
   // tli/src/commands/ml.rs

   #[derive(Parser)]
   pub enum MlCommand {
       // ... existing commands ...

       /// Train ML models on cloud GPU
       TrainCloud {
           /// GPU provider (runpod, aws, lambda)
           #[arg(long, default_value = "runpod")]
           provider: String,

           /// GPU type (rtx-4090, a100, h100)
           #[arg(long, default_value = "rtx-4090")]
           gpu: String,

           /// Model to train (mamba2, dqn, ppo, tft, all)
           #[arg(long)]
           model: String,

           /// Parquet file or asset symbol
           #[arg(long)]
           asset: String,

           /// Training epochs
           #[arg(long, default_value = "50")]
           epochs: usize,

           /// Keep instance running after training (for debugging)
           #[arg(long)]
           keep_alive: bool,
       },
   }
   ```

4. **End-to-End Workflow**
   ```bash
   # User executes TLI command
   tli ml train-cloud --model mamba2 --asset ES.FUT --epochs 50

   # TLI → ML Training Service → Cloud GPU Orchestrator:
   # 1. Provision RunPod RTX 4090 instance ($0.34/hr spot)
   # 2. Upload ES_FUT_180d.parquet to instance
   # 3. Run training command via SSH
   # 4. Stream logs to TLI (live progress)
   # 5. Download checkpoint when complete
   # 6. Terminate instance (stop billing)

   # Output:
   # ✅ Training complete in 2.3 hours
   # 💾 Model saved: ml/trained_models/mamba2_ES_FUT_epoch50.safetensors
   # 💰 Total cost: $0.78 (2.3h × $0.34/hr)
   ```

**Validation**:
- ✅ TLI command successfully provisions cloud GPU
- ✅ Training executes without manual SSH intervention
- ✅ Logs stream to TLI in real-time
- ✅ Checkpoints download automatically
- ✅ Instance terminates after training (no waste)

**Estimated Time**: 2-3 weeks (includes testing, error handling)

---

### Phase 3: Production Deployment (1 week)

**Goal**: Scheduled retraining on cloud GPU

**Implementation**:

1. **Add Scheduler to ML Training Service**
   ```rust
   // services/ml_training_service/src/scheduler.rs

   pub struct TrainingScheduler {
       schedule: CronSchedule,  // e.g., "0 2 * * 0" (2am every Sunday)
       orchestrator: Arc<CloudGpuOrchestrator>,
   }

   impl TrainingScheduler {
       pub async fn run(&self) -> Result<()> {
           // 1. Wait for next scheduled time
           // 2. Provision cloud GPU
           // 3. Train all 4 models
           // 4. Download checkpoints
           // 5. Update PostgreSQL metadata
           // 6. Terminate instance
           // 7. Send Slack notification
       }
   }
   ```

2. **Monitoring & Alerting**
   ```rust
   // Prometheus metrics
   training_duration_seconds{model="mamba2", gpu="rtx-4090"}
   training_cost_usd{model="mamba2", gpu="rtx-4090"}
   gpu_utilization_percent{gpu="rtx-4090"}
   instance_uptime_seconds{gpu="rtx-4090"}

   // Grafana alerts
   - Alert: Training failed
     Condition: training_duration_seconds > 86400 (24h timeout)

   - Alert: Instance not terminated
     Condition: instance_uptime_seconds > 43200 (12h max)

   - Alert: High cost
     Condition: training_cost_usd > 50 (safety threshold)
   ```

3. **Cost Tracking**
   ```sql
   -- PostgreSQL table for cost tracking
   CREATE TABLE ml_training_costs (
       id SERIAL PRIMARY KEY,
       model_name TEXT NOT NULL,
       gpu_type TEXT NOT NULL,
       provider TEXT NOT NULL,
       duration_seconds INTEGER NOT NULL,
       cost_usd DECIMAL(10, 2) NOT NULL,
       checkpoint_path TEXT NOT NULL,
       trained_at TIMESTAMP NOT NULL DEFAULT NOW()
   );

   -- Query monthly costs
   SELECT
       DATE_TRUNC('month', trained_at) AS month,
       SUM(cost_usd) AS total_cost,
       COUNT(*) AS training_runs
   FROM ml_training_costs
   WHERE trained_at >= NOW() - INTERVAL '1 year'
   GROUP BY month
   ORDER BY month DESC;
   ```

**Validation**:
- ✅ Weekly retraining runs automatically
- ✅ All 4 models update every week
- ✅ Costs track below $50/month budget
- ✅ Alerts trigger on failures or cost overruns
- ✅ Grafana dashboard shows training metrics

**Estimated Time**: 1 week (includes monitoring setup)

---

## 💡 Cost Optimization Tips

### 1. Use Spot Instances Aggressively

**Savings**: 40-70% vs on-demand

```bash
# RunPod: Spot instances are default (no flag needed)
# AWS: Add --spot flag to SageMaker training jobs
# GCP: Use --preemptible flag
```

**Risk Mitigation**:
- ✅ Checkpoint every 10 epochs (already implemented)
- ✅ Resume from last checkpoint on preemption
- ✅ Retry up to 3 times before falling back to on-demand

---

### 2. Auto-Shutdown After Training

**Savings**: 100% of idle costs

```bash
# RunPod: Enable auto-shutdown in config
auto_shutdown = true
max_idle_minutes = 10  # Shutdown after 10 min idle

# AWS: Use SageMaker Training Jobs (auto-terminate on completion)
# GCP: Use Cloud Functions to monitor instance state
```

**Implementation**:
```rust
// Cloud GPU Orchestrator
pub async fn monitor_training(&self, instance: &CloudInstance) -> Result<()> {
    loop {
        let status = self.get_training_status(instance).await?;

        match status {
            TrainingStatus::Complete => {
                self.download_models(instance).await?;
                self.terminate_instance(instance).await?;
                break;
            }
            TrainingStatus::Failed => {
                self.retry_training(instance).await?;
            }
            TrainingStatus::Running => {
                tokio::time::sleep(Duration::from_secs(60)).await;
            }
        }
    }
    Ok(())
}
```

---

### 3. Batch Multiple Training Runs

**Savings**: Reduce startup overhead (5-10 min per instance)

```bash
# Instead of:
# 1. Train MAMBA-2 → terminate → restart → train DQN → ...

# Do this:
# 1. Train all 4 models sequentially in one instance session

# Example:
ssh root@runpod-instance << 'EOF'
  cargo run --example train_mamba2_parquet --release --features cuda -- --epochs 50
  cargo run --example train_dqn --release --features cuda -- --epochs 100
  cargo run --example train_ppo_parquet --release --features cuda -- --epochs 50
  cargo run --example train_tft_parquet --release --features cuda -- --epochs 50
EOF
```

**Time Savings**:
- **Before**: 4 models × (2 min training + 5 min startup) = 28 min
- **After**: 1 startup (5 min) + 4 models × 2 min = 13 min
- **Savings**: 15 min = 54% reduction

**Cost Savings** (RunPod @ $0.34/hr):
- **Before**: 28 min × $0.34/hr ÷ 60 = $0.16
- **After**: 13 min × $0.34/hr ÷ 60 = $0.07
- **Savings**: $0.09 per training session (56% reduction)

---

### 4. Dataset Caching on Cloud Storage

**Savings**: Avoid re-uploading 100MB+ Parquet files every run

```bash
# Option 1: RunPod Network Storage (persistent volume)
# - $0.10/GB/month
# - 1GB Parquet files = $0.10/month
# - Saves 5-10 min upload time per run

# Option 2: S3 (if using AWS)
# - $0.023/GB/month (standard storage)
# - 1GB Parquet files = $0.023/month
# - Fast download from S3 to EC2 (same region)

# Implementation:
# 1. Upload Parquet files once to persistent storage
# 2. Mount volume on instance startup
# 3. No re-upload needed for subsequent runs
```

**Cost-Benefit Analysis**:
- **Storage Cost**: $0.10/month (RunPod) or $0.023/month (S3)
- **Upload Time Saved**: 5-10 min per run × $0.34/hr = $0.03-$0.06 per run
- **Break-Even**: 2-3 runs per month

**Verdict**: Worth it if retraining >2x per month.

---

### 5. Reserved Instances (Long-Term)

**Savings**: 30-50% vs on-demand (if using cloud GPU continuously)

| Provider | GPU | On-Demand | Reserved (1-year) | Savings |
|----------|-----|-----------|-------------------|---------|
| AWS | ml.g5.xlarge | $1.41/hr | $0.85/hr | 40% |
| Lambda Labs | A100 80GB | $3.29/hr | $2.00/hr | 39% |
| RunPod | RTX 4090 | $0.59/hr | N/A (spot only) | N/A |

**When to Use**:
- ✅ Training >100 hours/month consistently
- ✅ Multi-tenant workloads (shared GPU across projects)
- ❌ NOT for Foxhunt (infrequent training, <20h/month)

**Verdict**: Stick with spot instances for now.

---

### 6. GPU Selection Strategy

**Heuristic**: Match GPU to model memory requirements

| Model | GPU Memory | Recommended GPU | $/hour | Notes |
|-------|------------|-----------------|--------|-------|
| DQN | 6MB | RTX 3090 (24GB) | $0.27 | Smallest GPU works |
| PPO | 145MB | RTX 3090 (24GB) | $0.27 | Smallest GPU works |
| TFT-INT8 | 125MB | RTX 4090 (24GB) | $0.34 | Need CUDA 12.x for INT8 |
| MAMBA-2 | 164MB | RTX 4090 (24GB) | $0.34 | Best performance |

**Optimization**:
- Train DQN + PPO on RTX 3090 ($0.27/hr)
- Train TFT + MAMBA-2 on RTX 4090 ($0.34/hr)
- **Total Savings**: 20% vs using RTX 4090 for all models

**Verdict**: Micro-optimization, not worth the complexity. Stick with RTX 4090 for simplicity.

---

## 🔒 Security & Compliance

### Data Security

**Threat Model**:
- ✅ Parquet files contain **public market data** (no PII, no secrets)
- ⚠️ Trained models are **proprietary IP** (protect checkpoints)
- ⚠️ API keys (RunPod, AWS) must be **secured in Vault**

**Mitigations**:
1. **Encrypt Parquet Files at Rest** (S3 SSE or RunPod volume encryption)
2. **Encrypt Model Checkpoints** (AES-256 before download)
3. **Rotate API Keys** (every 90 days, stored in Vault)
4. **Audit Logs** (track all cloud GPU provisioning events)

### Compliance

**Foxhunt Requirements**:
- ✅ SOC2 Type II (if deploying to production)
- ✅ GDPR (not applicable - no EU customer data)
- ✅ PCI-DSS (not applicable - no payment card data)

**Provider Compliance**:

| Provider | SOC2 | ISO 27001 | GDPR | HIPAA | Notes |
|----------|------|-----------|------|-------|-------|
| AWS | ✅ | ✅ | ✅ | ✅ | Full compliance suite |
| GCP | ✅ | ✅ | ✅ | ✅ | Full compliance suite |
| Azure | ✅ | ✅ | ✅ | ✅ | Full compliance suite |
| Lambda Labs | ⚠️ | ❌ | ⚠️ | ❌ | Limited compliance |
| RunPod | ⚠️ | ❌ | ⚠️ | ❌ | Limited compliance |
| Vast.ai | ❌ | ❌ | ❌ | ❌ | No compliance certifications |

**Verdict**:
- **Development/Testing**: RunPod is fine (public data only)
- **Production (if regulated)**: Use AWS SageMaker for compliance

---

## 📈 Performance Validation

### Expected Speedup: RTX 4090 vs RTX 3050 Ti

**Hardware Comparison**:

| Metric | RTX 3050 Ti (Mobile) | RTX 4090 | Improvement |
|--------|----------------------|----------|-------------|
| **CUDA Cores** | 2,560 | 16,384 | **6.4x** |
| **Tensor Cores** | 80 (3rd Gen) | 512 (4th Gen) | **6.4x** |
| **VRAM** | 4GB GDDR6 | 24GB GDDR6X | **6.0x** |
| **Memory Bandwidth** | 112 GB/s | 1,008 GB/s | **9.0x** |
| **TDP** | 60W | 450W | 7.5x |
| **FP16 TFLOPS** | 9.0 | 82.6 | **9.2x** |
| **Architecture** | Ampere | Ada Lovelace | 1 gen newer |

**Estimated Training Time Reduction**:

| Model | RTX 3050 Ti (Current) | RTX 4090 (Estimated) | Speedup |
|-------|----------------------|---------------------|---------|
| **MAMBA-2** (30 epochs) | 1.9 min | **0.3-0.5 min** | **3.8-6.3x** |
| **DQN** (100 epochs) | 15 sec | **2-4 sec** | **3.8-7.5x** |
| **PPO** (30 epochs) | 7 sec | **1-2 sec** | **3.5-7.0x** |
| **TFT-INT8** (50 epochs) | 3.5 min | **0.5-1.0 min** | **3.5-7.0x** |
| **Total (4 models)** | 5.5 min | **1.0-1.5 min** | **3.7-5.5x** |

**Assumptions**:
- Speedup limited by memory bandwidth (9x theoretical → 4-6x practical)
- Rust `candle` framework scales well with CUDA cores
- INT8 quantization benefits from 4th Gen Tensor Cores

---

## 🎯 ROI Analysis: Cloud GPU vs Local GPU Upgrade

### Scenario 1: Buy RTX 4090 Local ($1,600)

**Upfront Cost**: $1,600 (GPU only, excluding PSU upgrade, cooling, installation)

**Ongoing Costs**:
- Electricity: ~450W × 10h/month × $0.12/kWh = $5.40/month
- Depreciation: $1,600 ÷ 24 months = $66.67/month
- **Total**: $72/month

**Pros**:
- ✅ Zero network latency
- ✅ Immediate access (no provisioning delay)
- ✅ Unlimited training time (no per-hour charges)

**Cons**:
- ❌ High upfront cost ($1,600)
- ❌ Obsolescence risk (GPU depreciates 50% in 2 years)
- ❌ Power/cooling requirements (450W TDP)
- ❌ Single GPU (no scaling to multi-GPU)

---

### Scenario 2: Use RunPod Cloud GPU ($34/month)

**Upfront Cost**: $0 (pay-as-you-go)

**Ongoing Costs**:
- Training: 10h/month × $0.34/hr = $3.40/month
- Storage: 1GB Parquet × $0.10/GB/month = $0.10/month
- **Total**: $3.50/month

**Pros**:
- ✅ Zero upfront cost
- ✅ Pay only for actual usage (no idle waste)
- ✅ Scalable (upgrade to A100/H100 if needed)
- ✅ No hardware maintenance

**Cons**:
- ❌ Network upload/download overhead (5-10 min)
- ❌ Spot instance preemption risk (rare)
- ❌ Requires cloud GPU orchestration code

---

### ROI Comparison (2-Year Horizon)

| Metric | Local RTX 4090 | RunPod Cloud GPU | Winner |
|--------|----------------|------------------|--------|
| **Upfront Cost** | $1,600 | $0 | ☁️ Cloud |
| **Year 1 Cost** | $1,600 + $65 = $1,665 | $42 | ☁️ Cloud |
| **Year 2 Cost** | $65/month × 12 = $780 | $42 | ☁️ Cloud |
| **2-Year Total** | $2,445 | $84 | ☁️ Cloud |
| **Savings** | - | **$2,361** | ☁️ Cloud |
| **ROI** | - | **2,815%** | ☁️ Cloud |
| **Performance** | 6.4x faster than RTX 3050 Ti | 6.4x faster than RTX 3050 Ti | 🟰 Tie |
| **Flexibility** | Single GPU, fixed | Upgrade to A100/H100 anytime | ☁️ Cloud |

**Verdict**: **Cloud GPU (RunPod) wins decisively** for infrequent training workloads (<100h/month).

---

## 🚨 Risk Assessment

### Technical Risks

| Risk | Likelihood | Impact | Mitigation |
|------|------------|--------|------------|
| **Spot instance preemption** | Medium | Low | Checkpoint every 10 epochs, auto-retry |
| **Network upload/download failures** | Low | Medium | Use rsync with resume, retry 3x |
| **RunPod API downtime** | Low | High | Fallback to local GPU, multi-provider support |
| **CUDA version mismatch** | Low | High | Pin CUDA 12.1 in Dockerfile, test before production |
| **Cost overrun** | Medium | Medium | Set hard cost limits ($50/month), alerts on Grafana |
| **Model checkpoint corruption** | Low | High | Validate checksums, keep last 3 checkpoints |

### Business Risks

| Risk | Likelihood | Impact | Mitigation |
|------|------------|--------|------------|
| **RunPod pricing increase** | Medium | Low | Multi-provider strategy (add AWS as backup) |
| **Vendor lock-in** | Low | Medium | Abstract cloud provider interface |
| **Compliance issues** | Low | High | Use AWS SageMaker for regulated workloads |
| **Data exfiltration** | Low | Critical | Encrypt checkpoints, audit access logs |

**Overall Risk Score**: **Low** (acceptable for production deployment)

---

## 📋 Implementation Checklist

### Phase 1: Manual Cloud Training (Week 1)

- [ ] Create RunPod account + add payment method
- [ ] Build Docker image with Rust + CUDA 12.1
- [ ] Push Docker image to DockerHub or RunPod registry
- [ ] Provision RTX 4090 spot instance
- [ ] Upload test Parquet files (ES_FUT_small.parquet)
- [ ] Run MAMBA-2 training via SSH
- [ ] Validate GPU utilization (>70%)
- [ ] Download checkpoint, verify integrity
- [ ] Terminate instance, confirm cost ($0.34/hr)
- [ ] Document workflow in `docs/cloud_gpu_manual_workflow.md`

### Phase 2: Automated Cloud Training (Weeks 2-4)

- [ ] Add `cloud_gpu` section to `config/config.toml`
- [ ] Implement `CloudGpuOrchestrator` in ML Training Service
- [ ] Add RunPod API integration (provision/terminate instances)
- [ ] Implement SSH command execution with log streaming
- [ ] Add rsync integration for data upload/download
- [ ] Create TLI command: `tli ml train-cloud`
- [ ] Add unit tests for cloud GPU orchestration
- [ ] Add integration test (end-to-end workflow)
- [ ] Document TLI usage in `ML_TRAINING_PARQUET_GUIDE.md`

### Phase 3: Production Deployment (Week 5)

- [ ] Add training scheduler (cron: every Sunday 2am)
- [ ] Implement Prometheus metrics (cost, duration, GPU utilization)
- [ ] Create Grafana dashboard for cloud GPU training
- [ ] Set up Slack/email alerts for failures
- [ ] Add PostgreSQL table for cost tracking
- [ ] Run dry-run production test (all 4 models, 50 epochs)
- [ ] Validate costs (<$50/month)
- [ ] Update CLAUDE.md with cloud GPU status
- [ ] Create production runbook (`docs/cloud_gpu_runbook.md`)

---

## 🎉 Success Criteria

### Technical Success

- ✅ All 4 models (MAMBA-2, DQN, PPO, TFT) train successfully on cloud GPU
- ✅ Training time **3-5x faster** than local RTX 3050 Ti
- ✅ Checkpoints download cleanly and validate via checksum
- ✅ GPU utilization **>70%** during training (no bottlenecks)
- ✅ Zero CUDA out-of-memory errors (24GB VRAM sufficient)

### Cost Success

- ✅ 4-model training session costs **<$10** (20 hours max)
- ✅ Monthly retraining costs **<$50/month**
- ✅ Cloud GPU **2,815% ROI** vs local GPU purchase ($84 vs $2,445 over 2 years)
- ✅ Spot instances used **>80%** of the time (vs on-demand)

### Operational Success

- ✅ TLI command (`tli ml train-cloud`) provisions GPU in **<5 min**
- ✅ Training logs stream to TLI in **real-time**
- ✅ Auto-shutdown after training (**zero idle costs**)
- ✅ Grafana alerts trigger on cost overruns or failures
- ✅ Runbook created for common issues (preemption, failures, cost spikes)

---

## 📚 References

### Documentation
- [ML_TRAINING_PARQUET_GUIDE.md](/home/jgrusewski/Work/foxhunt/ML_TRAINING_PARQUET_GUIDE.md) - Parquet training guide
- [CLAUDE.md](/home/jgrusewski/Work/foxhunt/CLAUDE.md) - System architecture
- [AGENT_33_TFT_INT8_QUANTIZATION_FIX.md](/home/jgrusewski/Work/foxhunt/AGENT_33_TFT_INT8_QUANTIZATION_FIX.md) - INT8 optimizations

### Cloud GPU Providers
- **RunPod**: https://www.runpod.io/pricing
- **Lambda Labs**: https://lambda.ai/pricing
- **Vast.ai**: https://vast.ai/pricing
- **AWS SageMaker**: https://aws.amazon.com/sagemaker/pricing/
- **GCP Vertex AI**: https://cloud.google.com/vertex-ai/pricing
- **Azure ML**: https://azure.microsoft.com/en-us/pricing/details/machine-learning/

### Research Sources
- [7 Cheapest Cloud GPU Providers in 2025](https://northflank.com/blog/cheapest-cloud-gpu-providers)
- [Cloud GPU Pricing Comparison 2025](https://datacrunch.io/blog/cloud-gpu-pricing-comparison)
- [NVIDIA A100 Pricing Showdown](https://www.thundercompute.com/blog/a100-gpu-pricing-showdown-2025-who-s-the-cheapest-for-deep-learning-workloads)
- [AWS SageMaker Pricing Guide](https://www.cloudzero.com/blog/sagemaker-pricing/)

---

## 🏁 Conclusion

**Final Recommendation**: **RunPod Community Cloud (RTX 4090 Spot Instances)**

**Why**:
- ✅ **Best Price**: $0.34/hr (68% cheaper than on-demand)
- ✅ **Perfect Fit**: 24GB VRAM handles all 4 models with 5.5x headroom
- ✅ **Performance**: 6.4x more CUDA cores than RTX 3050 Ti
- ✅ **Flexibility**: Per-second billing, 31 regions, auto-shutdown
- ✅ **ROI**: 2,815% return vs buying local RTX 4090 ($84 vs $2,445 over 2 years)
- ✅ **Integration**: Docker + SSH + rsync = minimal code changes

**Next Steps**:
1. **Immediate** (This Week): Sign up for RunPod, run manual training test
2. **Short-Term** (2-3 Weeks): Implement `tli ml train-cloud` command
3. **Long-Term** (4-6 Weeks): Deploy automated weekly retraining

**Total Implementation Time**: 5 weeks (Phase 1-3)
**Total Cost**: $3.40 - $6.80 (initial training) + $34/month (ongoing)
**Expected Speedup**: 3-5x faster training
**Expected Savings**: $2,361 over 2 years vs local GPU

---

**End of AGENT-35 Report**

---

## 📎 Appendix A: RunPod Quick Start Guide

```bash
# 1. Create RunPod account
https://www.runpod.io/signup

# 2. Add payment method
https://www.runpod.io/console/billing

# 3. Deploy RTX 4090 instance
# - GPU: RTX 4090 (24GB)
# - Type: Community Cloud (spot pricing)
# - Template: CUDA 12.1 + Ubuntu 22.04
# - Disk: 50GB (sufficient for Foxhunt codebase)

# 4. SSH into instance
ssh root@<instance-ip>

# 5. Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source $HOME/.cargo/env

# 6. Clone Foxhunt repo
git clone https://github.com/<your-org>/foxhunt.git
cd foxhunt

# 7. Upload Parquet files
# (On local machine)
rsync -avz --progress test_data/*.parquet root@<instance-ip>:~/foxhunt/test_data/

# 8. Build with CUDA
cargo build --release --features cuda -p ml

# 9. Train models
cargo run -p ml --example train_mamba2_parquet --release --features cuda -- \
  --parquet-file test_data/ES_FUT_180d.parquet --epochs 50

# 10. Download checkpoints
# (On local machine)
rsync -avz --progress root@<instance-ip>:~/foxhunt/ml/trained_models/ \
  ml/trained_models/

# 11. Terminate instance
# (RunPod web console)
# Click "Stop" → Confirm → Billing stops
```

---

## 📎 Appendix B: Cost Calculator

```python
#!/usr/bin/env python3
"""
Cloud GPU Cost Calculator for Foxhunt ML Training

Usage:
  python cost_calculator.py --hours 10 --gpu rtx-4090 --provider runpod
"""

PRICING = {
    "runpod": {
        "rtx-4090": {"spot": 0.34, "on_demand": 0.59},
        "rtx-3090": {"spot": 0.27, "on_demand": 0.43},
        "a100-80gb": {"spot": 1.64, "on_demand": 1.99},
    },
    "lambda": {
        "a100-80gb": {"on_demand": 3.29},
        "h100-80gb": {"on_demand": 3.29},
    },
    "aws": {
        "ml.g5.xlarge": {"spot": 0.42, "on_demand": 1.41},
        "ml.p4d.24xlarge": {"spot": 9.83, "on_demand": 32.77},
    },
}

def calculate_cost(hours, gpu, provider, instance_type="spot"):
    price = PRICING[provider][gpu][instance_type]
    total = hours * price
    return {
        "hours": hours,
        "gpu": gpu,
        "provider": provider,
        "instance_type": instance_type,
        "price_per_hour": price,
        "total_cost": round(total, 2),
    }

# Example: 10 hours on RunPod RTX 4090 (spot)
result = calculate_cost(10, "rtx-4090", "runpod", "spot")
print(f"Total cost: ${result['total_cost']} ({result['hours']}h × ${result['price_per_hour']}/hr)")

# Example: Monthly cost (10h/month for 12 months)
monthly = result['total_cost']
annual = monthly * 12
print(f"Monthly cost: ${monthly}")
print(f"Annual cost: ${annual}")
```

**Output**:
```
Total cost: $3.40 (10h × $0.34/hr)
Monthly cost: $3.40
Annual cost: $40.80
```