foxhunt/AGENT_87_HANDOFF.md

# Agent 87: Complete MAMBA-2 & TFT Benchmarks

**Handoff from**: Agent 86 (GPU Performance Benchmarking)
**Task**: Complete remaining benchmarks (MAMBA-2, TFT) to enable 4-6 week training decision

---

## Context

Agent 86 discovered that Wave 152 benchmark system **only tested 2 of 4 trainable models**:
- ✅ **DQN**: 0.149 ms/epoch, 135 MB VRAM, 2.5 min for 1K epochs
- ✅ **PPO**: 181.9 ms/epoch, 135 MB VRAM, 6.1 min for 2K epochs
- ❌ **MAMBA-2**: NOT TESTED (module exists, not called by coordinator)
- ❌ **TFT**: NOT TESTED (module exists, not called by coordinator)
- ❌ **TLOB**: EXCLUDED (inference-only, no training needed)

**Current Decision**: local_gpu ✅ (6.1 min << 24h) but **only for DQN+PPO**

**Missing Data**: Cannot validate 4-6 week training timeline without MAMBA-2/TFT benchmarks.

---

## Your Mission

**Complete GPU benchmark suite with all 4 trainable models** to enable informed training timeline decision.

**Expected Timeline**: 2 hours total
1. Update coordinator (15 min)
2. Run full benchmark (30-60 min)
3. Analyze results (30 min)

---

## Step 1: Update Benchmark Coordinator (15 min)

**File**: `/home/jgrusewski/Work/foxhunt/ml/examples/gpu_training_benchmark.rs`

### Required Changes

#### 1. Add Imports (top of file)
```rust
use ml::benchmark::{
    DqnBenchmarkResult, DqnBenchmarkRunner,
    PpoBenchmarkResult, PpoBenchmarkRunner,
    Mamba2BenchmarkResult, Mamba2BenchmarkRunner,  // ADD THIS
    TftBenchmarkResult, TftBenchmarkRunner,        // ADD THIS
    GpuHardwareManager,
};
```

#### 2. Update BenchmarkReport Struct (around line 147)
```rust
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BenchmarkReport {
    pub timestamp: String,
    pub gpu_info: GpuInfo,
    pub data_info: DataInfo,
    pub dqn_results: DqnBenchmarkResult,
    pub ppo_results: PpoBenchmarkResult,
    pub mamba2_results: Mamba2BenchmarkResult,  // ADD THIS
    pub tft_results: TftBenchmarkResult,        // ADD THIS
    pub aggregate_metrics: AggregateMetrics,
    pub decision: TrainingDecision,
}
```

#### 3. Add Benchmark Methods (after line 297)
```rust
/// Run MAMBA-2 benchmark
async fn run_mamba2_benchmark(&mut self) -> Result<Mamba2BenchmarkResult> {
    let mut runner = Mamba2BenchmarkRunner::new(self.gpu_manager.clone());
    runner
        .run_benchmark(self.opts.epochs)
        .await
        .context("MAMBA-2 benchmark failed")
}

/// Run TFT benchmark
async fn run_tft_benchmark(&mut self) -> Result<TftBenchmarkResult> {
    let mut runner = TftBenchmarkRunner::new(self.gpu_manager.clone());
    runner
        .run_benchmark(self.opts.epochs)
        .await
        .context("TFT benchmark failed")
}
```

#### 4. Update run() Method (around line 204)
```rust
// After PPO benchmark (line 220), add:

// Step 5: Run MAMBA-2 benchmark
info!("\n📊 Running MAMBA-2 Benchmark...");
let mamba2_results = self.run_mamba2_benchmark().await?;
info!(
    "✅ MAMBA-2 Complete: {:.2}s/epoch (peak: {:.1}MB VRAM)",
    mamba2_results.statistics.mean_seconds,
    mamba2_results.memory_peak_mb
);

// Step 6: Run TFT benchmark
info!("\n📊 Running TFT Benchmark...");
let tft_results = self.run_tft_benchmark().await?;
info!(
    "✅ TFT Complete: {:.2}s/epoch (peak: {:.1}MB VRAM)",
    tft_results.statistics.mean_seconds,
    tft_results.memory_peak_mb
);
```

#### 5. Update compute_aggregate_metrics() (line 299)
```rust
fn compute_aggregate_metrics(
    &self,
    dqn: &DqnBenchmarkResult,
    ppo: &PpoBenchmarkResult,
    mamba2: &Mamba2BenchmarkResult,  // ADD PARAM
    tft: &TftBenchmarkResult,        // ADD PARAM
) -> AggregateMetrics {
    // Training epochs (from GPU_TRAINING_BENCHMARK.md)
    let dqn_full_epochs = 1000.0;
    let ppo_full_epochs = 2000.0;
    let mamba2_full_epochs = 1000.0;  // ADD THIS
    let tft_full_epochs = 1500.0;     // ADD THIS

    let dqn_total_hours = (dqn.statistics.mean_seconds * dqn_full_epochs) / 3600.0;
    let ppo_total_hours = (ppo.statistics.mean_seconds * ppo_full_epochs) / 3600.0;
    let mamba2_total_hours = (mamba2.statistics.mean_seconds * mamba2_full_epochs) / 3600.0;  // ADD
    let tft_total_hours = (tft.statistics.mean_seconds * tft_full_epochs) / 3600.0;          // ADD

    let total_training_time_hours = dqn_total_hours + ppo_total_hours
                                    + mamba2_total_hours + tft_total_hours;  // UPDATE

    // Peak memory
    let total_memory_peak_mb = dqn.memory_peak_mb
        .max(ppo.memory_peak_mb)
        .max(mamba2.memory_peak_mb)   // ADD
        .max(tft.memory_peak_mb);     // ADD

    // All stable
    let all_stable = dqn.stability.is_stable
                     && ppo.stability.is_stable
                     && mamba2.stability.is_stable  // ADD
                     && tft.stability.is_stable;    // ADD

    AggregateMetrics {
        total_training_time_hours,
        total_memory_peak_mb,
        all_stable,
        models_tested: vec![
            "DQN".to_string(),
            "PPO".to_string(),
            "MAMBA-2".to_string(),  // ADD
            "TFT".to_string()       // ADD
        ],
    }
}
```

#### 6. Update print_summary() (line 418)
```rust
// After PPO results (line 454), add:

println!("\n--- MAMBA-2 Results ---");
println!(
    "  • Mean epoch time: {:.3}s (P50: {:.3}s, P95: {:.3}s)",
    report.mamba2_results.statistics.mean_seconds,
    report.mamba2_results.statistics.p50_median,
    report.mamba2_results.statistics.p95
);
println!("  • Peak memory: {:.1}MB", report.mamba2_results.memory_peak_mb);
println!("  • Training stable: {}", report.mamba2_results.stability.is_stable);

println!("\n--- TFT Results ---");
println!(
    "  • Mean epoch time: {:.3}s (P50: {:.3}s, P95: {:.3}s)",
    report.tft_results.statistics.mean_seconds,
    report.tft_results.statistics.p50_median,
    report.tft_results.statistics.p95
);
println!("  • Peak memory: {:.1}MB", report.tft_results.memory_peak_mb);
println!("  • Training stable: {}", report.tft_results.stability.is_stable);
```

#### 7. Update Report Generation (line 240)
```rust
let report = BenchmarkReport {
    timestamp: Utc::now().to_rfc3339(),
    gpu_info,
    data_info,
    dqn_results,
    ppo_results,
    mamba2_results,  // ADD
    tft_results,     // ADD
    aggregate_metrics,
    decision,
};
```

#### 8. Update Method Calls (line 223)
```rust
// Change from:
let aggregate_metrics = self.compute_aggregate_metrics(&dqn_results, &ppo_results);

// To:
let aggregate_metrics = self.compute_aggregate_metrics(
    &dqn_results,
    &ppo_results,
    &mamba2_results,
    &tft_results
);
```

---

## Step 2: Run Full Benchmark (30-60 min)

### Command

```bash
cd /home/jgrusewski/Work/foxhunt

# Compile first (verify no errors)
cargo build -p ml --example gpu_training_benchmark --release

# Run full benchmark (all 4 models, 500 epochs each)
cargo run -p ml --example gpu_training_benchmark --release -- \
  --epochs 500 \
  --verbose \
  --output ml/benchmark_results/gpu_benchmark_full_$(date +%Y%m%d_%H%M%S).json
```

### Expected Output

```
📊 Running DQN Benchmark...
✅ DQN Complete: 0.00s/epoch (peak: 135.0MB VRAM)

📊 Running PPO Benchmark...
✅ PPO Complete: 0.18s/epoch (peak: 135.0MB VRAM)

📊 Running MAMBA-2 Benchmark...
✅ MAMBA-2 Complete: 1.20s/epoch (peak: 300.0MB VRAM)  <-- ESTIMATE

📊 Running TFT Benchmark...
✅ TFT Complete: 0.50s/epoch (peak: 2000.0MB VRAM)     <-- ESTIMATE

📈 Aggregate Metrics: X.XX hours total, XXXX.XMB peak memory

🎯 Decision: LOCAL_GPU / CLOUD_GPU / EITHER
   Rationale: [decision reasoning]
   Local cost: $X.XX, Cloud cost: $X.XX

📄 Report saved to: ml/benchmark_results/gpu_benchmark_full_20251014_XXXXXX.json
```

### Expected Duration
- DQN: ~1 second (already fast)
- PPO: ~90 seconds (already measured)
- **MAMBA-2**: ~10-15 minutes (SSM complexity)
- **TFT**: ~4-6 minutes (transformer attention)
- **Total**: 30-60 minutes (including overhead)

### Monitoring

```bash
# Monitor GPU in separate terminal
watch -n 1 nvidia-smi

# Check for VRAM usage spikes (TFT expected to use ~2GB)
```

---

## Step 3: Analyze Results (30 min)

### 1. Read JSON Report

```bash
# Find latest report
ls -lt /home/jgrusewski/Work/foxhunt/ml/benchmark_results/ | head -5

# Pretty-print JSON
cat ml/benchmark_results/gpu_benchmark_full_XXXXXX.json | jq .
```

### 2. Extract Key Metrics

```bash
# Total training time
jq '.aggregate_metrics.total_training_time_hours' report.json

# Decision recommendation
jq '.decision.recommendation' report.json

# Peak VRAM per model
jq '{dqn: .dqn_results.memory_peak_mb, ppo: .ppo_results.memory_peak_mb, mamba2: .mamba2_results.memory_peak_mb, tft: .tft_results.memory_peak_mb}' report.json

# Stability per model
jq '{dqn: .dqn_results.stability.is_stable, ppo: .ppo_results.stability.is_stable, mamba2: .mamba2_results.stability.is_stable, tft: .tft_results.stability.is_stable}' report.json
```

### 3. Update Decision Analysis

Create `AGENT_87_FINAL_DECISION.md` with:
- Complete benchmark results (all 4 models)
- Total training time estimate (1K DQN + 2K PPO + 1K MAMBA-2 + 1.5K TFT epochs)
- Decision recommendation (local_gpu / cloud_gpu / either)
- Cost analysis (local electricity vs cloud GPU rental)
- Risk assessment (memory bottlenecks, stability issues)
- Next steps (production training or hyperparameter tuning)

---

## Success Criteria

✅ All 4 models benchmarked (DQN, PPO, MAMBA-2, TFT)
✅ JSON report generated with complete results
✅ Decision recommendation provided (local_gpu / cloud_gpu / either)
✅ Peak VRAM measured for each model (especially TFT)
✅ Stability validated for each model
✅ Statistical confidence >95% (from 500 epochs)
✅ Total training time estimate calculated

---

## Known Risks

### HIGH RISK: TFT Memory Bottleneck

**Issue**: TFT requires 1.5-2.5GB VRAM (37-61% of 4GB GPU)

**Symptoms**:
- CUDA out-of-memory error during TFT benchmark
- GPU utilization drops to 0%
- Process crashes

**Mitigation**:
1. TFT benchmark already constrains batch_size to max=4
2. If still OOM, reduce to batch_size=2 (2x slower training)
3. Enable gradient accumulation (effective_batch_size = 4-8)

**Fallback**: If TFT fails on RTX 3050 Ti, recommend cloud GPU for TFT only (AWS g4dn.xlarge with 16GB VRAM)

### MEDIUM RISK: DQN Divergence

**Issue**: DQN loss diverging (0.225 → 0.273) in existing benchmarks

**Impact**: Cannot deploy DQN to production without fixing

**Mitigation**: Flag in report, recommend Agent 88 debug task (1-2 days hyperparameter tuning)

---

## Expected Outcomes

### Scenario 1: Local GPU Viable (<24h)
**Decision**: local_gpu ✅
**Cost**: ~$0.50 electricity
**Timeline**: Execute production training immediately
**Next Agent**: Agent 89 (Production Training)

### Scenario 2: Gray Zone (24-48h)
**Decision**: either ⚠️
**Cost**: $1.08 local vs $12.62-$25.25 cloud
**Timeline**: User decision required
**Next Agent**: User choice, then Agent 89

### Scenario 3: Cloud GPU Required (>48h)
**Decision**: cloud_gpu ❌
**Cost**: >$25.25 (AWS p3.2xlarge V100 @ $3.06/hr)
**Timeline**: Provision cloud GPU, then production training
**Next Agent**: Agent 88 (Cloud GPU Setup) → Agent 89

---

## Deliverables

1. **Updated Coordinator**: `ml/examples/gpu_training_benchmark.rs` (all 4 models)
2. **Benchmark Report**: `ml/benchmark_results/gpu_benchmark_full_XXXXXX.json`
3. **Decision Analysis**: `AGENT_87_FINAL_DECISION.md`
4. **Summary**: `AGENT_87_BENCHMARK_COMPLETE.txt` (visual summary)

---

## Quick Reference

**Agent 86 Reports**:
- `/home/jgrusewski/Work/foxhunt/AGENT_86_GPU_BENCHMARK_ANALYSIS.md` (15KB)
- `/home/jgrusewski/Work/foxhunt/AGENT_86_LATEST_BENCHMARK.json` (26KB)
- `/home/jgrusewski/Work/foxhunt/AGENT_86_BENCHMARK_GAP_SUMMARY.txt` (12KB)

**Benchmark Modules**:
- `/home/jgrusewski/Work/foxhunt/ml/src/benchmark/dqn_benchmark.rs` ✅
- `/home/jgrusewski/Work/foxhunt/ml/src/benchmark/ppo_benchmark.rs` ✅
- `/home/jgrusewski/Work/foxhunt/ml/src/benchmark/mamba2_benchmark.rs` ✅ (ready, not called)
- `/home/jgrusewski/Work/foxhunt/ml/src/benchmark/tft_benchmark.rs` ✅ (ready, not called)

**GPU Status**: RTX 3050 Ti, 4GB VRAM, 0% utilization, 59°C, IDLE, READY

---

**Handoff Complete**: Agent 86 → Agent 87
**Estimated Time**: 2 hours
**Priority**: HIGH (blocks 4-6 week training decision)
**Next Agent**: Agent 88 (DQN Stability Fix) or Agent 89 (Production Training) depending on results