foxhunt/docs/archive/agents/AGENT_87_HANDOFF.md

Agent 87: Complete MAMBA-2 & TFT Benchmarks

Handoff from: Agent 86 (GPU Performance Benchmarking) Task: Complete remaining benchmarks (MAMBA-2, TFT) to enable 4-6 week training decision

---

Context

Agent 86 discovered that Wave 152 benchmark system only tested 2 of 4 trainable models:

• ✅ DQN: 0.149 ms/epoch, 135 MB VRAM, 2.5 min for 1K epochs
• ✅ PPO: 181.9 ms/epoch, 135 MB VRAM, 6.1 min for 2K epochs
• ❌ MAMBA-2: NOT TESTED (module exists, not called by coordinator)
• ❌ TFT: NOT TESTED (module exists, not called by coordinator)
• ❌ TLOB: EXCLUDED (inference-only, no training needed)

Current Decision: local_gpu ✅ (6.1 min << 24h) but only for DQN+PPO

Missing Data: Cannot validate 4-6 week training timeline without MAMBA-2/TFT benchmarks.

---

Your Mission

Complete GPU benchmark suite with all 4 trainable models to enable informed training timeline decision.

Expected Timeline: 2 hours total

1. Update coordinator (15 min)
2. Run full benchmark (30-60 min)
3. Analyze results (30 min)

---

Step 1: Update Benchmark Coordinator (15 min)

File: /home/jgrusewski/Work/foxhunt/ml/examples/gpu_training_benchmark.rs

Required Changes

1. Add Imports (top of file)

use ml::benchmark::{
    DqnBenchmarkResult, DqnBenchmarkRunner,
    PpoBenchmarkResult, PpoBenchmarkRunner,
    Mamba2BenchmarkResult, Mamba2BenchmarkRunner,  // ADD THIS
    TftBenchmarkResult, TftBenchmarkRunner,        // ADD THIS
    GpuHardwareManager,
};

2. Update BenchmarkReport Struct (around line 147)

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BenchmarkReport {
    pub timestamp: String,
    pub gpu_info: GpuInfo,
    pub data_info: DataInfo,
    pub dqn_results: DqnBenchmarkResult,
    pub ppo_results: PpoBenchmarkResult,
    pub mamba2_results: Mamba2BenchmarkResult,  // ADD THIS
    pub tft_results: TftBenchmarkResult,        // ADD THIS
    pub aggregate_metrics: AggregateMetrics,
    pub decision: TrainingDecision,
}

3. Add Benchmark Methods (after line 297)

/// Run MAMBA-2 benchmark
async fn run_mamba2_benchmark(&mut self) -> Result<Mamba2BenchmarkResult> {
    let mut runner = Mamba2BenchmarkRunner::new(self.gpu_manager.clone());
    runner
        .run_benchmark(self.opts.epochs)
        .await
        .context("MAMBA-2 benchmark failed")
}

/// Run TFT benchmark
async fn run_tft_benchmark(&mut self) -> Result<TftBenchmarkResult> {
    let mut runner = TftBenchmarkRunner::new(self.gpu_manager.clone());
    runner
        .run_benchmark(self.opts.epochs)
        .await
        .context("TFT benchmark failed")
}

4. Update run() Method (around line 204)

// After PPO benchmark (line 220), add:

// Step 5: Run MAMBA-2 benchmark
info!("\n📊 Running MAMBA-2 Benchmark...");
let mamba2_results = self.run_mamba2_benchmark().await?;
info!(
    "✅ MAMBA-2 Complete: {:.2}s/epoch (peak: {:.1}MB VRAM)",
    mamba2_results.statistics.mean_seconds,
    mamba2_results.memory_peak_mb
);

// Step 6: Run TFT benchmark
info!("\n📊 Running TFT Benchmark...");
let tft_results = self.run_tft_benchmark().await?;
info!(
    "✅ TFT Complete: {:.2}s/epoch (peak: {:.1}MB VRAM)",
    tft_results.statistics.mean_seconds,
    tft_results.memory_peak_mb
);

5. Update compute_aggregate_metrics() (line 299)

fn compute_aggregate_metrics(
    &self,
    dqn: &DqnBenchmarkResult,
    ppo: &PpoBenchmarkResult,
    mamba2: &Mamba2BenchmarkResult,  // ADD PARAM
    tft: &TftBenchmarkResult,        // ADD PARAM
) -> AggregateMetrics {
    // Training epochs (from GPU_TRAINING_BENCHMARK.md)
    let dqn_full_epochs = 1000.0;
    let ppo_full_epochs = 2000.0;
    let mamba2_full_epochs = 1000.0;  // ADD THIS
    let tft_full_epochs = 1500.0;     // ADD THIS

    let dqn_total_hours = (dqn.statistics.mean_seconds * dqn_full_epochs) / 3600.0;
    let ppo_total_hours = (ppo.statistics.mean_seconds * ppo_full_epochs) / 3600.0;
    let mamba2_total_hours = (mamba2.statistics.mean_seconds * mamba2_full_epochs) / 3600.0;  // ADD
    let tft_total_hours = (tft.statistics.mean_seconds * tft_full_epochs) / 3600.0;          // ADD

    let total_training_time_hours = dqn_total_hours + ppo_total_hours
                                    + mamba2_total_hours + tft_total_hours;  // UPDATE

    // Peak memory
    let total_memory_peak_mb = dqn.memory_peak_mb
        .max(ppo.memory_peak_mb)
        .max(mamba2.memory_peak_mb)   // ADD
        .max(tft.memory_peak_mb);     // ADD

    // All stable
    let all_stable = dqn.stability.is_stable
                     && ppo.stability.is_stable
                     && mamba2.stability.is_stable  // ADD
                     && tft.stability.is_stable;    // ADD

    AggregateMetrics {
        total_training_time_hours,
        total_memory_peak_mb,
        all_stable,
        models_tested: vec![
            "DQN".to_string(),
            "PPO".to_string(),
            "MAMBA-2".to_string(),  // ADD
            "TFT".to_string()       // ADD
        ],
    }
}

6. Update print_summary() (line 418)

// After PPO results (line 454), add:

println!("\n--- MAMBA-2 Results ---");
println!(
    "  • Mean epoch time: {:.3}s (P50: {:.3}s, P95: {:.3}s)",
    report.mamba2_results.statistics.mean_seconds,
    report.mamba2_results.statistics.p50_median,
    report.mamba2_results.statistics.p95
);
println!("  • Peak memory: {:.1}MB", report.mamba2_results.memory_peak_mb);
println!("  • Training stable: {}", report.mamba2_results.stability.is_stable);

println!("\n--- TFT Results ---");
println!(
    "  • Mean epoch time: {:.3}s (P50: {:.3}s, P95: {:.3}s)",
    report.tft_results.statistics.mean_seconds,
    report.tft_results.statistics.p50_median,
    report.tft_results.statistics.p95
);
println!("  • Peak memory: {:.1}MB", report.tft_results.memory_peak_mb);
println!("  • Training stable: {}", report.tft_results.stability.is_stable);

7. Update Report Generation (line 240)

let report = BenchmarkReport {
    timestamp: Utc::now().to_rfc3339(),
    gpu_info,
    data_info,
    dqn_results,
    ppo_results,
    mamba2_results,  // ADD
    tft_results,     // ADD
    aggregate_metrics,
    decision,
};

8. Update Method Calls (line 223)

// Change from:
let aggregate_metrics = self.compute_aggregate_metrics(&dqn_results, &ppo_results);

// To:
let aggregate_metrics = self.compute_aggregate_metrics(
    &dqn_results,
    &ppo_results,
    &mamba2_results,
    &tft_results
);

---

Step 2: Run Full Benchmark (30-60 min)

Command

cd /home/jgrusewski/Work/foxhunt

# Compile first (verify no errors)
cargo build -p ml --example gpu_training_benchmark --release

# Run full benchmark (all 4 models, 500 epochs each)
cargo run -p ml --example gpu_training_benchmark --release -- \
  --epochs 500 \
  --verbose \
  --output ml/benchmark_results/gpu_benchmark_full_$(date +%Y%m%d_%H%M%S).json

Expected Output

📊 Running DQN Benchmark...
✅ DQN Complete: 0.00s/epoch (peak: 135.0MB VRAM)

📊 Running PPO Benchmark...
✅ PPO Complete: 0.18s/epoch (peak: 135.0MB VRAM)

📊 Running MAMBA-2 Benchmark...
✅ MAMBA-2 Complete: 1.20s/epoch (peak: 300.0MB VRAM)  <-- ESTIMATE

📊 Running TFT Benchmark...
✅ TFT Complete: 0.50s/epoch (peak: 2000.0MB VRAM)     <-- ESTIMATE

📈 Aggregate Metrics: X.XX hours total, XXXX.XMB peak memory

🎯 Decision: LOCAL_GPU / CLOUD_GPU / EITHER
   Rationale: [decision reasoning]
   Local cost: $X.XX, Cloud cost: $X.XX

📄 Report saved to: ml/benchmark_results/gpu_benchmark_full_20251014_XXXXXX.json

Expected Duration

• DQN: ~1 second (already fast)
• PPO: ~90 seconds (already measured)
• MAMBA-2: ~10-15 minutes (SSM complexity)
• TFT: ~4-6 minutes (transformer attention)
• Total: 30-60 minutes (including overhead)

Monitoring

# Monitor GPU in separate terminal
watch -n 1 nvidia-smi

# Check for VRAM usage spikes (TFT expected to use ~2GB)

---

Step 3: Analyze Results (30 min)

1. Read JSON Report

# Find latest report
ls -lt /home/jgrusewski/Work/foxhunt/ml/benchmark_results/ | head -5

# Pretty-print JSON
cat ml/benchmark_results/gpu_benchmark_full_XXXXXX.json | jq .

2. Extract Key Metrics

# Total training time
jq '.aggregate_metrics.total_training_time_hours' report.json

# Decision recommendation
jq '.decision.recommendation' report.json

# Peak VRAM per model
jq '{dqn: .dqn_results.memory_peak_mb, ppo: .ppo_results.memory_peak_mb, mamba2: .mamba2_results.memory_peak_mb, tft: .tft_results.memory_peak_mb}' report.json

# Stability per model
jq '{dqn: .dqn_results.stability.is_stable, ppo: .ppo_results.stability.is_stable, mamba2: .mamba2_results.stability.is_stable, tft: .tft_results.stability.is_stable}' report.json

3. Update Decision Analysis

Create AGENT_87_FINAL_DECISION.md with:

• Complete benchmark results (all 4 models)
• Total training time estimate (1K DQN + 2K PPO + 1K MAMBA-2 + 1.5K TFT epochs)
• Decision recommendation (local_gpu / cloud_gpu / either)
• Cost analysis (local electricity vs cloud GPU rental)
• Risk assessment (memory bottlenecks, stability issues)
• Next steps (production training or hyperparameter tuning)

---

Success Criteria

✅ All 4 models benchmarked (DQN, PPO, MAMBA-2, TFT) ✅ JSON report generated with complete results ✅ Decision recommendation provided (local_gpu / cloud_gpu / either) ✅ Peak VRAM measured for each model (especially TFT) ✅ Stability validated for each model ✅ Statistical confidence >95% (from 500 epochs) ✅ Total training time estimate calculated

---

Known Risks

HIGH RISK: TFT Memory Bottleneck

Issue: TFT requires 1.5-2.5GB VRAM (37-61% of 4GB GPU)

Symptoms:

• CUDA out-of-memory error during TFT benchmark
• GPU utilization drops to 0%
• Process crashes

Mitigation:

1. TFT benchmark already constrains batch_size to max=4
2. If still OOM, reduce to batch_size=2 (2x slower training)
3. Enable gradient accumulation (effective_batch_size = 4-8)

Fallback: If TFT fails on RTX 3050 Ti, recommend cloud GPU for TFT only (AWS g4dn.xlarge with 16GB VRAM)

MEDIUM RISK: DQN Divergence

Issue: DQN loss diverging (0.225 → 0.273) in existing benchmarks

Impact: Cannot deploy DQN to production without fixing

Mitigation: Flag in report, recommend Agent 88 debug task (1-2 days hyperparameter tuning)

---

Expected Outcomes

Scenario 1: Local GPU Viable (<24h)

Decision: local_gpu ✅ Cost: ~$0.50 electricity Timeline: Execute production training immediately Next Agent: Agent 89 (Production Training)

Scenario 2: Gray Zone (24-48h)

Decision: either ⚠️ Cost: $1.08 local vs $12.62-$25.25 cloud Timeline: User decision required Next Agent: User choice, then Agent 89

Scenario 3: Cloud GPU Required (>48h)

Decision: cloud_gpu ❌ Cost: >$25.25 (AWS p3.2xlarge V100 @ $3.06/hr) Timeline: Provision cloud GPU, then production training Next Agent: Agent 88 (Cloud GPU Setup) → Agent 89

---

Deliverables

1. Updated Coordinator: ml/examples/gpu_training_benchmark.rs (all 4 models)
2. Benchmark Report: ml/benchmark_results/gpu_benchmark_full_XXXXXX.json
3. Decision Analysis: AGENT_87_FINAL_DECISION.md
4. Summary: AGENT_87_BENCHMARK_COMPLETE.txt (visual summary)

---

Quick Reference

Agent 86 Reports:

• /home/jgrusewski/Work/foxhunt/AGENT_86_GPU_BENCHMARK_ANALYSIS.md (15KB)
• /home/jgrusewski/Work/foxhunt/AGENT_86_LATEST_BENCHMARK.json (26KB)
• /home/jgrusewski/Work/foxhunt/AGENT_86_BENCHMARK_GAP_SUMMARY.txt (12KB)

Benchmark Modules:

• /home/jgrusewski/Work/foxhunt/ml/src/benchmark/dqn_benchmark.rs ✅
• /home/jgrusewski/Work/foxhunt/ml/src/benchmark/ppo_benchmark.rs ✅
• /home/jgrusewski/Work/foxhunt/ml/src/benchmark/mamba2_benchmark.rs ✅ (ready, not called)
• /home/jgrusewski/Work/foxhunt/ml/src/benchmark/tft_benchmark.rs ✅ (ready, not called)

GPU Status: RTX 3050 Ti, 4GB VRAM, 0% utilization, 59°C, IDLE, READY

---

Handoff Complete: Agent 86 → Agent 87 Estimated Time: 2 hours Priority: HIGH (blocks 4-6 week training decision) Next Agent: Agent 88 (DQN Stability Fix) or Agent 89 (Production Training) depending on results