foxhunt/AGENT_71_HANDOFF.md

# Agent 71 Handoff: Next Steps After Wave 160 Phase 3

**From**: Agent 70 (Wave 160 Phase 3 Completion Report)
**To**: Agent 71 (Model Validation & Next Steps)
**Date**: 2025-10-14
**Status**: 2/4 models production-ready, validation needed

---

## 🎯 Your Mission (Choose One)

### Option A: Model Validation (RECOMMENDED) - 1-2 hours
**Priority**: HIGH
**Goal**: Validate DQN and PPO models with backtesting before production deployment

### Option B: MAMBA-2 Fix - 4-6 hours
**Priority**: MEDIUM
**Goal**: Fix device mismatch to enable GPU training for MAMBA-2

### Option C: Documentation Update - 30 minutes
**Priority**: LOW
**Goal**: Update CLAUDE.md with Wave 160 Phase 3 status

---

## 📋 Option A: Model Validation (RECOMMENDED)

### Current Status
- ✅ DQN trained: 51 checkpoints, GPU-accelerated, 99.3% loss reduction
- ✅ PPO trained: 200 checkpoints, CPU-trained, zero NaN
- ⏳ Backtesting: NOT DONE
- ⏳ Performance metrics: NOT VALIDATED

### Your Tasks

#### Task 1: Backtest DQN (30-45 min)

**Command**:
```bash
cargo run -p backtesting_service --example backtest_dqn --release -- \
  --model ml/trained_models/production/dqn_real_data/dqn_final_epoch500.safetensors \
  --data test_data/real/databento/ml_training/6E.FUT_ohlcv-1m_2024-01-*.dbn \
  --output ml/backtest_results/dqn_validation.json \
  --initial-capital 100000 \
  --commission 0.0001
```

**Success Criteria**:
- ✅ Sharpe ratio > 1.0
- ✅ Max drawdown < 20%
- ✅ Win rate > 50%
- ✅ Total return > 0%

**Expected Output**:
```json
{
  "sharpe_ratio": 1.2,
  "max_drawdown": 0.15,
  "win_rate": 0.55,
  "total_return": 0.08,
  "num_trades": 150,
  "avg_trade_duration": "15m"
}
```

**If Backtesting Fails**:
1. Check if backtesting example exists: `ls ml/examples/backtest_dqn.rs`
2. If missing, create basic backtest script using model inference
3. Report findings in `AGENT_71_DQN_BACKTEST_REPORT.md`

---

#### Task 2: Backtest PPO (30-45 min)

**Command**:
```bash
cargo run -p backtesting_service --example backtest_ppo --release -- \
  --model ml/trained_models/production/ppo_checkpoint_epoch_500.safetensors \
  --data test_data/real/databento/ml_training/6E.FUT_ohlcv-1m_2024-01-*.dbn \
  --output ml/backtest_results/ppo_validation.json \
  --initial-capital 100000 \
  --commission 0.0001
```

**Success Criteria**: Same as DQN

**Expected Output**: Similar JSON metrics

**If Backtesting Fails**: Same process as DQN

---

#### Task 3: Compare Models (15-30 min)

**Analysis Questions**:
1. Which model has higher Sharpe ratio?
2. Which model has lower drawdown?
3. Which model has more trades?
4. Which model is more stable (lower variance)?

**Recommendation**:
- If DQN > PPO: Deploy DQN first, use PPO as backup
- If PPO > DQN: Deploy PPO first, use DQN as backup
- If similar: Deploy both for diversification

**Output**: Create `AGENT_71_MODEL_COMPARISON.md` with:
- Performance metrics table
- Risk-adjusted returns analysis
- Deployment recommendation

---

#### Task 4: Generate Report (15 min)

**Create**: `AGENT_71_MODEL_VALIDATION_REPORT.md`

**Contents**:
1. Executive summary (validation pass/fail)
2. DQN backtest results
3. PPO backtest results
4. Model comparison
5. Production deployment recommendation
6. Next steps (hyperparameter tuning, integration, etc.)

---

## 📋 Option B: MAMBA-2 Device Mismatch Fix

### Current Status
- ❌ MAMBA-2 training blocked: Device mismatch error
- ❌ Error: `device mismatch in matmul, lhs: Cuda { gpu_id: 0 }, rhs: Cpu`
- ⏳ Fix identified: Add `.to_device(&device)` to 20-30 locations

### Your Tasks

#### Task 1: Identify All Tensor Locations (1-2 hours)

**Search Pattern**:
```bash
# Find all tensor creation in MAMBA-2 modules
rg "Tensor::" ml/src/mamba/ -A 2 -B 2

# Find all Linear layer creations
rg "Linear::new|nn::linear" ml/src/mamba/ -A 2 -B 2

# Find all model components
rg "struct.*Layer|struct.*Module" ml/src/mamba/ -A 5
```

**Create Checklist**:
```markdown
# MAMBA-2 Device Migration Checklist

## ml/src/mamba/mod.rs
- [ ] Line 123: Linear layer weights
- [ ] Line 145: SSM state tensors
- [ ] Line 167: Projection matrices

## ml/src/mamba/ssd_layer.rs
- [ ] Line 78: SSD layer weights
- [ ] Line 92: State space matrices
- [ ] Line 105: Output projections

## ml/src/mamba/selective_state.rs
- [ ] Line 45: Selection weights
- [ ] Line 67: Gate parameters
- [ ] Line 89: Transformation matrices

## ml/src/mamba/hardware_optimizer.rs
- [ ] Line 34: Optimization buffers
- [ ] Line 56: Cache tensors
```

---

#### Task 2: Apply Device Migration (2-3 hours)

**Pattern to Apply**:
```rust
// BEFORE (CPU tensor)
let weights = Tensor::randn(0.0, 1.0, (input_dim, output_dim), &Device::Cpu)?;

// AFTER (Device-aware tensor)
let weights = Tensor::randn(0.0, 1.0, (input_dim, output_dim), &device)?;

// OR if tensor created elsewhere
let weights = weights.to_device(&device)?;
```

**Files to Modify**:
1. `ml/src/mamba/mod.rs`
2. `ml/src/mamba/ssd_layer.rs`
3. `ml/src/mamba/selective_state.rs`
4. `ml/src/mamba/hardware_optimizer.rs`

**Validation After Each File**:
```bash
cargo build -p ml --lib --release
cargo test -p ml test_mamba2 --release
```

---

#### Task 3: Test MAMBA-2 Training (30-45 min)

**Command**:
```bash
cargo run -p ml --example train_mamba2 --release --features cuda -- \
  --epochs 10 \
  --batch-size 8 \
  --seq-len 128 \
  --learning-rate 0.0001 \
  --output ml/trained_models/production/mamba2_real_data
```

**Success Criteria**:
- ✅ No device mismatch errors
- ✅ GPU utilization 30-50%
- ✅ 10 epochs complete successfully
- ✅ Checkpoints generated (>1KB each)
- ✅ Loss decreasing

**Expected Output**:
```
INFO ml::trainers::mamba2: Using CUDA device for MAMBA-2 training
INFO ml::trainers::mamba2: Loaded 6385 training sequences, 710 validation sequences
INFO ml::trainers::mamba2: Epoch 1/10: loss=0.250000, duration=2.5s
INFO ml::trainers::mamba2: Epoch 10/10: loss=0.050000, duration=2.3s
✅ Training completed successfully!
```

---

#### Task 4: Full Training (if 10 epochs succeed)

**Command**:
```bash
cargo run -p ml --example train_mamba2 --release --features cuda -- \
  --epochs 500 \
  --batch-size 8 \
  --seq-len 128 \
  --learning-rate 0.0001 \
  --output ml/trained_models/production/mamba2_real_data
```

**Expected Duration**: 15-25 minutes (500 epochs × ~2-3s per epoch)

**Output**: Create `AGENT_71_MAMBA2_FIX_REPORT.md`

---

## 📋 Option C: Documentation Update

### Current Status
- ⏳ CLAUDE.md not updated with Wave 160 Phase 3 status
- ✅ Update guide ready: `WAVE_160_CLAUDE_UPDATE.md`

### Your Tasks

#### Task 1: Update CLAUDE.md (20 min)

**File**: `/home/jgrusewski/Work/foxhunt/CLAUDE.md`

**Changes** (from `WAVE_160_CLAUDE_UPDATE.md`):
1. Production Readiness: 100% → 50% ML Models
2. ML Model Status: Add DQN/PPO complete, MAMBA-2/TFT blocked
3. Testing Status: Add ML Production Training 2/4
4. Next Priorities: Replace GPU Benchmark with Model Validation
5. Documentation: Add Wave 160 Phase 3 reports
6. GPU Configuration: Add training performance metrics
7. Wave 160 Achievements: New section

**Verification**:
```bash
# Check file size (should be similar to before)
wc -l CLAUDE.md

# Check no syntax errors
grep -n "```" CLAUDE.md | wc -l  # Should be even number

# Verify key sections exist
grep -n "Production Readiness" CLAUDE.md
grep -n "Wave 160 Achievements" CLAUDE.md
```

---

#### Task 2: Archive Wave 160 Reports (10 min)

**Move to docs/**:
```bash
mkdir -p docs/wave160
mv AGENT_63_DBN_PARSER_FIX.md docs/wave160/
mv AGENT_64_TFT_SHAPE_FIX.md docs/wave160/
mv AGENT_66_PRICE_SCALING_FIX.md docs/wave160/
mv AGENT_68_GPU_TRAINING_INVESTIGATION.md docs/wave160/
mv WAVE_160_PHASE3_COMPLETE.md docs/wave160/
mv WAVE_160_EXECUTIVE_SUMMARY.md docs/wave160/
mv WAVE_160_CLAUDE_UPDATE.md docs/wave160/
```

**Create Index**:
```bash
cat > docs/wave160/README.md <<'EOF'
# Wave 160: ML Training Infrastructure

## Phase 3 Reports (Agents 63-70)
- [Phase 3 Complete](WAVE_160_PHASE3_COMPLETE.md) - Comprehensive analysis
- [Executive Summary](WAVE_160_EXECUTIVE_SUMMARY.md) - 1-page summary
- [Agent 63: DBN Parser Fix](AGENT_63_DBN_PARSER_FIX.md)
- [Agent 64: TFT Shape Fix](AGENT_64_TFT_SHAPE_FIX.md)
- [Agent 66: Price Scaling Fix](AGENT_66_PRICE_SCALING_FIX.md)
- [Agent 68: GPU Training](AGENT_68_GPU_TRAINING_INVESTIGATION.md)
- [CLAUDE.md Updates](WAVE_160_CLAUDE_UPDATE.md)
EOF
```

---

## 🎯 Recommendation

**Choose Option A (Model Validation)** for these reasons:

1. **Immediate Value**: Validates 2/4 operational models before production
2. **Low Risk**: Backtesting is safe (no live trading)
3. **High Priority**: Deployment blockers have highest business impact
4. **Clear Success Criteria**: Pass/fail metrics (Sharpe, drawdown, win rate)
5. **Fast Iteration**: 1-2 hours vs 4-6 hours for MAMBA-2 fix

**Why Not Option B (MAMBA-2)**:
- 4-6 hours vs 1-2 hours
- Medium priority (vs HIGH for validation)
- 50% models (DQN, PPO) sufficient for initial deployment
- Can do after validation proves DQN/PPO work

**Why Not Option C (Documentation)**:
- Low priority vs validation
- Can be done anytime
- Validation results may change documentation needs

---

## 📊 Success Criteria

### Option A (Model Validation)
- ✅ DQN backtest complete (Sharpe > 1.0, drawdown < 20%)
- ✅ PPO backtest complete (Sharpe > 1.0, drawdown < 20%)
- ✅ Model comparison report generated
- ✅ Deployment recommendation provided

### Option B (MAMBA-2 Fix)
- ✅ Zero device mismatch errors
- ✅ 500 epochs complete successfully
- ✅ 50 checkpoints generated (>1KB each)
- ✅ GPU utilization 30-50%
- ✅ Loss reduction 80%+ (final < 0.05)

### Option C (Documentation)
- ✅ CLAUDE.md updated with Phase 3 status
- ✅ Wave 160 reports archived to docs/wave160/
- ✅ README.md index created

---

## 📁 Files to Reference

### Read First
1. **WAVE_160_EXECUTIVE_SUMMARY.md** - 1-page overview
2. **WAVE_160_PHASE3_COMPLETE.md** - Full details (1,200+ lines)

### Agent Reports
1. **AGENT_63_DBN_PARSER_FIX.md** - DBN parser migration
2. **AGENT_64_TFT_SHAPE_FIX.md** - TFT shape fix
3. **AGENT_66_PRICE_SCALING_FIX.md** - Price scaling fix
4. **AGENT_68_GPU_TRAINING_INVESTIGATION.md** - GPU validation

### Training Results
1. **agent54_ppo_production_training_report.md** - PPO training
2. **ml/trained_models/production/dqn_real_data/** - DQN checkpoints (51 files)
3. **ml/trained_models/production/ppo_checkpoint_epoch_*.safetensors** - PPO checkpoints (200 files)

---

## 🚀 Quick Start (Option A)

```bash
# 1. Check model files exist
ls -lh ml/trained_models/production/dqn_real_data/dqn_final_epoch500.safetensors
ls -lh ml/trained_models/production/ppo_checkpoint_epoch_500.safetensors

# 2. Check backtesting examples exist
ls ml/examples/backtest_*.rs

# 3. Run DQN backtest (if example exists)
cargo run -p ml --example backtest_dqn --release -- \
  --model ml/trained_models/production/dqn_real_data/dqn_final_epoch500.safetensors \
  --data test_data/real/databento/ml_training/6E.FUT_ohlcv-1m_2024-01-*.dbn

# 4. If no example, create minimal backtest script
# (See WAVE_160_PHASE3_COMPLETE.md Section: "Backtest Implementation Guide")
```

---

## 📞 Questions?

**Technical Details**: See `WAVE_160_PHASE3_COMPLETE.md` (comprehensive)
**Quick Overview**: See `WAVE_160_EXECUTIVE_SUMMARY.md` (1-page)
**Training Results**: See agent reports (AGENT_63-68)

**Need Help?**: All commands, file paths, and success criteria documented above.

---

**Handoff Complete**: Agent 70 → Agent 71
**Recommendation**: Choose Option A (Model Validation)
**Expected Duration**: 1-2 hours
**Priority**: HIGH (blocks production deployment)

Good luck! 🚀