foxhunt/AGENT_71_HANDOFF.md

Agent 71 Handoff: Next Steps After Wave 160 Phase 3

From: Agent 70 (Wave 160 Phase 3 Completion Report) To: Agent 71 (Model Validation & Next Steps) Date: 2025-10-14 Status: 2/4 models production-ready, validation needed

---

🎯 Your Mission (Choose One)

Option A: Model Validation (RECOMMENDED) - 1-2 hours

Priority: HIGH Goal: Validate DQN and PPO models with backtesting before production deployment

Option B: MAMBA-2 Fix - 4-6 hours

Priority: MEDIUM Goal: Fix device mismatch to enable GPU training for MAMBA-2

Option C: Documentation Update - 30 minutes

Priority: LOW Goal: Update CLAUDE.md with Wave 160 Phase 3 status

---

📋 Option A: Model Validation (RECOMMENDED)

Current Status

• ✅ DQN trained: 51 checkpoints, GPU-accelerated, 99.3% loss reduction
• ✅ PPO trained: 200 checkpoints, CPU-trained, zero NaN
• ⏳ Backtesting: NOT DONE
• ⏳ Performance metrics: NOT VALIDATED

Your Tasks

Task 1: Backtest DQN (30-45 min)

Command:

cargo run -p backtesting_service --example backtest_dqn --release -- \
  --model ml/trained_models/production/dqn_real_data/dqn_final_epoch500.safetensors \
  --data test_data/real/databento/ml_training/6E.FUT_ohlcv-1m_2024-01-*.dbn \
  --output ml/backtest_results/dqn_validation.json \
  --initial-capital 100000 \
  --commission 0.0001

Success Criteria:

• ✅ Sharpe ratio > 1.0
• ✅ Max drawdown < 20%
• ✅ Win rate > 50%
• ✅ Total return > 0%

Expected Output:

{
  "sharpe_ratio": 1.2,
  "max_drawdown": 0.15,
  "win_rate": 0.55,
  "total_return": 0.08,
  "num_trades": 150,
  "avg_trade_duration": "15m"
}

If Backtesting Fails:

1. Check if backtesting example exists: ls ml/examples/backtest_dqn.rs
2. If missing, create basic backtest script using model inference
3. Report findings in AGENT_71_DQN_BACKTEST_REPORT.md

---

Task 2: Backtest PPO (30-45 min)

Command:

cargo run -p backtesting_service --example backtest_ppo --release -- \
  --model ml/trained_models/production/ppo_checkpoint_epoch_500.safetensors \
  --data test_data/real/databento/ml_training/6E.FUT_ohlcv-1m_2024-01-*.dbn \
  --output ml/backtest_results/ppo_validation.json \
  --initial-capital 100000 \
  --commission 0.0001

Success Criteria: Same as DQN

Expected Output: Similar JSON metrics

If Backtesting Fails: Same process as DQN

---

Task 3: Compare Models (15-30 min)

Analysis Questions:

1. Which model has higher Sharpe ratio?
2. Which model has lower drawdown?
3. Which model has more trades?
4. Which model is more stable (lower variance)?

Recommendation:

• If DQN > PPO: Deploy DQN first, use PPO as backup
• If PPO > DQN: Deploy PPO first, use DQN as backup
• If similar: Deploy both for diversification

Output: Create AGENT_71_MODEL_COMPARISON.md with:

• Performance metrics table
• Risk-adjusted returns analysis
• Deployment recommendation

---

Task 4: Generate Report (15 min)

Create: AGENT_71_MODEL_VALIDATION_REPORT.md

Contents:

1. Executive summary (validation pass/fail)
2. DQN backtest results
3. PPO backtest results
4. Model comparison
5. Production deployment recommendation
6. Next steps (hyperparameter tuning, integration, etc.)

---

📋 Option B: MAMBA-2 Device Mismatch Fix

Current Status

• ❌ MAMBA-2 training blocked: Device mismatch error
• ❌ Error: device mismatch in matmul, lhs: Cuda { gpu_id: 0 }, rhs: Cpu
• ⏳ Fix identified: Add .to_device(&device) to 20-30 locations

Your Tasks

Task 1: Identify All Tensor Locations (1-2 hours)

Search Pattern:

# Find all tensor creation in MAMBA-2 modules
rg "Tensor::" ml/src/mamba/ -A 2 -B 2

# Find all Linear layer creations
rg "Linear::new|nn::linear" ml/src/mamba/ -A 2 -B 2

# Find all model components
rg "struct.*Layer|struct.*Module" ml/src/mamba/ -A 5

Create Checklist:

# MAMBA-2 Device Migration Checklist

## ml/src/mamba/mod.rs
- [ ] Line 123: Linear layer weights
- [ ] Line 145: SSM state tensors
- [ ] Line 167: Projection matrices

## ml/src/mamba/ssd_layer.rs
- [ ] Line 78: SSD layer weights
- [ ] Line 92: State space matrices
- [ ] Line 105: Output projections

## ml/src/mamba/selective_state.rs
- [ ] Line 45: Selection weights
- [ ] Line 67: Gate parameters
- [ ] Line 89: Transformation matrices

## ml/src/mamba/hardware_optimizer.rs
- [ ] Line 34: Optimization buffers
- [ ] Line 56: Cache tensors

---

Task 2: Apply Device Migration (2-3 hours)

Pattern to Apply:

// BEFORE (CPU tensor)
let weights = Tensor::randn(0.0, 1.0, (input_dim, output_dim), &Device::Cpu)?;

// AFTER (Device-aware tensor)
let weights = Tensor::randn(0.0, 1.0, (input_dim, output_dim), &device)?;

// OR if tensor created elsewhere
let weights = weights.to_device(&device)?;

Files to Modify:

1. ml/src/mamba/mod.rs
2. ml/src/mamba/ssd_layer.rs
3. ml/src/mamba/selective_state.rs
4. ml/src/mamba/hardware_optimizer.rs

Validation After Each File:

cargo build -p ml --lib --release
cargo test -p ml test_mamba2 --release

---

Task 3: Test MAMBA-2 Training (30-45 min)

Command:

cargo run -p ml --example train_mamba2 --release --features cuda -- \
  --epochs 10 \
  --batch-size 8 \
  --seq-len 128 \
  --learning-rate 0.0001 \
  --output ml/trained_models/production/mamba2_real_data

Success Criteria:

• ✅ No device mismatch errors
• ✅ GPU utilization 30-50%
• ✅ 10 epochs complete successfully
• ✅ Checkpoints generated (>1KB each)
• ✅ Loss decreasing

Expected Output:

INFO ml::trainers::mamba2: Using CUDA device for MAMBA-2 training
INFO ml::trainers::mamba2: Loaded 6385 training sequences, 710 validation sequences
INFO ml::trainers::mamba2: Epoch 1/10: loss=0.250000, duration=2.5s
INFO ml::trainers::mamba2: Epoch 10/10: loss=0.050000, duration=2.3s
✅ Training completed successfully!

---

Task 4: Full Training (if 10 epochs succeed)

Command:

cargo run -p ml --example train_mamba2 --release --features cuda -- \
  --epochs 500 \
  --batch-size 8 \
  --seq-len 128 \
  --learning-rate 0.0001 \
  --output ml/trained_models/production/mamba2_real_data

Expected Duration: 15-25 minutes (500 epochs × ~2-3s per epoch)

Output: Create AGENT_71_MAMBA2_FIX_REPORT.md

---

📋 Option C: Documentation Update

Current Status

• ⏳ CLAUDE.md not updated with Wave 160 Phase 3 status
• ✅ Update guide ready: WAVE_160_CLAUDE_UPDATE.md

Your Tasks

Task 1: Update CLAUDE.md (20 min)

File: /home/jgrusewski/Work/foxhunt/CLAUDE.md

Changes (from WAVE_160_CLAUDE_UPDATE.md):

1. Production Readiness: 100% → 50% ML Models
2. ML Model Status: Add DQN/PPO complete, MAMBA-2/TFT blocked
3. Testing Status: Add ML Production Training 2/4
4. Next Priorities: Replace GPU Benchmark with Model Validation
5. Documentation: Add Wave 160 Phase 3 reports
6. GPU Configuration: Add training performance metrics
7. Wave 160 Achievements: New section

Verification:

# Check file size (should be similar to before)
wc -l CLAUDE.md

# Check no syntax errors
grep -n "```" CLAUDE.md | wc -l  # Should be even number

# Verify key sections exist
grep -n "Production Readiness" CLAUDE.md
grep -n "Wave 160 Achievements" CLAUDE.md

---

Task 2: Archive Wave 160 Reports (10 min)

Move to docs/:

mkdir -p docs/wave160
mv AGENT_63_DBN_PARSER_FIX.md docs/wave160/
mv AGENT_64_TFT_SHAPE_FIX.md docs/wave160/
mv AGENT_66_PRICE_SCALING_FIX.md docs/wave160/
mv AGENT_68_GPU_TRAINING_INVESTIGATION.md docs/wave160/
mv WAVE_160_PHASE3_COMPLETE.md docs/wave160/
mv WAVE_160_EXECUTIVE_SUMMARY.md docs/wave160/
mv WAVE_160_CLAUDE_UPDATE.md docs/wave160/

Create Index:

cat > docs/wave160/README.md <<'EOF'
# Wave 160: ML Training Infrastructure

## Phase 3 Reports (Agents 63-70)
- [Phase 3 Complete](WAVE_160_PHASE3_COMPLETE.md) - Comprehensive analysis
- [Executive Summary](WAVE_160_EXECUTIVE_SUMMARY.md) - 1-page summary
- [Agent 63: DBN Parser Fix](AGENT_63_DBN_PARSER_FIX.md)
- [Agent 64: TFT Shape Fix](AGENT_64_TFT_SHAPE_FIX.md)
- [Agent 66: Price Scaling Fix](AGENT_66_PRICE_SCALING_FIX.md)
- [Agent 68: GPU Training](AGENT_68_GPU_TRAINING_INVESTIGATION.md)
- [CLAUDE.md Updates](WAVE_160_CLAUDE_UPDATE.md)
EOF

---

🎯 Recommendation

Choose Option A (Model Validation) for these reasons:

1. Immediate Value: Validates 2/4 operational models before production
2. Low Risk: Backtesting is safe (no live trading)
3. High Priority: Deployment blockers have highest business impact
4. Clear Success Criteria: Pass/fail metrics (Sharpe, drawdown, win rate)
5. Fast Iteration: 1-2 hours vs 4-6 hours for MAMBA-2 fix

Why Not Option B (MAMBA-2):

• 4-6 hours vs 1-2 hours
• Medium priority (vs HIGH for validation)
• 50% models (DQN, PPO) sufficient for initial deployment
• Can do after validation proves DQN/PPO work

Why Not Option C (Documentation):

• Low priority vs validation
• Can be done anytime
• Validation results may change documentation needs

---

📊 Success Criteria

Option A (Model Validation)

• ✅ DQN backtest complete (Sharpe > 1.0, drawdown < 20%)
• ✅ PPO backtest complete (Sharpe > 1.0, drawdown < 20%)
• ✅ Model comparison report generated
• ✅ Deployment recommendation provided

Option B (MAMBA-2 Fix)

• ✅ Zero device mismatch errors
• ✅ 500 epochs complete successfully
• ✅ 50 checkpoints generated (>1KB each)
• ✅ GPU utilization 30-50%
• ✅ Loss reduction 80%+ (final < 0.05)

Option C (Documentation)

• ✅ CLAUDE.md updated with Phase 3 status
• ✅ Wave 160 reports archived to docs/wave160/
• ✅ README.md index created

---

📁 Files to Reference

Read First

1. WAVE_160_EXECUTIVE_SUMMARY.md - 1-page overview
2. WAVE_160_PHASE3_COMPLETE.md - Full details (1,200+ lines)

Agent Reports

1. AGENT_63_DBN_PARSER_FIX.md - DBN parser migration
2. AGENT_64_TFT_SHAPE_FIX.md - TFT shape fix
3. AGENT_66_PRICE_SCALING_FIX.md - Price scaling fix
4. AGENT_68_GPU_TRAINING_INVESTIGATION.md - GPU validation

Training Results

1. agent54_ppo_production_training_report.md - PPO training
2. ml/trained_models/production/dqn_real_data/ - DQN checkpoints (51 files)
3. ml/trained_models/production/ppo_checkpoint_epoch_*.safetensors - PPO checkpoints (200 files)

---

🚀 Quick Start (Option A)

# 1. Check model files exist
ls -lh ml/trained_models/production/dqn_real_data/dqn_final_epoch500.safetensors
ls -lh ml/trained_models/production/ppo_checkpoint_epoch_500.safetensors

# 2. Check backtesting examples exist
ls ml/examples/backtest_*.rs

# 3. Run DQN backtest (if example exists)
cargo run -p ml --example backtest_dqn --release -- \
  --model ml/trained_models/production/dqn_real_data/dqn_final_epoch500.safetensors \
  --data test_data/real/databento/ml_training/6E.FUT_ohlcv-1m_2024-01-*.dbn

# 4. If no example, create minimal backtest script
# (See WAVE_160_PHASE3_COMPLETE.md Section: "Backtest Implementation Guide")

---

📞 Questions?

Technical Details: See WAVE_160_PHASE3_COMPLETE.md (comprehensive) Quick Overview: See WAVE_160_EXECUTIVE_SUMMARY.md (1-page) Training Results: See agent reports (AGENT_63-68)

Need Help?: All commands, file paths, and success criteria documented above.

---

Handoff Complete: Agent 70 → Agent 71 Recommendation: Choose Option A (Model Validation) Expected Duration: 1-2 hours Priority: HIGH (blocks production deployment)

Good luck! 🚀