foxhunt/PHASE_2_QUICK_START.md

Phase 2 Quick Start - 225-Feature Integration

⏱️ Time: 30 minutes validation OR 8-12 hours full integration 🎯 Goal: Verify/fix 225-feature extraction in ML training pipeline

---

🚨 Critical Finding from Phase 1

Problem: Models trained on 85% zero-padded junk data

Evidence:

• DQN features_to_state(): Only 10 real features + 215 zeros
• File: /home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn.rs:668-681
• All 4 models configured for 225 dimensions but receive junk data

Impact: Production-ready architecture, but unusable training data

---

⚡ Quick Validation (30 minutes)

Step 1: Test Feature Extraction (10 min)

cd /home/jgrusewski/Work/foxhunt

# Test 225-feature extraction
cargo test -p ml integration_wave_d_features --release -- --nocapture

# Expected: ✅ PASS (225 real features)
# Actual if broken: ❌ FAIL (zero padding detected)

Step 2: Check Integration (10 min)

# Verify common::features exists
grep -r "FeatureVector225" common/src/

# Check ml::features fallback
grep -r "extract_unified_features" ml/src/features/

# Inspect DQN feature extraction
grep -A 20 "features_to_state" ml/src/trainers/dqn.rs

# Look for: zero-padding logic (BAD) or extract_225_features() call (GOOD)

Step 3: Smoke Test (10 min)

# Train 1 epoch with verbose logging
cargo run -p ml --example train_dqn --release -- \
  --epochs 1 \
  --verbose

# Look for in logs:
# ✅ GOOD: "Extracted 225 features from bar"
# ❌ BAD: "Padding features to 225"

---

🔧 If Validation Fails: Integration Fix (4-6 hours)

Priority 1: DQN (2 hours)

File: /home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn.rs

Changes Required:

1. Add imports (top of file):

   use common::features::{FeatureVector225, FeatureExtractor};
   use common::regime_detection::RegimeDetector;
   

2. Replace features_to_state() method (lines 663-695):

   fn features_to_state(&self,
       ohlcv: &OHLCVBar,
       regime_detector: &RegimeDetector,
   ) -> Result<TradingState> {
       // Extract all 225 features (Wave C + Wave D)
       let feature_vector = FeatureExtractor::extract_225_features(
           ohlcv,
           regime_detector,
       )?;
   
       // Convert to TradingState (no padding!)
       Ok(TradingState::from_feature_vector_225(feature_vector))
   }
   

3. Update train() method (line 169):

   pub async fn train<F>(...) -> Result<TrainingMetrics> {
       // Add regime detector
       let mut regime_detector = RegimeDetector::new(100, 0.05)?;
   
       // In training loop, update regime state before feature extraction
       for bar in dbn_loader.iter() {
           regime_detector.update(&bar)?;
           let state = self.features_to_state(&bar, &regime_detector)?;
           // ... rest of training logic
       }
   }
   

Priority 2: PPO (2 hours)

File: /home/jgrusewski/Work/foxhunt/ml/src/trainers/ppo.rs

• Same changes as DQN
• Add regime_detector parameter
• Wire extract_225_features()

Priority 3: MAMBA-2 & TFT (2 hours)

Files:

• /home/jgrusewski/Work/foxhunt/ml/src/trainers/mamba2.rs

• /home/jgrusewski/Work/foxhunt/ml/src/trainers/tft.rs

• Same pattern as DQN/PPO

• MAMBA-2: Extract 225 features per sequence step

• TFT: 225 base + 20 time encodings = 245 (expected)

---

✅ Testing After Integration (1 hour)

# Test 1: Feature extraction
cargo test -p ml integration_wave_d_features --release

# Test 2: Trainer integration
cargo test -p ml test_dqn_225_features --release
cargo test -p ml test_ppo_225_features --release

# Test 3: End-to-end
cargo run -p ml --example train_dqn --release -- --epochs 1 --verbose

# Verify logs show:
# ✅ "Extracted 225 features"
# ✅ Wave C (201) + Wave D (24)
# ✅ NO zero-padding warnings

---

🏋️ Retrain Models (2-4 hours)

# Once integration validated, retrain all 4 models

# DQN (100 epochs, ~3 min)
cargo run -p ml --example train_dqn --release

# PPO (20 epochs, ~7 min)
cargo run -p ml --example train_ppo --release

# MAMBA-2 (50 epochs with tuning, ~5 min)
cargo run -p ml --example train_mamba2_dbn --release -- \
  --learning-rate 0.001 \
  --n-layers 4 \
  --d-model 512

# TFT (20 epochs with reduced arch, ~10 min)
cargo run -p ml --example train_tft_dbn --release -- \
  --hidden-dim 128 \
  --num-attention-heads 4 \
  --lstm-layers 1 \
  --batch-size 16

Expected Improvements:

• DQN loss: 0.045 → 0.020-0.030 (33-55% better)
• MAMBA-2: Diverged (1e+38) → Converged (0.1-1.0)
• Backtest Sharpe: 0.5-0.8 → 1.5-2.0 (150-300% gain)

---

📊 Phase 3: Backtest Validation (30 min)

# Run Wave Comparison Backtest
cargo run -p backtesting_service --example wave_comparison --release

# Expected metrics:
# ✅ Sharpe: 1.5-2.0 (target ≥1.5)
# ✅ Win Rate: 55-60% (target ≥55%)
# ✅ Drawdown: 15-20% (target ≤20%)

# If Sharpe ≥ 1.5:
#   → Deploy to paper trading (1 week)
#
# If Sharpe < 1.5:
#   → Purchase extended data ($2-$4)
#   → Retrain with 90-180 days

---

📋 Decision Flow

START: Phase 2
   ↓
Step 1: Validation (10 min)
   ↓
   ├─→ Tests PASS? → Step 2
   └─→ Tests FAIL? → Integration Fix (4-6h)
       ↓
Step 2: Integration Check (10 min)
   ↓
   ├─→ Integration EXISTS? → Step 3
   └─→ Integration MISSING? → Integration Fix (4-6h)
       ↓
Step 3: Smoke Test (10 min)
   ↓
   ├─→ Real Features? → Phase 3 Backtest
   └─→ Zero Padding? → Integration Fix (4-6h)
       ↓
Integration Fix (4-6h)
   ↓
Retrain Models (2-4h)
   ↓
Phase 3: Backtest (30 min)
   ↓
   ├─→ Sharpe ≥ 1.5? → Paper Trading (1 week)
   └─→ Sharpe < 1.5? → Extended Data ($2-$4)

---

🎯 Success Criteria

Phase 2 Complete When:

• All 225 features extracted (no zero-padding)
• Regime detection operational
• DQN trains with real features (loss <0.03)
• MAMBA-2 converges (loss 0.1-1.0, not 1e+38)
• No "padding" warnings in logs
• Backtest Sharpe ≥ 1.5

---

🚀 Next Command

# Start here:
cd /home/jgrusewski/Work/foxhunt
cargo test -p ml integration_wave_d_features --release -- --nocapture

Expected Time:

• Best case: 30 min (integration exists)
• Worst case: 12 hours (full integration + retrain)
• Most likely: 6 hours (partial fix + retrain)

---

Document: Quick Start Guide Created: 2025-10-20 See Also: PHASE_2_INTEGRATION_PLAN.md (full details)