foxhunt/PHASE_2_QUICK_START.md

# Phase 2 Quick Start - 225-Feature Integration

**⏱️ Time**: 30 minutes validation OR 8-12 hours full integration
**🎯 Goal**: Verify/fix 225-feature extraction in ML training pipeline

---

## 🚨 Critical Finding from Phase 1

**Problem**: Models trained on **85% zero-padded junk data**

**Evidence**:
- DQN `features_to_state()`: Only 10 real features + 215 zeros
- File: `/home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn.rs:668-681`
- All 4 models configured for 225 dimensions but receive junk data

**Impact**: Production-ready architecture, but **unusable training data**

---

## ⚡ Quick Validation (30 minutes)

### Step 1: Test Feature Extraction (10 min)

```bash
cd /home/jgrusewski/Work/foxhunt

# Test 225-feature extraction
cargo test -p ml integration_wave_d_features --release -- --nocapture

# Expected: ✅ PASS (225 real features)
# Actual if broken: ❌ FAIL (zero padding detected)
```

### Step 2: Check Integration (10 min)

```bash
# Verify common::features exists
grep -r "FeatureVector225" common/src/

# Check ml::features fallback
grep -r "extract_unified_features" ml/src/features/

# Inspect DQN feature extraction
grep -A 20 "features_to_state" ml/src/trainers/dqn.rs

# Look for: zero-padding logic (BAD) or extract_225_features() call (GOOD)
```

### Step 3: Smoke Test (10 min)

```bash
# Train 1 epoch with verbose logging
cargo run -p ml --example train_dqn --release -- \
  --epochs 1 \
  --verbose

# Look for in logs:
# ✅ GOOD: "Extracted 225 features from bar"
# ❌ BAD: "Padding features to 225"
```

---

## 🔧 If Validation Fails: Integration Fix (4-6 hours)

### Priority 1: DQN (2 hours)

**File**: `/home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn.rs`

**Changes Required**:

1. **Add imports** (top of file):
   ```rust
   use common::features::{FeatureVector225, FeatureExtractor};
   use common::regime_detection::RegimeDetector;
   ```

2. **Replace `features_to_state()` method** (lines 663-695):
   ```rust
   fn features_to_state(&self,
       ohlcv: &OHLCVBar,
       regime_detector: &RegimeDetector,
   ) -> Result<TradingState> {
       // Extract all 225 features (Wave C + Wave D)
       let feature_vector = FeatureExtractor::extract_225_features(
           ohlcv,
           regime_detector,
       )?;

       // Convert to TradingState (no padding!)
       Ok(TradingState::from_feature_vector_225(feature_vector))
   }
   ```

3. **Update `train()` method** (line 169):
   ```rust
   pub async fn train<F>(...) -> Result<TrainingMetrics> {
       // Add regime detector
       let mut regime_detector = RegimeDetector::new(100, 0.05)?;

       // In training loop, update regime state before feature extraction
       for bar in dbn_loader.iter() {
           regime_detector.update(&bar)?;
           let state = self.features_to_state(&bar, &regime_detector)?;
           // ... rest of training logic
       }
   }
   ```

### Priority 2: PPO (2 hours)

**File**: `/home/jgrusewski/Work/foxhunt/ml/src/trainers/ppo.rs`

- Same changes as DQN
- Add regime_detector parameter
- Wire extract_225_features()

### Priority 3: MAMBA-2 & TFT (2 hours)

**Files**:
- `/home/jgrusewski/Work/foxhunt/ml/src/trainers/mamba2.rs`
- `/home/jgrusewski/Work/foxhunt/ml/src/trainers/tft.rs`

- Same pattern as DQN/PPO
- MAMBA-2: Extract 225 features per sequence step
- TFT: 225 base + 20 time encodings = 245 (expected)

---

## ✅ Testing After Integration (1 hour)

```bash
# Test 1: Feature extraction
cargo test -p ml integration_wave_d_features --release

# Test 2: Trainer integration
cargo test -p ml test_dqn_225_features --release
cargo test -p ml test_ppo_225_features --release

# Test 3: End-to-end
cargo run -p ml --example train_dqn --release -- --epochs 1 --verbose

# Verify logs show:
# ✅ "Extracted 225 features"
# ✅ Wave C (201) + Wave D (24)
# ✅ NO zero-padding warnings
```

---

## 🏋️ Retrain Models (2-4 hours)

```bash
# Once integration validated, retrain all 4 models

# DQN (100 epochs, ~3 min)
cargo run -p ml --example train_dqn --release

# PPO (20 epochs, ~7 min)
cargo run -p ml --example train_ppo --release

# MAMBA-2 (50 epochs with tuning, ~5 min)
cargo run -p ml --example train_mamba2_dbn --release -- \
  --learning-rate 0.001 \
  --n-layers 4 \
  --d-model 512

# TFT (20 epochs with reduced arch, ~10 min)
cargo run -p ml --example train_tft_dbn --release -- \
  --hidden-dim 128 \
  --num-attention-heads 4 \
  --lstm-layers 1 \
  --batch-size 16
```

**Expected Improvements**:
- DQN loss: 0.045 → 0.020-0.030 (33-55% better)
- MAMBA-2: Diverged (1e+38) → Converged (0.1-1.0)
- Backtest Sharpe: 0.5-0.8 → 1.5-2.0 (150-300% gain)

---

## 📊 Phase 3: Backtest Validation (30 min)

```bash
# Run Wave Comparison Backtest
cargo run -p backtesting_service --example wave_comparison --release

# Expected metrics:
# ✅ Sharpe: 1.5-2.0 (target ≥1.5)
# ✅ Win Rate: 55-60% (target ≥55%)
# ✅ Drawdown: 15-20% (target ≤20%)

# If Sharpe ≥ 1.5:
#   → Deploy to paper trading (1 week)
#
# If Sharpe < 1.5:
#   → Purchase extended data ($2-$4)
#   → Retrain with 90-180 days
```

---

## 📋 Decision Flow

```
START: Phase 2
   ↓
Step 1: Validation (10 min)
   ↓
   ├─→ Tests PASS? → Step 2
   └─→ Tests FAIL? → Integration Fix (4-6h)
       ↓
Step 2: Integration Check (10 min)
   ↓
   ├─→ Integration EXISTS? → Step 3
   └─→ Integration MISSING? → Integration Fix (4-6h)
       ↓
Step 3: Smoke Test (10 min)
   ↓
   ├─→ Real Features? → Phase 3 Backtest
   └─→ Zero Padding? → Integration Fix (4-6h)
       ↓
Integration Fix (4-6h)
   ↓
Retrain Models (2-4h)
   ↓
Phase 3: Backtest (30 min)
   ↓
   ├─→ Sharpe ≥ 1.5? → Paper Trading (1 week)
   └─→ Sharpe < 1.5? → Extended Data ($2-$4)
```

---

## 🎯 Success Criteria

### Phase 2 Complete When:

- [ ] All 225 features extracted (no zero-padding)
- [ ] Regime detection operational
- [ ] DQN trains with real features (loss <0.03)
- [ ] MAMBA-2 converges (loss 0.1-1.0, not 1e+38)
- [ ] No "padding" warnings in logs
- [ ] Backtest Sharpe ≥ 1.5

---

## 🚀 Next Command

```bash
# Start here:
cd /home/jgrusewski/Work/foxhunt
cargo test -p ml integration_wave_d_features --release -- --nocapture
```

**Expected Time**:
- Best case: 30 min (integration exists)
- Worst case: 12 hours (full integration + retrain)
- Most likely: 6 hours (partial fix + retrain)

---

**Document**: Quick Start Guide
**Created**: 2025-10-20
**See Also**: `PHASE_2_INTEGRATION_PLAN.md` (full details)