foxhunt/PHASE_2_INTEGRATION_PLAN.md

# Phase 2: 225-Feature Integration Plan - Detailed Analysis & Action Items

**Date**: 2025-10-20
**Based On**: Phase 1 Training Results
**Decision Point**: Integration Status Assessment
**Next Steps**: Concrete code changes required

---

## Executive Summary

**FINDING**: 🔴 **225-Feature Integration is INCOMPLETE**

### Current Status (Phase 1 Findings)

✅ **What Works**:
- Model architectures configured for 225 input dimensions (DQN: line 130, PPO: line 69)
- All 4 models compile and train successfully
- DQN & PPO: Production ready with basic features
- MAMBA-2: Needs hyperparameter tuning only
- TFT: Needs architecture reduction only

❌ **Critical Gap**:
- **NO actual 225-feature extraction during training**
- Training uses placeholder/padded features (6 basic OHLCV features + 219 zeros)
- `features_to_state()` padding logic: Lines 668-681 in `dqn.rs`
- **Models trained on junk data** (85% zeros)

### Impact Assessment

| Metric | Current Reality | Expected with Real 225 Features |
|--------|----------------|----------------------------------|
| **Training Quality** | ❌ Poor (85% zero padding) | ✅ High (Wave C + D features) |
| **Model Performance** | ⚠️ Sharpe 0.5-0.8 (guessing) | ✅ Sharpe 2.0+ (informed) |
| **Win Rate** | ⚠️ 48-52% (random) | ✅ 60%+ (strategic) |
| **Production Ready** | ❌ NO (junk training data) | ✅ YES (full feature set) |

---

## Phase 2 Decision: Skip to Integration Layer

### Option 1: Quick Validation (RECOMMENDED) ✅
**Time**: 30 minutes
**Risk**: Low
**Goal**: Confirm integration status

### Option 2: Full Integration (IF validation fails)
**Time**: 4-6 hours
**Risk**: Medium
**Goal**: Wire 225-feature extraction into all 4 trainers

### Option 3: Data Purchase First (NOT RECOMMENDED)
**Time**: 1 week + $4
**Risk**: High (wasting money on broken pipeline)
**Goal**: N/A (premature)

**DECISION**: Execute **Option 1**, then decide based on results.

---

## Phase 2: Integration Validation (30 minutes)

### Step 1: Verify Feature Extraction Works (10 minutes)

```bash
# Check if 225-feature extraction exists
cd /home/jgrusewski/Work/foxhunt

# Test 1: Check for existing 225-feature tests
cargo test -p ml test_225 --release -- --nocapture

# Test 2: Validate regime detection features (Wave D)
cargo run -p ml --example validate_regime_features --release

# Test 3: Check feature extraction benchmark
cargo bench -p ml bench_feature_extraction --release

# Expected output:
# ✅ 225 features extracted per bar
# ✅ Wave C (201) + Wave D (24) = 225
# ✅ Performance: <50μs per bar (target met)
```

**Success Criteria**:
- All 225 features extracted (no zero padding)
- Regime detection operational (CUSUM, ADX, transition probabilities)
- Performance: <50μs per bar

**If Tests Pass**: ✅ Integration exists → Proceed to Step 2
**If Tests Fail**: ❌ Integration missing → Execute Phase 2B (Full Integration)

---

### Step 2: Validate Trainer Integration (10 minutes)

```bash
# Check if trainers use real feature extraction
cd /home/jgrusewski/Work/foxhunt

# Test 1: DQN with 225 features
cargo run -p ml --example validate_dqn_225_features --release

# Test 2: PPO with 225 features
cargo test -p ml test_ppo_225_features --release -- --nocapture

# Test 3: Check data loader integration
cargo test -p ml dbn_feature_config_test --release -- --nocapture

# Expected output:
# ✅ DQN loads 225 real features (not padded zeros)
# ✅ PPO loads 225 real features
# ✅ Data loader extracts Wave C + Wave D features
```

**Success Criteria**:
- No zero-padding in feature vectors
- All 225 features have real values (not 0.0)
- Feature extraction called during training loop

**If Tests Pass**: ✅ Full integration exists → Proceed to Phase 3 (Backtest)
**If Tests Fail**: ❌ Partial integration → Execute Phase 2B (Wire Trainers)

---

### Step 3: Smoke Test with Real Training (10 minutes)

```bash
# Run 1-epoch training with feature logging
cd /home/jgrusewski/Work/foxhunt

# DQN: 1 epoch, verbose logging
cargo run -p ml --example train_dqn --release -- \
  --epochs 1 \
  --verbose \
  --data-dir test_data/real/databento/ml_training

# Check logs for feature extraction
# Expected output:
# ✅ "Extracting 225 features from OHLCV bar"
# ✅ "Wave C features (201): [0.45, 0.78, ...]"
# ✅ "Wave D features (24): [0.12, 0.34, ...]"
# ❌ "Padding features to 225" (BAD - means zero-padding)
```

**Success Criteria**:
- Log contains "225 features extracted"
- No "padding" or "zero-fill" warnings
- Feature values are diverse (not 85% zeros)

**If Logs Show Real Features**: ✅ Proceed to Phase 3
**If Logs Show Padding**: ❌ Execute Phase 2B

---

## Phase 2B: Full Integration Layer (4-6 hours)

### If Validation Fails: Wire 225-Feature Extraction

#### Problem Analysis

**File**: `/home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn.rs`
**Lines**: 668-681
**Issue**: Placeholder features with zero-padding

```rust
// CURRENT (BROKEN):
fn features_to_state(&self, features: &FinancialFeatures) -> Result<TradingState> {
    // Extract 4 prices + 6 basic indicators = 10 features
    let technical_indicators: Vec<f32> = features
        .technical_indicators
        .values()
        .map(|&v| v as f32)
        .collect();

    // Pad to 221 with ZEROS (this is the problem!)
    let mut tech_indicators_padded = technical_indicators;
    while tech_indicators_padded.len() < 221 {
        tech_indicators_padded.push(0.0);  // ❌ JUNK DATA
    }
    tech_indicators_padded.truncate(221);

    // Total: 4 prices + 221 tech = 225 (but 219 are zeros!)
    Ok(TradingState::new(
        price_features,
        tech_indicators_padded,  // ❌ 85% ZEROS
        market_features,
        portfolio_features,
    ))
}
```

---

### Solution 1: Wire Common Feature Extraction (RECOMMENDED)

**Prerequisite Check**:
```bash
# Verify common::features exists
grep -r "FeatureVector225" /home/jgrusewski/Work/foxhunt/common/src/
grep -r "extract_225_features" /home/jgrusewski/Work/foxhunt/common/src/

# If found: Integration path exists ✅
# If not found: Feature extraction still in ml/ crate (needs migration)
```

**Code Changes** (if common::features exists):

**File 1**: `/home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn.rs`

```rust
// ADD at top:
use common::features::{FeatureVector225, FeatureExtractor};
use common::regime_detection::{RegimeDetector, RegimeType};

// REPLACE features_to_state() method:
fn features_to_state(&self,
    ohlcv: &OHLCVBar,
    regime_detector: &RegimeDetector,
) -> Result<TradingState> {
    // Extract all 225 features (Wave C + Wave D)
    let feature_vector = FeatureExtractor::extract_225_features(
        ohlcv,
        regime_detector,
    )?;

    // Convert to TradingState (no padding needed!)
    Ok(TradingState::from_feature_vector_225(feature_vector))
}

// UPDATE train() method to create RegimeDetector:
pub async fn train<F>(
    &mut self,
    dbn_data_dir: &str,
    mut checkpoint_callback: F,
) -> Result<TrainingMetrics>
where
    F: FnMut(usize, Vec<u8>) -> Result<String> + Send,
{
    // ADD regime detector
    let mut regime_detector = RegimeDetector::new(
        100,  // lookback window
        0.05, // volatility threshold
    )?;

    // Load DBN data
    let dbn_loader = DbnSequenceLoader::new(dbn_data_dir)?;

    for epoch in 0..self.hyperparams.epochs {
        for bar in dbn_loader.iter() {
            // Update regime state
            regime_detector.update(&bar)?;

            // Extract 225 features (Wave C + Wave D)
            let state = self.features_to_state(&bar, &regime_detector)?;

            // Select action
            let action = self.select_action(&state).await?;

            // Calculate reward
            let reward = self.calculate_reward(&bar, &action);

            // Get next state
            let next_bar = dbn_loader.peek_next()?;
            regime_detector.update(&next_bar)?;
            let next_state = self.features_to_state(&next_bar, &regime_detector)?;

            // Store experience
            self.store_experience(state, action, reward, next_state).await?;

            // Train on batch
            if self.can_train().await? {
                let (loss, q_value, grad_norm) = self.train_step().await?;
                // ... metrics logging
            }
        }

        // Save checkpoint
        if epoch % self.hyperparams.checkpoint_frequency == 0 {
            self.save_checkpoint(epoch, &mut checkpoint_callback).await?;
        }
    }

    Ok(self.get_metrics().await)
}
```

**Estimated Time**: 2 hours (DQN)

---

**File 2**: `/home/jgrusewski/Work/foxhunt/ml/src/trainers/ppo.rs`

```rust
// Similar changes:
// 1. Import common::features::FeatureVector225
// 2. Add regime_detector to train() method
// 3. Replace feature extraction with extract_225_features()
// 4. Remove zero-padding logic
```

**Estimated Time**: 2 hours (PPO)

---

**File 3**: `/home/jgrusewski/Work/foxhunt/ml/src/trainers/mamba2.rs`

```rust
// Similar changes for MAMBA-2
// Note: MAMBA-2 uses sequence modeling, so:
// 1. Extract 225 features for each bar in sequence
// 2. Pass [batch_size, seq_len, 225] tensor to model
// 3. Update regime state for each sequence step
```

**Estimated Time**: 1.5 hours (MAMBA-2)

---

**File 4**: `/home/jgrusewski/Work/foxhunt/ml/src/trainers/tft.rs`

```rust
// Similar changes for TFT
// Note: TFT adds 20 time/positional encodings
// Total: 225 + 20 = 245 features (expected)
// Fix existing "245 vs 225" mismatch warning
```

**Estimated Time**: 1.5 hours (TFT)

---

### Solution 2: Use ML-Local Feature Extraction (FALLBACK)

**If common::features doesn't exist**:

```bash
# Check if ml crate has feature extraction
ls -la /home/jgrusewski/Work/foxhunt/ml/src/features/
grep -r "extract_225" /home/jgrusewski/Work/foxhunt/ml/src/features/

# Expected files:
# - unified.rs (Wave C + Wave D unified extraction)
# - extraction.rs (main extraction logic)
# - adx_features.rs (Wave D ADX features)
# - config.rs (feature configuration)
```

**Code Changes**:

```rust
// File: ml/src/trainers/dqn.rs

use crate::features::{extract_unified_features, FeatureConfig};
use crate::regime_detection::RegimeOrchestrator;

fn features_to_state(&self,
    ohlcv: &OHLCVBar,
    regime_orchestrator: &mut RegimeOrchestrator,
) -> Result<TradingState> {
    // Configure 225-feature extraction
    let config = FeatureConfig::wave_d_full();  // 201 + 24 = 225

    // Extract features
    let feature_vector = extract_unified_features(
        ohlcv,
        regime_orchestrator,
        &config,
    )?;

    assert_eq!(feature_vector.len(), 225, "Expected 225 features");

    // Convert to TradingState
    Ok(TradingState::from_vec(feature_vector))
}
```

**Estimated Time**: 3 hours (all 4 models)

---

## Phase 2C: Testing & Validation (1 hour)

### After Integration Changes

```bash
# Test 1: Verify 225-feature extraction
cargo test -p ml integration_wave_d_features --release -- --nocapture

# Expected output:
# ✅ test_extract_225_features ... ok
# ✅ test_wave_c_201_features ... ok
# ✅ test_wave_d_24_features ... ok
# ✅ test_regime_detection_integration ... ok

# Test 2: Train 1 epoch with feature logging
cargo run -p ml --example train_dqn --release -- \
  --epochs 1 \
  --verbose

# Expected output:
# ✅ "Extracted 225 features from bar 1"
# ✅ "Wave C features (201): [min=0.12, max=0.98, mean=0.45]"
# ✅ "Wave D features (24): [min=0.05, max=0.87, mean=0.32]"
# ❌ NO "padding" or "zero-fill" warnings

# Test 3: Verify checkpoint dimensions
cargo run -p ml --example validate_dqn_225_features --release

# Expected output:
# ✅ "Model input dimension: 225"
# ✅ "Checkpoint compatible: true"
# ✅ "Feature extraction tested: PASS"
```

---

## Phase 2D: Retrain Models with Real Features (2-4 hours)

### Once Integration is Validated

```bash
# Retrain DQN (100 epochs, ~3 minutes)
cargo run -p ml --example train_dqn --release -- \
  --epochs 100 \
  --output-dir ml/trained_models_225_features

# Retrain PPO (20 epochs, ~7 minutes)
cargo run -p ml --example train_ppo --release -- \
  --epochs 20 \
  --output-dir ml/trained_models_225_features

# Retrain MAMBA-2 (50 epochs with tuning, ~5 minutes)
cargo run -p ml --example train_mamba2_dbn --release -- \
  --epochs 50 \
  --learning-rate 0.001 \
  --n-layers 4 \
  --d-model 512 \
  --output-dir ml/trained_models_225_features

# Retrain TFT (20 epochs with reduced arch, ~10 minutes)
cargo run -p ml --example train_tft_dbn --release -- \
  --epochs 20 \
  --hidden-dim 128 \
  --num-attention-heads 4 \
  --lstm-layers 1 \
  --batch-size 16 \
  --output-dir ml/trained_models_225_features
```

**Expected Improvements** (vs Phase 1 broken training):

| Metric | Phase 1 (Junk Data) | Phase 2 (Real 225 Features) | Improvement |
|--------|--------------------|-----------------------------|-------------|
| **DQN Loss** | 0.045 | 0.020-0.030 | 33-55% better |
| **DQN Convergence** | Epoch 70 | Epoch 40-50 | 30% faster |
| **PPO Convergence** | Epoch 20 | Epoch 12-15 | 25% faster |
| **MAMBA-2 Loss** | 1.4e+38 (diverged) | 0.1-1.0 (stable) | 100% fixed |
| **Backtest Sharpe** | 0.5-0.8 | 1.5-2.0 | 150-300% gain |

---

## Decision Tree Summary

```
Phase 2 Start
    │
    ├─→ Step 1: Run validation tests (10 min)
    │       │
    │       ├─→ Tests PASS → Step 2
    │       └─→ Tests FAIL → Phase 2B (Full Integration, 4-6h)
    │
    ├─→ Step 2: Check trainer integration (10 min)
    │       │
    │       ├─→ Integration EXISTS → Step 3
    │       └─→ Integration MISSING → Phase 2B
    │
    ├─→ Step 3: Smoke test 1-epoch training (10 min)
    │       │
    │       ├─→ Real features extracted → Phase 3 (Backtest)
    │       └─→ Zero-padding detected → Phase 2B
    │
    └─→ Phase 2B: Full integration (4-6h)
            │
            ├─→ Wire common::features → 4h
            │   └─→ Test → Phase 2C (1h)
            │       └─→ Retrain → Phase 2D (2-4h)
            │
            └─→ Use ml::features → 3h
                └─→ Test → Phase 2C (1h)
                    └─→ Retrain → Phase 2D (2-4h)
```

---

## Time Estimates

### Best Case (Integration Exists)
- Phase 2 Validation: 30 minutes
- Phase 3 Backtest: 30 minutes
- **Total**: 1 hour → Ready for production deployment

### Worst Case (Integration Missing)
- Phase 2 Validation: 30 minutes
- Phase 2B Integration: 4-6 hours
- Phase 2C Testing: 1 hour
- Phase 2D Retraining: 2-4 hours
- Phase 3 Backtest: 30 minutes
- **Total**: 8-12 hours → Ready for production deployment

### Most Likely (Partial Integration)
- Phase 2 Validation: 30 minutes
- Phase 2B Partial Fix: 2-3 hours
- Phase 2C Testing: 1 hour
- Phase 2D Retraining: 2 hours
- Phase 3 Backtest: 30 minutes
- **Total**: 6 hours → Ready for production deployment

---

## Next Actions (Priority Order)

### Immediate (Next 10 minutes)

1. **Run validation test suite**:
   ```bash
   cargo test -p ml test_225 --release -- --nocapture
   ```

2. **Check for common::features**:
   ```bash
   grep -r "FeatureVector225" /home/jgrusewski/Work/foxhunt/common/src/
   ```

3. **Inspect DQN feature extraction**:
   ```bash
   grep -A 20 "features_to_state" /home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn.rs
   ```

### Short-Term (If integration missing, next 4-6 hours)

4. **Implement Solution 1 or Solution 2** (see Phase 2B above)

5. **Run integration tests** (Phase 2C)

6. **Retrain all 4 models** (Phase 2D)

### Medium-Term (After integration validated, next 1 week)

7. **Run Wave Comparison Backtest** (Phase 3):
   ```bash
   cargo run -p backtesting_service --example wave_comparison --release
   ```

8. **If Sharpe ≥ 1.5**: Deploy to paper trading (1 week)

9. **If Sharpe < 1.5**: Purchase extended data ($2-$4) and retrain

---

## Risk Mitigation

### Risk 1: Integration Completely Missing
**Probability**: 60%
**Impact**: HIGH (8-12 hours delay)
**Mitigation**: Execute Phase 2B immediately, prioritize DQN+PPO first

### Risk 2: Integration Exists but Broken
**Probability**: 30%
**Impact**: MEDIUM (4-6 hours debug)
**Mitigation**: Use git blame to find original implementation, check Wave D docs

### Risk 3: Feature Extraction Performance Issues
**Probability**: 10%
**Impact**: LOW (1-2 hours optimization)
**Mitigation**: Use existing benchmarks (target: <50μs per bar, already validated)

---

## Success Criteria

### Phase 2 Complete When:

✅ **Validation Tests**:
- [ ] All 225 features extracted (no zero-padding)
- [ ] Regime detection operational
- [ ] Performance: <50μs per bar

✅ **Integration Tests**:
- [ ] DQN trains with real 225 features
- [ ] PPO trains with real 225 features
- [ ] MAMBA-2 trains with real 225 features
- [ ] TFT trains with real 225 features (245 = 225 + 20 time encodings)

✅ **Training Quality**:
- [ ] DQN loss: <0.03 (not 0.045)
- [ ] MAMBA-2 loss: 0.1-1.0 (not 1e+38)
- [ ] No "padding" or "zero-fill" warnings in logs
- [ ] Feature diversity: No more than 10% zeros

✅ **Checkpoint Validation**:
- [ ] All checkpoints have 225-dimensional input layer
- [ ] Models load successfully in inference mode
- [ ] Feature extraction test passes

---

## Conclusion

**Status**: 🟡 **INTEGRATION INCOMPLETE** (95% confidence)

**Evidence**:
1. DQN `features_to_state()` uses zero-padding (lines 668-681)
2. Only 10 real features + 215 zeros = 225 "features"
3. Phase 1 training succeeded too easily (no feature extraction errors)
4. MAMBA-2 divergence suggests low-quality training data

**Recommendation**:
1. **Execute Phase 2 validation** (30 min) to confirm status
2. **If validation fails**: Execute Phase 2B integration (4-6 hours)
3. **If validation passes**: Proceed directly to Phase 3 backtest

**Expected Outcome**:
- **With real 225 features**: Sharpe 1.5-2.0, Win Rate 60%, Drawdown 15%
- **With junk features**: Sharpe 0.5-0.8, Win Rate 48-52%, Drawdown 25%

**Next Command**:
```bash
cargo test -p ml integration_wave_d_features --release -- --nocapture
```

---

**Document Version**: 1.0
**Created**: 2025-10-20
**Status**: READY TO EXECUTE
**Estimated Completion**: 30 minutes (validation) or 8-12 hours (full integration)