foxhunt/PHASE_2_INTEGRATION_PLAN.md

Phase 2: 225-Feature Integration Plan - Detailed Analysis & Action Items

Date: 2025-10-20 Based On: Phase 1 Training Results Decision Point: Integration Status Assessment Next Steps: Concrete code changes required

---

Executive Summary

FINDING: 🔴 225-Feature Integration is INCOMPLETE

Current Status (Phase 1 Findings)

✅ What Works:

• Model architectures configured for 225 input dimensions (DQN: line 130, PPO: line 69)
• All 4 models compile and train successfully
• DQN & PPO: Production ready with basic features
• MAMBA-2: Needs hyperparameter tuning only
• TFT: Needs architecture reduction only

❌ Critical Gap:

• NO actual 225-feature extraction during training
• Training uses placeholder/padded features (6 basic OHLCV features + 219 zeros)
• features_to_state() padding logic: Lines 668-681 in dqn.rs
• Models trained on junk data (85% zeros)

Impact Assessment

Metric	Current Reality	Expected with Real 225 Features
Training Quality	❌ Poor (85% zero padding)	✅ High (Wave C + D features)
Model Performance	⚠️ Sharpe 0.5-0.8 (guessing)	✅ Sharpe 2.0+ (informed)
Win Rate	⚠️ 48-52% (random)	✅ 60%+ (strategic)
Production Ready	❌ NO (junk training data)	✅ YES (full feature set)

---

Phase 2 Decision: Skip to Integration Layer

Option 1: Quick Validation (RECOMMENDED) ✅

Time: 30 minutes Risk: Low Goal: Confirm integration status

Option 2: Full Integration (IF validation fails)

Time: 4-6 hours Risk: Medium Goal: Wire 225-feature extraction into all 4 trainers

Option 3: Data Purchase First (NOT RECOMMENDED)

Time: 1 week + $4 Risk: High (wasting money on broken pipeline) Goal: N/A (premature)

DECISION: Execute Option 1, then decide based on results.

---

Phase 2: Integration Validation (30 minutes)

Step 1: Verify Feature Extraction Works (10 minutes)

# Check if 225-feature extraction exists
cd /home/jgrusewski/Work/foxhunt

# Test 1: Check for existing 225-feature tests
cargo test -p ml test_225 --release -- --nocapture

# Test 2: Validate regime detection features (Wave D)
cargo run -p ml --example validate_regime_features --release

# Test 3: Check feature extraction benchmark
cargo bench -p ml bench_feature_extraction --release

# Expected output:
# ✅ 225 features extracted per bar
# ✅ Wave C (201) + Wave D (24) = 225
# ✅ Performance: <50μs per bar (target met)

Success Criteria:

• All 225 features extracted (no zero padding)
• Regime detection operational (CUSUM, ADX, transition probabilities)
• Performance: <50μs per bar

If Tests Pass: ✅ Integration exists → Proceed to Step 2 If Tests Fail: ❌ Integration missing → Execute Phase 2B (Full Integration)

---

Step 2: Validate Trainer Integration (10 minutes)

# Check if trainers use real feature extraction
cd /home/jgrusewski/Work/foxhunt

# Test 1: DQN with 225 features
cargo run -p ml --example validate_dqn_225_features --release

# Test 2: PPO with 225 features
cargo test -p ml test_ppo_225_features --release -- --nocapture

# Test 3: Check data loader integration
cargo test -p ml dbn_feature_config_test --release -- --nocapture

# Expected output:
# ✅ DQN loads 225 real features (not padded zeros)
# ✅ PPO loads 225 real features
# ✅ Data loader extracts Wave C + Wave D features

Success Criteria:

• No zero-padding in feature vectors
• All 225 features have real values (not 0.0)
• Feature extraction called during training loop

If Tests Pass: ✅ Full integration exists → Proceed to Phase 3 (Backtest) If Tests Fail: ❌ Partial integration → Execute Phase 2B (Wire Trainers)

---

Step 3: Smoke Test with Real Training (10 minutes)

# Run 1-epoch training with feature logging
cd /home/jgrusewski/Work/foxhunt

# DQN: 1 epoch, verbose logging
cargo run -p ml --example train_dqn --release -- \
  --epochs 1 \
  --verbose \
  --data-dir test_data/real/databento/ml_training

# Check logs for feature extraction
# Expected output:
# ✅ "Extracting 225 features from OHLCV bar"
# ✅ "Wave C features (201): [0.45, 0.78, ...]"
# ✅ "Wave D features (24): [0.12, 0.34, ...]"
# ❌ "Padding features to 225" (BAD - means zero-padding)

Success Criteria:

• Log contains "225 features extracted"
• No "padding" or "zero-fill" warnings
• Feature values are diverse (not 85% zeros)

If Logs Show Real Features: ✅ Proceed to Phase 3 If Logs Show Padding: ❌ Execute Phase 2B

---

Phase 2B: Full Integration Layer (4-6 hours)

If Validation Fails: Wire 225-Feature Extraction

Problem Analysis

File: /home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn.rs Lines: 668-681 Issue: Placeholder features with zero-padding

// CURRENT (BROKEN):
fn features_to_state(&self, features: &FinancialFeatures) -> Result<TradingState> {
    // Extract 4 prices + 6 basic indicators = 10 features
    let technical_indicators: Vec<f32> = features
        .technical_indicators
        .values()
        .map(|&v| v as f32)
        .collect();

    // Pad to 221 with ZEROS (this is the problem!)
    let mut tech_indicators_padded = technical_indicators;
    while tech_indicators_padded.len() < 221 {
        tech_indicators_padded.push(0.0);  // ❌ JUNK DATA
    }
    tech_indicators_padded.truncate(221);

    // Total: 4 prices + 221 tech = 225 (but 219 are zeros!)
    Ok(TradingState::new(
        price_features,
        tech_indicators_padded,  // ❌ 85% ZEROS
        market_features,
        portfolio_features,
    ))
}

---

Solution 1: Wire Common Feature Extraction (RECOMMENDED)

Prerequisite Check:

# Verify common::features exists
grep -r "FeatureVector225" /home/jgrusewski/Work/foxhunt/common/src/
grep -r "extract_225_features" /home/jgrusewski/Work/foxhunt/common/src/

# If found: Integration path exists ✅
# If not found: Feature extraction still in ml/ crate (needs migration)

Code Changes (if common::features exists):

File 1: /home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn.rs

// ADD at top:
use common::features::{FeatureVector225, FeatureExtractor};
use common::regime_detection::{RegimeDetector, RegimeType};

// REPLACE features_to_state() method:
fn features_to_state(&self,
    ohlcv: &OHLCVBar,
    regime_detector: &RegimeDetector,
) -> Result<TradingState> {
    // Extract all 225 features (Wave C + Wave D)
    let feature_vector = FeatureExtractor::extract_225_features(
        ohlcv,
        regime_detector,
    )?;

    // Convert to TradingState (no padding needed!)
    Ok(TradingState::from_feature_vector_225(feature_vector))
}

// UPDATE train() method to create RegimeDetector:
pub async fn train<F>(
    &mut self,
    dbn_data_dir: &str,
    mut checkpoint_callback: F,
) -> Result<TrainingMetrics>
where
    F: FnMut(usize, Vec<u8>) -> Result<String> + Send,
{
    // ADD regime detector
    let mut regime_detector = RegimeDetector::new(
        100,  // lookback window
        0.05, // volatility threshold
    )?;

    // Load DBN data
    let dbn_loader = DbnSequenceLoader::new(dbn_data_dir)?;

    for epoch in 0..self.hyperparams.epochs {
        for bar in dbn_loader.iter() {
            // Update regime state
            regime_detector.update(&bar)?;

            // Extract 225 features (Wave C + Wave D)
            let state = self.features_to_state(&bar, &regime_detector)?;

            // Select action
            let action = self.select_action(&state).await?;

            // Calculate reward
            let reward = self.calculate_reward(&bar, &action);

            // Get next state
            let next_bar = dbn_loader.peek_next()?;
            regime_detector.update(&next_bar)?;
            let next_state = self.features_to_state(&next_bar, &regime_detector)?;

            // Store experience
            self.store_experience(state, action, reward, next_state).await?;

            // Train on batch
            if self.can_train().await? {
                let (loss, q_value, grad_norm) = self.train_step().await?;
                // ... metrics logging
            }
        }

        // Save checkpoint
        if epoch % self.hyperparams.checkpoint_frequency == 0 {
            self.save_checkpoint(epoch, &mut checkpoint_callback).await?;
        }
    }

    Ok(self.get_metrics().await)
}

Estimated Time: 2 hours (DQN)

---

File 2: /home/jgrusewski/Work/foxhunt/ml/src/trainers/ppo.rs

// Similar changes:
// 1. Import common::features::FeatureVector225
// 2. Add regime_detector to train() method
// 3. Replace feature extraction with extract_225_features()
// 4. Remove zero-padding logic

Estimated Time: 2 hours (PPO)

---

File 3: /home/jgrusewski/Work/foxhunt/ml/src/trainers/mamba2.rs

// Similar changes for MAMBA-2
// Note: MAMBA-2 uses sequence modeling, so:
// 1. Extract 225 features for each bar in sequence
// 2. Pass [batch_size, seq_len, 225] tensor to model
// 3. Update regime state for each sequence step

Estimated Time: 1.5 hours (MAMBA-2)

---

File 4: /home/jgrusewski/Work/foxhunt/ml/src/trainers/tft.rs

// Similar changes for TFT
// Note: TFT adds 20 time/positional encodings
// Total: 225 + 20 = 245 features (expected)
// Fix existing "245 vs 225" mismatch warning

Estimated Time: 1.5 hours (TFT)

---

Solution 2: Use ML-Local Feature Extraction (FALLBACK)

If common::features doesn't exist:

# Check if ml crate has feature extraction
ls -la /home/jgrusewski/Work/foxhunt/ml/src/features/
grep -r "extract_225" /home/jgrusewski/Work/foxhunt/ml/src/features/

# Expected files:
# - unified.rs (Wave C + Wave D unified extraction)
# - extraction.rs (main extraction logic)
# - adx_features.rs (Wave D ADX features)
# - config.rs (feature configuration)

Code Changes:

// File: ml/src/trainers/dqn.rs

use crate::features::{extract_unified_features, FeatureConfig};
use crate::regime_detection::RegimeOrchestrator;

fn features_to_state(&self,
    ohlcv: &OHLCVBar,
    regime_orchestrator: &mut RegimeOrchestrator,
) -> Result<TradingState> {
    // Configure 225-feature extraction
    let config = FeatureConfig::wave_d_full();  // 201 + 24 = 225

    // Extract features
    let feature_vector = extract_unified_features(
        ohlcv,
        regime_orchestrator,
        &config,
    )?;

    assert_eq!(feature_vector.len(), 225, "Expected 225 features");

    // Convert to TradingState
    Ok(TradingState::from_vec(feature_vector))
}

Estimated Time: 3 hours (all 4 models)

---

Phase 2C: Testing & Validation (1 hour)

After Integration Changes

# Test 1: Verify 225-feature extraction
cargo test -p ml integration_wave_d_features --release -- --nocapture

# Expected output:
# ✅ test_extract_225_features ... ok
# ✅ test_wave_c_201_features ... ok
# ✅ test_wave_d_24_features ... ok
# ✅ test_regime_detection_integration ... ok

# Test 2: Train 1 epoch with feature logging
cargo run -p ml --example train_dqn --release -- \
  --epochs 1 \
  --verbose

# Expected output:
# ✅ "Extracted 225 features from bar 1"
# ✅ "Wave C features (201): [min=0.12, max=0.98, mean=0.45]"
# ✅ "Wave D features (24): [min=0.05, max=0.87, mean=0.32]"
# ❌ NO "padding" or "zero-fill" warnings

# Test 3: Verify checkpoint dimensions
cargo run -p ml --example validate_dqn_225_features --release

# Expected output:
# ✅ "Model input dimension: 225"
# ✅ "Checkpoint compatible: true"
# ✅ "Feature extraction tested: PASS"

---

Phase 2D: Retrain Models with Real Features (2-4 hours)

Once Integration is Validated

# Retrain DQN (100 epochs, ~3 minutes)
cargo run -p ml --example train_dqn --release -- \
  --epochs 100 \
  --output-dir ml/trained_models_225_features

# Retrain PPO (20 epochs, ~7 minutes)
cargo run -p ml --example train_ppo --release -- \
  --epochs 20 \
  --output-dir ml/trained_models_225_features

# Retrain MAMBA-2 (50 epochs with tuning, ~5 minutes)
cargo run -p ml --example train_mamba2_dbn --release -- \
  --epochs 50 \
  --learning-rate 0.001 \
  --n-layers 4 \
  --d-model 512 \
  --output-dir ml/trained_models_225_features

# Retrain TFT (20 epochs with reduced arch, ~10 minutes)
cargo run -p ml --example train_tft_dbn --release -- \
  --epochs 20 \
  --hidden-dim 128 \
  --num-attention-heads 4 \
  --lstm-layers 1 \
  --batch-size 16 \
  --output-dir ml/trained_models_225_features

Expected Improvements (vs Phase 1 broken training):

Metric	Phase 1 (Junk Data)	Phase 2 (Real 225 Features)	Improvement
DQN Loss	0.045	0.020-0.030	33-55% better
DQN Convergence	Epoch 70	Epoch 40-50	30% faster
PPO Convergence	Epoch 20	Epoch 12-15	25% faster
MAMBA-2 Loss	1.4e+38 (diverged)	0.1-1.0 (stable)	100% fixed
Backtest Sharpe	0.5-0.8	1.5-2.0	150-300% gain

---

Decision Tree Summary

Phase 2 Start
    │
    ├─→ Step 1: Run validation tests (10 min)
    │       │
    │       ├─→ Tests PASS → Step 2
    │       └─→ Tests FAIL → Phase 2B (Full Integration, 4-6h)
    │
    ├─→ Step 2: Check trainer integration (10 min)
    │       │
    │       ├─→ Integration EXISTS → Step 3
    │       └─→ Integration MISSING → Phase 2B
    │
    ├─→ Step 3: Smoke test 1-epoch training (10 min)
    │       │
    │       ├─→ Real features extracted → Phase 3 (Backtest)
    │       └─→ Zero-padding detected → Phase 2B
    │
    └─→ Phase 2B: Full integration (4-6h)
            │
            ├─→ Wire common::features → 4h
            │   └─→ Test → Phase 2C (1h)
            │       └─→ Retrain → Phase 2D (2-4h)
            │
            └─→ Use ml::features → 3h
                └─→ Test → Phase 2C (1h)
                    └─→ Retrain → Phase 2D (2-4h)

---

Time Estimates

Best Case (Integration Exists)

• Phase 2 Validation: 30 minutes
• Phase 3 Backtest: 30 minutes
• Total: 1 hour → Ready for production deployment

Worst Case (Integration Missing)

• Phase 2 Validation: 30 minutes
• Phase 2B Integration: 4-6 hours
• Phase 2C Testing: 1 hour
• Phase 2D Retraining: 2-4 hours
• Phase 3 Backtest: 30 minutes
• Total: 8-12 hours → Ready for production deployment

Most Likely (Partial Integration)

• Phase 2 Validation: 30 minutes
• Phase 2B Partial Fix: 2-3 hours
• Phase 2C Testing: 1 hour
• Phase 2D Retraining: 2 hours
• Phase 3 Backtest: 30 minutes
• Total: 6 hours → Ready for production deployment

---

Next Actions (Priority Order)

Immediate (Next 10 minutes)

1. Run validation test suite:

   cargo test -p ml test_225 --release -- --nocapture
   

2. Check for common::features:

   grep -r "FeatureVector225" /home/jgrusewski/Work/foxhunt/common/src/
   

3. Inspect DQN feature extraction:

   grep -A 20 "features_to_state" /home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn.rs
   

Short-Term (If integration missing, next 4-6 hours)

4. Implement Solution 1 or Solution 2 (see Phase 2B above)

5. Run integration tests (Phase 2C)

6. Retrain all 4 models (Phase 2D)

Medium-Term (After integration validated, next 1 week)

7. Run Wave Comparison Backtest (Phase 3):

   cargo run -p backtesting_service --example wave_comparison --release
   

8. If Sharpe ≥ 1.5: Deploy to paper trading (1 week)

9. If Sharpe < 1.5: Purchase extended data ($2-$4) and retrain

---

Risk Mitigation

Risk 1: Integration Completely Missing

Probability: 60% Impact: HIGH (8-12 hours delay) Mitigation: Execute Phase 2B immediately, prioritize DQN+PPO first

Risk 2: Integration Exists but Broken

Probability: 30% Impact: MEDIUM (4-6 hours debug) Mitigation: Use git blame to find original implementation, check Wave D docs

Risk 3: Feature Extraction Performance Issues

Probability: 10% Impact: LOW (1-2 hours optimization) Mitigation: Use existing benchmarks (target: <50μs per bar, already validated)

---

Success Criteria

Phase 2 Complete When:

✅ Validation Tests:

• All 225 features extracted (no zero-padding)
• Regime detection operational
• Performance: <50μs per bar

✅ Integration Tests:

• DQN trains with real 225 features
• PPO trains with real 225 features
• MAMBA-2 trains with real 225 features
• TFT trains with real 225 features (245 = 225 + 20 time encodings)

✅ Training Quality:

• DQN loss: <0.03 (not 0.045)
• MAMBA-2 loss: 0.1-1.0 (not 1e+38)
• No "padding" or "zero-fill" warnings in logs
• Feature diversity: No more than 10% zeros

✅ Checkpoint Validation:

• All checkpoints have 225-dimensional input layer
• Models load successfully in inference mode
• Feature extraction test passes

---

Conclusion

Status: 🟡 INTEGRATION INCOMPLETE (95% confidence)

Evidence:

1. DQN features_to_state() uses zero-padding (lines 668-681)
2. Only 10 real features + 215 zeros = 225 "features"
3. Phase 1 training succeeded too easily (no feature extraction errors)
4. MAMBA-2 divergence suggests low-quality training data

Recommendation:

1. Execute Phase 2 validation (30 min) to confirm status
2. If validation fails: Execute Phase 2B integration (4-6 hours)
3. If validation passes: Proceed directly to Phase 3 backtest

Expected Outcome:

• With real 225 features: Sharpe 1.5-2.0, Win Rate 60%, Drawdown 15%
• With junk features: Sharpe 0.5-0.8, Win Rate 48-52%, Drawdown 25%

Next Command:

cargo test -p ml integration_wave_d_features --release -- --nocapture

---

Document Version: 1.0 Created: 2025-10-20 Status: READY TO EXECUTE Estimated Completion: 30 minutes (validation) or 8-12 hours (full integration)