foxhunt/IMPLEMENTATION_GUIDE_WAVE_C.md

ML Training Service - Implementation Guide for Wave C Integration

Quick Reference: Feature Extraction Touch Points

1. Real-Time Inference (26 features) - WORKING

When: Every order/prediction in trading service
File: /home/jgrusewski/Work/foxhunt/common/src/ml_strategy.rs (Lines 170-897)

// MLFeatureExtractor::extract_features()
let mut features = Vec::new();
features.push(price_return);           // Index 0
features.push(short_ma_ratio);         // Index 1
// ... through Index 25 (macd_signal)
Ok(features)  // Returns Vec<f64> with exactly 26 elements

Call Stack:

SharedMLStrategy::get_ensemble_prediction()
  ↓
MLFeatureExtractor::extract_features(price, volume, timestamp)
  ↓
SimpleDQNAdapter::predict(&features) // Expects 26 features

---

2. Training Data Loading (256 features padding) - NEEDS FIXING

When: During model training (MAMBA-2, DQN, PPO, TFT)
File: /home/jgrusewski/Work/foxhunt/ml/src/data_loaders/dbn_sequence_loader.rs (Lines 664-804)

fn extract_features(&self, msg: &ProcessedMessage) -> Result<Vec<f32>> {
    match msg {
        ProcessedMessage::Ohlcv { open, high, low, close, volume, .. } => {
            // Lines 675-703: 31 actual features
            let base_features = [o, h, l, c, v, range, body, upper_wick, lower_wick];
            
            // Lines 704-763: Expand to 256 via padding
            for _ in 0..25 {
                features.extend_from_slice(&base_features);  // REPETITION!
            }
            Ok(features)  // Returns Vec<f32> with 256 elements
        }
    }
}

Problem: Only 31 unique features + 225 padding (9×25 repetition)

---

3. Where 256 Gets Hardcoded

File: ml/examples/train_mamba2_dbn.rs (Line 292)

// HARDCODED - Should be configurable
let mut loader = DbnSequenceLoader::new(config.seq_len, config.d_model)
    .await?;  // d_model = 256 (hardcoded in config)

// Results in:
// - Input tensors: [batch=1, seq_len=60, d_model=256]
// - Actual features: 31 real + 225 padding

Related: ml/examples/train_ppo.rs, ml/examples/train_dqn.rs, ml/examples/train_tft_dbn.rs

---

Code Locations by Task

Task 1: See Current 26 Features

File: /home/jgrusewski/Work/foxhunt/common/src/ml_strategy.rs
Lines: 170-897
Search for: pub fn extract_features(&mut self, price: f64, volume: f64

Features extracted in order:

• 0-2: Price features (return, MA ratio, volatility)
• 3-4: Volume features
• 5-6: Time features
• 7-17: Technical indicators
• 18-25: Wave 19 additions

---

Task 2: See Current 256 Feature Extraction (Padding)

File: /home/jgrusewski/Work/foxhunt/ml/src/data_loaders/dbn_sequence_loader.rs
Lines: 664-804
Search for: fn extract_features(&self, msg: &ProcessedMessage)

Actual feature computation:

• Lines 675-703: 31 real features
• Lines 704-758: Padding to 256

---

Task 3: Where DQN Expects 26 Features

File: /home/jgrusewski/Work/foxhunt/common/src/ml_strategy.rs
Lines: 914-1018
Search for: pub struct SimpleDQNAdapter

pub struct SimpleDQNAdapter {
    weights: Vec<f64>,  // Line 917: Must be exactly 26
    // ...
}

// Line 966: Assertion
assert_eq!(weights.len(), 26, "SimpleDQNAdapter must have exactly 26 weights");

// Line 979: Validates input
if features.len() != self.weights.len() {
    return Err(anyhow::anyhow!("Feature dimension mismatch: expected {}, got {}",
        self.weights.len(), features.len()));
}

---

Task 4: Where MAMBA-2 Loads Data

File: /home/jgrusewski/Work/foxhunt/ml/examples/train_mamba2_dbn.rs
Lines: 290-350

// Line 291-292: Data loading
let mut loader = DbnSequenceLoader::new(config.seq_len, config.d_model)
    .await
    .context("Failed to create DBN sequence loader")?;

// Line 296-299: Load sequences
let (train_data, val_data) = loader
    .load_sequences(&config.data_dir, 0.8)
    .await
    .context("Failed to load DBN sequences")?;

// Line 308-373: Validate shapes
// This is where [1, 60, 256] tensors are created

---

Task 5: Where Features Are Configured in DbnSequenceLoader

File: /home/jgrusewski/Work/foxhunt/ml/src/data_loaders/dbn_sequence_loader.rs
Lines: 101-171

// Line 50: Feature dimension stored
pub struct DbnSequenceLoader {
    pub seq_len: usize,
    pub d_model: usize,  // Currently only used for tensor shape (256)
}

// Line 110: Constructor
pub async fn new(seq_len: usize, d_model: usize) -> Result<Self> {
    // No feature config - just stores d_model
}

// Line 292 (train_mamba2_dbn.rs): Always 256
let mut loader = DbnSequenceLoader::new(config.seq_len, config.d_model)

---

Change Summary for Wave C Integration

Change 1: Make Feature Count Configurable

Target File: ml/src/data_loaders/dbn_sequence_loader.rs

- pub async fn new(seq_len: usize, d_model: usize) -> Result<Self> {
+ pub async fn new(seq_len: usize, d_model: usize) -> Result<Self> {
+ pub async fn with_feature_config(seq_len: usize, config: FeatureConfig) -> Result<Self> {
      let parser = DbnParser::new()...
+     // Validate feature count matches config
+     let actual_features = config.compute_total_features();
+     info!("Feature config: {} total features", actual_features);
  }

---

Change 2: Replace Padding with Actual Features

Target File: ml/src/data_loaders/dbn_sequence_loader.rs (Lines 675-804)

- // 7. Tile base 9 features 25 times to reach 256
- for _ in 0..25 {
-     features.extend_from_slice(&base_features);
- }

+ // 7. Add Wave B features (alternative bars, barrier optimization)
+ if config.include_alternative_bars {
+     // Dollar bars: 5 features
+     // Volume bars: 5 features
+ }
+ 
+ // 8. Add Wave C features (fractional diff, meta-labels)
+ if config.include_fractional_diff {
+     // Fractional differentiation: 10+ features
+ }
+ 
+ // 9. Add structural break features
+ if config.include_struct_breaks {
+     // CUSUM: 5 features
+ }

---

Change 3: Create FeatureConfig System

New File: ml/src/config/feature_config.rs

pub enum WaveLevel {
    WaveA,  // 26 features (current)
    WaveB,  // 26 + 10 = 36 features
    WaveC,  // 36 + 20+ = 65+ features
}

pub struct FeatureConfig {
    pub wave_level: WaveLevel,
    pub include_alternative_bars: bool,
    pub alternative_bars_type: BarType,  // Dollar, Volume, etc.
    pub include_fractional_diff: bool,
    pub include_meta_labels: bool,
    pub include_struct_breaks: bool,
}

impl FeatureConfig {
    pub fn compute_total_features(&self) -> usize {
        let mut count = 26;  // Base Wave A
        if self.include_alternative_bars { count += 10; }
        if self.include_fractional_diff { count += 20; }
        if self.include_meta_labels { count += 15; }
        if self.include_struct_breaks { count += 5; }
        count
    }
}

---

Change 4: Update SimpleDQNAdapter

Target File: common/src/ml_strategy.rs

  pub struct SimpleDQNAdapter {
      model_id: String,
      weights: Vec<f64>,
+     feature_config: Option<FeatureConfig>,
  }
  
  impl SimpleDQNAdapter {
      pub fn new(model_id: String) -> Self {
          // Backward compatible: 26 weights
          let weights = vec![...];  // 26 weights
+         SimpleDQNAdapter { model_id, weights, feature_config: None }
+     }
+     
+     pub fn with_config(model_id: String, config: FeatureConfig) -> Self {
+         let feature_count = config.compute_total_features();
+         let weights = generate_random_weights(feature_count);  // Dynamic size
+         SimpleDQNAdapter { model_id, weights, feature_config: Some(config) }
      }
  }

---

Change 5: Update Training Scripts

Target Files: All 4 ml/examples/train_*.rs

  // train_mamba2_dbn.rs (Line 292)
- let mut loader = DbnSequenceLoader::new(config.seq_len, config.d_model).await?;

+ let feature_config = FeatureConfig {
+     wave_level: WaveLevel::WaveC,
+     include_alternative_bars: true,
+     alternative_bars_type: BarType::Dollar(1_000_000),
+     include_fractional_diff: true,
+     include_meta_labels: true,
+ };
+ let mut loader = DbnSequenceLoader::with_feature_config(
+     config.seq_len,
+     feature_config
+ ).await?;
+
+ let actual_feature_count = feature_config.compute_total_features();
+ let mut mamba_config = Mamba2Config {
+     d_model: actual_feature_count,  // NOT 256 - dynamic
+     // ...
+ };

---

Verification Checklist

After implementing Wave C integration:

• Feature Count: assert_eq!(features.len(), config.compute_total_features())
• No Padding: Features 31-255 are NOT repetitions of earlier features
• All 4 Scripts: DQN, PPO, MAMBA-2, TFT all use configurable feature count
• SimpleDQNAdapter: Weights match feature count dynamically
• Tests Pass: All existing tests still pass with Wave A config
• New Tests: Add tests for Wave B (36 features) and Wave C (65+ features)
• Performance: Feature extraction latency <1ms per bar
• Backtest: Win rate improves by 5-15% vs Wave A baseline

---

Files to Monitor

Critical Dependencies:

1. /home/jgrusewski/Work/foxhunt/ml/src/data_loaders/dbn_sequence_loader.rs

   • DbnSequenceLoader::extract_features()
   • DbnSequenceLoader::create_sequences()

2. /home/jgrusewski/Work/foxhunt/common/src/ml_strategy.rs

   • MLFeatureExtractor (must stay 26 for inference)
   • SimpleDQNAdapter (must be dynamic for training)

3. /home/jgrusewski/Work/foxhunt/ml/examples/train_mamba2_dbn.rs

   • Line 292: Data loader initialization
   • Line 388: MAMBA-2 config

4. /home/jgrusewski/Work/foxhunt/ml/src/features/alternative_bars.rs

   • Required for Wave B integration

---

Timeline Estimate

Phase	Task	Hours	Blocker
1	Create FeatureConfig system	2	None
2	Update DbnSequenceLoader	3	Phase 1
3	Fix SimpleDQNAdapter	1	Phase 2
4	Update 4 training scripts	2	Phase 2
5	Add Wave B/C features	4	Phase 1
6	Integration testing	3	Phase 5
7	Backtest validation	4	Phase 6
Total		19 hours

---

Success Criteria

1. Inference Still Works: SharedMLStrategy::get_ensemble_prediction() returns 26 features
2. Training Improved: 65+ real features instead of 256 with padding
3. All Models Train: DQN, PPO, MAMBA-2, TFT all use new feature config
4. Performance Stable: <1ms feature extraction, no GPU memory regression
5. Backtest Positive: 5-25% Sharpe improvement vs Wave A