# ML Training Service - Implementation Guide for Wave C Integration

## Quick Reference: Feature Extraction Touch Points

### 1. Real-Time Inference (26 features) - WORKING
**When**: Every order/prediction in trading service  
**File**: `/home/jgrusewski/Work/foxhunt/common/src/ml_strategy.rs` (Lines 170-897)

```rust
// MLFeatureExtractor::extract_features()
let mut features = Vec::new();
features.push(price_return);           // Index 0
features.push(short_ma_ratio);         // Index 1
// ... through Index 25 (macd_signal)
Ok(features)  // Returns Vec<f64> with exactly 26 elements
```

**Call Stack**:
```
SharedMLStrategy::get_ensemble_prediction()
  ↓
MLFeatureExtractor::extract_features(price, volume, timestamp)
  ↓
SimpleDQNAdapter::predict(&features) // Expects 26 features
```

---

### 2. Training Data Loading (256 features padding) - NEEDS FIXING
**When**: During model training (MAMBA-2, DQN, PPO, TFT)  
**File**: `/home/jgrusewski/Work/foxhunt/ml/src/data_loaders/dbn_sequence_loader.rs` (Lines 664-804)

```rust
fn extract_features(&self, msg: &ProcessedMessage) -> Result<Vec<f32>> {
    match msg {
        ProcessedMessage::Ohlcv { open, high, low, close, volume, .. } => {
            // Lines 675-703: 31 actual features
            let base_features = [o, h, l, c, v, range, body, upper_wick, lower_wick];
            
            // Lines 704-763: Expand to 256 via padding
            for _ in 0..25 {
                features.extend_from_slice(&base_features);  // REPETITION!
            }
            Ok(features)  // Returns Vec<f32> with 256 elements
        }
    }
}
```

**Problem**: Only 31 unique features + 225 padding (9×25 repetition)

---

### 3. Where 256 Gets Hardcoded
**File**: `ml/examples/train_mamba2_dbn.rs` (Line 292)

```rust
// HARDCODED - Should be configurable
let mut loader = DbnSequenceLoader::new(config.seq_len, config.d_model)
    .await?;  // d_model = 256 (hardcoded in config)

// Results in:
// - Input tensors: [batch=1, seq_len=60, d_model=256]
// - Actual features: 31 real + 225 padding
```

**Related**: `ml/examples/train_ppo.rs`, `ml/examples/train_dqn.rs`, `ml/examples/train_tft_dbn.rs`

---

## Code Locations by Task

### Task 1: See Current 26 Features
**File**: `/home/jgrusewski/Work/foxhunt/common/src/ml_strategy.rs`  
**Lines**: 170-897  
**Search for**: `pub fn extract_features(&mut self, price: f64, volume: f64`

Features extracted in order:
- 0-2: Price features (return, MA ratio, volatility)
- 3-4: Volume features
- 5-6: Time features
- 7-17: Technical indicators
- 18-25: Wave 19 additions

---

### Task 2: See Current 256 Feature Extraction (Padding)
**File**: `/home/jgrusewski/Work/foxhunt/ml/src/data_loaders/dbn_sequence_loader.rs`  
**Lines**: 664-804  
**Search for**: `fn extract_features(&self, msg: &ProcessedMessage)`

Actual feature computation:
- Lines 675-703: 31 real features
- Lines 704-758: Padding to 256

---

### Task 3: Where DQN Expects 26 Features
**File**: `/home/jgrusewski/Work/foxhunt/common/src/ml_strategy.rs`  
**Lines**: 914-1018  
**Search for**: `pub struct SimpleDQNAdapter`

```rust
pub struct SimpleDQNAdapter {
    weights: Vec<f64>,  // Line 917: Must be exactly 26
    // ...
}

// Line 966: Assertion
assert_eq!(weights.len(), 26, "SimpleDQNAdapter must have exactly 26 weights");

// Line 979: Validates input
if features.len() != self.weights.len() {
    return Err(anyhow::anyhow!("Feature dimension mismatch: expected {}, got {}",
        self.weights.len(), features.len()));
}
```

---

### Task 4: Where MAMBA-2 Loads Data
**File**: `/home/jgrusewski/Work/foxhunt/ml/examples/train_mamba2_dbn.rs`  
**Lines**: 290-350

```rust
// Line 291-292: Data loading
let mut loader = DbnSequenceLoader::new(config.seq_len, config.d_model)
    .await
    .context("Failed to create DBN sequence loader")?;

// Line 296-299: Load sequences
let (train_data, val_data) = loader
    .load_sequences(&config.data_dir, 0.8)
    .await
    .context("Failed to load DBN sequences")?;

// Line 308-373: Validate shapes
// This is where [1, 60, 256] tensors are created
```

---

### Task 5: Where Features Are Configured in DbnSequenceLoader
**File**: `/home/jgrusewski/Work/foxhunt/ml/src/data_loaders/dbn_sequence_loader.rs`  
**Lines**: 101-171

```rust
// Line 50: Feature dimension stored
pub struct DbnSequenceLoader {
    pub seq_len: usize,
    pub d_model: usize,  // Currently only used for tensor shape (256)
}

// Line 110: Constructor
pub async fn new(seq_len: usize, d_model: usize) -> Result<Self> {
    // No feature config - just stores d_model
}

// Line 292 (train_mamba2_dbn.rs): Always 256
let mut loader = DbnSequenceLoader::new(config.seq_len, config.d_model)
```

---

## Change Summary for Wave C Integration

### Change 1: Make Feature Count Configurable
**Target File**: `ml/src/data_loaders/dbn_sequence_loader.rs`

```diff
- pub async fn new(seq_len: usize, d_model: usize) -> Result<Self> {
+ pub async fn new(seq_len: usize, d_model: usize) -> Result<Self> {
+ pub async fn with_feature_config(seq_len: usize, config: FeatureConfig) -> Result<Self> {
      let parser = DbnParser::new()...
+     // Validate feature count matches config
+     let actual_features = config.compute_total_features();
+     info!("Feature config: {} total features", actual_features);
  }
```

---

### Change 2: Replace Padding with Actual Features
**Target File**: `ml/src/data_loaders/dbn_sequence_loader.rs` (Lines 675-804)

```diff
- // 7. Tile base 9 features 25 times to reach 256
- for _ in 0..25 {
-     features.extend_from_slice(&base_features);
- }

+ // 7. Add Wave B features (alternative bars, barrier optimization)
+ if config.include_alternative_bars {
+     // Dollar bars: 5 features
+     // Volume bars: 5 features
+ }
+ 
+ // 8. Add Wave C features (fractional diff, meta-labels)
+ if config.include_fractional_diff {
+     // Fractional differentiation: 10+ features
+ }
+ 
+ // 9. Add structural break features
+ if config.include_struct_breaks {
+     // CUSUM: 5 features
+ }
```

---

### Change 3: Create FeatureConfig System
**New File**: `ml/src/config/feature_config.rs`

```rust
pub enum WaveLevel {
    WaveA,  // 26 features (current)
    WaveB,  // 26 + 10 = 36 features
    WaveC,  // 36 + 20+ = 65+ features
}

pub struct FeatureConfig {
    pub wave_level: WaveLevel,
    pub include_alternative_bars: bool,
    pub alternative_bars_type: BarType,  // Dollar, Volume, etc.
    pub include_fractional_diff: bool,
    pub include_meta_labels: bool,
    pub include_struct_breaks: bool,
}

impl FeatureConfig {
    pub fn compute_total_features(&self) -> usize {
        let mut count = 26;  // Base Wave A
        if self.include_alternative_bars { count += 10; }
        if self.include_fractional_diff { count += 20; }
        if self.include_meta_labels { count += 15; }
        if self.include_struct_breaks { count += 5; }
        count
    }
}
```

---

### Change 4: Update SimpleDQNAdapter
**Target File**: `common/src/ml_strategy.rs`

```diff
  pub struct SimpleDQNAdapter {
      model_id: String,
      weights: Vec<f64>,
+     feature_config: Option<FeatureConfig>,
  }
  
  impl SimpleDQNAdapter {
      pub fn new(model_id: String) -> Self {
          // Backward compatible: 26 weights
          let weights = vec![...];  // 26 weights
+         SimpleDQNAdapter { model_id, weights, feature_config: None }
+     }
+     
+     pub fn with_config(model_id: String, config: FeatureConfig) -> Self {
+         let feature_count = config.compute_total_features();
+         let weights = generate_random_weights(feature_count);  // Dynamic size
+         SimpleDQNAdapter { model_id, weights, feature_config: Some(config) }
      }
  }
```

---

### Change 5: Update Training Scripts
**Target Files**: All 4 `ml/examples/train_*.rs`

```diff
  // train_mamba2_dbn.rs (Line 292)
- let mut loader = DbnSequenceLoader::new(config.seq_len, config.d_model).await?;

+ let feature_config = FeatureConfig {
+     wave_level: WaveLevel::WaveC,
+     include_alternative_bars: true,
+     alternative_bars_type: BarType::Dollar(1_000_000),
+     include_fractional_diff: true,
+     include_meta_labels: true,
+ };
+ let mut loader = DbnSequenceLoader::with_feature_config(
+     config.seq_len,
+     feature_config
+ ).await?;
+
+ let actual_feature_count = feature_config.compute_total_features();
+ let mut mamba_config = Mamba2Config {
+     d_model: actual_feature_count,  // NOT 256 - dynamic
+     // ...
+ };
```

---

## Verification Checklist

After implementing Wave C integration:

- [ ] **Feature Count**: `assert_eq!(features.len(), config.compute_total_features())`
- [ ] **No Padding**: Features 31-255 are NOT repetitions of earlier features
- [ ] **All 4 Scripts**: DQN, PPO, MAMBA-2, TFT all use configurable feature count
- [ ] **SimpleDQNAdapter**: Weights match feature count dynamically
- [ ] **Tests Pass**: All existing tests still pass with Wave A config
- [ ] **New Tests**: Add tests for Wave B (36 features) and Wave C (65+ features)
- [ ] **Performance**: Feature extraction latency <1ms per bar
- [ ] **Backtest**: Win rate improves by 5-15% vs Wave A baseline

---

## Files to Monitor

**Critical Dependencies**:
1. `/home/jgrusewski/Work/foxhunt/ml/src/data_loaders/dbn_sequence_loader.rs`
   - DbnSequenceLoader::extract_features()
   - DbnSequenceLoader::create_sequences()

2. `/home/jgrusewski/Work/foxhunt/common/src/ml_strategy.rs`
   - MLFeatureExtractor (must stay 26 for inference)
   - SimpleDQNAdapter (must be dynamic for training)

3. `/home/jgrusewski/Work/foxhunt/ml/examples/train_mamba2_dbn.rs`
   - Line 292: Data loader initialization
   - Line 388: MAMBA-2 config

4. `/home/jgrusewski/Work/foxhunt/ml/src/features/alternative_bars.rs`
   - Required for Wave B integration

---

## Timeline Estimate

| Phase | Task | Hours | Blocker |
|-------|------|-------|---------|
| 1 | Create FeatureConfig system | 2 | None |
| 2 | Update DbnSequenceLoader | 3 | Phase 1 |
| 3 | Fix SimpleDQNAdapter | 1 | Phase 2 |
| 4 | Update 4 training scripts | 2 | Phase 2 |
| 5 | Add Wave B/C features | 4 | Phase 1 |
| 6 | Integration testing | 3 | Phase 5 |
| 7 | Backtest validation | 4 | Phase 6 |
| **Total** | | **19 hours** | |

---

## Success Criteria

1. **Inference Still Works**: `SharedMLStrategy::get_ensemble_prediction()` returns 26 features
2. **Training Improved**: 65+ real features instead of 256 with padding
3. **All Models Train**: DQN, PPO, MAMBA-2, TFT all use new feature config
4. **Performance Stable**: <1ms feature extraction, no GPU memory regression
5. **Backtest Positive**: 5-25% Sharpe improvement vs Wave A