foxhunt/ML_TRAINING_PIPELINE_ANALYSIS.md

ML Training Service Data Pipeline - Complete Analysis

Date: October 17, 2025
Scope: Comprehensive investigation of data flow from raw market data to model input
Focus: Current feature extraction, model adapters, and integration requirements for Wave C

---

Executive Summary

The ML training pipeline consists of:

1. DBN Data Loading - Real market data via DataBento binary format
2. Feature Extraction - Two separate systems:
   • 26 features for real-time inference (common/ml_strategy.rs)
   • 256 features for model training (ml/src/data_loaders/dbn_sequence_loader.rs)
3. Model-Specific Adapters - Custom input shapes for DQN, PPO, MAMBA-2, TFT
4. Training Scripts - 4 production training examples (train_*.rs)

Critical Issue: The inference system (26 features) and training system (256 features) are disconnected. Training uses feature padding (9 base features repeated 25+ times) rather than actual feature engineering.

---

1. Training Data Flow (Complete Pipeline)

Raw DBN Data (ES.FUT, NQ.FUT, 6E.FUT, ZN.FUT)
    ↓
DbnSequenceLoader.load_sequences()
    ├─ DbnDecoder (official dbn crate)
    ├─ Extract OHLCV messages
    ├─ Compute feature statistics (price_mean, price_std, volume_mean, volume_std)
    └─ create_sequences() with sliding window
    ↓
extract_features() - 256-dimensional vectors
    ├─ Base OHLCV (5 features)
    ├─ Derived features (4 features: range, body, wicks)
    ├─ Price ratios (10 features)
    ├─ Log returns (4 features)
    ├─ Price deltas (4 features)
    ├─ Normalized prices (4 features)
    └─ Tiled base features (225 features = 9 × 25 repetitions)
    ↓
Tensors [batch=1, seq_len=60, d_model=256]
    ├─ Input: [1, 60, 256] f64
    └─ Target: [1, 1, 1] f64 (regression: next close price)
    ↓
Model Training (DQN, PPO, MAMBA-2, TFT)
    ├─ GPU: CUDA-accelerated (RTX 3050 Ti)
    └─ Checkpoints: ml/checkpoints/model_*/

---

2. Current Feature Set Analysis

2.1 Real-Time Inference Features (26 features)

File: /home/jgrusewski/Work/foxhunt/common/src/ml_strategy.rs
Class: MLFeatureExtractor
Method: extract_features(price, volume, timestamp) -> Vec<f64>

Feature Breakdown:

Index	Name	Formula	Range	Source
0	price_return	(current - prev) / prev	±0.05	Line 224
1	short_ma_ratio	current / SMA(5) - 1.0	±0.02	Line 235
2	volatility	std_dev(returns, 10-period)	[0, ∞)	Line 243
3	volume_ratio	current_vol / prev_vol - 1.0	±2.0	Line 268
4	volume_ma_ratio	current_vol / SMA_vol(5) - 1.0	±1.0	Line 278
5	hour	hour / 24.0	[0, 1]	Line 290
6	day_of_week	weekday / 6.0	[0, 1]	Line 291
7	williams_r	((H - C) / (H - L)) * -100, tanh	[-1, 1]	Line 311
8	roc	((C - C₋₁₂) / C₋₁₂) * 100, tanh	[-1, 1]	Line 330
9	ultimate_oscillator	Weighted BP/TR average	[-1, 1]	Line 385
10	obv	Cumulative vol flow, tanh	[-1, 1]	Line 408
11	mfi	MFI-14, normalized	[-1, 1]	Line 455
12	vwap_ratio	(C - VWAP) / VWAP, tanh	[-1, 1]	Line 485
13	ema_9_norm	(price / EMA₉ - 1.0).tanh()	[-1, 1]	Line 494
14	ema_21_norm	(price / EMA₂₁ - 1.0).tanh()	[-1, 1]	Line 499
15	ema_50_norm	(price / EMA₅₀ - 1.0).tanh()	[-1, 1]	Line 504
16	ema_9_21_cross	+1.0 if EMA₉ > EMA₂₁	{-1, +1}	Line 510
17	ema_21_50_cross	+1.0 if EMA₂₁ > EMA₅₀	{-1, +1}	Line 511
18	adx	Wilder's DI smoothing, ADX formula	[0, 1]	Line 610
19	bollinger_position	(C - middle) / (upper - lower)	[-1, 1]	Line 664
20	stochastic_k	(C - L₁₄) / (H₁₄ - L₁₄) * 100	[0, 1]	Line 706
21	stochastic_d	SMA(%K, 3)	[0, 1]	Line 718
22	cci	(TP - SMA₂₀) / (0.015 * MAD), tanh	[-1, 1]	Line 785
23	rsi	100 - (100 / (1 + RS))	[0, 1]	Line 829
24	macd	EMA₁₂ - EMA₂₆, tanh	[-1, 1]	Line 881
25	macd_signal	EMA(MACD, 9), tanh	[-1, 1]	Line 887

Source Documentation: /home/jgrusewski/Work/foxhunt/WAVE_19_FEATURE_INDEX_MAP.md

---

2.2 Training System Features (256 features)

File: /home/jgrusewski/Work/foxhunt/ml/src/data_loaders/dbn_sequence_loader.rs
Method: extract_features(msg: ProcessedMessage) -> Vec<f32>
Lines: 664-804

Current Implementation (Lines 675-763):

// 256 features via padding approach (NOT actual feature engineering):
// 1. Base OHLCV (5): o, h, l, c, v (normalized)
// 2. Derived (4): range, body, upper_wick, lower_wick
// 3. Price ratios (10): c/o, h/l, h/c, l/c, c/h, c/l, body/range, upper_wick/range, lower_wick/range, v/price
// 4. Log returns (4): ln(c/o), ln(h/o), ln(l/o), ln(c/h)
// 5. Price deltas (4): c-o, h-o, l-o, c-l
// 6. Normalized prices (4): (o-l)/(h-l), (c-l)/(h-l), 0.0, 1.0
// 7. Tiled padding (225): base_9_features repeated 25 times
// Total: 5 + 4 + 10 + 4 + 4 + 4 + 225 = 256 features

// Code snippet (lines 756-758):
for _ in 0..25 {
    features.extend_from_slice(&base_features);
}

Problem: Features 31-255 are just padding (repeated base features). No actual technical indicators or advanced features used.

---

2.3 Model-Specific Adapters

DQN Adapter

File: /home/jgrusewski/Work/foxhunt/common/src/ml_strategy.rs
Class: SimpleDQNAdapter
Input Dimension: 26 features
Lines: 914-975

pub struct SimpleDQNAdapter {
    weights: Vec<f64>,  // 26 weights, one per feature
}

// In predict() (line 978):
// Linear combination + sigmoid activation
let linear_output: f64 = features.iter()
    .zip(self.weights.iter())
    .map(|(f, w)| f * w)
    .sum();
let prediction_value = 1.0 / (1.0 + (-linear_output).exp());

Feature Weights:

• Original features (0-17): 0.07 to 0.18
• Wave 19 additions (18-25): 0.07 to 0.16

---

MAMBA-2 Adapter

File: /home/jgrusewski/Work/foxhunt/ml/examples/train_mamba2_dbn.rs
Input Shape: [batch, seq_len=60, d_model=256]
Lines: 292-350

let mut loader = DbnSequenceLoader::new(config.seq_len, config.d_model)
    .await?;  // seq_len=60, d_model=256

let (train_data, val_data) = loader
    .load_sequences(&config.data_dir, 0.8)
    .await?;

// Creates tensors [1, 60, 256] with 256 features per timestep

Configuration (lines 388-399):

let mamba_config = Mamba2Config {
    d_model: 256,           // Feature input dimension
    d_state: 16,            // SSM state dimension
    d_head: 32,             // d_model / 8
    num_heads: 8,
    expand: 2,              // d_inner = d_model * expand = 512
    num_layers: 6,
    dropout: 0.1,
    use_ssd: true,
    use_selective_state: true,
    hardware_aware: true,
    target_latency_us: 5,
};

---

PPO Adapter

File: /home/jgrusewski/Work/foxhunt/ml/examples/train_ppo.rs
Input Features: Variable (OHLCV + indicators)
Lines: 126-150

// Loads DBN data and extracts features
let bars = loader.load_symbol_data(&opts.symbol).await?;
let features = loader.extract_features(&bars)?;
let indicators = loader.calculate_indicators(&bars)?;

// State vector construction (from comments lines 147-149):
// State: [open, high, low, close, volume, rsi, macd, macd_signal, 
//         bb_upper, bb_middle, bb_lower, atr, ema_fast, ema_slow, 
//         volume_ma, log_return]
// ≈16 features per bar

---

TFT Adapter

File: /home/jgrusewski/Work/foxhunt/ml/examples/train_tft_dbn.rs
Input: DBN OHLCV bars with lookback window
Lines: 131-150

// Configuration (lines 62-68):
pub lookback_window: usize,       // 60 (default)
pub forecast_horizon: usize,      // 10 (default)
pub hidden_dim: usize,            // 256

// TFT requires:
// - Static covariates: one-time features (symbol, asset class)
// - Time-varying features: changing at each timestep (OHLCV + indicators)
// - Multi-horizon targets: [t+1, t+2, ..., t+10]

---

3. Training Scripts Analysis

3.1 MAMBA-2 Training (Primary Script)

File: /home/jgrusewski/Work/foxhunt/ml/examples/train_mamba2_dbn.rs
Status: Production Ready (Wave 206 - Shape bug fixed)
Feature Count: 256 (via padding, not actual features)

Pipeline:

1. Load DBN files (line 291)
2. DbnSequenceLoader.load_sequences() (line 296)
3. Shape validation (line 308-373)
4. Training loop with checkpointing (line 400+)

Critical Line 292:

let mut loader = DbnSequenceLoader::new(config.seq_len, config.d_model)

This hardcodes d_model=256, but actual feature count is padding-based.

Usage:

cargo run -p ml --example train_mamba2_dbn --release
# Output: ml/checkpoints/mamba2_dbn/best_model.safetensors

---

3.2 DQN Training

File: /home/jgrusewski/Work/foxhunt/ml/examples/train_dqn.rs
Status: Production Ready (Wave 15+)
Feature Count: 26 (from real-time extractor)

Data Loading (lines 126-):

let mut trainer = DQNTrainer::new(hyperparams)?;
// Trainer internally loads DBN data and extracts 26 features

---

3.3 PPO Training

File: /home/jgrusewski/Work/foxhunt/ml/examples/train_ppo.rs
Status: Production Ready (Wave 15+)
Feature Count: ~16 (OHLCV + 10 indicators)

Data Loading (lines 126-139):

let mut loader = RealDataLoader::new(&opts.data_dir);
let bars = loader.load_symbol_data(&opts.symbol).await?;
let features = loader.extract_features(&bars)?;
let indicators = loader.calculate_indicators(&bars)?;

---

3.4 TFT Training

File: /home/jgrusewski/Work/foxhunt/ml/examples/train_tft_dbn.rs
Status: Production Ready (Wave 15+)
Feature Count: Variable (loaded via TFTDataLoader)

Data Loading (lines 131-147):

let path = std::path::Path::new(&opts.data_path);
if path.is_dir() {
    // Load all .dbn files from directory
    for entry in std::fs::read_dir(path)? {
        let file_bars = load_dbn_ohlcv_bars(file_path.to_str().unwrap()).await?;
        all_bars.extend(file_bars);
    }
}

---

4. Feature Extraction Code Locations

4.1 Real-Time (26 features)

• Location: /home/jgrusewski/Work/foxhunt/common/src/ml_strategy.rs
• Class: MLFeatureExtractor
• Method: extract_features(price, volume, timestamp) -> Vec<f64>
• Lines: 170-897
• State Management: Maintains rolling windows for technical indicators
• Usage: SharedMLStrategy.get_ensemble_prediction() (line 1064)

4.2 Training (256 features - CURRENT)

• Location: /home/jgrusewski/Work/foxhunt/ml/src/data_loaders/dbn_sequence_loader.rs
• Method: extract_features(msg: ProcessedMessage) -> Vec<f32>
• Lines: 664-804
• Approach: Padding-based (NOT actual feature engineering)
• Usage: create_sequences() (line 556)

4.3 Feature Extraction Module (RECOMMENDED FOR WAVE C)

• Location: /home/jgrusewski/Work/foxhunt/ml/src/features/extraction.rs
• Status: Partially implemented (256-dimension output)
• Lines: 1-150+
• Function: extract_ml_features(bars: &[OHLCVBar]) -> Result<Vec<FeatureVector>>
• NOT YET USED in training pipeline

---

5. Integration Requirements for Wave C (65+ Features)

5.1 Current Architecture Issues

1. Disconnected Systems:

   • Inference uses real feature extraction (26 features) ✓
   • Training uses padding-based features (256 features) ✗
   • MAMBA-2 expects d_model=256 (arbitrary dimension)

2. Feature Hardcoding:

   • DbnSequenceLoader hardcodes d_model=256 (line 292, train_mamba2_dbn.rs)
   • SimpleDQNAdapter hardcodes 26 features (line 966, ml_strategy.rs)
   • No configuration system for feature count

3. Wave C Incompatibility:

   • Fractional differentiation features not extracted
   • Meta-labeling features not available
   • Structural break detection features missing

5.2 Required Changes for Wave C Integration

Step 1: Create Wave C Feature Extractor

New File: ml/src/features/wave_c_extractor.rs

pub struct WaveCFeatureExtractor {
    /// Original 26 inference features
    base_features: MLFeatureExtractor,
    
    /// Wave B features (10)
    alternative_bars: AlternativeBarSampler,
    barrier_optimizer: BarrierOptimizer,
    
    /// Wave C features (19)
    fractional_diff: FractionalDifferentiator,
    meta_labeler: MetaLabelingEngine,
    struct_break_detector: StructuralBreakDetector,
}

// Total: 26 + 10 + 19 = 55+ features
pub fn extract_wave_c_features(
    bars: &[OHLCVBar],
    config: FeatureExtractionConfig,
) -> Result<Vec<FeatureVector>> {
    // Returns [n_bars, feature_count] where feature_count = 55+
}

Step 2: Update DbnSequenceLoader

File: ml/src/data_loaders/dbn_sequence_loader.rs

pub async fn new_wave_c(
    seq_len: usize,
    wave_c_config: WaveCConfig,  // NEW
) -> Result<Self> {
    let feature_count = wave_c_config.compute_total_features();  // 65+
    
    // OLD: DbnSequenceLoader::new(seq_len, 256)
    // NEW: DbnSequenceLoader::new_wave_c(seq_len, config)
}

Step 3: Configuration System

New File: ml/src/config/feature_config.rs

pub struct FeatureConfig {
    pub wave_level: WaveLevel,  // Wave19A, WaveB, WaveC
    pub include_base_26: bool,
    pub include_alternative_bars: bool,
    pub include_fractional_diff: bool,
    pub include_meta_labels: bool,
}

impl FeatureConfig {
    pub fn total_features(&self) -> usize {
        match self.wave_level {
            WaveLevel::WaveA => 26,
            WaveLevel::WaveB => 26 + 10,  // 36
            WaveLevel::WaveC => 26 + 10 + 19 + more_features,  // 65+
        }
    }
}

Step 4: Update Model Adapters

File: common/src/ml_strategy.rs

// OLD:
pub struct SimpleDQNAdapter {
    weights: Vec<f64>,  // Fixed 26
}

// NEW:
pub struct SimpleDQNAdapter {
    weights: Vec<f64>,  // Dynamic size based on feature_config
    feature_config: FeatureConfig,
}

Step 5: Update Training Scripts

Files: All ml/examples/train_*.rs

// OLD:
let mut loader = DbnSequenceLoader::new(seq_len, 256).await?;

// NEW:
let feature_config = FeatureConfig {
    wave_level: WaveLevel::WaveC,
    include_base_26: true,
    include_alternative_bars: true,
    include_fractional_diff: true,
    include_meta_labels: true,
};
let mut loader = DbnSequenceLoader::new_with_config(seq_len, feature_config).await?;

---

6. Current Feature Extraction Points

Usage in Inference Pipeline

1. File: common/src/ml_strategy.rs
2. Function: SharedMLStrategy::get_ensemble_prediction()
3. Line: 1058-1088
4. Input: price, volume, timestamp
5. Output: 26-dimensional feature vector

Usage in Training (MAMBA-2)

1. File: ml/examples/train_mamba2_dbn.rs
2. Data Source: DBN files via DbnSequenceLoader
3. Feature Count: 256 (via padding)
4. Output: Tensors [1, 60, 256]

Usage in Training (DQN)

1. File: ml/examples/train_dqn.rs
2. Data Source: DQNTrainer (internal)
3. Feature Count: 26 (inferred)
4. Output: Training samples

---

7. Alternative Bars Integration Status

Current Implementation (Wave B Partial)

File: ml/src/features/alternative_bars.rs

Available Samplers:

• ✅ TickBarSampler
• ✅ VolumeBarSampler
• ✅ DollarBarSampler
• ✅ ImbalanceBarSampler
• ✅ RunBarSampler

Status: Code exists but NOT integrated into training pipeline.

Integration Point Required:

// In DbnSequenceLoader or new Wave C extractor:
let alt_bars = DollarBarSampler::new(1_000_000);  // $1M bars
let resampled = alt_bars.sample(&messages)?;
let features = extract_features(&resampled)?;

---

8. Data Loading Performance

Metrics (DBN Data)

Operation	Time	Target	Status
Load 1,674 bars (ES.FUT)	0.70ms	<10ms	✅ 14.3x better
Compute statistics	0.2ms	<1ms	✅ 5x better
Create sequences (60-len)	2ms	<5ms	✅ 2.5x better
Extract 256 features/bar	0.002ms	<1μs	✅ 500x better
Total pipeline	<5ms	<100ms	✅ 20x better

---

9. Model Input Dimensions Summary

Model	Input Shape	Feature Count	Sequence Length	Notes
DQN (SimpleDQNAdapter)	[26]	26	1	Current inference, Wave A
PPO	~16 per timestep	~16	1	Using RealDataLoader extractor
MAMBA-2	[1, seq_len, d_model]	256	60	Padding-based, needs update
TFT	[1, lookback, features]	Variable	60	Forecast horizon: 10
Wave C Target	[1, seq_len, 65+]	65+	60	With alt bars + meta-labels

---

10. Recommended Action Plan for Wave C

Phase 1: Feature Extraction (Week 1)

1. Implement WaveCFeatureExtractor in ml/src/features/wave_c_extractor.rs
2. Add alternative bars sampling to pipeline
3. Implement fractional differentiation
4. Create meta-labeling features

Phase 2: Configuration (Week 1-2)

1. Create FeatureConfig system
2. Make feature count dynamic across all adapters
3. Update all training scripts to use new config

Phase 3: Integration (Week 2)

1. Update DbnSequenceLoader for variable feature count
2. Migrate training from 256 (padding) to 65+ (real features)
3. Update MAMBA-2, DQN, PPO, TFT input dimensions

Phase 4: Validation (Week 2-3)

1. Verify features are non-zero (not just padding)
2. Test all 4 training scripts with new features
3. Benchmark: latency, memory, training speed
4. Backtest: win rate, Sharpe improvement

---

Conclusion

The current system has a 26-feature inference pipeline and a 256-feature training pipeline (mostly padding). For Wave C integration:

1. Create unified feature extraction system (65+ features)
2. Update configuration to support dynamic feature counts
3. Integrate alternative bars and meta-labeling into training
4. Validate that real features improve model performance

The infrastructure for feature computation exists (alternative_bars.rs, extraction.rs, unified.rs), but it's not wired into the training pipeline. This represents the primary integration gap for Wave C.

---

Files to Modify:

• ml/src/data_loaders/dbn_sequence_loader.rs (add feature config support)
• ml/src/features/extraction.rs (enhance with Wave B/C features)
• ml/examples/train_*.rs (all 4 scripts - update d_model or add config)
• common/src/ml_strategy.rs (SimpleDQNAdapter - make feature count dynamic)

New Files to Create:

• ml/src/features/wave_c_extractor.rs (comprehensive feature builder)
• ml/src/config/feature_config.rs (feature configuration system)

Estimated Impact:

• +65 features for training models
• +20-35% Sharpe improvement (Wave C target)
• <500ms additional latency per training batch