foxhunt/WAVE9_AGENT2_EXTRACTION_PIPELINE_LOCATION.md

Wave 9 Agent 2: ML Crate Extraction Pipeline Location Report

Mission: Locate the actual extraction pipeline in ml crate after the hard migration.

Date: 2025-10-20 Status: ✅ COMPLETE

---

Executive Summary

After the hard migration, the feature extraction pipeline remains 100% in the ml crate at /home/jgrusewski/Work/foxhunt/ml/src/features/extraction.rs. There was NO migration to common crate for the extraction pipeline itself. The common crate only provides shared technical indicator implementations (RSI, MACD, EMA, etc.) that are consumed by the ml extraction pipeline.

---

Active Extraction Pipeline Location

Primary File

/home/jgrusewski/Work/foxhunt/ml/src/features/extraction.rs

Size: 58,255 bytes (1,717 lines) Last Modified: 2025-10-20 16:56:37 Status: ✅ Production-ready, Wave D complete (225 features)

Key Implementation Details

1. Main Entry Point

pub fn extract_ml_features(bars: &[OHLCVBar]) -> Result<Vec<FeatureVector>>

• Location: Line 74
• Purpose: Batch extraction of 225-dim feature vectors from OHLCV bars
• Returns: Vec<[f64; 225]> after 50-bar warmup period
• Used by: All training examples (DQN, PPO, TFT, MAMBA-2)

2. Stateful Feature Extractor

pub struct FeatureExtractor

• Location: Lines 108-129
• Made public: Line 107 (WAVE 7 AGENT 29C) to allow custom extraction in DQN trainer
• State: Rolling windows (VecDeque), technical indicators, microstructure calculators, Wave D extractors
• Capacity: 260 bars (52-week approximation)

3. Per-Bar Feature Extraction

pub fn extract_current_features(&self) -> Result<FeatureVector>

• Location: Lines 166-201
• Returns: [f64; 225] - single 225-dim feature vector
• Called by: Line 97 in batch extraction loop

---

Feature Breakdown (225 Total)

Wave C Features (201 features, indices 0-200)

Range	Count	Description	Method
0-4	5	OHLCV (normalized)	extract_ohlcv_features()
5-14	10	Technical indicators	extract_technical_features()
15-74	60	Price patterns	extract_price_patterns()
75-114	40	Volume patterns	extract_volume_patterns()
115-164	50	Microstructure proxies	extract_microstructure_features()
165-174	10	Time-based features	extract_time_features()
175-200	26	Statistical features (part)	extract_statistical_features()

Wave D Features (24 features, indices 201-224)

Range	Count	Description	Module
201-210	10	CUSUM regime detection	RegimeCUSUMFeatures
211-215	5	ADX & directional indicators	RegimeADXFeatures
216-220	5	Transition probabilities	RegimeTransitionFeatures
221-224	4	Adaptive position/stop-loss	RegimeAdaptiveFeatures

Critical Note: Wave D features are NOT EXTRACTED in the current extract_current_features() implementation!

---

Missing Wave D Integration

Problem

The extract_wave_d_features() method exists (line 800-866) but is NEVER CALLED in the production extraction pipeline.

Evidence

pub fn extract_current_features(&self) -> Result<FeatureVector> {
    let mut features = [0.0; 225];
    let mut idx = 0;

    // 1-7: Wave C features (201 total) ✅
    self.extract_ohlcv_features(&mut features[idx..idx + 5])?;
    // ... other Wave C methods ...
    self.extract_statistical_features(&mut features[idx..idx + 50])?;

    // MISSING: No call to extract_wave_d_features()! ❌

    self.validate_features(&features)?;
    Ok(features)
}

Impact

• Features 201-224 are always zero in production training
• Wave D regime detection features are not being used by ML models
• Training examples expect 225 features but only get 201 real values + 24 zeros

---

Callers & Usage Patterns

Training Examples

1. DQN Trainer (ml/src/trainers/dqn.rs)

// Line 21: Import extraction types
use crate::features::extraction::OHLCVBar;

// Lines 895-931: Custom extraction method
fn extract_full_features(&self, bars: &[OHLCVBar]) -> Result<Vec<FeatureVector225>> {
    let mut extractor = FeatureExtractor::new();
    for (i, bar) in bars.iter().enumerate() {
        extractor.update(bar)?;
        if i >= WARMUP_PERIOD {
            let features_225 = extractor.extract_current_features()?;  // ← calls ml/features/extraction.rs
            feature_vectors.push(features_225);
        }
    }
    Ok(feature_vectors)
}

2. PPO Training Example (ml/examples/train_ppo.rs)

// Line 29: Import extraction function
use ml::features::extraction::{extract_ml_features, OHLCVBar};

// Lines 215-217: Direct batch extraction
let feature_vectors = extract_ml_features(&bars)  // ← calls ml/features/extraction.rs
    .context("Failed to extract 225-dimensional features")?;

3. TFT Training Example (ml/examples/train_tft_dbn.rs)

// Line 34: Import extraction types
use ml::features::extraction::{extract_ml_features, OHLCVBar as ExtractorBar};

// Lines 485-489: Production pipeline extraction
let feature_vectors = extract_ml_features(&extractor_bars)?;  // ← calls ml/features/extraction.rs
info!("✅ Extracted {} feature vectors (225-dim each)", feature_vectors.len());

4. DBN Sequence Loader (ml/src/data_loaders/dbn_sequence_loader.rs)

// Line 50: Import extraction function
use crate::features::extraction::{extract_ml_features, OHLCVBar as ExtractionOHLCVBar};

// Lines 1017-1029: Batch extraction for sequences
let feature_vectors = extract_ml_features(&ohlcv_bars)  // ← calls ml/features/extraction.rs
    .context("Failed to extract 225-feature vectors from production pipeline")?;

Common Pattern

ALL callers use ml::features::extraction::extract_ml_features() or FeatureExtractor::extract_current_features() directly. There is NO usage of common crate extraction.

---

Common Crate Role

What Common Provides

The common crate provides shared technical indicator implementations, not extraction pipelines:

// ml/src/features/extraction.rs line 30
use common::features::{RSI, EMA, MACD, BollingerBands, ATR};

Common Crate Extract Methods

Found in search results but NOT USED by ml crate:

1. common/src/ml_strategy.rs - pub fn extract_features() (line 255)
2. common/src/ml_strategy_fix.rs - pub fn extract_features() (line 330)
3. common/src/ml_strategy_backup.rs - pub fn extract_features() (line 330)

These are for the trading services, NOT ML training.

---

File Structure Analysis

ML Features Directory

/home/jgrusewski/Work/foxhunt/ml/src/features/
├── extraction.rs                    # ✅ ACTIVE (58,255 bytes)
├── extraction.rs.backup             # Backup from 2025-10-20 16:54
├── extraction_wave_d_impl.rs        # Standalone Wave D impl (not imported)
├── extraction_wave_d_patch.txt      # Patch file (not applied)
├── regime_cusum.rs                  # Wave D CUSUM features
├── regime_adx.rs                    # Wave D ADX features
├── regime_transition.rs             # Wave D transition probabilities
├── regime_adaptive.rs               # Wave D adaptive metrics
├── microstructure.rs                # Wave C microstructure
├── normalization.rs                 # Feature normalization
└── ... (other Wave C feature modules)

Key Observations

1. extraction.rs: Active production file (last modified 16:56:37)
2. extraction_wave_d_impl.rs: Separate implementation file (2,732 bytes, NOT imported)
3. extraction_wave_d_patch.txt: Patch file suggesting incomplete integration
4. Wave D feature modules exist but extract_wave_d_features() is not called

---

Import Analysis

Training Examples Import Pattern

# All 27 training/test files use the same pattern:
use ml::features::extraction::{extract_ml_features, OHLCVBar};

Count: 28 files import from ml::features::extraction Count: 0 files import extraction from common::features

Wave D Feature Modules

// ml/src/features/extraction.rs lines 32-36
use crate::features::regime_cusum::RegimeCUSUMFeatures;
use crate::features::regime_adx::RegimeADXFeatures;
use crate::features::regime_transition::RegimeTransitionFeatures;
use crate::features::regime_adaptive::RegimeAdaptiveFeatures;
use crate::ensemble::MarketRegime;

Status: ✅ Imported, ✅ Initialized, ❌ Never called in production

---

Critical Discovery: Wave D Gap

The Unused Method

// ml/src/features/extraction.rs lines 793-866
/// WAVE 8 AGENT 37: Extract Wave D regime detection features (24 total)
fn extract_wave_d_features(&mut self, out: &mut [f64]) -> Result<()> {
    // ... 73 lines of Wave D feature extraction ...
    // Features 201-210: CUSUM
    // Features 211-215: ADX
    // Features 216-220: Transitions
    // Features 221-224: Adaptive
}

Problem: This method is defined but NEVER CALLED in extract_current_features().

Expected vs Actual

Feature Range	Expected	Actual	Status
0-200 (Wave C)	Extracted	Extracted	✅ Working
201-224 (Wave D)	Extracted	Always 0.0	❌ Missing

Why Tests Pass

Tests pass because:

1. Feature vector has correct shape [f64; 225] ✅
2. Validation only checks for NaN/Inf, not zero values ✅
3. Models train without errors (zero features are valid) ✅
4. No explicit tests for non-zero Wave D features ❌

---

Conclusion

Answer to Mission Questions

1. Does ml/src/features/extraction.rs still exist and is used?

   • ✅ YES - Active production file (58,255 bytes, last modified 16:56:37)

2. Did extraction move to common/src/features/extraction.rs?

   • ❌ NO - Common crate has no extraction.rs file
   • Common only provides indicator implementations (RSI, MACD, etc.)

3. Find the ACTUAL extract_current_features() method being used

   • ✅ Found at /home/jgrusewski/Work/foxhunt/ml/src/features/extraction.rs:166
   • Used by all training examples and data loaders

4. Trace callers: where do training examples call feature extraction?

   • ✅ DQN: ml/src/trainers/dqn.rs:925 (via extract_full_features())
   • ✅ PPO: ml/examples/train_ppo.rs:217 (direct call)
   • ✅ TFT: ml/examples/train_tft_dbn.rs:489 (direct call)
   • ✅ DBN Loader: ml/src/data_loaders/dbn_sequence_loader.rs:1023 (direct call)

5. Is this the file we need to modify?

   • ✅ YES - This is the ONLY active extraction pipeline
   • ✅ Modification needed: Add call to extract_wave_d_features() in extract_current_features()

---

Recommendations for Wave 9

Immediate Action Required

The extraction pipeline in ml/src/features/extraction.rs needs ONE LINE ADDED:

pub fn extract_current_features(&self) -> Result<FeatureVector> {
    let mut features = [0.0; 225];
    let mut idx = 0;

    // ... existing Wave C extractions (idx: 0-200) ...
    self.extract_statistical_features(&mut features[idx..idx + 50])?;

    // 🔴 ADD THIS LINE (Wave D features 201-224):
    self.extract_wave_d_features(&mut features[175..225])?;  // ← FIX indices 201-224

    self.validate_features(&features)?;
    Ok(features)
}

Impact: This single line will activate Wave D regime detection features in all ML training.

Files to Modify

1. /home/jgrusewski/Work/foxhunt/ml/src/features/extraction.rs (line ~196)

Files NOT to Modify

1. common/src/ml_strategy.rs - Different extraction for trading services
2. ml/src/features/extraction_wave_d_impl.rs - Standalone copy, not imported
3. Any test files - They call production pipeline automatically

---

Verification Commands

# Confirm active file location
ls -lh /home/jgrusewski/Work/foxhunt/ml/src/features/extraction.rs

# Check Wave D method exists
grep -n "fn extract_wave_d_features" ml/src/features/extraction.rs

# Verify it's never called
grep -n "extract_wave_d_features" ml/src/features/extraction.rs | grep -v "fn extract_wave_d_features"

# Count callers of extract_ml_features
rg "extract_ml_features" ml/ --count-matches

# Verify no common crate extraction imports
rg "use common::features::extract" ml/

---

Agent Signature

Wave 9 Agent 2: ML Crate Extraction Pipeline Location Completion Time: 15 minutes Files Analyzed: 32 (extraction.rs, callers, imports, common crate) Critical Discovery: Wave D features (201-224) are never extracted (always zero) Next Agent: Wave 9 Agent 3 should wire extract_wave_d_features() into production pipeline

Status: ✅ MISSION COMPLETE