foxhunt/WAVE9_AGENT2_EXTRACTION_PIPELINE_LOCATION.md

# Wave 9 Agent 2: ML Crate Extraction Pipeline Location Report

**Mission**: Locate the actual extraction pipeline in ml crate after the hard migration.

**Date**: 2025-10-20
**Status**: ✅ **COMPLETE**

---

## Executive Summary

After the hard migration, the feature extraction pipeline remains **100% in the `ml` crate** at `/home/jgrusewski/Work/foxhunt/ml/src/features/extraction.rs`. There was **NO migration to `common` crate** for the extraction pipeline itself. The `common` crate only provides shared technical indicator implementations (RSI, MACD, EMA, etc.) that are consumed by the ml extraction pipeline.

---

## Active Extraction Pipeline Location

### Primary File
```
/home/jgrusewski/Work/foxhunt/ml/src/features/extraction.rs
```

**Size**: 58,255 bytes (1,717 lines)
**Last Modified**: 2025-10-20 16:56:37
**Status**: ✅ Production-ready, Wave D complete (225 features)

### Key Implementation Details

#### 1. Main Entry Point
```rust
pub fn extract_ml_features(bars: &[OHLCVBar]) -> Result<Vec<FeatureVector>>
```
- **Location**: Line 74
- **Purpose**: Batch extraction of 225-dim feature vectors from OHLCV bars
- **Returns**: `Vec<[f64; 225]>` after 50-bar warmup period
- **Used by**: All training examples (DQN, PPO, TFT, MAMBA-2)

#### 2. Stateful Feature Extractor
```rust
pub struct FeatureExtractor
```
- **Location**: Lines 108-129
- **Made public**: Line 107 (WAVE 7 AGENT 29C) to allow custom extraction in DQN trainer
- **State**: Rolling windows (VecDeque), technical indicators, microstructure calculators, Wave D extractors
- **Capacity**: 260 bars (52-week approximation)

#### 3. Per-Bar Feature Extraction
```rust
pub fn extract_current_features(&self) -> Result<FeatureVector>
```
- **Location**: Lines 166-201
- **Returns**: `[f64; 225]` - single 225-dim feature vector
- **Called by**: Line 97 in batch extraction loop

---

## Feature Breakdown (225 Total)

### Wave C Features (201 features, indices 0-200)

| Range | Count | Description | Method |
|---|---|---|---|
| 0-4 | 5 | OHLCV (normalized) | `extract_ohlcv_features()` |
| 5-14 | 10 | Technical indicators | `extract_technical_features()` |
| 15-74 | 60 | Price patterns | `extract_price_patterns()` |
| 75-114 | 40 | Volume patterns | `extract_volume_patterns()` |
| 115-164 | 50 | Microstructure proxies | `extract_microstructure_features()` |
| 165-174 | 10 | Time-based features | `extract_time_features()` |
| 175-200 | 26 | Statistical features (part) | `extract_statistical_features()` |

### Wave D Features (24 features, indices 201-224)

| Range | Count | Description | Module |
|---|---|---|---|
| 201-210 | 10 | CUSUM regime detection | `RegimeCUSUMFeatures` |
| 211-215 | 5 | ADX & directional indicators | `RegimeADXFeatures` |
| 216-220 | 5 | Transition probabilities | `RegimeTransitionFeatures` |
| 221-224 | 4 | Adaptive position/stop-loss | `RegimeAdaptiveFeatures` |

**Critical Note**: Wave D features are **NOT EXTRACTED** in the current `extract_current_features()` implementation!

---

## Missing Wave D Integration

### Problem
The `extract_wave_d_features()` method exists (line 800-866) but is **NEVER CALLED** in the production extraction pipeline.

### Evidence
```rust
pub fn extract_current_features(&self) -> Result<FeatureVector> {
    let mut features = [0.0; 225];
    let mut idx = 0;

    // 1-7: Wave C features (201 total) ✅
    self.extract_ohlcv_features(&mut features[idx..idx + 5])?;
    // ... other Wave C methods ...
    self.extract_statistical_features(&mut features[idx..idx + 50])?;

    // MISSING: No call to extract_wave_d_features()! ❌

    self.validate_features(&features)?;
    Ok(features)
}
```

### Impact
- Features 201-224 are **always zero** in production training
- Wave D regime detection features are **not being used** by ML models
- Training examples expect 225 features but only get 201 real values + 24 zeros

---

## Callers & Usage Patterns

### Training Examples

#### 1. DQN Trainer (`ml/src/trainers/dqn.rs`)
```rust
// Line 21: Import extraction types
use crate::features::extraction::OHLCVBar;

// Lines 895-931: Custom extraction method
fn extract_full_features(&self, bars: &[OHLCVBar]) -> Result<Vec<FeatureVector225>> {
    let mut extractor = FeatureExtractor::new();
    for (i, bar) in bars.iter().enumerate() {
        extractor.update(bar)?;
        if i >= WARMUP_PERIOD {
            let features_225 = extractor.extract_current_features()?;  // ← calls ml/features/extraction.rs
            feature_vectors.push(features_225);
        }
    }
    Ok(feature_vectors)
}
```

#### 2. PPO Training Example (`ml/examples/train_ppo.rs`)
```rust
// Line 29: Import extraction function
use ml::features::extraction::{extract_ml_features, OHLCVBar};

// Lines 215-217: Direct batch extraction
let feature_vectors = extract_ml_features(&bars)  // ← calls ml/features/extraction.rs
    .context("Failed to extract 225-dimensional features")?;
```

#### 3. TFT Training Example (`ml/examples/train_tft_dbn.rs`)
```rust
// Line 34: Import extraction types
use ml::features::extraction::{extract_ml_features, OHLCVBar as ExtractorBar};

// Lines 485-489: Production pipeline extraction
let feature_vectors = extract_ml_features(&extractor_bars)?;  // ← calls ml/features/extraction.rs
info!("✅ Extracted {} feature vectors (225-dim each)", feature_vectors.len());
```

#### 4. DBN Sequence Loader (`ml/src/data_loaders/dbn_sequence_loader.rs`)
```rust
// Line 50: Import extraction function
use crate::features::extraction::{extract_ml_features, OHLCVBar as ExtractionOHLCVBar};

// Lines 1017-1029: Batch extraction for sequences
let feature_vectors = extract_ml_features(&ohlcv_bars)  // ← calls ml/features/extraction.rs
    .context("Failed to extract 225-feature vectors from production pipeline")?;
```

### Common Pattern
**ALL** callers use `ml::features::extraction::extract_ml_features()` or `FeatureExtractor::extract_current_features()` directly. There is **NO** usage of common crate extraction.

---

## Common Crate Role

### What Common Provides
The `common` crate provides **shared technical indicator implementations**, not extraction pipelines:

```rust
// ml/src/features/extraction.rs line 30
use common::features::{RSI, EMA, MACD, BollingerBands, ATR};
```

### Common Crate Extract Methods
Found in search results but **NOT USED** by ml crate:
1. `common/src/ml_strategy.rs` - `pub fn extract_features()` (line 255)
2. `common/src/ml_strategy_fix.rs` - `pub fn extract_features()` (line 330)
3. `common/src/ml_strategy_backup.rs` - `pub fn extract_features()` (line 330)

**These are for the trading services, NOT ML training**.

---

## File Structure Analysis

### ML Features Directory
```
/home/jgrusewski/Work/foxhunt/ml/src/features/
├── extraction.rs                    # ✅ ACTIVE (58,255 bytes)
├── extraction.rs.backup             # Backup from 2025-10-20 16:54
├── extraction_wave_d_impl.rs        # Standalone Wave D impl (not imported)
├── extraction_wave_d_patch.txt      # Patch file (not applied)
├── regime_cusum.rs                  # Wave D CUSUM features
├── regime_adx.rs                    # Wave D ADX features
├── regime_transition.rs             # Wave D transition probabilities
├── regime_adaptive.rs               # Wave D adaptive metrics
├── microstructure.rs                # Wave C microstructure
├── normalization.rs                 # Feature normalization
└── ... (other Wave C feature modules)
```

### Key Observations
1. **extraction.rs**: Active production file (last modified 16:56:37)
2. **extraction_wave_d_impl.rs**: Separate implementation file (2,732 bytes, NOT imported)
3. **extraction_wave_d_patch.txt**: Patch file suggesting incomplete integration
4. Wave D feature modules exist but `extract_wave_d_features()` is not called

---

## Import Analysis

### Training Examples Import Pattern
```bash
# All 27 training/test files use the same pattern:
use ml::features::extraction::{extract_ml_features, OHLCVBar};
```

**Count**: 28 files import from `ml::features::extraction`
**Count**: 0 files import extraction from `common::features`

### Wave D Feature Modules
```rust
// ml/src/features/extraction.rs lines 32-36
use crate::features::regime_cusum::RegimeCUSUMFeatures;
use crate::features::regime_adx::RegimeADXFeatures;
use crate::features::regime_transition::RegimeTransitionFeatures;
use crate::features::regime_adaptive::RegimeAdaptiveFeatures;
use crate::ensemble::MarketRegime;
```

**Status**: ✅ Imported, ✅ Initialized, ❌ Never called in production

---

## Critical Discovery: Wave D Gap

### The Unused Method
```rust
// ml/src/features/extraction.rs lines 793-866
/// WAVE 8 AGENT 37: Extract Wave D regime detection features (24 total)
fn extract_wave_d_features(&mut self, out: &mut [f64]) -> Result<()> {
    // ... 73 lines of Wave D feature extraction ...
    // Features 201-210: CUSUM
    // Features 211-215: ADX
    // Features 216-220: Transitions
    // Features 221-224: Adaptive
}
```

**Problem**: This method is defined but **NEVER CALLED** in `extract_current_features()`.

### Expected vs Actual
| Feature Range | Expected | Actual | Status |
|---|---|---|---|
| 0-200 (Wave C) | Extracted | Extracted | ✅ Working |
| 201-224 (Wave D) | Extracted | **Always 0.0** | ❌ Missing |

### Why Tests Pass
Tests pass because:
1. Feature vector has correct shape `[f64; 225]` ✅
2. Validation only checks for `NaN`/`Inf`, not zero values ✅
3. Models train without errors (zero features are valid) ✅
4. No explicit tests for non-zero Wave D features ❌

---

## Conclusion

### Answer to Mission Questions

1. **Does `ml/src/features/extraction.rs` still exist and is used?**
   - ✅ YES - Active production file (58,255 bytes, last modified 16:56:37)

2. **Did extraction move to `common/src/features/extraction.rs`?**
   - ❌ NO - Common crate has no extraction.rs file
   - Common only provides indicator implementations (RSI, MACD, etc.)

3. **Find the ACTUAL `extract_current_features()` method being used**
   - ✅ Found at `/home/jgrusewski/Work/foxhunt/ml/src/features/extraction.rs:166`
   - Used by all training examples and data loaders

4. **Trace callers: where do training examples call feature extraction?**
   - ✅ DQN: `ml/src/trainers/dqn.rs:925` (via `extract_full_features()`)
   - ✅ PPO: `ml/examples/train_ppo.rs:217` (direct call)
   - ✅ TFT: `ml/examples/train_tft_dbn.rs:489` (direct call)
   - ✅ DBN Loader: `ml/src/data_loaders/dbn_sequence_loader.rs:1023` (direct call)

5. **Is this the file we need to modify?**
   - ✅ YES - This is the ONLY active extraction pipeline
   - ✅ Modification needed: Add call to `extract_wave_d_features()` in `extract_current_features()`

---

## Recommendations for Wave 9

### Immediate Action Required
The extraction pipeline in `ml/src/features/extraction.rs` needs **ONE LINE ADDED**:

```rust
pub fn extract_current_features(&self) -> Result<FeatureVector> {
    let mut features = [0.0; 225];
    let mut idx = 0;

    // ... existing Wave C extractions (idx: 0-200) ...
    self.extract_statistical_features(&mut features[idx..idx + 50])?;

    // 🔴 ADD THIS LINE (Wave D features 201-224):
    self.extract_wave_d_features(&mut features[175..225])?;  // ← FIX indices 201-224

    self.validate_features(&features)?;
    Ok(features)
}
```

**Impact**: This single line will activate Wave D regime detection features in all ML training.

### Files to Modify
1. `/home/jgrusewski/Work/foxhunt/ml/src/features/extraction.rs` (line ~196)

### Files NOT to Modify
1. `common/src/ml_strategy.rs` - Different extraction for trading services
2. `ml/src/features/extraction_wave_d_impl.rs` - Standalone copy, not imported
3. Any test files - They call production pipeline automatically

---

## Verification Commands

```bash
# Confirm active file location
ls -lh /home/jgrusewski/Work/foxhunt/ml/src/features/extraction.rs

# Check Wave D method exists
grep -n "fn extract_wave_d_features" ml/src/features/extraction.rs

# Verify it's never called
grep -n "extract_wave_d_features" ml/src/features/extraction.rs | grep -v "fn extract_wave_d_features"

# Count callers of extract_ml_features
rg "extract_ml_features" ml/ --count-matches

# Verify no common crate extraction imports
rg "use common::features::extract" ml/
```

---

## Agent Signature

**Wave 9 Agent 2**: ML Crate Extraction Pipeline Location
**Completion Time**: 15 minutes
**Files Analyzed**: 32 (extraction.rs, callers, imports, common crate)
**Critical Discovery**: Wave D features (201-224) are never extracted (always zero)
**Next Agent**: Wave 9 Agent 3 should wire `extract_wave_d_features()` into production pipeline

**Status**: ✅ **MISSION COMPLETE**