# Feature Extraction Pipeline Fix

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Fix 6 broken features, eliminate multi-symbol mixing, and ensure all 42 features are properly normalized and carry meaningful signal for the DQN model.

**Architecture:** Two independent fixes: (1) symbol filtering in precompute to ensure single-instrument data, (2) feature extraction fixes to normalize raw-price indicators into dimensionless values. Both must be done — symbol filtering alone doesn't fix the Bollinger/MACD/autocorrelation bugs, and feature fixes alone don't fix cross-symbol contamination.

**Tech Stack:** Rust, `crates/ml/src/features/extraction.rs` (main extraction), `crates/common/src/features/technical_indicators.rs` (MACD, Bollinger, ATR), `crates/ml/examples/precompute_features.rs` (precompute entry point)

---

## Bug Inventory

| Index | Feature | Bug | Fix |
|-------|---------|-----|-----|
| 6 | MACD histogram | Raw dollars (±5–100), clipped to ±3 → near-binary | Normalize: `histogram / atr` (dimensionless) |
| 7 | Bollinger upper | Raw price (~20000), clipped to +3.0 → dead constant | Change to: `(close - upper) / (upper - lower)` (dimensionless distance) |
| 8 | Bollinger lower | Raw price (~20000), clipped to +3.0 → dead constant | Change to: `(close - lower) / (upper - lower)` (dimensionless distance) |
| 15 | LR slope(20) | Raw dollars/bar, saturated for ES/NQ | Normalize: `slope / atr` or `slope / close` (dimensionless) |
| 31-33 | Autocorr lag-1,5,10 | Computed on price levels → always ~1.0 | Compute on log returns instead of raw prices |
| ALL | Multi-symbol mix | `collect_dbn_files_recursive` loads ES+NQ+ZN+6E | Filter by `--symbol`, pass only matching files |

## File Map

| File | Changes |
|------|---------|
| `crates/ml/src/features/extraction.rs` | Fix features 6, 7, 8, 15, 31-33 |
| `crates/ml/examples/precompute_features.rs` | Filter DBN files by `--symbol`, validate feature ranges |
| `crates/ml/src/trainers/dqn/data_loading.rs` | `collect_dbn_files_recursive` → accept symbol filter |
| `crates/ml/src/features/extraction.rs` (tests) | Add range validation tests for all 42 features |

---

### Task 1: Symbol Filtering in Precompute

**Files:**
- Modify: `crates/ml/examples/precompute_features.rs:214-224`
- Modify: `crates/ml/src/trainers/dqn/data_loading.rs` (`collect_dbn_files_recursive`)

- [ ] **Step 1: Write failing test — symbol filter**

In `crates/ml/src/trainers/dqn/data_loading.rs`, add a test:

```rust
#[test]
fn test_collect_dbn_files_filters_by_symbol() {
    use std::path::PathBuf;
    use tempfile::TempDir;
    
    let dir = TempDir::new().unwrap();
    let es_dir = dir.path().join("ES.FUT");
    let nq_dir = dir.path().join("NQ.FUT");
    std::fs::create_dir_all(&es_dir).unwrap();
    std::fs::create_dir_all(&nq_dir).unwrap();
    std::fs::write(es_dir.join("ES_Q1.dbn.zst"), b"").unwrap();
    std::fs::write(nq_dir.join("NQ_Q1.dbn.zst"), b"").unwrap();
    
    let all = collect_dbn_files_recursive(dir.path());
    assert_eq!(all.len(), 2);
    
    let filtered = collect_dbn_files_filtered(dir.path(), Some("ES.FUT"));
    assert_eq!(filtered.len(), 1);
    assert!(filtered[0].to_str().unwrap().contains("ES.FUT"));
}
```

- [ ] **Step 2: Run test to verify it fails**

Run: `SQLX_OFFLINE=true cargo test -p ml --lib -- test_collect_dbn_files_filters_by_symbol`
Expected: FAIL — `collect_dbn_files_filtered` doesn't exist

- [ ] **Step 3: Implement `collect_dbn_files_filtered`**

In `crates/ml/src/trainers/dqn/data_loading.rs`:

```rust
/// Collect DBN files, optionally filtering to a specific symbol subdirectory.
pub fn collect_dbn_files_filtered(dir: &Path, symbol: Option<&str>) -> Vec<PathBuf> {
    match symbol {
        Some(sym) => {
            let symbol_dir = dir.join(sym);
            if symbol_dir.is_dir() {
                collect_dbn_files_recursive(&symbol_dir)
            } else {
                // No subdirectory — filter files by name prefix
                collect_dbn_files_recursive(dir)
                    .into_iter()
                    .filter(|p| {
                        p.file_name()
                            .and_then(|n| n.to_str())
                            .map(|n| n.starts_with(sym) || n.contains(sym))
                            .unwrap_or(false)
                    })
                    .collect()
            }
        }
        None => collect_dbn_files_recursive(dir),
    }
}
```

- [ ] **Step 4: Run test to verify it passes**

Run: `SQLX_OFFLINE=true cargo test -p ml --lib -- test_collect_dbn_files_filters_by_symbol`
Expected: PASS

- [ ] **Step 5: Wire `--symbol` into precompute**

In `crates/ml/examples/precompute_features.rs`, replace:

```rust
let mut dbn_files = collect_dbn_files_recursive(&data_dir);
```

with:

```rust
let mut dbn_files = collect_dbn_files_filtered(&data_dir, Some(&opts.symbol));
if dbn_files.is_empty() {
    anyhow::bail!(
        "No DBN files found for symbol '{}' in {}",
        opts.symbol, data_dir.display()
    );
}
```

Update the import to include `collect_dbn_files_filtered`.

- [ ] **Step 6: Verify precompute only loads ES.FUT**

Run: `echo "yes" | SQLX_OFFLINE=true cargo run -p ml --example precompute_features -- --data-dir test_data/futures-baseline --symbol ES.FUT 2>&1 | grep -E 'Found|Loaded|bars'`

Expected: Should show ~670 bars (ES only), NOT 4M bars (all symbols)

- [ ] **Step 7: Commit**

```bash
git add crates/ml/src/trainers/dqn/data_loading.rs crates/ml/examples/precompute_features.rs
git commit -m "fix: filter DBN files by --symbol in precompute (was mixing ES+NQ+ZN+6E)"
```

---

### Task 2: Fix MACD Histogram (Feature 6) — Normalize by ATR

**Files:**
- Modify: `crates/ml/src/features/extraction.rs:274`

- [ ] **Step 1: Write failing test**

In `crates/ml/src/features/extraction.rs` test module:

```rust
#[test]
fn test_macd_histogram_is_dimensionless() {
    // Create bars with ES-like prices (~5500)
    let bars = make_price_series(100, 5500.0, 0.001); // 100 bars, base 5500, 0.1% volatility
    let mut extractor = FeatureExtractor::new();
    for bar in &bars {
        extractor.update(bar).unwrap();
    }
    let features = extractor.extract_current_features_v2().unwrap();
    let macd_hist = features[6];
    // Must be in [-3, 3] AND not always at the boundary
    assert!(macd_hist.abs() <= 3.0, "MACD histogram out of range: {}", macd_hist);
    assert!(macd_hist.abs() < 2.9, "MACD histogram saturated at boundary: {}", macd_hist);
}
```

- [ ] **Step 2: Run test, verify it fails (saturated at ±3.0)**

- [ ] **Step 3: Fix extraction — normalize MACD by ATR**

In `extract_technical_features_v2`, change the MACD histogram line from:

```rust
out[1] = safe_clip(last_macd.2, -3.0, 3.0);
```

to:

```rust
// Normalize MACD histogram by ATR to make it dimensionless.
// Raw histogram is in dollars (ES ~±5, NQ ~±50) — dividing by ATR
// produces a unitless ratio in approximately [-3, +3].
let atr = self.indicators.get_atr().unwrap_or(1.0).max(1e-8);
out[1] = safe_clip(last_macd.2 / atr, -3.0, 3.0);
```

- [ ] **Step 4: Run test, verify it passes**

- [ ] **Step 5: Commit**

```bash
git commit -m "fix: MACD histogram normalized by ATR (was raw dollars, saturated to ±3)"
```

---

### Task 3: Fix Bollinger Bands (Features 7-8) — Dimensionless Distance

**Files:**
- Modify: `crates/ml/src/features/extraction.rs:275-276`

- [ ] **Step 1: Write failing test**

```rust
#[test]
fn test_bollinger_bands_are_dimensionless() {
    let bars = make_price_series(100, 5500.0, 0.001);
    let mut extractor = FeatureExtractor::new();
    for bar in &bars {
        extractor.update(bar).unwrap();
    }
    let features = extractor.extract_current_features_v2().unwrap();
    let bb_upper = features[7];
    let bb_lower = features[8];
    // Must NOT be constant 3.0 (the old saturated value)
    assert!(bb_upper < 2.9, "BB upper saturated: {}", bb_upper);
    assert!(bb_lower < 2.9, "BB lower saturated: {}", bb_lower);
    // Must be in reasonable range
    assert!(bb_upper.abs() <= 3.0);
    assert!(bb_lower.abs() <= 3.0);
}
```

- [ ] **Step 2: Run test, verify it fails**

- [ ] **Step 3: Fix extraction — use dimensionless band distance**

Replace the Bollinger band lines:

```rust
out[2] = safe_clip(last_bollinger.1, -3.0, 3.0); // upper band raw price
out[3] = safe_clip(last_bollinger.2, -3.0, 3.0); // lower band raw price
```

with:

```rust
// Bollinger band position: where is close relative to the bands?
// Returns [-1, +1] where -1 = at lower band, +1 = at upper band, 0 = at middle.
let (middle, upper, lower) = last_bollinger;
let band_width = (upper - lower).max(1e-8);
let bb_position = safe_clip((close - middle) / (band_width * 0.5), -3.0, 3.0);
// Bandwidth as fraction of middle (volatility proxy)
let bb_width = safe_clip(band_width / middle.max(1e-8), 0.0, 0.2);
out[2] = bb_position;  // dimensionless position within bands
out[3] = safe_normalize(bb_width, 0.0, 0.2);  // normalized bandwidth [0,1]
```

Note: `close` must be available — get it from `self.bars.back().unwrap().close`.

- [ ] **Step 4: Run test, verify it passes**

- [ ] **Step 5: Commit**

```bash
git commit -m "fix: Bollinger bands use dimensionless position/width (was raw prices → constant 3.0)"
```

---

### Task 4: Fix Linear Regression Slope (Feature 15) — Normalize by Price

**Files:**
- Modify: `crates/ml/src/features/extraction.rs:329`

- [ ] **Step 1: Write failing test**

```rust
#[test]
fn test_lr_slope_not_saturated_for_es() {
    let bars = make_price_series(100, 5500.0, 0.001);
    let mut extractor = FeatureExtractor::new();
    for bar in &bars {
        extractor.update(bar).unwrap();
    }
    let features = extractor.extract_current_features_v2().unwrap();
    let slope = features[15];
    assert!(slope.abs() < 0.09, "LR slope saturated at boundary: {}", slope);
}
```

- [ ] **Step 2: Run test, verify it fails (saturated at ±0.1)**

- [ ] **Step 3: Fix — normalize slope by close price**

Change:

```rust
out[5] = safe_clip(slope, -0.1, 0.1);
```

to:

```rust
// Normalize slope by close price to make dimensionless (% change per bar).
let close = self.bars.back().map(|b| b.close).unwrap_or(1.0).max(1e-8);
out[5] = safe_clip(slope / close, -0.01, 0.01);
```

The new range `[-0.01, +0.01]` means ±1% per bar, which is generous for 1-min bars.

- [ ] **Step 4: Run test, verify it passes**

- [ ] **Step 5: Commit**

```bash
git commit -m "fix: LR slope normalized by close price (was raw dollars/bar, saturated for ES/NQ)"
```

---

### Task 5: Fix Autocorrelation (Features 31-33) — Use Returns Not Prices

**Files:**
- Modify: `crates/ml/src/features/extraction.rs` — `compute_autocorr` function (~line 649)

- [ ] **Step 1: Write failing test**

```rust
#[test]
fn test_autocorrelation_uses_returns() {
    // Random walk prices should have near-zero autocorrelation of RETURNS
    let bars = make_random_walk(200, 5500.0, 0.001);
    let mut extractor = FeatureExtractor::new();
    for bar in &bars {
        extractor.update(bar).unwrap();
    }
    let features = extractor.extract_current_features_v2().unwrap();
    let ac1 = features[31];
    let ac5 = features[32];
    let ac10 = features[33];
    // Random walk returns have near-zero autocorrelation — not near 1.0
    assert!(ac1.abs() < 0.5, "Autocorr lag-1 too high for random walk: {}", ac1);
    assert!(ac5.abs() < 0.5, "Autocorr lag-5 too high: {}", ac5);
    assert!(ac10.abs() < 0.5, "Autocorr lag-10 too high: {}", ac10);
}
```

- [ ] **Step 2: Run test, verify it fails (all ~1.0)**

- [ ] **Step 3: Fix — compute autocorrelation on log returns**

Replace `compute_autocorr`:

```rust
fn compute_autocorr(&self, lag: usize) -> f64 {
    let n = self.bars.len();
    if n < lag + 2 { return 0.0; }
    
    // Compute log returns first
    let returns: Vec<f64> = (1..n)
        .map(|i| safe_log_return(self.bars[i].close, self.bars[i-1].close))
        .collect();
    
    let m = returns.len();
    if m < lag + 1 { return 0.0; }
    
    let mean = returns.iter().sum::<f64>() / m as f64;
    let var: f64 = returns.iter().map(|r| (r - mean).powi(2)).sum::<f64>();
    if var < 1e-12 { return 0.0; }
    
    let mut cov = 0.0;
    for i in 0..(m - lag) {
        cov += (returns[i] - mean) * (returns[i + lag] - mean);
    }
    
    (cov / var).clamp(-1.0, 1.0)
}
```

- [ ] **Step 4: Run test, verify it passes**

- [ ] **Step 5: Commit**

```bash
git commit -m "fix: autocorrelation computed on log returns (was raw prices → always ~1.0)"
```

---

### Task 6: Feature Range Validation Test

**Files:**
- Create: `crates/ml/src/features/extraction_tests.rs` (or add to existing test module)

- [ ] **Step 1: Write comprehensive range validation test**

```rust
#[test]
fn test_all_42_features_bounded() {
    // Test with ES-like prices
    let bars = make_price_series(200, 5500.0, 0.002);
    let mut extractor = FeatureExtractor::new();
    for bar in &bars {
        extractor.update(bar).unwrap();
    }
    let features = extractor.extract_current_features_v2().unwrap();
    
    let bounds: [(f64, f64); 42] = [
        // OHLCV (0-4): log returns — generous bounds
        (-0.1, 0.1), (-0.1, 0.1), (-0.1, 0.1), (-0.1, 0.1), (0.0, 1.0),
        // Technical (5-9)
        (0.0, 1.0),   // RSI [0,1]
        (-3.0, 3.0),  // MACD/ATR
        (-3.0, 3.0),  // BB position
        (0.0, 1.0),   // BB width
        (0.0, 1.0),   // ATR normalized
        // Price patterns (10-15)
        (-0.1, 0.1), (-0.1, 0.1), (-0.1, 0.1),  // returns
        (-0.5, 0.5), (-0.5, 0.5),                 // close/SMA ratios
        (-0.01, 0.01),                              // LR slope/price
        // Volume (16-21)
        (-2.0, 2.0), (0.0, 1.0), (-0.1, 0.1), (-0.1, 0.1), (-0.1, 0.1), (-1.0, 1.0),
        // Time (22-26)
        (0.0, 1.0), (0.0, 1.0), (0.0, 1.0), (0.0, 1.0), (0.0, 1.0),
        // Statistical (27-39)
        (-3.0, 3.0), (-3.0, 3.0),   // z-scores
        (0.0, 1.0), (0.0, 1.0),     // percentile ranks
        (-1.0, 1.0), (-1.0, 1.0), (-1.0, 1.0),  // autocorrelations
        (-3.0, 3.0), (-3.0, 3.0), (-3.0, 3.0),  // skewness
        (-3.0, 3.0), (-3.0, 3.0), (-3.0, 3.0),  // kurtosis
        // Regime (40-41)
        (0.0, 1.0),   // ADX
        (-1.0, 1.0),  // CUSUM
    ];
    
    for (i, &val) in features.iter().enumerate() {
        let (lo, hi) = bounds[i];
        assert!(
            val >= lo && val <= hi,
            "Feature[{}] = {:.6} outside expected range [{}, {}]",
            i, val, lo, hi
        );
    }
}

#[test]
fn test_features_not_saturated_at_boundaries() {
    // Run 200 bars and check that features have variance (not constant)
    let bars = make_price_series(200, 5500.0, 0.002);
    let mut extractor = FeatureExtractor::new();
    let mut all_features: Vec<[f64; 42]> = Vec::new();
    for (i, bar) in bars.iter().enumerate() {
        extractor.update(bar).unwrap();
        if i >= 50 {
            all_features.push(extractor.extract_current_features_v2().unwrap());
        }
    }
    
    // Check each feature has non-trivial variance
    for feat_idx in 0..42 {
        let vals: Vec<f64> = all_features.iter().map(|f| f[feat_idx]).collect();
        let mean = vals.iter().sum::<f64>() / vals.len() as f64;
        let var = vals.iter().map(|v| (v - mean).powi(2)).sum::<f64>() / vals.len() as f64;
        assert!(
            var > 1e-12,
            "Feature[{}] has zero variance (saturated at {:.4})",
            feat_idx, mean
        );
    }
}
```

- [ ] **Step 2: Add helper `make_price_series` and `make_random_walk`**

```rust
fn make_price_series(n: usize, base: f64, volatility: f64) -> Vec<OHLCVBar> {
    use chrono::{TimeZone, Utc};
    let mut bars = Vec::with_capacity(n);
    let mut price = base;
    let mut rng_state = 42u64; // deterministic pseudo-random
    for i in 0..n {
        rng_state = rng_state.wrapping_mul(6364136223846793005).wrapping_add(1442695040888963407);
        let r = ((rng_state >> 33) as f64 / u32::MAX as f64 - 0.5) * 2.0 * volatility;
        price *= 1.0 + r;
        let high = price * (1.0 + volatility.abs());
        let low = price * (1.0 - volatility.abs());
        let ts = Utc.timestamp_opt(1704067200 + (i as i64 * 60), 0).unwrap(); // 2024-01-01 + i minutes
        bars.push(OHLCVBar {
            timestamp: ts,
            open: price * (1.0 + r * 0.3),
            high,
            low,
            close: price,
            volume: 1000.0 + (rng_state % 5000) as f64,
        });
    }
    bars
}

fn make_random_walk(n: usize, base: f64, step_std: f64) -> Vec<OHLCVBar> {
    make_price_series(n, base, step_std)
}
```

- [ ] **Step 3: Run all tests, verify they pass**

Run: `SQLX_OFFLINE=true cargo test -p ml --lib -- test_all_42_features_bounded test_features_not_saturated -v`

- [ ] **Step 4: Commit**

```bash
git commit -m "test: comprehensive feature range validation (42 features × bounds + variance)"
```

---

### Task 7: Regenerate fxcache and Validate End-to-End

**Files:**
- No code changes — validation only

- [ ] **Step 1: Delete old fxcache**

```bash
rm -f test_data/feature-cache/*.fxcache test_data/feature-cache/*.json
```

- [ ] **Step 2: Regenerate with symbol filter**

```bash
echo "yes" | SQLX_OFFLINE=true cargo run -p ml --example precompute_features -- \
    --data-dir test_data/futures-baseline --symbol ES.FUT
```

Expected: loads only ES.FUT files, produces ~670 bars (not 4M)

- [ ] **Step 3: Validate feature ranges in the fxcache**

```python
import struct, numpy as np
path = "test_data/feature-cache/<hash>.fxcache"
with open(path, "rb") as f:
    f.read(8); f.read(8)
    bar_count = struct.unpack('<Q', f.read(8))[0]; f.read(40)
    raw = np.frombuffer(f.read(min(1000, bar_count) * 54 * 8), dtype=np.float64).reshape(-1, 54)
    feat = raw[:, :42]
    print(f"mean={feat.mean():.4f} std={feat.std():.4f} min={feat.min():.4f} max={feat.max():.4f}")
    assert feat.max() < 10.0, f"Features still have raw prices: max={feat.max()}"
    assert feat.min() > -10.0, f"Features still have raw prices: min={feat.min()}"
```

- [ ] **Step 4: Run production smoketest with full dataset**

Remove the `.min(1000)` cap in `training_stability.rs` and run:

```bash
SQLX_OFFLINE=true cargo test -p ml --lib -- test_production_training_stability --ignored --nocapture
```

Expected: 10 epochs, finite loss, no NaN, completes in <10s

- [ ] **Step 5: Run compute-sanitizer to verify 0 errors**

```bash
SQLX_OFFLINE=true /usr/local/cuda/bin/compute-sanitizer --tool memcheck --error-exitcode 0 \
    target/debug/deps/ml-* test_production_training_stability --ignored --nocapture --test-threads=1
```

Expected: `ERROR SUMMARY: 0 errors`

- [ ] **Step 6: Commit final state**

```bash
git commit -m "feat: feature extraction pipeline fixed — all 42 features bounded, single-symbol, no NaN"
```

---

### Task 8: Update Argo Precompute Workflow

**Files:**
- Modify: `infra/k8s/argo/precompute-features.yaml` (or equivalent)

- [ ] **Step 1: Add `--symbol ES.FUT` to the Argo precompute job**

The Argo workflow that generates the fxcache on the PVC must pass `--symbol ES.FUT` to the precompute binary. Find the workflow template and add the flag.

- [ ] **Step 2: Verify the PVC data layout matches `--symbol` expectations**

Check that `/mnt/training-data/futures-baseline/ES.FUT/` contains the ES DBN files on the PVC.

- [ ] **Step 3: Commit**

```bash
git commit -m "fix: Argo precompute passes --symbol ES.FUT (was loading all symbols)"
```
