foxhunt/docs/superpowers/plans/2026-04-03-feature-extraction-pipeline-fix.md

Feature Extraction Pipeline Fix

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Fix 6 broken features, eliminate multi-symbol mixing, and ensure all 42 features are properly normalized and carry meaningful signal for the DQN model.

Architecture: Two independent fixes: (1) symbol filtering in precompute to ensure single-instrument data, (2) feature extraction fixes to normalize raw-price indicators into dimensionless values. Both must be done — symbol filtering alone doesn't fix the Bollinger/MACD/autocorrelation bugs, and feature fixes alone don't fix cross-symbol contamination.

Tech Stack: Rust, crates/ml/src/features/extraction.rs (main extraction), crates/common/src/features/technical_indicators.rs (MACD, Bollinger, ATR), crates/ml/examples/precompute_features.rs (precompute entry point)

---

Bug Inventory

Index	Feature	Bug	Fix
6	MACD histogram	Raw dollars (±5–100), clipped to ±3 → near-binary	Normalize: histogram / atr (dimensionless)
7	Bollinger upper	Raw price (~20000), clipped to +3.0 → dead constant	Change to: (close - upper) / (upper - lower) (dimensionless distance)
8	Bollinger lower	Raw price (~20000), clipped to +3.0 → dead constant	Change to: (close - lower) / (upper - lower) (dimensionless distance)
15	LR slope(20)	Raw dollars/bar, saturated for ES/NQ	Normalize: slope / atr or slope / close (dimensionless)
31-33	Autocorr lag-1,5,10	Computed on price levels → always ~1.0	Compute on log returns instead of raw prices
ALL	Multi-symbol mix	collect_dbn_files_recursive loads ES+NQ+ZN+6E	Filter by --symbol, pass only matching files

File Map

File	Changes
crates/ml/src/features/extraction.rs	Fix features 6, 7, 8, 15, 31-33
crates/ml/examples/precompute_features.rs	Filter DBN files by --symbol, validate feature ranges
crates/ml/src/trainers/dqn/data_loading.rs	collect_dbn_files_recursive → accept symbol filter
crates/ml/src/features/extraction.rs (tests)	Add range validation tests for all 42 features

---

Task 1: Symbol Filtering in Precompute

Files:

• Modify: crates/ml/examples/precompute_features.rs:214-224

• Modify: crates/ml/src/trainers/dqn/data_loading.rs (collect_dbn_files_recursive)

• Step 1: Write failing test — symbol filter

In crates/ml/src/trainers/dqn/data_loading.rs, add a test:

#[test]
fn test_collect_dbn_files_filters_by_symbol() {
    use std::path::PathBuf;
    use tempfile::TempDir;
    
    let dir = TempDir::new().unwrap();
    let es_dir = dir.path().join("ES.FUT");
    let nq_dir = dir.path().join("NQ.FUT");
    std::fs::create_dir_all(&es_dir).unwrap();
    std::fs::create_dir_all(&nq_dir).unwrap();
    std::fs::write(es_dir.join("ES_Q1.dbn.zst"), b"").unwrap();
    std::fs::write(nq_dir.join("NQ_Q1.dbn.zst"), b"").unwrap();
    
    let all = collect_dbn_files_recursive(dir.path());
    assert_eq!(all.len(), 2);
    
    let filtered = collect_dbn_files_filtered(dir.path(), Some("ES.FUT"));
    assert_eq!(filtered.len(), 1);
    assert!(filtered[0].to_str().unwrap().contains("ES.FUT"));
}

• Step 2: Run test to verify it fails

Run: SQLX_OFFLINE=true cargo test -p ml --lib -- test_collect_dbn_files_filters_by_symbol Expected: FAIL — collect_dbn_files_filtered doesn't exist

• Step 3: Implement collect_dbn_files_filtered

In crates/ml/src/trainers/dqn/data_loading.rs:

/// Collect DBN files, optionally filtering to a specific symbol subdirectory.
pub fn collect_dbn_files_filtered(dir: &Path, symbol: Option<&str>) -> Vec<PathBuf> {
    match symbol {
        Some(sym) => {
            let symbol_dir = dir.join(sym);
            if symbol_dir.is_dir() {
                collect_dbn_files_recursive(&symbol_dir)
            } else {
                // No subdirectory — filter files by name prefix
                collect_dbn_files_recursive(dir)
                    .into_iter()
                    .filter(|p| {
                        p.file_name()
                            .and_then(|n| n.to_str())
                            .map(|n| n.starts_with(sym) || n.contains(sym))
                            .unwrap_or(false)
                    })
                    .collect()
            }
        }
        None => collect_dbn_files_recursive(dir),
    }
}

• Step 4: Run test to verify it passes

Run: SQLX_OFFLINE=true cargo test -p ml --lib -- test_collect_dbn_files_filters_by_symbol Expected: PASS

• Step 5: Wire --symbol into precompute

In crates/ml/examples/precompute_features.rs, replace:

let mut dbn_files = collect_dbn_files_recursive(&data_dir);

with:

let mut dbn_files = collect_dbn_files_filtered(&data_dir, Some(&opts.symbol));
if dbn_files.is_empty() {
    anyhow::bail!(
        "No DBN files found for symbol '{}' in {}",
        opts.symbol, data_dir.display()
    );
}

Update the import to include collect_dbn_files_filtered.

• Step 6: Verify precompute only loads ES.FUT

Run: echo "yes" | SQLX_OFFLINE=true cargo run -p ml --example precompute_features -- --data-dir test_data/futures-baseline --symbol ES.FUT 2>&1 | grep -E 'Found|Loaded|bars'

Expected: Should show ~670 bars (ES only), NOT 4M bars (all symbols)

• Step 7: Commit

git add crates/ml/src/trainers/dqn/data_loading.rs crates/ml/examples/precompute_features.rs
git commit -m "fix: filter DBN files by --symbol in precompute (was mixing ES+NQ+ZN+6E)"

---

Task 2: Fix MACD Histogram (Feature 6) — Normalize by ATR

Files:

• Modify: crates/ml/src/features/extraction.rs:274

• Step 1: Write failing test

In crates/ml/src/features/extraction.rs test module:

#[test]
fn test_macd_histogram_is_dimensionless() {
    // Create bars with ES-like prices (~5500)
    let bars = make_price_series(100, 5500.0, 0.001); // 100 bars, base 5500, 0.1% volatility
    let mut extractor = FeatureExtractor::new();
    for bar in &bars {
        extractor.update(bar).unwrap();
    }
    let features = extractor.extract_current_features_v2().unwrap();
    let macd_hist = features[6];
    // Must be in [-3, 3] AND not always at the boundary
    assert!(macd_hist.abs() <= 3.0, "MACD histogram out of range: {}", macd_hist);
    assert!(macd_hist.abs() < 2.9, "MACD histogram saturated at boundary: {}", macd_hist);
}

• Step 2: Run test, verify it fails (saturated at ±3.0)

• Step 3: Fix extraction — normalize MACD by ATR

In extract_technical_features_v2, change the MACD histogram line from:

out[1] = safe_clip(last_macd.2, -3.0, 3.0);

to:

// Normalize MACD histogram by ATR to make it dimensionless.
// Raw histogram is in dollars (ES ~±5, NQ ~±50) — dividing by ATR
// produces a unitless ratio in approximately [-3, +3].
let atr = self.indicators.get_atr().unwrap_or(1.0).max(1e-8);
out[1] = safe_clip(last_macd.2 / atr, -3.0, 3.0);

• Step 4: Run test, verify it passes

• Step 5: Commit

git commit -m "fix: MACD histogram normalized by ATR (was raw dollars, saturated to ±3)"

---

Task 3: Fix Bollinger Bands (Features 7-8) — Dimensionless Distance

Files:

• Modify: crates/ml/src/features/extraction.rs:275-276

• Step 1: Write failing test

#[test]
fn test_bollinger_bands_are_dimensionless() {
    let bars = make_price_series(100, 5500.0, 0.001);
    let mut extractor = FeatureExtractor::new();
    for bar in &bars {
        extractor.update(bar).unwrap();
    }
    let features = extractor.extract_current_features_v2().unwrap();
    let bb_upper = features[7];
    let bb_lower = features[8];
    // Must NOT be constant 3.0 (the old saturated value)
    assert!(bb_upper < 2.9, "BB upper saturated: {}", bb_upper);
    assert!(bb_lower < 2.9, "BB lower saturated: {}", bb_lower);
    // Must be in reasonable range
    assert!(bb_upper.abs() <= 3.0);
    assert!(bb_lower.abs() <= 3.0);
}

• Step 2: Run test, verify it fails

• Step 3: Fix extraction — use dimensionless band distance

Replace the Bollinger band lines:

out[2] = safe_clip(last_bollinger.1, -3.0, 3.0); // upper band raw price
out[3] = safe_clip(last_bollinger.2, -3.0, 3.0); // lower band raw price

with:

// Bollinger band position: where is close relative to the bands?
// Returns [-1, +1] where -1 = at lower band, +1 = at upper band, 0 = at middle.
let (middle, upper, lower) = last_bollinger;
let band_width = (upper - lower).max(1e-8);
let bb_position = safe_clip((close - middle) / (band_width * 0.5), -3.0, 3.0);
// Bandwidth as fraction of middle (volatility proxy)
let bb_width = safe_clip(band_width / middle.max(1e-8), 0.0, 0.2);
out[2] = bb_position;  // dimensionless position within bands
out[3] = safe_normalize(bb_width, 0.0, 0.2);  // normalized bandwidth [0,1]

Note: close must be available — get it from self.bars.back().unwrap().close.

• Step 4: Run test, verify it passes

• Step 5: Commit

git commit -m "fix: Bollinger bands use dimensionless position/width (was raw prices → constant 3.0)"

---

Task 4: Fix Linear Regression Slope (Feature 15) — Normalize by Price

Files:

• Modify: crates/ml/src/features/extraction.rs:329

• Step 1: Write failing test

#[test]
fn test_lr_slope_not_saturated_for_es() {
    let bars = make_price_series(100, 5500.0, 0.001);
    let mut extractor = FeatureExtractor::new();
    for bar in &bars {
        extractor.update(bar).unwrap();
    }
    let features = extractor.extract_current_features_v2().unwrap();
    let slope = features[15];
    assert!(slope.abs() < 0.09, "LR slope saturated at boundary: {}", slope);
}

• Step 2: Run test, verify it fails (saturated at ±0.1)

• Step 3: Fix — normalize slope by close price

Change:

out[5] = safe_clip(slope, -0.1, 0.1);

to:

// Normalize slope by close price to make dimensionless (% change per bar).
let close = self.bars.back().map(|b| b.close).unwrap_or(1.0).max(1e-8);
out[5] = safe_clip(slope / close, -0.01, 0.01);

The new range [-0.01, +0.01] means ±1% per bar, which is generous for 1-min bars.

• Step 4: Run test, verify it passes

• Step 5: Commit

git commit -m "fix: LR slope normalized by close price (was raw dollars/bar, saturated for ES/NQ)"

---

Task 5: Fix Autocorrelation (Features 31-33) — Use Returns Not Prices

Files:

• Modify: crates/ml/src/features/extraction.rs — compute_autocorr function (~line 649)

• Step 1: Write failing test

#[test]
fn test_autocorrelation_uses_returns() {
    // Random walk prices should have near-zero autocorrelation of RETURNS
    let bars = make_random_walk(200, 5500.0, 0.001);
    let mut extractor = FeatureExtractor::new();
    for bar in &bars {
        extractor.update(bar).unwrap();
    }
    let features = extractor.extract_current_features_v2().unwrap();
    let ac1 = features[31];
    let ac5 = features[32];
    let ac10 = features[33];
    // Random walk returns have near-zero autocorrelation — not near 1.0
    assert!(ac1.abs() < 0.5, "Autocorr lag-1 too high for random walk: {}", ac1);
    assert!(ac5.abs() < 0.5, "Autocorr lag-5 too high: {}", ac5);
    assert!(ac10.abs() < 0.5, "Autocorr lag-10 too high: {}", ac10);
}

• Step 2: Run test, verify it fails (all ~1.0)

• Step 3: Fix — compute autocorrelation on log returns

Replace compute_autocorr:

fn compute_autocorr(&self, lag: usize) -> f64 {
    let n = self.bars.len();
    if n < lag + 2 { return 0.0; }
    
    // Compute log returns first
    let returns: Vec<f64> = (1..n)
        .map(|i| safe_log_return(self.bars[i].close, self.bars[i-1].close))
        .collect();
    
    let m = returns.len();
    if m < lag + 1 { return 0.0; }
    
    let mean = returns.iter().sum::<f64>() / m as f64;
    let var: f64 = returns.iter().map(|r| (r - mean).powi(2)).sum::<f64>();
    if var < 1e-12 { return 0.0; }
    
    let mut cov = 0.0;
    for i in 0..(m - lag) {
        cov += (returns[i] - mean) * (returns[i + lag] - mean);
    }
    
    (cov / var).clamp(-1.0, 1.0)
}

• Step 4: Run test, verify it passes

• Step 5: Commit

git commit -m "fix: autocorrelation computed on log returns (was raw prices → always ~1.0)"

---

Task 6: Feature Range Validation Test

Files:

• Create: crates/ml/src/features/extraction_tests.rs (or add to existing test module)

• Step 1: Write comprehensive range validation test

#[test]
fn test_all_42_features_bounded() {
    // Test with ES-like prices
    let bars = make_price_series(200, 5500.0, 0.002);
    let mut extractor = FeatureExtractor::new();
    for bar in &bars {
        extractor.update(bar).unwrap();
    }
    let features = extractor.extract_current_features_v2().unwrap();
    
    let bounds: [(f64, f64); 42] = [
        // OHLCV (0-4): log returns — generous bounds
        (-0.1, 0.1), (-0.1, 0.1), (-0.1, 0.1), (-0.1, 0.1), (0.0, 1.0),
        // Technical (5-9)
        (0.0, 1.0),   // RSI [0,1]
        (-3.0, 3.0),  // MACD/ATR
        (-3.0, 3.0),  // BB position
        (0.0, 1.0),   // BB width
        (0.0, 1.0),   // ATR normalized
        // Price patterns (10-15)
        (-0.1, 0.1), (-0.1, 0.1), (-0.1, 0.1),  // returns
        (-0.5, 0.5), (-0.5, 0.5),                 // close/SMA ratios
        (-0.01, 0.01),                              // LR slope/price
        // Volume (16-21)
        (-2.0, 2.0), (0.0, 1.0), (-0.1, 0.1), (-0.1, 0.1), (-0.1, 0.1), (-1.0, 1.0),
        // Time (22-26)
        (0.0, 1.0), (0.0, 1.0), (0.0, 1.0), (0.0, 1.0), (0.0, 1.0),
        // Statistical (27-39)
        (-3.0, 3.0), (-3.0, 3.0),   // z-scores
        (0.0, 1.0), (0.0, 1.0),     // percentile ranks
        (-1.0, 1.0), (-1.0, 1.0), (-1.0, 1.0),  // autocorrelations
        (-3.0, 3.0), (-3.0, 3.0), (-3.0, 3.0),  // skewness
        (-3.0, 3.0), (-3.0, 3.0), (-3.0, 3.0),  // kurtosis
        // Regime (40-41)
        (0.0, 1.0),   // ADX
        (-1.0, 1.0),  // CUSUM
    ];
    
    for (i, &val) in features.iter().enumerate() {
        let (lo, hi) = bounds[i];
        assert!(
            val >= lo && val <= hi,
            "Feature[{}] = {:.6} outside expected range [{}, {}]",
            i, val, lo, hi
        );
    }
}

#[test]
fn test_features_not_saturated_at_boundaries() {
    // Run 200 bars and check that features have variance (not constant)
    let bars = make_price_series(200, 5500.0, 0.002);
    let mut extractor = FeatureExtractor::new();
    let mut all_features: Vec<[f64; 42]> = Vec::new();
    for (i, bar) in bars.iter().enumerate() {
        extractor.update(bar).unwrap();
        if i >= 50 {
            all_features.push(extractor.extract_current_features_v2().unwrap());
        }
    }
    
    // Check each feature has non-trivial variance
    for feat_idx in 0..42 {
        let vals: Vec<f64> = all_features.iter().map(|f| f[feat_idx]).collect();
        let mean = vals.iter().sum::<f64>() / vals.len() as f64;
        let var = vals.iter().map(|v| (v - mean).powi(2)).sum::<f64>() / vals.len() as f64;
        assert!(
            var > 1e-12,
            "Feature[{}] has zero variance (saturated at {:.4})",
            feat_idx, mean
        );
    }
}

• Step 2: Add helper make_price_series and make_random_walk

fn make_price_series(n: usize, base: f64, volatility: f64) -> Vec<OHLCVBar> {
    use chrono::{TimeZone, Utc};
    let mut bars = Vec::with_capacity(n);
    let mut price = base;
    let mut rng_state = 42u64; // deterministic pseudo-random
    for i in 0..n {
        rng_state = rng_state.wrapping_mul(6364136223846793005).wrapping_add(1442695040888963407);
        let r = ((rng_state >> 33) as f64 / u32::MAX as f64 - 0.5) * 2.0 * volatility;
        price *= 1.0 + r;
        let high = price * (1.0 + volatility.abs());
        let low = price * (1.0 - volatility.abs());
        let ts = Utc.timestamp_opt(1704067200 + (i as i64 * 60), 0).unwrap(); // 2024-01-01 + i minutes
        bars.push(OHLCVBar {
            timestamp: ts,
            open: price * (1.0 + r * 0.3),
            high,
            low,
            close: price,
            volume: 1000.0 + (rng_state % 5000) as f64,
        });
    }
    bars
}

fn make_random_walk(n: usize, base: f64, step_std: f64) -> Vec<OHLCVBar> {
    make_price_series(n, base, step_std)
}

• Step 3: Run all tests, verify they pass

Run: SQLX_OFFLINE=true cargo test -p ml --lib -- test_all_42_features_bounded test_features_not_saturated -v

• Step 4: Commit

git commit -m "test: comprehensive feature range validation (42 features × bounds + variance)"

---

Task 7: Regenerate fxcache and Validate End-to-End

Files:

• No code changes — validation only

• Step 1: Delete old fxcache

rm -f test_data/feature-cache/*.fxcache test_data/feature-cache/*.json

• Step 2: Regenerate with symbol filter

echo "yes" | SQLX_OFFLINE=true cargo run -p ml --example precompute_features -- \
    --data-dir test_data/futures-baseline --symbol ES.FUT

Expected: loads only ES.FUT files, produces ~670 bars (not 4M)

• Step 3: Validate feature ranges in the fxcache

import struct, numpy as np
path = "test_data/feature-cache/<hash>.fxcache"
with open(path, "rb") as f:
    f.read(8); f.read(8)
    bar_count = struct.unpack('<Q', f.read(8))[0]; f.read(40)
    raw = np.frombuffer(f.read(min(1000, bar_count) * 54 * 8), dtype=np.float64).reshape(-1, 54)
    feat = raw[:, :42]
    print(f"mean={feat.mean():.4f} std={feat.std():.4f} min={feat.min():.4f} max={feat.max():.4f}")
    assert feat.max() < 10.0, f"Features still have raw prices: max={feat.max()}"
    assert feat.min() > -10.0, f"Features still have raw prices: min={feat.min()}"

• Step 4: Run production smoketest with full dataset

Remove the .min(1000) cap in training_stability.rs and run:

SQLX_OFFLINE=true cargo test -p ml --lib -- test_production_training_stability --ignored --nocapture

Expected: 10 epochs, finite loss, no NaN, completes in <10s

• Step 5: Run compute-sanitizer to verify 0 errors

SQLX_OFFLINE=true /usr/local/cuda/bin/compute-sanitizer --tool memcheck --error-exitcode 0 \
    target/debug/deps/ml-* test_production_training_stability --ignored --nocapture --test-threads=1

Expected: ERROR SUMMARY: 0 errors

• Step 6: Commit final state

git commit -m "feat: feature extraction pipeline fixed — all 42 features bounded, single-symbol, no NaN"

---

Task 8: Update Argo Precompute Workflow

Files:

• Modify: infra/k8s/argo/precompute-features.yaml (or equivalent)

• Step 1: Add --symbol ES.FUT to the Argo precompute job

The Argo workflow that generates the fxcache on the PVC must pass --symbol ES.FUT to the precompute binary. Find the workflow template and add the flag.

• Step 2: Verify the PVC data layout matches --symbol expectations

Check that /mnt/training-data/futures-baseline/ES.FUT/ contains the ES DBN files on the PVC.

• Step 3: Commit

git commit -m "fix: Argo precompute passes --symbol ES.FUT (was loading all symbols)"