foxhunt/docs/archive/wave_abc/WAVE_C_FEATURE_NORMALIZATION_DESIGN.md

Wave C: Feature Normalization and Scaling Strategy

Date: 2025-10-17 Mission: Design production-ready normalization pipeline for 256-dimension ML features Scope: Online/incremental normalization for streaming HFT data Status: 📋 DESIGN COMPLETE (awaiting implementation)

---

🎯 Overview

Feature normalization is critical for ML model convergence and prediction quality. This design specifies:

1. Normalization methods for each feature category (price, volume, technical, microstructure, time, statistical)
2. Online/incremental algorithms for streaming data (no batch recomputation)
3. Rolling window strategies for mean/std calculation
4. NaN/Inf handling (imputation vs filtering)
5. Outlier clipping (±3σ thresholds)

Performance Targets:

• Latency: <10μs per 256-feature normalization
• Memory: <2KB per symbol (rolling statistics)
• Stability: No NaN/Inf in output features
• Accuracy: <1% error vs batch normalization after warmup

---

📊 Feature Categories & Normalization Methods

1. Price Features (Indices 0-4, 15-74 in 256-dim vector)

Features: Returns, log returns, price ratios, moving average ratios, price extremes

Normalization Method: Z-Score Normalization (mean=0, std=1)

normalized = (raw_value - rolling_mean) / (rolling_std + epsilon)
clipped = normalized.clamp(-3.0, 3.0)  // ±3σ outlier removal

Rationale:

• Price features are unbounded and Gaussian-distributed (approximately)
• Z-score centers data at zero, scales by volatility
• Handles non-stationarity via rolling windows

Rolling Window Sizes:

• Fast regime (intraday): 20 bars (~1-2 hours for 5-min bars)
• Medium regime (daily): 50 bars (~10 days)
• Slow regime (weekly): 260 bars (~52 weeks)
• Recommendation: Use 50 bars for HFT (balances responsiveness vs stability)

Implementation:

// Rolling mean/std with Welford's online algorithm (O(1) memory)
struct RollingZScore {
    window_size: usize,
    values: VecDeque<f64>,  // Last N values
    mean: f64,
    m2: f64,  // Sum of squared deviations (for std)
    count: usize,
}

impl RollingZScore {
    fn update(&mut self, value: f64) -> f64 {
        // Add new value
        self.values.push_back(value);

        if self.values.len() > self.window_size {
            // Remove oldest value
            let old_val = self.values.pop_front().unwrap();

            // Update statistics (Welford's algorithm)
            let delta = value - old_val;
            self.mean += delta / self.count as f64;
            self.m2 += delta * (value - self.mean + old_val - self.mean);
        } else {
            // Warmup phase: incremental update
            self.count = self.values.len();
            let delta = value - self.mean;
            self.mean += delta / self.count as f64;
            let delta2 = value - self.mean;
            self.m2 += delta * delta2;
        }

        // Compute normalized value
        let std = (self.m2 / (self.count - 1) as f64).sqrt();
        let normalized = (value - self.mean) / (std + 1e-8);
        normalized.clamp(-3.0, 3.0)
    }
}

NaN/Inf Handling:

• Input validation: Skip NaN/Inf values, use last valid value
• Division by zero: Add epsilon (1e-8) to std denominator
• Warmup period: Return 0.0 for first 10 bars (insufficient data)

---

2. Volume Features (Indices 3-4, 75-114 in 256-dim vector)

Features: Volume change, volume ratios, volume MA, OBV, volume momentum

Normalization Method: Percentile Rank Normalization (0-1)

normalized = rank(value) / total_count

Rationale:

• Volume data is highly skewed (log-normal distribution)
• Percentile rank is robust to outliers and non-parametric
• Maps to [0, 1] range naturally (0 = min, 1 = max)

Rolling Window Size: 50 bars (same as price for consistency)

Implementation:

struct RollingPercentileRank {
    window_size: usize,
    values: VecDeque<f64>,
}

impl RollingPercentileRank {
    fn update(&mut self, value: f64) -> f64 {
        // Add new value
        self.values.push_back(value);

        if self.values.len() > self.window_size {
            self.values.pop_front();
        }

        // Compute percentile rank (O(n) but n=50 is small)
        let rank = self.values.iter()
            .filter(|&&v| v < value)
            .count();

        let normalized = rank as f64 / self.values.len() as f64;
        normalized.clamp(0.0, 1.0)
    }
}

Optimization: Use sorted data structure (BTreeSet) for O(log n) rank calculation if needed

NaN/Inf Handling:

• Input validation: Skip NaN/Inf, use last valid volume
• Zero volume: Map to 0.0 percentile (minimum)
• Warmup period: Return 0.5 (median) for first 10 bars

---

3. Technical Indicators (Indices 5-17 in 26-dim vector, 5-14 in 256-dim)

Features: RSI, MACD, Bollinger, ATR, EMA, Stochastic, ADX, CCI, Williams %R

Normalization Method: Already Normalized (0-1 or -1 to +1)

Implementation: No additional normalization needed

Existing Ranges:

• RSI: [0, 100] → Already normalized to [0, 1] in extraction.rs (line 200)
• MACD: Unbounded → Already normalized with tanh (lines 203-205)
• Bollinger: [lower, upper] → Position normalized to [-1, 1] (lines 206-208)
• ATR: [0, ∞) → Already normalized to [0, 1] (line 210)
• Stochastic: [0, 100] → Normalized to [0, 1]
• ADX: [0, 100] → Normalized to [0, 1]
• CCI: Unbounded → Normalized with tanh to [-1, 1]

Validation: Ensure no indicator exceeds [-1, 1] range

NaN/Inf Handling:

• Insufficient data: Return neutral value (0.0 for [-1,1], 0.5 for [0,1])
• Division by zero: Add epsilon in indicator calculation
• Output validation: Assert all values in expected range

---

4. Microstructure Features (Indices 115-164 in 256-dim vector)

Features: Roll spread, Amihud illiquidity, Corwin-Schultz spread, order flow proxies

Normalization Method: Log Transform + Z-Score

// Step 1: Log transform (handles skewed distributions)
log_value = if value > 0.0 {
    (value * scale_factor).ln()
} else {
    -10.0  // Map zero/negative to minimum
};

// Step 2: Z-score normalization
normalized = (log_value - rolling_mean) / (rolling_std + epsilon);
clipped = normalized.clamp(-3.0, 3.0);

Rationale:

• Microstructure features are highly skewed (e.g., Amihud: 1e-9 to 1e-5)
• Log transform stabilizes variance and makes distribution more Gaussian
• Z-score after log transform provides consistent scale

Rolling Window Size: 20 bars (faster adaptation for microstructure regime changes)

Scale Factors (map to reasonable log range):

• Roll spread: 1.0 (already in price units, ~0.01-10.0)
• Amihud illiquidity: 1e8 (map 1e-8 → 1.0 for ln)
• Corwin-Schultz spread: 100.0 (map 0.01 → 1.0 for ln)

Implementation:

struct LogZScoreNormalizer {
    scale_factor: f64,
    zscore: RollingZScore,  // Reuse from price features
}

impl LogZScoreNormalizer {
    fn update(&mut self, value: f64) -> f64 {
        // Log transform
        let log_val = if value > 0.0 {
            (value * self.scale_factor).ln()
        } else {
            -10.0  // Minimum sentinel for zero/negative
        };

        // Z-score normalization
        self.zscore.update(log_val)
    }
}

NaN/Inf Handling:

• Zero values: Map to -10.0 after log (extreme negative, clipped to -3σ)
• Negative values: Map to -10.0 (shouldn't happen for spread/illiquidity)
• Inf values: Clamp to ±3σ after z-score

---

5. Time Features (Indices 165-174 in 256-dim vector)

Features: Hour, day of week, market hours, session indicators

Normalization Method: Cyclical Encoding (already normalized)

Implementation: No additional normalization needed

Current Encoding (from extraction.rs lines 640-654):

• Hour: Normalized to [0, 1] via hour / 24.0
• Day of week: Normalized to [0, 1] via weekday / 6.0
• Market hours: Binary {0, 1} indicators
• Minutes since open/close: Normalized to [0, 1] via division by 420 (7 hours)

Validation: Ensure all time features ∈ [0, 1]

NaN/Inf Handling: Not applicable (time features always valid)

---

6. Statistical Features (Indices 175-255 in 256-dim vector)

Features: Rolling mean/std/percentiles, autocorrelations, skewness, kurtosis, volatility

Normalization Method: Mixed Approach

6a. Z-Scores (already normalized, lines 672-678)

• Features: Z-scores relative to rolling windows
• No additional normalization needed (already mean=0, std=1)

6b. Percentile Ranks (already normalized, lines 674)

• Features: Percentile rank features
• No additional normalization needed (already [0, 1])

6c. Correlations (already normalized, lines 686-780)

• Features: Autocorrelations, cross-correlations
• No additional normalization needed (correlations ∈ [-1, 1])

6d. Skewness/Kurtosis (lines 696-713)

• Normalization Method: Clipping to [-3, 3]
• Already implemented (line 1244, 1260)

6e. Volatility (lines 740-752)

• Normalization Method: Log Transform + Z-Score (similar to microstructure)
• Rationale: Volatility is non-negative and skewed

Implementation: Statistical features are already well-normalized in extraction.rs

---

🔄 Online/Incremental Normalization Architecture

Design Pattern: Stateful Normalizer per Feature Category

pub struct FeatureNormalizer {
    // Price features (60 features: 15-74)
    price_normalizers: Vec<RollingZScore>,

    // Volume features (40 features: 75-114)
    volume_normalizers: Vec<RollingPercentileRank>,

    // Microstructure features (50 features: 115-164)
    microstructure_normalizers: Vec<LogZScoreNormalizer>,

    // No normalizers needed for:
    // - OHLCV (indices 0-4): Already log returns / normalized
    // - Technical indicators (5-14): Already normalized
    // - Time features (165-174): Already cyclical encoded
    // - Statistical features (175-255): Already normalized
}

impl FeatureNormalizer {
    pub fn new() -> Self {
        Self {
            price_normalizers: (0..60).map(|_| RollingZScore::new(50)).collect(),
            volume_normalizers: (0..40).map(|_| RollingPercentileRank::new(50)).collect(),
            microstructure_normalizers: vec![
                LogZScoreNormalizer::new(1.0, 20),      // Roll spread
                LogZScoreNormalizer::new(1e8, 20),      // Amihud illiquidity
                LogZScoreNormalizer::new(100.0, 20),    // Corwin-Schultz
                // ... 47 more microstructure features
            ],
        }
    }

    pub fn normalize(&mut self, features: &mut [f64; 256]) -> Result<()> {
        // 1. Validate input (no NaN/Inf)
        for (i, &val) in features.iter().enumerate() {
            if !val.is_finite() {
                // Option A: Skip normalization, return error
                anyhow::bail!("Feature {} is non-finite: {}", i, val);

                // Option B: Impute with neutral value (safer for production)
                // features[i] = 0.0;
            }
        }

        // 2. Normalize OHLCV (indices 0-4)
        // ALREADY NORMALIZED (log returns, safe_normalize)

        // 3. Normalize Technical Indicators (indices 5-14)
        // ALREADY NORMALIZED (0-1 or -1 to +1 ranges)

        // 4. Normalize Price Patterns (indices 15-74)
        for i in 15..75 {
            let idx = i - 15;
            features[i] = self.price_normalizers[idx].update(features[i]);
        }

        // 5. Normalize Volume Patterns (indices 75-114)
        for i in 75..115 {
            let idx = i - 75;
            features[i] = self.volume_normalizers[idx].update(features[i]);
        }

        // 6. Normalize Microstructure (indices 115-164)
        for i in 115..165 {
            let idx = i - 115;
            features[i] = self.microstructure_normalizers[idx].update(features[i]);
        }

        // 7. Time features (165-174): ALREADY NORMALIZED

        // 8. Statistical features (175-255): ALREADY NORMALIZED

        // 9. Final validation
        for (i, &val) in features.iter().enumerate() {
            if !val.is_finite() {
                anyhow::bail!("Normalized feature {} is non-finite: {}", i, val);
            }
        }

        Ok(())
    }

    pub fn reset(&mut self) {
        // Reset all normalizers (useful for backtesting)
        for norm in &mut self.price_normalizers {
            norm.reset();
        }
        for norm in &mut self.volume_normalizers {
            norm.reset();
        }
        for norm in &mut self.microstructure_normalizers {
            norm.reset();
        }
    }
}

---

🪟 Rolling Window Strategies

Window Size Selection Criteria

Trade-offs:

• Small windows (10-20 bars): Fast adaptation to regime changes, more noise
• Medium windows (50 bars): Balance responsiveness vs stability
• Large windows (200+ bars): Stable statistics, slow adaptation

Recommended Sizes:

Feature Category	Window Size	Rationale
Price features	50 bars	Balances intraday regime changes vs stability
Volume features	50 bars	Consistent with price (same market regime)
Microstructure	20 bars	Faster adaptation for liquidity regime changes
Statistical features	5-50 bars	Already handled in feature extraction

Multi-Regime Approach (Optional Enhancement)

For adaptive normalization across market regimes:

pub struct AdaptiveNormalizer {
    fast: RollingZScore,   // 20 bars
    medium: RollingZScore, // 50 bars
    slow: RollingZScore,   // 260 bars
    regime: MarketRegime,  // High/Medium/Low volatility
}

impl AdaptiveNormalizer {
    fn update(&mut self, value: f64) -> f64 {
        // Detect regime based on recent volatility
        let vol = self.compute_volatility();
        self.regime = if vol > 0.05 {
            MarketRegime::HighVolatility
        } else if vol > 0.02 {
            MarketRegime::MediumVolatility
        } else {
            MarketRegime::LowVolatility
        };

        // Use appropriate window size
        match self.regime {
            MarketRegime::HighVolatility => self.fast.update(value),
            MarketRegime::MediumVolatility => self.medium.update(value),
            MarketRegime::LowVolatility => self.slow.update(value),
        }
    }
}

Recommendation: Start with fixed 50-bar windows, add adaptive logic if backtesting shows regime-specific performance

---

🛡️ NaN/Inf Handling Strategy

Input Validation (Pre-Normalization)

Strategy: Imputation with Last Valid Value

pub struct NaNHandler {
    last_valid: [f64; 256],
    nan_count: [u32; 256],
}

impl NaNHandler {
    fn handle_input(&mut self, features: &mut [f64; 256]) {
        for (i, val) in features.iter_mut().enumerate() {
            if !val.is_finite() {
                // Impute with last valid value
                *val = self.last_valid[i];
                self.nan_count[i] += 1;

                // Log warning if excessive NaNs
                if self.nan_count[i] % 100 == 0 {
                    warn!("Feature {} has {} NaN occurrences", i, self.nan_count[i]);
                }
            } else {
                // Update last valid value
                self.last_valid[i] = *val;
                self.nan_count[i] = 0;  // Reset counter
            }
        }
    }
}

Rationale:

• Filtering (removing bars with NaN) → Data loss, training gaps
• Zero imputation → Bias toward zero, incorrect signal
• Last valid value → Preserves continuity, minimal distortion

Output Validation (Post-Normalization)

Strategy: Assert + Error

fn validate_normalized_features(features: &[f64; 256]) -> Result<()> {
    for (i, &val) in features.iter().enumerate() {
        if !val.is_finite() {
            anyhow::bail!("Normalized feature {} is non-finite: {}", i, val);
        }
    }
    Ok(())
}

Rationale: If normalization produces NaN/Inf, it indicates a bug in the normalizer → Fail fast

---

✂️ Feature Clipping (Outlier Handling)

Z-Score Clipping: ±3σ

Rationale:

• 99.7% of Gaussian data falls within ±3σ
• Outliers beyond ±3σ are likely errors or extreme events
• Clipping prevents ML model saturation from rare extreme values

Implementation: Already integrated in RollingZScore normalizer (line 28)

normalized.clamp(-3.0, 3.0)

Percentile Clipping: [0, 1]

Rationale: Percentile rank is naturally bounded to [0, 1]

Implementation: Already integrated in RollingPercentileRank normalizer

normalized.clamp(0.0, 1.0)

Technical Indicator Validation

Rationale: Technical indicators should never exceed design ranges

Implementation:

// Assert RSI ∈ [0, 1]
debug_assert!(features[23] >= 0.0 && features[23] <= 1.0, "RSI out of range");

// Assert MACD ∈ [-1, 1] (tanh normalized)
debug_assert!(features[24] >= -1.0 && features[24] <= 1.0, "MACD out of range");

---

📈 Performance Optimization

Memory Efficiency

Target: <2KB per symbol

Breakdown:

• Price normalizers (60 × 50 values): 60 × 50 × 8 bytes = 24KB (exceeds target)
• Optimization: Use online algorithms (Welford's) instead of storing full window

Optimized Memory:

• RollingZScore: 3 × f64 (24 bytes) + VecDeque header
• RollingPercentileRank: 50 × f64 (400 bytes) + VecDeque header
• LogZScoreNormalizer: 1 × f64 + RollingZScore (32 bytes)

Total Memory:

• Price normalizers: 60 × 24 bytes = 1,440 bytes
• Volume normalizers: 40 × 400 bytes = 16,000 bytes ⚠️ EXCEEDS TARGET

Solution: Use approximate percentile rank with fixed-size sorted buffer (10-20 values) instead of full 50-value window

Latency Optimization

Target: <10μs per 256-feature normalization

Current Estimate:

• OHLCV (5 features): 0μs (already normalized)
• Technical indicators (10 features): 0μs (already normalized)
• Price features (60 features): 60 × 0.1μs = 6μs
• Volume features (40 features): 40 × 0.5μs = 20μs ⚠️ EXCEEDS TARGET
• Microstructure (50 features): 50 × 0.2μs = 10μs ⚠️ EXCEEDS TARGET
• Time features (10 features): 0μs (already normalized)
• Statistical features (81 features): 0μs (already normalized)

Total: 36μs (exceeds 10μs target)

Optimization:

1. Reduce percentile rank complexity: Use approximate rank (sorted buffer)
2. SIMD vectorization: Process 4-8 features in parallel
3. Skip already-normalized features: Don't iterate over indices 5-14, 165-255

Revised Estimate:

• Price features: 60 × 0.05μs = 3μs (SIMD)
• Volume features: 40 × 0.1μs = 4μs (approximate rank)
• Microstructure: 50 × 0.1μs = 5μs (SIMD)
• Total: 12μs ⚠️ Still slightly over target

Final Optimization: Lazy normalization (normalize on-demand, cache results)

---

🧪 Testing Strategy

Unit Tests

1. RollingZScore: Verify mean=0, std=1 after warmup (50 bars)
2. RollingPercentileRank: Verify output ∈ [0, 1], monotonic with rank
3. LogZScoreNormalizer: Verify log transform + z-score correctness
4. NaN Handling: Verify last-valid-value imputation
5. Clipping: Verify ±3σ bounds enforced

Integration Tests

1. End-to-End Pipeline: Raw bars → Extraction → Normalization → Validation
2. Batch vs Online: Compare online normalization vs batch normalization (after warmup)
3. Performance Benchmark: Measure latency (<10μs target)
4. Memory Benchmark: Measure memory usage (<2KB target)

Stress Tests

1. Extreme Values: Test with price spikes, volume surges, zero volume
2. NaN Injection: Inject NaN at random indices, verify no propagation
3. Long Sequences: Test with 10,000+ bars, verify no memory leaks
4. Regime Changes: Test with volatile → calm → volatile transitions

---

🚀 Implementation Plan (Wave C)

Phase 1: Core Normalizers (1-2 days)

1. ✅ Design specification (this document)
2. ⏳ Implement RollingZScore with Welford's algorithm
3. ⏳ Implement RollingPercentileRank with approximate rank
4. ⏳ Implement LogZScoreNormalizer
5. ⏳ Unit tests (15 tests)

Phase 2: Integration (1 day)

1. ⏳ Implement FeatureNormalizer wrapper
2. ⏳ Integrate with extract_ml_features() function
3. ⏳ Add NaN handling (NaNHandler)
4. ⏳ Integration tests (6 tests)

Phase 3: Optimization (1 day)

1. ⏳ SIMD vectorization for z-score computation
2. ⏳ Approximate percentile rank algorithm
3. ⏳ Memory profiling (<2KB per symbol)
4. ⏳ Latency benchmarking (<10μs target)

Phase 4: Validation (1 day)

1. ⏳ Backtest with ES.FUT/NQ.FUT (win rate comparison)
2. ⏳ Compare online vs batch normalization (accuracy within 1%)
3. ⏳ Stress testing (NaN injection, extreme values)
4. ⏳ Production readiness checklist

Total Estimated Time: 4-5 days

---

📝 Configuration File

Create normalization_config.yaml for tunable parameters:

normalization:
  # Rolling window sizes
  windows:
    price_features: 50
    volume_features: 50
    microstructure_features: 20

  # Clipping thresholds
  clipping:
    z_score_sigma: 3.0  # ±3σ
    percentile_min: 0.0
    percentile_max: 1.0

  # NaN handling
  nan_handling:
    strategy: "last_valid_value"  # Options: last_valid_value, zero, median
    warning_threshold: 100  # Warn after N consecutive NaNs

  # Microstructure scale factors
  microstructure_scales:
    roll_spread: 1.0
    amihud_illiquidity: 1.0e8
    corwin_schultz_spread: 100.0

  # Performance
  performance:
    max_latency_us: 10
    max_memory_bytes: 2048

---

🔬 Alternative Approaches Considered

1. MinMax Normalization (rejected)

normalized = (value - min) / (max - min)

• Pros: Simple, bounded [0, 1]
• Cons: Sensitive to outliers, not suitable for streaming data
• Reason rejected: HFT data has frequent outliers (flash crashes, fat-finger trades)

2. Robust Scaling (considered for future)

normalized = (value - median) / (Q3 - Q1)

• Pros: Robust to outliers, uses IQR instead of std
• Cons: Higher computational cost for online median/IQR
• Reason deferred: Good alternative if z-score proves unstable

3. Batch Normalization (rejected for online)

normalized = (value - batch_mean) / (batch_std + epsilon)

• Pros: Standard in deep learning, proven effective
• Cons: Requires full batch (incompatible with streaming)
• Reason rejected: HFT requires online/incremental processing

---

📚 References

1. Welford's Online Algorithm (1962): Numerically stable variance computation

   • Paper: "Note on a method for calculating corrected sums of squares and products"
   • Used in: RollingZScore implementation

2. Lopez de Prado (2018): "Advances in Financial Machine Learning"

   • Chapter 20: Feature Engineering for ML
   • Emphasis on stationarity and normalization

3. MLFinLab Documentation: Feature Engineering Best Practices

   • https://mlfinlab.readthedocs.io/en/latest/feature_engineering/feature_engineering.html

4. Wave B: Alternative Bar Sampling (WAVE_B_COMPLETION_SUMMARY.md)

   • Tick, dollar, volume, imbalance, run bars
   • EWMA threshold adaptation

5. Wave 19: Feature Index Map (WAVE_19_FEATURE_INDEX_MAP.md)

   • 26-dimension feature vector specification
   • Technical indicator ranges

---

✅ Acceptance Criteria

Functional Requirements

• ✅ Z-score normalization for price features (mean=0, std=1)
• ✅ Percentile rank normalization for volume features (0-1)
• ✅ Log-transform + z-score for microstructure features
• ✅ No additional normalization for technical indicators (already normalized)
• ✅ No additional normalization for time features (cyclical encoding)
• ✅ No additional normalization for statistical features (already normalized)

Non-Functional Requirements

• ✅ Online/incremental updates (no batch recomputation)
• ✅ Latency: <10μs per 256-feature normalization
• ✅ Memory: <2KB per symbol (rolling statistics)
• ✅ Stability: No NaN/Inf in output features
• ✅ Accuracy: <1% error vs batch normalization after warmup

Testing Requirements

• ✅ 15+ unit tests (normalizers)
• ✅ 6+ integration tests (end-to-end pipeline)
• ✅ Performance benchmarks (latency, memory)
• ✅ Stress tests (NaN injection, extreme values)

---

Last Updated: 2025-10-17 Status: 📋 DESIGN COMPLETE (ready for implementation) Next Milestone: Phase 1 implementation (core normalizers) Production Readiness: 0% (design only)