foxhunt/WAVE_C_NORMALIZATION_SUMMARY.md

Wave C: Feature Normalization Strategy - Executive Summary

Date: 2025-10-17 Mission: Design production-ready normalization pipeline for 256-dimension ML features Status: ✅ DESIGN COMPLETE (ready for implementation)

---

🎯 Overview

Comprehensive normalization strategy designed for online/incremental processing of streaming HFT data. Ensures ML models receive stable, normalized features without batch recomputation overhead.

Key Design Principles:

1. Online algorithms: No batch recomputation (streaming-compatible)
2. Category-specific methods: Tailored to each feature type
3. Robust to outliers: ±3σ clipping, percentile ranks
4. Production-grade: <10μs latency, <2KB memory per symbol

---

📊 Normalization Methods by Category

1. Price Features (60 features: indices 15-74)

Method: Z-Score Normalization (mean=0, std=1)

normalized = (value - rolling_mean) / (rolling_std + epsilon)
clipped = normalized.clamp(-3.0, 3.0)

• Window: 50 bars (balances responsiveness vs stability)
• Algorithm: Welford's online algorithm (O(1) memory)
• Rationale: Price features are unbounded and Gaussian-distributed

2. Volume Features (40 features: indices 75-114)

Method: Percentile Rank Normalization (0-1)

normalized = rank(value) / total_count

• Window: 50 bars
• Algorithm: Sorted buffer with approximate rank (O(log n))
• Rationale: Volume is highly skewed (log-normal), percentile rank is robust to outliers

3. Technical Indicators (10 features: indices 5-14)

Method: None (already normalized)

• RSI, Stochastic, ADX: Already [0, 1]
• MACD, Bollinger, CCI: Already [-1, 1] via tanh
• No additional normalization needed

4. Microstructure Features (50 features: indices 115-164)

Method: Log Transform + Z-Score

log_value = (value * scale_factor).ln()
normalized = (log_value - rolling_mean) / (rolling_std + epsilon)
clipped = normalized.clamp(-3.0, 3.0)

• Window: 20 bars (faster adaptation for liquidity regime changes)
• Scale Factors:
  • Roll spread: 1.0
  • Amihud illiquidity: 1e8
  • Corwin-Schultz: 100.0
• Rationale: Microstructure features are highly skewed (log-normal)

5. Time Features (10 features: indices 165-174)

Method: None (already cyclical encoded)

• Hour, day: Already normalized to [0, 1]
• Market hours: Binary indicators {0, 1}

6. Statistical Features (81 features: indices 175-255)

Method: None (already normalized)

• Z-scores: Already mean=0, std=1
• Percentile ranks: Already [0, 1]
• Correlations: Already [-1, 1]

---

🔄 Online/Incremental Architecture

Core Design Pattern

pub struct FeatureNormalizer {
    price_normalizers: Vec<RollingZScore>,           // 60 normalizers
    volume_normalizers: Vec<RollingPercentileRank>,  // 40 normalizers
    microstructure_normalizers: Vec<LogZScoreNormalizer>, // 50 normalizers
}

impl FeatureNormalizer {
    pub fn normalize(&mut self, features: &mut [f64; 256]) -> Result<()> {
        // 1. Validate input (no NaN/Inf)
        // 2. Normalize price features (15-74)
        // 3. Normalize volume features (75-114)
        // 4. Normalize microstructure features (115-164)
        // 5. Skip already-normalized: OHLCV, technical, time, statistical
        // 6. Final validation
    }
}

Key Components

RollingZScore (Welford's Algorithm)

struct RollingZScore {
    window_size: usize,
    values: VecDeque<f64>,
    mean: f64,
    m2: f64,  // Sum of squared deviations
    count: usize,
}
// Memory: 24 bytes (3 × f64)
// Latency: <0.1μs per update

RollingPercentileRank

struct RollingPercentileRank {
    window_size: usize,
    values: VecDeque<f64>,
}
// Memory: 400 bytes (50 × f64)
// Latency: <0.5μs per update (approximate rank)

LogZScoreNormalizer

struct LogZScoreNormalizer {
    scale_factor: f64,
    zscore: RollingZScore,
}
// Memory: 32 bytes
// Latency: <0.2μs per update

---

🪟 Rolling Window Sizes

Feature Category	Window Size	Rationale
Price features	50 bars	Balances intraday regime changes vs stability
Volume features	50 bars	Consistent with price (same market regime)
Microstructure	20 bars	Faster adaptation for liquidity regime changes
Statistical	5-50 bars	Already handled in feature extraction

Trade-offs:

• Small windows (10-20): Fast regime adaptation, more noise
• Medium windows (50): Balance responsiveness vs stability ✅ RECOMMENDED
• Large windows (200+): Stable statistics, slow adaptation

---

🛡️ NaN/Inf Handling Strategy

Input Validation (Pre-Normalization)

Strategy: Last Valid Value Imputation

if !val.is_finite() {
    *val = self.last_valid[i];  // Use last valid value
    self.nan_count[i] += 1;     // Track occurrences
}

Rationale: Preserves continuity, minimal distortion (vs zero imputation or filtering)

Output Validation (Post-Normalization)

Strategy: Assert + Error

for (i, &val) in features.iter().enumerate() {
    if !val.is_finite() {
        anyhow::bail!("Normalized feature {} is non-finite: {}", i, val);
    }
}

Rationale: Fail-fast on normalization bugs

Edge Cases

• Zero volume: Map to 0.0 percentile (minimum)
• Zero price: Use last valid price
• Division by zero: Add epsilon (1e-8)
• Log of zero/negative: Map to -10.0 (extreme negative, clipped to -3σ)

---

✂️ Outlier Clipping

Z-Score Clipping: ±3σ

normalized.clamp(-3.0, 3.0)

• Rationale: 99.7% of Gaussian data within ±3σ
• Prevents: ML model saturation from extreme events

Percentile Clipping: [0, 1]

normalized.clamp(0.0, 1.0)

• Rationale: Percentile rank naturally bounded

Technical Indicator Validation

debug_assert!(features[23] >= 0.0 && features[23] <= 1.0, "RSI out of range");

• Rationale: Indicators should never exceed design ranges

---

📈 Performance Targets

Latency

• Target: <10μs per 256-feature normalization
• Breakdown:
  • Price features (60): 3μs (SIMD)
  • Volume features (40): 4μs (approximate rank)
  • Microstructure (50): 5μs (SIMD)
  • Total: 12μs ⚠️ Slightly over target
• Optimization: Lazy normalization (normalize on-demand)

Memory

• Target: <2KB per symbol
• Breakdown:
  • Price normalizers (60): 1,440 bytes
  • Volume normalizers (40): 16,000 bytes ⚠️ Exceeds target
  • Microstructure normalizers (50): 1,600 bytes
• Optimization: Approximate percentile rank (reduce to 10-20 values instead of 50)

---

🧪 Testing Strategy

Unit Tests (15 tests)

1. RollingZScore: Verify mean=0, std=1 after warmup
2. RollingPercentileRank: Verify output ∈ [0, 1], monotonic
3. LogZScoreNormalizer: Verify log + z-score correctness
4. NaN Handling: Verify last-valid-value imputation
5. Clipping: Verify ±3σ bounds enforced

Integration Tests (6 tests)

1. E2E Pipeline: Raw bars → Extraction → Normalization → Validation
2. Batch vs Online: Compare online vs batch (accuracy within 1%)
3. Performance: Measure latency (<10μs)
4. Memory: Measure memory (<2KB)

Stress Tests (3 tests)

1. Extreme Values: Price spikes, volume surges, zero volume
2. NaN Injection: Random NaN insertion, verify no propagation
3. Regime Changes: Volatile → calm → volatile transitions

---

🚀 Implementation Plan (4-5 days)

Phase 1: Core Normalizers (1-2 days)

• Implement RollingZScore with Welford's algorithm
• Implement RollingPercentileRank with approximate rank
• Implement LogZScoreNormalizer
• Unit tests (15 tests)

Phase 2: Integration (1 day)

• Implement FeatureNormalizer wrapper
• Integrate with extract_ml_features()
• Add NaNHandler
• Integration tests (6 tests)

Phase 3: Optimization (1 day)

• SIMD vectorization for z-score
• Approximate percentile rank algorithm
• Memory profiling (<2KB)
• Latency benchmarking (<10μs)

Phase 4: Validation (1 day)

• Backtest with ES.FUT/NQ.FUT
• Online vs batch accuracy comparison
• Stress testing
• Production readiness checklist

---

📋 Configuration

Create normalization_config.yaml:

normalization:
  windows:
    price_features: 50
    volume_features: 50
    microstructure_features: 20
  clipping:
    z_score_sigma: 3.0
    percentile_min: 0.0
    percentile_max: 1.0
  nan_handling:
    strategy: "last_valid_value"
    warning_threshold: 100
  microstructure_scales:
    roll_spread: 1.0
    amihud_illiquidity: 1.0e8
    corwin_schultz_spread: 100.0

---

✅ Acceptance Criteria

Functional Requirements

• ✅ Z-score normalization for price features
• ✅ Percentile rank for volume features
• ✅ Log-transform + z-score for microstructure
• ✅ Skip already-normalized features (technical, time, statistical)
• ✅ NaN/Inf handling (last-valid-value imputation)
• ✅ ±3σ outlier clipping

Non-Functional Requirements

• ✅ Online/incremental updates (no batch)
• ✅ Latency: <10μs per 256 features (12μs estimated, optimization needed)
• ✅ Memory: <2KB per symbol (16KB estimated, optimization needed)
• ✅ Stability: No NaN/Inf in output
• ✅ Accuracy: <1% error vs batch after warmup

Testing Requirements

• ✅ 15+ unit tests
• ✅ 6+ integration tests
• ✅ Performance benchmarks
• ✅ Stress tests

---

🔗 References

1. Full Design Document: WAVE_C_FEATURE_NORMALIZATION_DESIGN.md (2,500+ lines)
2. Wave B: Alternative Bar Sampling (WAVE_B_COMPLETION_SUMMARY.md)
3. Wave 19: Feature Index Map (WAVE_19_FEATURE_INDEX_MAP.md)
4. Feature Extraction: ml/src/features/extraction.rs (1,538 lines)
5. Welford's Algorithm (1962): Online variance computation

---

Last Updated: 2025-10-17 Status: ✅ DESIGN COMPLETE (ready for implementation) Next Milestone: Phase 1 implementation (core normalizers) Estimated Timeline: 4-5 days to production-ready implementation