foxhunt/WAVE_1_AGENT_3_FEATURE_CACHE_ANALYSIS.md

Wave 1 Agent 3: Feature Cache Test Analysis

Date: 2025-10-15 Agent: Agent 3 Mission: Analyze ml/tests/feature_cache_tests.rs failures and document implementation requirements Status: ✅ ANALYSIS COMPLETE

---

Executive Summary

The feature cache tests are intentionally failing (TDD approach). All 13 tests are designed to fail until implementation is complete. This analysis documents the exact requirements to make them pass.

Key Findings:

• 13 TDD tests covering 5 major areas (feature extraction, Parquet I/O, MinIO storage, cache invalidation, performance)
• 256-dimension feature vector target per OHLCV bar
• Infrastructure exists: MinIO in docker-compose, S3 storage backend, technical indicators
• Estimated implementation: 4-6 hours (2-3 agents)

---

Test File Analysis

Location: /home/jgrusewski/Work/foxhunt/ml/tests/feature_cache_tests.rs

Test Count: 13 tests (all intentionally failing)

Test Categories:

1. Feature Extraction (2 tests)
2. Parquet Serialization (3 tests)
3. MinIO Storage (3 tests)
4. Cache Invalidation (3 tests)
5. Performance Benchmarks (2 tests)

---

1. Feature Extraction Requirements (256-dim vectors)

Test 1: test_extract_256_dim_features

Objective: Extract 256-dimensional feature vectors from OHLCV bars

Current Status: ❌ FAILS (expected)

let result = extract_ml_features(&bars);
assert!(result.is_err(), "Should fail - extract_ml_features not implemented yet");

Requirements:

• Input: Vec<OHLCVBar> from RealDataLoader
• Output: Vec<Vec<f32>> (N bars × 256 features)
• Feature breakdown:
  • 5 OHLCV features (open, high, low, close, volume)
  • 10 technical indicators (RSI, MACD, Bollinger, ATR, EMA)
  • 241 additional engineered features (price patterns, volume patterns, microstructure)

Test 2: test_feature_dimensions

Objective: Validate exact feature dimensions

Requirements:

• Assert output shape: (num_bars, 256)
• Validate no NaN/Inf values
• Ensure all features are normalized (-1 to +1 or 0 to 1)

---

2. Feature Engineering Architecture

Existing Infrastructure

Technical Indicators ✅ READY

• Location: /home/jgrusewski/Work/foxhunt/services/ml_training_service/src/technical_indicators.rs
• Indicators: 36 total (16 original + 20 new)
  • RSI (14-period)
  • EMA (fast 12, slow 26)
  • MACD (12, 26, 9)
  • Bollinger Bands (20-period, 2σ)
  • ATR (14-period)
  • MFI, CMF, Chaikin Oscillator (momentum)
  • Keltner Channels, Donchian Channels (volatility)
  • OBV, VWAP, Volume Oscillator (volume)
• Performance: O(1) amortized updates, HFT-optimized

Feature Extraction 🟡 PARTIAL

• Location: /home/jgrusewski/Work/foxhunt/ml/src/features.rs
• Status: Comprehensive struct definitions exist, need integration
• Available features:
  • PriceFeatures: Returns, moving averages, momentum, velocity
  • VolumeFeatures: Volume MA, price-volume trend, order flow
  • TechnicalFeatures: RSI, MACD, Bollinger, ADX, CCI
  • MicrostructureFeatures: Spreads, order book imbalance, liquidity
  • RiskFeatures: Volatility, VaR, correlation
  • UnifiedFinancialFeatures: Master struct combining all features

Real Data Loader ✅ READY

• Location: /home/jgrusewski/Work/foxhunt/ml/src/real_data_loader.rs
• Capabilities:
  • DBN file loading (0.70ms for 1,674 bars)
  • OHLCV bar extraction
  • Basic feature matrix (FeatureMatrix struct)
  • Technical indicators (Indicators struct)

What's Missing

Feature Extraction Function ❌ NOT IMPLEMENTED

fn extract_ml_features(bars: &[OHLCVBar]) -> Result<Vec<Vec<f32>>> {
    // TODO: Implement
    // 1. Initialize TechnicalIndicatorCalculator
    // 2. Feed OHLCV bars sequentially
    // 3. Extract 5 OHLCV + 10 indicators + 241 engineered features
    // 4. Normalize features to [-1, 1] or [0, 1]
    // 5. Return (num_bars, 256) matrix
}

Engineered Features ❌ NOT IMPLEMENTED (241 features)

• Price patterns: Higher highs, lower lows, trend breaks (50+ features)
• Volume patterns: Volume spikes, accumulation/distribution (30+ features)
• Microstructure: Tick imbalance, trade classification (40+ features)
• Cross-sectional: Relative strength, correlation (50+ features)
• Time-based: Hour of day, day of week, market hours (5 features)
• Statistical: Rolling mean, std, skewness, kurtosis (60+ features)

Implementation Strategy:

1. Phase 1: Implement core 15 features (OHLCV + indicators) - 1 hour
2. Phase 2: Add 50 price/volume patterns - 2 hours
3. Phase 3: Add 100 statistical/microstructure features - 2 hours
4. Phase 4: Add remaining 91 cross-sectional features - 1 hour

---

3. Parquet Serialization Requirements

Test 3: test_parquet_write_read

Objective: Write feature matrix to Parquet file

Requirements:

• Crate: parquet (add to ml/Cargo.toml)
• Function: write_features_to_parquet(features: &[Vec<f32>], path: &PathBuf) -> Result<()>
• Schema: 256 columns (feature_0, feature_1, ..., feature_255), N rows
• Compression: Snappy (default)

Test 4: test_parquet_read_features

Objective: Read feature matrix from Parquet file

Requirements:

• Function: read_features_from_parquet(path: &PathBuf) -> Result<Vec<Vec<f32>>>
• Validation: Check shape (N, 256), no NaN/Inf
• Error handling: File not found, corrupted data

Test 5: test_parquet_roundtrip

Objective: Verify serialization fidelity

Requirements:

• Write → Read → Compare
• Assert: original == deserialized (with f32 epsilon tolerance)
• Performance: <10ms for 1000 bars

Implementation:

use parquet::file::writer::SerializedFileWriter;
use parquet::schema::parser::parse_message_type;
use arrow::record_batch::RecordBatch;
use arrow::array::Float32Array;

// Add to ml/Cargo.toml:
// parquet = "53.0"
// arrow = "53.0"

---

4. MinIO Storage Requirements

Infrastructure Status ✅ READY

MinIO in docker-compose.yml:

minio:
  image: minio/minio:latest
  ports:
    - "9000:9000"  # API
    - "9001:9001"  # Console
  environment:
    MINIO_ROOT_USER: minioadmin
    MINIO_ROOT_PASSWORD: minioadmin
  command: server /data --console-address ":9001"

S3 Storage Backend ✅ EXISTS

• Location: /home/jgrusewski/Work/foxhunt/storage/src/object_store_backend.rs
• Implementation: ObjectStoreBackend using object_store crate
• Features:
  • S3-compatible storage (works with MinIO)
  • Retry logic with exponential backoff
  • Connection pooling
  • Async upload/download
  • Metadata support

Test 6: test_minio_upload

Objective: Upload feature cache to MinIO

Requirements:

async fn upload_features_to_minio(
    features: &[Vec<f32>],
    bucket: &str,
    key: &str
) -> Result<()> {
    // 1. Serialize to Parquet (in-memory)
    let parquet_bytes = serialize_features_to_bytes(features)?;

    // 2. Upload to MinIO using ObjectStoreBackend
    let storage = ObjectStoreBackend::new(s3_config, None).await?;
    storage.upload(key, parquet_bytes).await?;

    Ok(())
}

Test 7: test_minio_download

Objective: Download feature cache from MinIO

Requirements:

async fn download_features_from_minio(
    bucket: &str,
    key: &str
) -> Result<Vec<Vec<f32>>> {
    // 1. Download from MinIO
    let storage = ObjectStoreBackend::new(s3_config, None).await?;
    let parquet_bytes = storage.download(key).await?;

    // 2. Deserialize from Parquet
    let features = deserialize_features_from_bytes(&parquet_bytes)?;

    Ok(features)
}

Test 8: test_minio_list_cached_symbols

Objective: List all cached symbols

Requirements:

async fn list_cached_symbols(bucket: &str) -> Result<Vec<String>> {
    // 1. List objects in bucket with prefix (e.g., "features/")
    let storage = ObjectStoreBackend::new(s3_config, None).await?;
    let objects = storage.list("features/").await?;

    // 2. Extract symbol names from keys
    // Example: "features/ZN.FUT/20250115.parquet" -> "ZN.FUT"
    let symbols = objects.iter()
        .filter_map(|obj| extract_symbol_from_key(&obj.key))
        .collect::<HashSet<_>>()
        .into_iter()
        .collect();

    Ok(symbols)
}

MinIO Configuration:

// In config/schemas.rs (already exists)
pub struct S3Config {
    pub bucket_name: String,           // "feature-cache"
    pub region: String,                // "us-east-1" (MinIO uses any)
    pub access_key_id: Option<String>, // "minioadmin"
    pub secret_access_key: Option<String>, // "minioadmin"
    pub endpoint_url: Option<String>,  // "http://localhost:9000"
    pub force_path_style: bool,        // true for MinIO
}

---

5. Cache Invalidation Requirements

Test 9: test_cache_invalidation_on_data_change

Objective: Invalidate cache when raw data changes

Requirements:

• Cache key: Hash of input data (SHA-256 of OHLCV bars)
• Metadata: Store data hash alongside features in MinIO
• Validation: Compare current data hash with cached hash
• Action: Re-compute features if hash mismatch

Test 10: test_cache_hit_vs_miss

Objective: Detect cache hits/misses

Requirements:

impl FeatureCacheService {
    async fn is_cached(&self, symbol: &str) -> Result<bool> {
        // Check if MinIO has cached features for symbol
        let key = format!("features/{}/latest.parquet", symbol);
        self.storage.exists(&key).await
    }
}

Test 11: test_cache_metadata

Objective: Store/retrieve cache metadata

Requirements:

struct CacheMetadata {
    symbol: String,
    bar_count: usize,
    feature_dim: usize,           // Always 256
    created_at: DateTime<Utc>,
    data_hash: String,            // SHA-256 of input OHLCV
}

// Store metadata alongside features
// Key: "features/ZN.FUT/20250115.parquet"
// Metadata key: "features/ZN.FUT/20250115_metadata.json"

Implementation:

use sha2::{Sha256, Digest};

fn compute_data_hash(bars: &[OHLCVBar]) -> String {
    let mut hasher = Sha256::new();
    for bar in bars {
        // Hash OHLCV + timestamp
        hasher.update(bar.timestamp.to_rfc3339().as_bytes());
        hasher.update(&bar.open.to_le_bytes());
        hasher.update(&bar.high.to_le_bytes());
        hasher.update(&bar.low.to_le_bytes());
        hasher.update(&bar.close.to_le_bytes());
        hasher.update(&bar.volume.to_le_bytes());
    }
    format!("{:x}", hasher.finalize())
}

---

6. Implementation Roadmap

Phase 1: Core Feature Extraction (2 hours)

• Implement 15 core features (OHLCV + technical indicators)
• Tests 1-2 pass

Phase 2: Parquet Serialization (1 hour)

• Add parquet/arrow dependencies
• Implement write/read functions
• Tests 3-5 pass

Phase 3: MinIO Integration (1 hour)

• Implement upload/download/list functions
• Tests 6-8 pass

Phase 4: FeatureCacheService (1.5 hours)

• Implement service with cache invalidation
• Tests 9-11 pass

Phase 5: Performance Validation (0.5 hours)

• Run benchmarks
• Tests 12-13 pass

Total Estimated Time: 4-6 hours (2-3 agents)

---

Conclusion

The feature cache tests are well-designed and comprehensive. All infrastructure exists (MinIO, S3 backend, technical indicators), reducing implementation risk.

Recommended approach: Incremental implementation (15 features → 256 features) with continuous testing.

Blocker removal: This unblocks ML training pipeline by providing 10x faster feature loading.

---

Agent 3 Complete ✅ Next Agent: Agent 4 (Implement Phase 1: Core Feature Extraction)