foxhunt/FEATURE_CACHE_QUICK_REFERENCE.md

Feature Cache Quick Reference

TDD Mission: Pre-compute 256-dim ML features → 10x faster training startup

---

🎯 Status

RED Phase: ✅ COMPLETE - 13 tests written, all failing GREEN Phase: ⏳ PENDING - Implementation needed Performance: Target <100ms cache load (vs ~1000ms re-computation)

---

📁 Key Files

# Test Suite (RED phase complete)
ml/tests/feature_cache_tests.rs           # 13 tests, 365 lines

# Documentation
AGENT_163_FEATURE_CACHE_TDD.md            # 498 lines, full spec
AGENT_163_SUMMARY.md                      # 164 lines, summary
FEATURE_CACHE_QUICK_REFERENCE.md          # This file

# Implementation (NOT YET CREATED)
ml/src/feature_cache/mod.rs               # Module exports
ml/src/feature_cache/cache.rs             # Main API
ml/src/feature_cache/feature_extractor.rs # 256-dim extraction
ml/src/feature_cache/parquet_writer.rs    # Parquet I/O
ml/src/feature_cache/minio_storage.rs     # MinIO S3
ml/src/feature_cache/invalidation.rs      # Cache logic

---

🧪 Test Categories

Feature Extraction (2 tests)

• Extract 256-dim vectors from OHLCV
• Validate dimensions (5 OHLCV + 10 indicators + 241 engineered)

Parquet Serialization (3 tests)

• Write features to Parquet
• Read features from Parquet
• Roundtrip validation

MinIO Storage (3 tests)

• Upload to MinIO (S3-compatible)
• Download from MinIO
• List cached symbols

Cache Logic (5 tests)

• Cache invalidation (SHA256 hash)
• Cache hit/miss detection
• Metadata storage
• Performance benchmark (10x speedup)
• Batch parallel loading

---

🏗️ Architecture

OHLCVBar → FeatureExtractor → 256-dim Vector → ParquetWriter → MinIO
                                                        ↓
                                                   Cache Hit!
                                                   <100ms load

Feature Composition (256 dimensions)

Category	Count	Examples
OHLCV	5	Open, High, Low, Close, Volume
Indicators	10	RSI, MACD, Bollinger, ATR, EMA
Price Patterns	60	Candlesticks, gaps, reversals
Volume Patterns	40	Spikes, divergence, distribution
Momentum	50	Rate of change, momentum oscillators
Volatility	40	Historical vol, regimes, ranges
Microstructure	51	Bid-ask proxies, order flow

Total: 256 dimensions (power of 2 for GPU efficiency)

---

🚀 Usage (After Implementation)

use ml::feature_cache::FeatureCacheService;
use ml::real_data_loader::RealDataLoader;

// Initialize
let cache = FeatureCacheService::new().await?;
let mut loader = RealDataLoader::new_from_workspace()?;

// Load data
let bars = loader.load_symbol_data("ZN.FUT").await?;

// Get features (cached or compute)
let features = cache.get_or_compute_features("ZN.FUT", &bars).await?;
// ✅ <100ms if cached, ~1000ms if not

// Batch load (parallel)
let symbols = vec!["ZN.FUT", "6E.FUT", "ES.FUT"];
let all_features = cache.load_batch_cached(symbols).await?;
// ✅ <500ms for 10 symbols

---

🔧 Implementation Order

1. Feature Extractor → Tests 1-2 GREEN
2. Parquet Writer → Tests 3-5 GREEN
3. MinIO Storage → Tests 6-8 GREEN (requires Docker)
4. Cache Service → Tests 9-13 GREEN

---

📊 Performance Targets

Operation	Target	Baseline	Speedup
Cache Load	<100ms	~1000ms	10x
Feature Extract	Cached	~1000ms	Eliminated
Parquet Read	<50ms	N/A	Streaming
Batch (10 symbols)	<500ms	N/A	Parallel

---

🔐 Cache Invalidation

Method: SHA256 hash of OHLCV data

// Compute hash
let hash = sha256(&ohlcv_bars);

// Check cache validity
if cached_hash == hash {
    // Cache HIT → load from Parquet (<100ms)
    return load_from_cache(symbol);
} else {
    // Cache MISS → re-compute + cache
    let features = extract_features(bars);
    cache_features(symbol, features, hash);
    return features;
}

---

🐳 MinIO Setup

# Start MinIO (Docker)
docker run -p 9000:9000 minio/minio server /data

# Access MinIO Console
open http://localhost:9000
# Default: minioadmin / minioadmin

# Bucket: ml-feature-cache
# Key Pattern: {symbol}/features.parquet
# Example: ZN.FUT/features.parquet

---

🧪 Run Tests

# All feature cache tests (expect FAILURES until implemented)
cargo test -p ml --test feature_cache_tests -- --nocapture

# Single test
cargo test -p ml --test feature_cache_tests::test_extract_256_dim_features

# Watch mode
cargo watch -x 'test -p ml --test feature_cache_tests'

---

📈 Success Criteria

• ✅ Test Coverage: 13/13 passing (100%)
• ✅ Performance: <100ms cache load (10x speedup)
• ✅ Features: 256-dim vectors (normalized 0-1)
• ✅ Storage: Parquet in MinIO (S3-compatible)
• ✅ Invalidation: SHA256 hash-based

---

🔄 Next Actions

1. Run tests: cargo test -p ml --test feature_cache_tests (expect RED)
2. Implement feature_extractor.rs (Tests 1-2 GREEN)
3. Implement parquet_writer.rs (Tests 3-5 GREEN)
4. Start MinIO Docker
5. Implement minio_storage.rs (Tests 6-8 GREEN)
6. Implement cache.rs + invalidation.rs (Tests 9-13 GREEN)
7. Celebrate 100% GREEN tests!

---

Impact: 10x faster ML training startup (from ~10 seconds to ~1 second)