foxhunt/FEATURE_CACHE_QUICK_REFERENCE.md

# Feature Cache Quick Reference

**TDD Mission**: Pre-compute 256-dim ML features → 10x faster training startup

---

## 🎯 Status

**RED Phase**: ✅ COMPLETE - 13 tests written, all failing
**GREEN Phase**: ⏳ PENDING - Implementation needed
**Performance**: Target <100ms cache load (vs ~1000ms re-computation)

---

## 📁 Key Files

```bash
# Test Suite (RED phase complete)
ml/tests/feature_cache_tests.rs           # 13 tests, 365 lines

# Documentation
AGENT_163_FEATURE_CACHE_TDD.md            # 498 lines, full spec
AGENT_163_SUMMARY.md                      # 164 lines, summary
FEATURE_CACHE_QUICK_REFERENCE.md          # This file

# Implementation (NOT YET CREATED)
ml/src/feature_cache/mod.rs               # Module exports
ml/src/feature_cache/cache.rs             # Main API
ml/src/feature_cache/feature_extractor.rs # 256-dim extraction
ml/src/feature_cache/parquet_writer.rs    # Parquet I/O
ml/src/feature_cache/minio_storage.rs     # MinIO S3
ml/src/feature_cache/invalidation.rs      # Cache logic
```

---

## 🧪 Test Categories

### Feature Extraction (2 tests)
- Extract 256-dim vectors from OHLCV
- Validate dimensions (5 OHLCV + 10 indicators + 241 engineered)

### Parquet Serialization (3 tests)
- Write features to Parquet
- Read features from Parquet
- Roundtrip validation

### MinIO Storage (3 tests)
- Upload to MinIO (S3-compatible)
- Download from MinIO
- List cached symbols

### Cache Logic (5 tests)
- Cache invalidation (SHA256 hash)
- Cache hit/miss detection
- Metadata storage
- Performance benchmark (10x speedup)
- Batch parallel loading

---

## 🏗️ Architecture

```
OHLCVBar → FeatureExtractor → 256-dim Vector → ParquetWriter → MinIO
                                                        ↓
                                                   Cache Hit!
                                                   <100ms load
```

### Feature Composition (256 dimensions)

| Category | Count | Examples |
|----------|-------|----------|
| OHLCV | 5 | Open, High, Low, Close, Volume |
| Indicators | 10 | RSI, MACD, Bollinger, ATR, EMA |
| Price Patterns | 60 | Candlesticks, gaps, reversals |
| Volume Patterns | 40 | Spikes, divergence, distribution |
| Momentum | 50 | Rate of change, momentum oscillators |
| Volatility | 40 | Historical vol, regimes, ranges |
| Microstructure | 51 | Bid-ask proxies, order flow |

**Total**: 256 dimensions (power of 2 for GPU efficiency)

---

## 🚀 Usage (After Implementation)

```rust
use ml::feature_cache::FeatureCacheService;
use ml::real_data_loader::RealDataLoader;

// Initialize
let cache = FeatureCacheService::new().await?;
let mut loader = RealDataLoader::new_from_workspace()?;

// Load data
let bars = loader.load_symbol_data("ZN.FUT").await?;

// Get features (cached or compute)
let features = cache.get_or_compute_features("ZN.FUT", &bars).await?;
// ✅ <100ms if cached, ~1000ms if not

// Batch load (parallel)
let symbols = vec!["ZN.FUT", "6E.FUT", "ES.FUT"];
let all_features = cache.load_batch_cached(symbols).await?;
// ✅ <500ms for 10 symbols
```

---

## 🔧 Implementation Order

1. **Feature Extractor** → Tests 1-2 GREEN
2. **Parquet Writer** → Tests 3-5 GREEN
3. **MinIO Storage** → Tests 6-8 GREEN (requires Docker)
4. **Cache Service** → Tests 9-13 GREEN

---

## 📊 Performance Targets

| Operation | Target | Baseline | Speedup |
|-----------|--------|----------|---------|
| Cache Load | <100ms | ~1000ms | **10x** |
| Feature Extract | Cached | ~1000ms | Eliminated |
| Parquet Read | <50ms | N/A | Streaming |
| Batch (10 symbols) | <500ms | N/A | Parallel |

---

## 🔐 Cache Invalidation

**Method**: SHA256 hash of OHLCV data

```rust
// Compute hash
let hash = sha256(&ohlcv_bars);

// Check cache validity
if cached_hash == hash {
    // Cache HIT → load from Parquet (<100ms)
    return load_from_cache(symbol);
} else {
    // Cache MISS → re-compute + cache
    let features = extract_features(bars);
    cache_features(symbol, features, hash);
    return features;
}
```

---

## 🐳 MinIO Setup

```bash
# Start MinIO (Docker)
docker run -p 9000:9000 minio/minio server /data

# Access MinIO Console
open http://localhost:9000
# Default: minioadmin / minioadmin

# Bucket: ml-feature-cache
# Key Pattern: {symbol}/features.parquet
# Example: ZN.FUT/features.parquet
```

---

## 🧪 Run Tests

```bash
# All feature cache tests (expect FAILURES until implemented)
cargo test -p ml --test feature_cache_tests -- --nocapture

# Single test
cargo test -p ml --test feature_cache_tests::test_extract_256_dim_features

# Watch mode
cargo watch -x 'test -p ml --test feature_cache_tests'
```

---

## 📈 Success Criteria

- ✅ **Test Coverage**: 13/13 passing (100%)
- ✅ **Performance**: <100ms cache load (10x speedup)
- ✅ **Features**: 256-dim vectors (normalized 0-1)
- ✅ **Storage**: Parquet in MinIO (S3-compatible)
- ✅ **Invalidation**: SHA256 hash-based

---

## 🔄 Next Actions

1. Run tests: `cargo test -p ml --test feature_cache_tests` (expect RED)
2. Implement `feature_extractor.rs` (Tests 1-2 GREEN)
3. Implement `parquet_writer.rs` (Tests 3-5 GREEN)
4. Start MinIO Docker
5. Implement `minio_storage.rs` (Tests 6-8 GREEN)
6. Implement `cache.rs` + `invalidation.rs` (Tests 9-13 GREEN)
7. Celebrate 100% GREEN tests!

---

**Impact**: 10x faster ML training startup (from ~10 seconds to ~1 second)