# Feature Pre-Compute Pipeline Design

**Goal**: Pre-compute all DQN training features (42-dim OHLCV + 4-dim targets + 8-dim OFI) on a cheap CPU node and write a flat binary cache file (`.fxcache`) so the H100 skips all feature extraction and starts training immediately.

**Architecture**: A new Rust binary (`precompute_features`) streams DBN source data, runs the full feature extraction pipeline, and writes a single `.fxcache` binary per symbol. An Argo WorkflowTemplate runs this on a `ci-compile-cpu` node (32 cores, EUR 0.85/hr) after data downloads complete. The H100 training workflow reads the cache via `mmap()` with zero deserialization overhead.

**Tech Stack**: Rust (ml crate, rayon parallel), Argo Workflows, flat binary format, mmap loading.

---

## Binary Cache Format (`.fxcache`)

Single file per symbol/date-range. Two versions sharing the same header:

### Header (64 bytes, fixed)

| Offset | Field | Type | Description |
|--------|-------|------|-------------|
| 0 | magic | `[u8; 8]` | `b"FXCACHE\0"` |
| 8 | version | `u16` | 1 = f64, 2 = bf16 |
| 10 | feature_dims | `u16` | 42 |
| 12 | target_dims | `u16` | 4 |
| 14 | ofi_dims | `u16` | 8 |
| 16 | bar_count | `u64` | Number of bars |
| 24 | cache_key | `[u8; 32]` | SHA256 of source files + extraction code |
| 56 | reserved | `[u8; 8]` | Zeroed, future use |

### Body

**Version 1 (f64)**: `bar_count` × `[f64; 54]` = 432 bytes/bar. No padding needed (f64 is 8-byte aligned).

**Version 2 (bf16)**: `bar_count` × `[bf16; 56]` = 112 bytes/bar. 54 data values + 2 zero-padding to match GPU state_dim alignment `(54 + 7) & !7 = 56`.

Record layout (both versions):
- `[0..42]` — 42-dim feature vector (OHLCV + technical + statistical + regime)
- `[42..46]` — 4-dim targets `[preproc_close, preproc_next, raw_close, raw_next]`
- `[46..54]` — 8-dim OFI `[ofi_l1, ofi_l5, depth_imbalance, vpin, kyle_lambda, bid_slope, ask_slope, trade_imbalance]`

### Loading on H100

Version 1: `mmap()` → skip 64-byte header → reinterpret as `&[[f64; 54]]` → convert f64→f32→bf16 → VRAM upload.

Version 2: `mmap()` → skip 64-byte header → reinterpret as `&[[bf16; 56]]` → direct `memcpy` to VRAM, zero conversion.

---

## `precompute_features` Binary

**Location**: `crates/ml/examples/precompute_features.rs`

### CLI

```
precompute_features \
  --data-dir /data/futures-baseline \
  --mbp10-data-dir /data/futures-baseline-mbp10 \
  --trades-data-dir /data/futures-baseline-trades \
  --output-dir /data/feature-cache \
  --symbol ES.FUT \
  --bf16              # optional: write version 2 (bf16)
  --yes               # skip confirmation
```

### Pipeline

1. Discover all `.dbn.zst` files in data-dir (OHLCV), mbp10-data-dir, trades-data-dir
2. Stream OHLCV DBN → extract 42-dim features via `FeatureExtractor` (sequential, stateful rolling windows)
3. Stream MBP-10 + trades DBN → compute 8-dim OFI via `OFICalculator` with trade enrichment (rayon parallel per quarterly file)
4. Merge features + targets + OFI per bar, aligned by timestamp
5. Compute SHA256 cache key from source file paths + mtimes + extraction code hash
6. Write `.fxcache` to `output-dir/SYMBOL/features_CACHEKEY.fxcache`
7. Print summary: bar count, file size, time elapsed

### Output Size Estimate

~50K bars (2 years of 1-minute OHLCV for ES.FUT) × 432 bytes/bar ≈ **21 MB** (version 1) or **5.6 MB** (version 2). Negligible compared to source data.

---

## Loader Integration

### New functions in `crates/ml/src/feature_cache.rs`

```rust
pub fn load_fxcache(path: &Path) -> Result<FxCacheData>
pub fn save_fxcache(path: &Path, data: &FxCacheData, bf16: bool) -> Result<()>

pub struct FxCacheData {
    pub features: Vec<[f64; 42]>,
    pub targets: Vec<[f64; 4]>,
    pub ofi: Vec<[f64; 8]>,
    pub cache_key: [u8; 32],
}
```

### Integration in `data_loading.rs`

`load_training_data()` checks `feature_cache_dir` for a `.fxcache` file matching the current cache key:
- **Hit**: return features + targets, set `self.ofi_features` from OFI array. Zero DBN parsing.
- **Miss**: fall back to existing DBN streaming path (unchanged).

This replaces the current Parquet-based feature cache and the `/tmp/.foxhunt_feature_cache/` binary cache with a single unified format that includes OFI.

---

## Argo Integration

### New WorkflowTemplate: `precompute-features`

**File**: `infra/k8s/argo/precompute-features-template.yaml`

**Parameters**:
- `symbol` (default: `ES.FUT`)
- `data-dir` (default: `/data/futures-baseline`)
- `mbp10-dir` (default: `/data/futures-baseline-mbp10`)
- `trades-dir` (default: `/data/futures-baseline-trades`)
- `output-dir` (default: `/data/feature-cache`)
- `node-pool` (default: `ci-compile-cpu`)
- `bf16` (default: `"false"`)

Binary pre-uploaded to PVC at `/data/bin/precompute_features`.

### New CLI script: `scripts/argo-precompute.sh`

```bash
./scripts/argo-precompute.sh ES.FUT                    # defaults
./scripts/argo-precompute.sh ES.FUT --bf16             # bf16 output
./scripts/argo-precompute.sh ES.FUT --pool platform    # smaller node
```

### Update training templates

`train-dqn-template.yaml` gets a new parameter:

```yaml
- name: feature-cache-dir
  value: /data/feature-cache
```

Passed to the training binary. The trainer checks for a `.fxcache` file. Hit → skip feature extraction. Miss → existing path.

### Pipeline flow

```
argo-download.sh mbp-10 → (data on PVC) → argo-precompute.sh ES.FUT → (cache on PVC) → argo-train.sh dqn (reads cache)
```

These are separate one-off workflows, not DAG-chained. Precompute once after data changes, train many times.

---

## Local Validation

### Step 1: Copy subset from PVC

After MBP-10 download completes, pull 1-2 quarters per schema:

```bash
# Script or manual kubectl cp from a temporary pod
# OHLCV: test_data/futures-baseline/ES.FUT/ES.FUT_2024-Q1.dbn.zst (already local)
# MBP-10: test_data/futures-baseline-mbp10/ES.FUT/ES.FUT_2024-Q1.dbn.zst
# Trades: test_data/futures-baseline-trades/ES.FUT/ES.FUT_2024-Q1.dbn.zst (copy from PVC)
```

### Step 2: Run precompute locally

```bash
cargo run -p ml --example precompute_features --release -- \
  --data-dir test_data/futures-baseline \
  --mbp10-data-dir test_data/futures-baseline-mbp10 \
  --trades-data-dir test_data/futures-baseline-trades \
  --output-dir test_data/feature-cache \
  --symbol ES.FUT --yes
```

### Step 3: Validate cache file

- Check header: magic, version, bar_count, dims
- Verify bar_count matches OHLCV bars minus warmup period (50)
- Bit-exact comparison: load cache vs fresh DBN extraction for first/last 10 bars
- Run with `--bf16`, verify version 2 output
- Verify cache key changes when source data changes

### Step 4: Smoke test training against cache

New test: load `.fxcache` → run 2 epochs of DQN → verify training loop completes with same loss trajectory as uncached path.

```bash
FOXHUNT_TEST_DATA=test_data/futures-baseline \
  cargo test -p ml --lib -- fxcache_smoke --ignored --nocapture
```

---

## Compute Node Selection

| Task | Recommended Node | Specs | Cost | Why |
|------|-----------------|-------|------|-----|
| Pre-compute features | `ci-compile-cpu` (POP2-HC-32C-64G) | 32 cores, 64G RAM | EUR 0.85/hr | Rayon parallel MBP-10 parsing, ~30 min for full dataset |
| Alternative | `POP2-HC-16C-32G` | 16 cores, 32G RAM | EUR 0.43/hr | Cheaper, ~60 min, sufficient if not time-sensitive |
| Local dev | RTX 3050 workstation | — | free | Subset validation only |

No SIMD needed — the feature extraction is floating-point arithmetic with rolling windows. The bottleneck is MBP-10 DBN decompression + parsing, which parallelizes well with rayon across quarterly files.
