foxhunt/docs/superpowers/specs/2026-03-31-feature-precompute-pipeline-design.md

Feature Pre-Compute Pipeline Design

Goal: Pre-compute all DQN training features (42-dim OHLCV + 4-dim targets + 8-dim OFI) on a cheap CPU node and write a flat binary cache file (.fxcache) so the H100 skips all feature extraction and starts training immediately.

Architecture: A new Rust binary (precompute_features) streams DBN source data, runs the full feature extraction pipeline, and writes a single .fxcache binary per symbol. An Argo WorkflowTemplate runs this on a ci-compile-cpu node (32 cores, EUR 0.85/hr) after data downloads complete. The H100 training workflow reads the cache via mmap() with zero deserialization overhead.

Tech Stack: Rust (ml crate, rayon parallel), Argo Workflows, flat binary format, mmap loading.

---

Binary Cache Format (.fxcache)

Single file per symbol/date-range. Two versions sharing the same header:

Header (64 bytes, fixed)

Offset	Field	Type	Description
0	magic	[u8; 8]	b"FXCACHE\0"
8	version	u16	1 = f64, 2 = bf16
10	feature_dims	u16	42
12	target_dims	u16	4
14	ofi_dims	u16	8
16	bar_count	u64	Number of bars
24	cache_key	[u8; 32]	SHA256 of source files + extraction code
56	reserved	[u8; 8]	Zeroed, future use

Body

Version 1 (f64): bar_count × [f64; 54] = 432 bytes/bar. No padding needed (f64 is 8-byte aligned).

Version 2 (bf16): bar_count × [bf16; 56] = 112 bytes/bar. 54 data values + 2 zero-padding to match GPU state_dim alignment (54 + 7) & !7 = 56.

Record layout (both versions):

• [0..42] — 42-dim feature vector (OHLCV + technical + statistical + regime)
• [42..46] — 4-dim targets [preproc_close, preproc_next, raw_close, raw_next]
• [46..54] — 8-dim OFI [ofi_l1, ofi_l5, depth_imbalance, vpin, kyle_lambda, bid_slope, ask_slope, trade_imbalance]

Loading on H100

Version 1: mmap() → skip 64-byte header → reinterpret as &[[f64; 54]] → convert f64→f32→bf16 → VRAM upload.

Version 2: mmap() → skip 64-byte header → reinterpret as &[[bf16; 56]] → direct memcpy to VRAM, zero conversion.

---

precompute_features Binary

Location: crates/ml/examples/precompute_features.rs

CLI

precompute_features \
  --data-dir /data/futures-baseline \
  --mbp10-data-dir /data/futures-baseline-mbp10 \
  --trades-data-dir /data/futures-baseline-trades \
  --output-dir /data/feature-cache \
  --symbol ES.FUT \
  --bf16              # optional: write version 2 (bf16)
  --yes               # skip confirmation

Pipeline

1. Discover all .dbn.zst files in data-dir (OHLCV), mbp10-data-dir, trades-data-dir
2. Stream OHLCV DBN → extract 42-dim features via FeatureExtractor (sequential, stateful rolling windows)
3. Stream MBP-10 + trades DBN → compute 8-dim OFI via OFICalculator with trade enrichment (rayon parallel per quarterly file)
4. Merge features + targets + OFI per bar, aligned by timestamp
5. Compute SHA256 cache key from source file paths + mtimes + extraction code hash
6. Write .fxcache to output-dir/SYMBOL/features_CACHEKEY.fxcache
7. Print summary: bar count, file size, time elapsed

Output Size Estimate

~50K bars (2 years of 1-minute OHLCV for ES.FUT) × 432 bytes/bar ≈ 21 MB (version 1) or 5.6 MB (version 2). Negligible compared to source data.

---

Loader Integration

New functions in crates/ml/src/feature_cache.rs

pub fn load_fxcache(path: &Path) -> Result<FxCacheData>
pub fn save_fxcache(path: &Path, data: &FxCacheData, bf16: bool) -> Result<()>

pub struct FxCacheData {
    pub features: Vec<[f64; 42]>,
    pub targets: Vec<[f64; 4]>,
    pub ofi: Vec<[f64; 8]>,
    pub cache_key: [u8; 32],
}

Integration in data_loading.rs

load_training_data() checks feature_cache_dir for a .fxcache file matching the current cache key:

• Hit: return features + targets, set self.ofi_features from OFI array. Zero DBN parsing.
• Miss: fall back to existing DBN streaming path (unchanged).

This replaces the current Parquet-based feature cache and the /tmp/.foxhunt_feature_cache/ binary cache with a single unified format that includes OFI.

---

Argo Integration

New WorkflowTemplate: precompute-features

File: infra/k8s/argo/precompute-features-template.yaml

Parameters:

• symbol (default: ES.FUT)
• data-dir (default: /data/futures-baseline)
• mbp10-dir (default: /data/futures-baseline-mbp10)
• trades-dir (default: /data/futures-baseline-trades)
• output-dir (default: /data/feature-cache)
• node-pool (default: ci-compile-cpu)
• bf16 (default: "false")

Binary pre-uploaded to PVC at /data/bin/precompute_features.

New CLI script: scripts/argo-precompute.sh

./scripts/argo-precompute.sh ES.FUT                    # defaults
./scripts/argo-precompute.sh ES.FUT --bf16             # bf16 output
./scripts/argo-precompute.sh ES.FUT --pool platform    # smaller node

Update training templates

train-dqn-template.yaml gets a new parameter:

- name: feature-cache-dir
  value: /data/feature-cache

Passed to the training binary. The trainer checks for a .fxcache file. Hit → skip feature extraction. Miss → existing path.

Pipeline flow

argo-download.sh mbp-10 → (data on PVC) → argo-precompute.sh ES.FUT → (cache on PVC) → argo-train.sh dqn (reads cache)

These are separate one-off workflows, not DAG-chained. Precompute once after data changes, train many times.

---

Local Validation

Step 1: Copy subset from PVC

After MBP-10 download completes, pull 1-2 quarters per schema:

# Script or manual kubectl cp from a temporary pod
# OHLCV: test_data/futures-baseline/ES.FUT/ES.FUT_2024-Q1.dbn.zst (already local)
# MBP-10: test_data/futures-baseline-mbp10/ES.FUT/ES.FUT_2024-Q1.dbn.zst
# Trades: test_data/futures-baseline-trades/ES.FUT/ES.FUT_2024-Q1.dbn.zst (copy from PVC)

Step 2: Run precompute locally

cargo run -p ml --example precompute_features --release -- \
  --data-dir test_data/futures-baseline \
  --mbp10-data-dir test_data/futures-baseline-mbp10 \
  --trades-data-dir test_data/futures-baseline-trades \
  --output-dir test_data/feature-cache \
  --symbol ES.FUT --yes

Step 3: Validate cache file

• Check header: magic, version, bar_count, dims
• Verify bar_count matches OHLCV bars minus warmup period (50)
• Bit-exact comparison: load cache vs fresh DBN extraction for first/last 10 bars
• Run with --bf16, verify version 2 output
• Verify cache key changes when source data changes

Step 4: Smoke test training against cache

New test: load .fxcache → run 2 epochs of DQN → verify training loop completes with same loss trajectory as uncached path.

FOXHUNT_TEST_DATA=test_data/futures-baseline \
  cargo test -p ml --lib -- fxcache_smoke --ignored --nocapture

---

Compute Node Selection

Task	Recommended Node	Specs	Cost	Why
Pre-compute features	ci-compile-cpu (POP2-HC-32C-64G)	32 cores, 64G RAM	EUR 0.85/hr	Rayon parallel MBP-10 parsing, ~30 min for full dataset
Alternative	POP2-HC-16C-32G	16 cores, 32G RAM	EUR 0.43/hr	Cheaper, ~60 min, sufficient if not time-sensitive
Local dev	RTX 3050 workstation	—	free	Subset validation only

No SIMD needed — the feature extraction is floating-point arithmetic with rolling windows. The bottleneck is MBP-10 DBN decompression + parsing, which parallelizes well with rayon across quarterly files.