From a643652d0e13635dac0f6ac598ca987a46338dac Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Tue, 31 Mar 2026 23:30:42 +0200 Subject: [PATCH] docs: feature pre-compute pipeline design spec Flat binary .fxcache format (f64 + bf16 versions), precompute_features binary, Argo workflow integration, local validation plan. Co-Authored-By: Claude Opus 4.6 (1M context) --- ...3-31-feature-precompute-pipeline-design.md | 203 ++++++++++++++++++ 1 file changed, 203 insertions(+) create mode 100644 docs/superpowers/specs/2026-03-31-feature-precompute-pipeline-design.md diff --git a/docs/superpowers/specs/2026-03-31-feature-precompute-pipeline-design.md b/docs/superpowers/specs/2026-03-31-feature-precompute-pipeline-design.md new file mode 100644 index 000000000..902fbec42 --- /dev/null +++ b/docs/superpowers/specs/2026-03-31-feature-precompute-pipeline-design.md @@ -0,0 +1,203 @@ +# Feature Pre-Compute Pipeline Design + +**Goal**: Pre-compute all DQN training features (42-dim OHLCV + 4-dim targets + 8-dim OFI) on a cheap CPU node and write a flat binary cache file (`.fxcache`) so the H100 skips all feature extraction and starts training immediately. + +**Architecture**: A new Rust binary (`precompute_features`) streams DBN source data, runs the full feature extraction pipeline, and writes a single `.fxcache` binary per symbol. An Argo WorkflowTemplate runs this on a `ci-compile-cpu` node (32 cores, EUR 0.85/hr) after data downloads complete. The H100 training workflow reads the cache via `mmap()` with zero deserialization overhead. + +**Tech Stack**: Rust (ml crate, rayon parallel), Argo Workflows, flat binary format, mmap loading. + +--- + +## Binary Cache Format (`.fxcache`) + +Single file per symbol/date-range. Two versions sharing the same header: + +### Header (64 bytes, fixed) + +| Offset | Field | Type | Description | +|--------|-------|------|-------------| +| 0 | magic | `[u8; 8]` | `b"FXCACHE\0"` | +| 8 | version | `u16` | 1 = f64, 2 = bf16 | +| 10 | feature_dims | `u16` | 42 | +| 12 | target_dims | `u16` | 4 | +| 14 | ofi_dims | `u16` | 8 | +| 16 | bar_count | `u64` | Number of bars | +| 24 | cache_key | `[u8; 32]` | SHA256 of source files + extraction code | +| 56 | reserved | `[u8; 8]` | Zeroed, future use | + +### Body + +**Version 1 (f64)**: `bar_count` × `[f64; 54]` = 432 bytes/bar. No padding needed (f64 is 8-byte aligned). + +**Version 2 (bf16)**: `bar_count` × `[bf16; 56]` = 112 bytes/bar. 54 data values + 2 zero-padding to match GPU state_dim alignment `(54 + 7) & !7 = 56`. + +Record layout (both versions): +- `[0..42]` — 42-dim feature vector (OHLCV + technical + statistical + regime) +- `[42..46]` — 4-dim targets `[preproc_close, preproc_next, raw_close, raw_next]` +- `[46..54]` — 8-dim OFI `[ofi_l1, ofi_l5, depth_imbalance, vpin, kyle_lambda, bid_slope, ask_slope, trade_imbalance]` + +### Loading on H100 + +Version 1: `mmap()` → skip 64-byte header → reinterpret as `&[[f64; 54]]` → convert f64→f32→bf16 → VRAM upload. + +Version 2: `mmap()` → skip 64-byte header → reinterpret as `&[[bf16; 56]]` → direct `memcpy` to VRAM, zero conversion. + +--- + +## `precompute_features` Binary + +**Location**: `crates/ml/examples/precompute_features.rs` + +### CLI + +``` +precompute_features \ + --data-dir /data/futures-baseline \ + --mbp10-data-dir /data/futures-baseline-mbp10 \ + --trades-data-dir /data/futures-baseline-trades \ + --output-dir /data/feature-cache \ + --symbol ES.FUT \ + --bf16 # optional: write version 2 (bf16) + --yes # skip confirmation +``` + +### Pipeline + +1. Discover all `.dbn.zst` files in data-dir (OHLCV), mbp10-data-dir, trades-data-dir +2. Stream OHLCV DBN → extract 42-dim features via `FeatureExtractor` (sequential, stateful rolling windows) +3. Stream MBP-10 + trades DBN → compute 8-dim OFI via `OFICalculator` with trade enrichment (rayon parallel per quarterly file) +4. Merge features + targets + OFI per bar, aligned by timestamp +5. Compute SHA256 cache key from source file paths + mtimes + extraction code hash +6. Write `.fxcache` to `output-dir/SYMBOL/features_CACHEKEY.fxcache` +7. Print summary: bar count, file size, time elapsed + +### Output Size Estimate + +~50K bars (2 years of 1-minute OHLCV for ES.FUT) × 432 bytes/bar ≈ **21 MB** (version 1) or **5.6 MB** (version 2). Negligible compared to source data. + +--- + +## Loader Integration + +### New functions in `crates/ml/src/feature_cache.rs` + +```rust +pub fn load_fxcache(path: &Path) -> Result +pub fn save_fxcache(path: &Path, data: &FxCacheData, bf16: bool) -> Result<()> + +pub struct FxCacheData { + pub features: Vec<[f64; 42]>, + pub targets: Vec<[f64; 4]>, + pub ofi: Vec<[f64; 8]>, + pub cache_key: [u8; 32], +} +``` + +### Integration in `data_loading.rs` + +`load_training_data()` checks `feature_cache_dir` for a `.fxcache` file matching the current cache key: +- **Hit**: return features + targets, set `self.ofi_features` from OFI array. Zero DBN parsing. +- **Miss**: fall back to existing DBN streaming path (unchanged). + +This replaces the current Parquet-based feature cache and the `/tmp/.foxhunt_feature_cache/` binary cache with a single unified format that includes OFI. + +--- + +## Argo Integration + +### New WorkflowTemplate: `precompute-features` + +**File**: `infra/k8s/argo/precompute-features-template.yaml` + +**Parameters**: +- `symbol` (default: `ES.FUT`) +- `data-dir` (default: `/data/futures-baseline`) +- `mbp10-dir` (default: `/data/futures-baseline-mbp10`) +- `trades-dir` (default: `/data/futures-baseline-trades`) +- `output-dir` (default: `/data/feature-cache`) +- `node-pool` (default: `ci-compile-cpu`) +- `bf16` (default: `"false"`) + +Binary pre-uploaded to PVC at `/data/bin/precompute_features`. + +### New CLI script: `scripts/argo-precompute.sh` + +```bash +./scripts/argo-precompute.sh ES.FUT # defaults +./scripts/argo-precompute.sh ES.FUT --bf16 # bf16 output +./scripts/argo-precompute.sh ES.FUT --pool platform # smaller node +``` + +### Update training templates + +`train-dqn-template.yaml` gets a new parameter: + +```yaml +- name: feature-cache-dir + value: /data/feature-cache +``` + +Passed to the training binary. The trainer checks for a `.fxcache` file. Hit → skip feature extraction. Miss → existing path. + +### Pipeline flow + +``` +argo-download.sh mbp-10 → (data on PVC) → argo-precompute.sh ES.FUT → (cache on PVC) → argo-train.sh dqn (reads cache) +``` + +These are separate one-off workflows, not DAG-chained. Precompute once after data changes, train many times. + +--- + +## Local Validation + +### Step 1: Copy subset from PVC + +After MBP-10 download completes, pull 1-2 quarters per schema: + +```bash +# Script or manual kubectl cp from a temporary pod +# OHLCV: test_data/futures-baseline/ES.FUT/ES.FUT_2024-Q1.dbn.zst (already local) +# MBP-10: test_data/futures-baseline-mbp10/ES.FUT/ES.FUT_2024-Q1.dbn.zst +# Trades: test_data/futures-baseline-trades/ES.FUT/ES.FUT_2024-Q1.dbn.zst (copy from PVC) +``` + +### Step 2: Run precompute locally + +```bash +cargo run -p ml --example precompute_features --release -- \ + --data-dir test_data/futures-baseline \ + --mbp10-data-dir test_data/futures-baseline-mbp10 \ + --trades-data-dir test_data/futures-baseline-trades \ + --output-dir test_data/feature-cache \ + --symbol ES.FUT --yes +``` + +### Step 3: Validate cache file + +- Check header: magic, version, bar_count, dims +- Verify bar_count matches OHLCV bars minus warmup period (50) +- Bit-exact comparison: load cache vs fresh DBN extraction for first/last 10 bars +- Run with `--bf16`, verify version 2 output +- Verify cache key changes when source data changes + +### Step 4: Smoke test training against cache + +New test: load `.fxcache` → run 2 epochs of DQN → verify training loop completes with same loss trajectory as uncached path. + +```bash +FOXHUNT_TEST_DATA=test_data/futures-baseline \ + cargo test -p ml --lib -- fxcache_smoke --ignored --nocapture +``` + +--- + +## Compute Node Selection + +| Task | Recommended Node | Specs | Cost | Why | +|------|-----------------|-------|------|-----| +| Pre-compute features | `ci-compile-cpu` (POP2-HC-32C-64G) | 32 cores, 64G RAM | EUR 0.85/hr | Rayon parallel MBP-10 parsing, ~30 min for full dataset | +| Alternative | `POP2-HC-16C-32G` | 16 cores, 32G RAM | EUR 0.43/hr | Cheaper, ~60 min, sufficient if not time-sensitive | +| Local dev | RTX 3050 workstation | — | free | Subset validation only | + +No SIMD needed — the feature extraction is floating-point arithmetic with rolling windows. The bottleneck is MBP-10 DBN decompression + parsing, which parallelizes well with rayon across quarterly files.