docs: feature pre-compute pipeline design spec
Flat binary .fxcache format (f64 + bf16 versions), precompute_features binary, Argo workflow integration, local validation plan. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -0,0 +1,203 @@
|
||||
# Feature Pre-Compute Pipeline Design
|
||||
|
||||
**Goal**: Pre-compute all DQN training features (42-dim OHLCV + 4-dim targets + 8-dim OFI) on a cheap CPU node and write a flat binary cache file (`.fxcache`) so the H100 skips all feature extraction and starts training immediately.
|
||||
|
||||
**Architecture**: A new Rust binary (`precompute_features`) streams DBN source data, runs the full feature extraction pipeline, and writes a single `.fxcache` binary per symbol. An Argo WorkflowTemplate runs this on a `ci-compile-cpu` node (32 cores, EUR 0.85/hr) after data downloads complete. The H100 training workflow reads the cache via `mmap()` with zero deserialization overhead.
|
||||
|
||||
**Tech Stack**: Rust (ml crate, rayon parallel), Argo Workflows, flat binary format, mmap loading.
|
||||
|
||||
---
|
||||
|
||||
## Binary Cache Format (`.fxcache`)
|
||||
|
||||
Single file per symbol/date-range. Two versions sharing the same header:
|
||||
|
||||
### Header (64 bytes, fixed)
|
||||
|
||||
| Offset | Field | Type | Description |
|
||||
|--------|-------|------|-------------|
|
||||
| 0 | magic | `[u8; 8]` | `b"FXCACHE\0"` |
|
||||
| 8 | version | `u16` | 1 = f64, 2 = bf16 |
|
||||
| 10 | feature_dims | `u16` | 42 |
|
||||
| 12 | target_dims | `u16` | 4 |
|
||||
| 14 | ofi_dims | `u16` | 8 |
|
||||
| 16 | bar_count | `u64` | Number of bars |
|
||||
| 24 | cache_key | `[u8; 32]` | SHA256 of source files + extraction code |
|
||||
| 56 | reserved | `[u8; 8]` | Zeroed, future use |
|
||||
|
||||
### Body
|
||||
|
||||
**Version 1 (f64)**: `bar_count` × `[f64; 54]` = 432 bytes/bar. No padding needed (f64 is 8-byte aligned).
|
||||
|
||||
**Version 2 (bf16)**: `bar_count` × `[bf16; 56]` = 112 bytes/bar. 54 data values + 2 zero-padding to match GPU state_dim alignment `(54 + 7) & !7 = 56`.
|
||||
|
||||
Record layout (both versions):
|
||||
- `[0..42]` — 42-dim feature vector (OHLCV + technical + statistical + regime)
|
||||
- `[42..46]` — 4-dim targets `[preproc_close, preproc_next, raw_close, raw_next]`
|
||||
- `[46..54]` — 8-dim OFI `[ofi_l1, ofi_l5, depth_imbalance, vpin, kyle_lambda, bid_slope, ask_slope, trade_imbalance]`
|
||||
|
||||
### Loading on H100
|
||||
|
||||
Version 1: `mmap()` → skip 64-byte header → reinterpret as `&[[f64; 54]]` → convert f64→f32→bf16 → VRAM upload.
|
||||
|
||||
Version 2: `mmap()` → skip 64-byte header → reinterpret as `&[[bf16; 56]]` → direct `memcpy` to VRAM, zero conversion.
|
||||
|
||||
---
|
||||
|
||||
## `precompute_features` Binary
|
||||
|
||||
**Location**: `crates/ml/examples/precompute_features.rs`
|
||||
|
||||
### CLI
|
||||
|
||||
```
|
||||
precompute_features \
|
||||
--data-dir /data/futures-baseline \
|
||||
--mbp10-data-dir /data/futures-baseline-mbp10 \
|
||||
--trades-data-dir /data/futures-baseline-trades \
|
||||
--output-dir /data/feature-cache \
|
||||
--symbol ES.FUT \
|
||||
--bf16 # optional: write version 2 (bf16)
|
||||
--yes # skip confirmation
|
||||
```
|
||||
|
||||
### Pipeline
|
||||
|
||||
1. Discover all `.dbn.zst` files in data-dir (OHLCV), mbp10-data-dir, trades-data-dir
|
||||
2. Stream OHLCV DBN → extract 42-dim features via `FeatureExtractor` (sequential, stateful rolling windows)
|
||||
3. Stream MBP-10 + trades DBN → compute 8-dim OFI via `OFICalculator` with trade enrichment (rayon parallel per quarterly file)
|
||||
4. Merge features + targets + OFI per bar, aligned by timestamp
|
||||
5. Compute SHA256 cache key from source file paths + mtimes + extraction code hash
|
||||
6. Write `.fxcache` to `output-dir/SYMBOL/features_CACHEKEY.fxcache`
|
||||
7. Print summary: bar count, file size, time elapsed
|
||||
|
||||
### Output Size Estimate
|
||||
|
||||
~50K bars (2 years of 1-minute OHLCV for ES.FUT) × 432 bytes/bar ≈ **21 MB** (version 1) or **5.6 MB** (version 2). Negligible compared to source data.
|
||||
|
||||
---
|
||||
|
||||
## Loader Integration
|
||||
|
||||
### New functions in `crates/ml/src/feature_cache.rs`
|
||||
|
||||
```rust
|
||||
pub fn load_fxcache(path: &Path) -> Result<FxCacheData>
|
||||
pub fn save_fxcache(path: &Path, data: &FxCacheData, bf16: bool) -> Result<()>
|
||||
|
||||
pub struct FxCacheData {
|
||||
pub features: Vec<[f64; 42]>,
|
||||
pub targets: Vec<[f64; 4]>,
|
||||
pub ofi: Vec<[f64; 8]>,
|
||||
pub cache_key: [u8; 32],
|
||||
}
|
||||
```
|
||||
|
||||
### Integration in `data_loading.rs`
|
||||
|
||||
`load_training_data()` checks `feature_cache_dir` for a `.fxcache` file matching the current cache key:
|
||||
- **Hit**: return features + targets, set `self.ofi_features` from OFI array. Zero DBN parsing.
|
||||
- **Miss**: fall back to existing DBN streaming path (unchanged).
|
||||
|
||||
This replaces the current Parquet-based feature cache and the `/tmp/.foxhunt_feature_cache/` binary cache with a single unified format that includes OFI.
|
||||
|
||||
---
|
||||
|
||||
## Argo Integration
|
||||
|
||||
### New WorkflowTemplate: `precompute-features`
|
||||
|
||||
**File**: `infra/k8s/argo/precompute-features-template.yaml`
|
||||
|
||||
**Parameters**:
|
||||
- `symbol` (default: `ES.FUT`)
|
||||
- `data-dir` (default: `/data/futures-baseline`)
|
||||
- `mbp10-dir` (default: `/data/futures-baseline-mbp10`)
|
||||
- `trades-dir` (default: `/data/futures-baseline-trades`)
|
||||
- `output-dir` (default: `/data/feature-cache`)
|
||||
- `node-pool` (default: `ci-compile-cpu`)
|
||||
- `bf16` (default: `"false"`)
|
||||
|
||||
Binary pre-uploaded to PVC at `/data/bin/precompute_features`.
|
||||
|
||||
### New CLI script: `scripts/argo-precompute.sh`
|
||||
|
||||
```bash
|
||||
./scripts/argo-precompute.sh ES.FUT # defaults
|
||||
./scripts/argo-precompute.sh ES.FUT --bf16 # bf16 output
|
||||
./scripts/argo-precompute.sh ES.FUT --pool platform # smaller node
|
||||
```
|
||||
|
||||
### Update training templates
|
||||
|
||||
`train-dqn-template.yaml` gets a new parameter:
|
||||
|
||||
```yaml
|
||||
- name: feature-cache-dir
|
||||
value: /data/feature-cache
|
||||
```
|
||||
|
||||
Passed to the training binary. The trainer checks for a `.fxcache` file. Hit → skip feature extraction. Miss → existing path.
|
||||
|
||||
### Pipeline flow
|
||||
|
||||
```
|
||||
argo-download.sh mbp-10 → (data on PVC) → argo-precompute.sh ES.FUT → (cache on PVC) → argo-train.sh dqn (reads cache)
|
||||
```
|
||||
|
||||
These are separate one-off workflows, not DAG-chained. Precompute once after data changes, train many times.
|
||||
|
||||
---
|
||||
|
||||
## Local Validation
|
||||
|
||||
### Step 1: Copy subset from PVC
|
||||
|
||||
After MBP-10 download completes, pull 1-2 quarters per schema:
|
||||
|
||||
```bash
|
||||
# Script or manual kubectl cp from a temporary pod
|
||||
# OHLCV: test_data/futures-baseline/ES.FUT/ES.FUT_2024-Q1.dbn.zst (already local)
|
||||
# MBP-10: test_data/futures-baseline-mbp10/ES.FUT/ES.FUT_2024-Q1.dbn.zst
|
||||
# Trades: test_data/futures-baseline-trades/ES.FUT/ES.FUT_2024-Q1.dbn.zst (copy from PVC)
|
||||
```
|
||||
|
||||
### Step 2: Run precompute locally
|
||||
|
||||
```bash
|
||||
cargo run -p ml --example precompute_features --release -- \
|
||||
--data-dir test_data/futures-baseline \
|
||||
--mbp10-data-dir test_data/futures-baseline-mbp10 \
|
||||
--trades-data-dir test_data/futures-baseline-trades \
|
||||
--output-dir test_data/feature-cache \
|
||||
--symbol ES.FUT --yes
|
||||
```
|
||||
|
||||
### Step 3: Validate cache file
|
||||
|
||||
- Check header: magic, version, bar_count, dims
|
||||
- Verify bar_count matches OHLCV bars minus warmup period (50)
|
||||
- Bit-exact comparison: load cache vs fresh DBN extraction for first/last 10 bars
|
||||
- Run with `--bf16`, verify version 2 output
|
||||
- Verify cache key changes when source data changes
|
||||
|
||||
### Step 4: Smoke test training against cache
|
||||
|
||||
New test: load `.fxcache` → run 2 epochs of DQN → verify training loop completes with same loss trajectory as uncached path.
|
||||
|
||||
```bash
|
||||
FOXHUNT_TEST_DATA=test_data/futures-baseline \
|
||||
cargo test -p ml --lib -- fxcache_smoke --ignored --nocapture
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Compute Node Selection
|
||||
|
||||
| Task | Recommended Node | Specs | Cost | Why |
|
||||
|------|-----------------|-------|------|-----|
|
||||
| Pre-compute features | `ci-compile-cpu` (POP2-HC-32C-64G) | 32 cores, 64G RAM | EUR 0.85/hr | Rayon parallel MBP-10 parsing, ~30 min for full dataset |
|
||||
| Alternative | `POP2-HC-16C-32G` | 16 cores, 32G RAM | EUR 0.43/hr | Cheaper, ~60 min, sufficient if not time-sensitive |
|
||||
| Local dev | RTX 3050 workstation | — | free | Subset validation only |
|
||||
|
||||
No SIMD needed — the feature extraction is floating-point arithmetic with rolling windows. The bottleneck is MBP-10 DBN decompression + parsing, which parallelizes well with rayon across quarterly files.
|
||||
Reference in New Issue
Block a user