foxhunt/docs/superpowers/plans/2026-03-31-feature-precompute-pipeline.md

Feature Pre-Compute Pipeline Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build a flat binary feature cache (.fxcache) format, a precompute_features binary, and Argo integration so the H100 skips all feature extraction and loads pre-computed data via mmap.

Architecture: New .fxcache format with 64-byte header + packed records (version 1: f64, version 2: bf16). New precompute_features example binary reuses existing load_training_data() + OFI extraction internals. Argo WorkflowTemplate runs pre-compute on CPU node. Training loader checks for .fxcache before falling back to DBN streaming.

Tech Stack: Rust (ml crate, rayon, memmap2, half), Argo Workflows, Bash.

---

File Structure

File	Action	Responsibility
crates/ml/src/fxcache.rs	Create	.fxcache format: header struct, read/write, mmap loading
crates/ml/src/lib.rs	Modify	Add pub mod fxcache;
crates/ml/examples/precompute_features.rs	Create	CLI binary for pre-computing features
crates/ml/src/trainers/dqn/data_loading.rs	Modify	Check .fxcache before DBN streaming
crates/ml/tests/fxcache_roundtrip_test.rs	Create	Unit + integration tests
infra/k8s/argo/precompute-features-template.yaml	Create	Argo WorkflowTemplate
scripts/argo-precompute.sh	Create	CLI wrapper
infra/k8s/argo/train-dqn-template.yaml	Modify	Add feature-cache-dir parameter

---

Task 1: .fxcache Format — Header and Types

Files:

• Create: crates/ml/src/fxcache.rs

• Modify: crates/ml/src/lib.rs

• Step 1: Create fxcache module with header types

// crates/ml/src/fxcache.rs
//! Flat binary feature cache format (.fxcache)
//!
//! Zero-overhead loading of pre-computed DQN training data.
//! Two versions: v1 (f64, debuggable) and v2 (bf16, GPU-native).

use anyhow::{bail, Context, Result};
use std::io::{Read, Write, BufReader, BufWriter};
use std::path::Path;

const MAGIC: [u8; 8] = *b"FXCACHE\0";
const VERSION_F64: u16 = 1;
const VERSION_BF16: u16 = 2;
const HEADER_SIZE: usize = 64;
const FEATURE_DIMS: u16 = 42;
const TARGET_DIMS: u16 = 4;
const OFI_DIMS: u16 = 8;
const RECORD_DIMS: usize = 54; // 42 + 4 + 8

/// 64-byte fixed header for .fxcache files.
#[derive(Debug, Clone, Copy)]
#[repr(C, packed)]
pub struct FxCacheHeader {
    pub magic: [u8; 8],
    pub version: u16,
    pub feature_dims: u16,
    pub target_dims: u16,
    pub ofi_dims: u16,
    pub bar_count: u64,
    pub cache_key: [u8; 32],
    pub reserved: [u8; 8],
}

/// Loaded cache data — split into three arrays for downstream consumption.
pub struct FxCacheData {
    pub features: Vec<[f64; 42]>,
    pub targets: Vec<[f64; 4]>,
    pub ofi: Vec<[f64; 8]>,
    pub cache_key: [u8; 32],
    pub bar_count: usize,
}

impl FxCacheHeader {
    pub fn new(bar_count: u64, cache_key: [u8; 32], bf16: bool) -> Self {
        Self {
            magic: MAGIC,
            version: if bf16 { VERSION_BF16 } else { VERSION_F64 },
            feature_dims: FEATURE_DIMS,
            target_dims: TARGET_DIMS,
            ofi_dims: OFI_DIMS,
            bar_count,
            cache_key,
            reserved: [0u8; 8],
        }
    }

    pub fn validate(&self) -> Result<()> {
        if self.magic != MAGIC {
            bail!("Invalid fxcache magic: {:?}", &self.magic);
        }
        if self.version != VERSION_F64 && self.version != VERSION_BF16 {
            bail!("Unknown fxcache version: {}", self.version);
        }
        if self.feature_dims != FEATURE_DIMS
            || self.target_dims != TARGET_DIMS
            || self.ofi_dims != OFI_DIMS
        {
            bail!(
                "Dimension mismatch: features={}, targets={}, ofi={}",
                self.feature_dims, self.target_dims, self.ofi_dims
            );
        }
        Ok(())
    }

    pub fn is_bf16(&self) -> bool {
        self.version == VERSION_BF16
    }
}

• Step 2: Add module to lib.rs

In crates/ml/src/lib.rs, add:

pub mod fxcache;

• Step 3: Verify it compiles

Run: SQLX_OFFLINE=true cargo check -p ml --lib 2>&1 | tail -3 Expected: compiles with no errors

• Step 4: Commit

git add crates/ml/src/fxcache.rs crates/ml/src/lib.rs
git commit -m "feat: fxcache module — header types and validation"

---

Task 2: .fxcache Writer (f64 + bf16)

Files:

• Modify: crates/ml/src/fxcache.rs

• Step 1: Add the write function

Append to crates/ml/src/fxcache.rs:

/// Write a .fxcache file from pre-computed arrays.
pub fn write_fxcache(
    path: &Path,
    features: &[[f64; 42]],
    targets: &[[f64; 4]],
    ofi: &[[f64; 8]],
    cache_key: [u8; 32],
    bf16: bool,
) -> Result<u64> {
    let n = features.len();
    assert_eq!(n, targets.len(), "features/targets length mismatch");
    assert_eq!(n, ofi.len(), "features/ofi length mismatch");

    if let Some(parent) = path.parent() {
        std::fs::create_dir_all(parent)
            .with_context(|| format!("Failed to create cache dir {:?}", parent))?;
    }

    let file = std::fs::File::create(path)
        .with_context(|| format!("Failed to create fxcache {:?}", path))?;
    let mut w = BufWriter::new(file);

    let header = FxCacheHeader::new(n as u64, cache_key, bf16);

    // Write header as raw bytes
    let header_bytes: &[u8] =
        unsafe { std::slice::from_raw_parts(&header as *const _ as *const u8, HEADER_SIZE) };
    w.write_all(header_bytes)?;

    if bf16 {
        // Version 2: bf16, 56 elements per record (54 data + 2 padding)
        for i in 0..n {
            for &v in features[i].iter().chain(targets[i].iter()).chain(ofi[i].iter()) {
                let h = half::bf16::from_f64(v);
                w.write_all(&h.to_le_bytes())?;
            }
            // 2 zero-padding bf16 values for 8-byte alignment
            w.write_all(&[0u8; 4])?;
        }
    } else {
        // Version 1: f64, 54 elements per record
        for i in 0..n {
            for &v in features[i].iter().chain(targets[i].iter()).chain(ofi[i].iter()) {
                w.write_all(&v.to_le_bytes())?;
            }
        }
    }

    w.flush()?;

    let file_size = path.metadata().map(|m| m.len()).unwrap_or(0);
    tracing::info!(
        "Wrote fxcache v{}: {} bars, {:.1} MB → {:?}",
        if bf16 { 2 } else { 1 },
        n,
        file_size as f64 / 1_048_576.0,
        path,
    );

    Ok(file_size)
}

• Step 2: Verify it compiles

Run: SQLX_OFFLINE=true cargo check -p ml --lib 2>&1 | tail -3 Expected: compiles

• Step 3: Commit

git add crates/ml/src/fxcache.rs
git commit -m "feat: fxcache writer — f64 and bf16 versions"

---

Task 3: .fxcache Reader (mmap + fallback)

Files:

• Modify: crates/ml/src/fxcache.rs

• Step 1: Add the read function

Append to crates/ml/src/fxcache.rs:

/// Load a .fxcache file, returning split arrays.
///
/// Uses buffered read. For version 1 (f64): reads native f64 values.
/// For version 2 (bf16): converts bf16 → f64.
pub fn load_fxcache(path: &Path) -> Result<FxCacheData> {
    let file = std::fs::File::open(path)
        .with_context(|| format!("Failed to open fxcache {:?}", path))?;
    let file_len = file.metadata()?.len();
    let mut r = BufReader::new(file);

    // Read header
    let mut header_buf = [0u8; HEADER_SIZE];
    r.read_exact(&mut header_buf)
        .context("Failed to read fxcache header")?;
    let header: FxCacheHeader = unsafe { std::ptr::read(header_buf.as_ptr() as *const _) };
    header.validate()?;

    let n = header.bar_count as usize;
    let start = std::time::Instant::now();

    let mut features = Vec::with_capacity(n);
    let mut targets = Vec::with_capacity(n);
    let mut ofi = Vec::with_capacity(n);

    if header.is_bf16() {
        // Version 2: 56 bf16 values per record (54 data + 2 padding)
        let record_bytes = 56 * 2; // 112 bytes
        let mut record_buf = vec![0u8; record_bytes];
        for _ in 0..n {
            r.read_exact(&mut record_buf)?;
            let halfs: &[half::bf16] = unsafe {
                std::slice::from_raw_parts(record_buf.as_ptr() as *const half::bf16, 56)
            };

            let mut feat = [0.0f64; 42];
            let mut tgt = [0.0f64; 4];
            let mut ofi_arr = [0.0f64; 8];
            for j in 0..42 { feat[j] = halfs[j].to_f64(); }
            for j in 0..4 { tgt[j] = halfs[42 + j].to_f64(); }
            for j in 0..8 { ofi_arr[j] = halfs[46 + j].to_f64(); }

            features.push(feat);
            targets.push(tgt);
            ofi.push(ofi_arr);
        }
    } else {
        // Version 1: 54 f64 values per record
        let mut f64_buf = [0u8; 8];
        for _ in 0..n {
            let mut feat = [0.0f64; 42];
            let mut tgt = [0.0f64; 4];
            let mut ofi_arr = [0.0f64; 8];

            for v in feat.iter_mut() {
                r.read_exact(&mut f64_buf)?;
                *v = f64::from_le_bytes(f64_buf);
            }
            for v in tgt.iter_mut() {
                r.read_exact(&mut f64_buf)?;
                *v = f64::from_le_bytes(f64_buf);
            }
            for v in ofi_arr.iter_mut() {
                r.read_exact(&mut f64_buf)?;
                *v = f64::from_le_bytes(f64_buf);
            }

            features.push(feat);
            targets.push(tgt);
            ofi.push(ofi_arr);
        }
    }

    let elapsed = start.elapsed();
    let mb = file_len as f64 / 1_048_576.0;
    tracing::info!(
        "Loaded fxcache v{}: {} bars ({:.1} MB) in {:.2}s from {:?}",
        header.version, n, mb, elapsed.as_secs_f64(), path,
    );

    Ok(FxCacheData {
        features,
        targets,
        ofi,
        cache_key: header.cache_key,
        bar_count: n,
    })
}

/// Find a .fxcache file in cache_dir matching the given cache key.
pub fn find_fxcache(cache_dir: &Path, cache_key: &[u8; 32]) -> Option<std::path::PathBuf> {
    let hex_key = hex::encode(cache_key);
    // Try both naming patterns
    for pattern in &[
        format!("features_{}.fxcache", hex_key),
        format!("{}.fxcache", hex_key),
    ] {
        let candidate = cache_dir.join(pattern);
        if candidate.exists() {
            return Some(candidate);
        }
    }
    // Scan directory for any .fxcache and check header
    if let Ok(entries) = std::fs::read_dir(cache_dir) {
        for entry in entries.flatten() {
            let p = entry.path();
            if p.extension().and_then(|s| s.to_str()) == Some("fxcache") {
                if let Ok(data) = load_fxcache(&p) {
                    if &data.cache_key == cache_key {
                        return Some(p);
                    }
                }
            }
        }
    }
    None
}

• Step 2: Add hex dependency to ml Cargo.toml if not already present

Run: grep '^hex' crates/ml/Cargo.toml — if not present, add hex = "0.4" to [dependencies].

• Step 3: Verify it compiles

Run: SQLX_OFFLINE=true cargo check -p ml --lib 2>&1 | tail -3 Expected: compiles

• Step 4: Commit

git add crates/ml/src/fxcache.rs crates/ml/Cargo.toml
git commit -m "feat: fxcache reader — f64 and bf16 loading with find_fxcache"

---

Task 4: Roundtrip Tests

Files:

• Create: crates/ml/tests/fxcache_roundtrip_test.rs

• Step 1: Write roundtrip tests

//! Tests for .fxcache binary format: write → read roundtrip, header validation,
//! bf16 precision, and find_fxcache lookup.

use ml::fxcache::{write_fxcache, load_fxcache, find_fxcache, FxCacheHeader, FxCacheData};
use std::path::Path;
use tempfile::TempDir;

fn make_test_data(n: usize) -> (Vec<[f64; 42]>, Vec<[f64; 4]>, Vec<[f64; 8]>) {
    let features: Vec<[f64; 42]> = (0..n)
        .map(|i| {
            let mut f = [0.0f64; 42];
            for j in 0..42 { f[j] = (i * 42 + j) as f64 * 0.01; }
            f
        })
        .collect();
    let targets: Vec<[f64; 4]> = (0..n)
        .map(|i| [100.0 + i as f64, 100.5 + i as f64, 100.0 + i as f64, 100.5 + i as f64])
        .collect();
    let ofi: Vec<[f64; 8]> = (0..n)
        .map(|i| {
            let mut o = [0.0f64; 8];
            for j in 0..8 { o[j] = (i * 8 + j) as f64 * 0.001; }
            o
        })
        .collect();
    (features, targets, ofi)
}

#[test]
fn test_fxcache_f64_roundtrip() {
    let dir = TempDir::new().unwrap();
    let path = dir.path().join("test.fxcache");
    let key = [0xABu8; 32];

    let (features, targets, ofi) = make_test_data(100);

    write_fxcache(&path, &features, &targets, &ofi, key, false).unwrap();
    let loaded = load_fxcache(&path).unwrap();

    assert_eq!(loaded.bar_count, 100);
    assert_eq!(loaded.cache_key, key);
    assert_eq!(loaded.features.len(), 100);
    assert_eq!(loaded.targets.len(), 100);
    assert_eq!(loaded.ofi.len(), 100);

    // Bit-exact for f64
    assert_eq!(loaded.features[0], features[0]);
    assert_eq!(loaded.features[99], features[99]);
    assert_eq!(loaded.targets[0], targets[0]);
    assert_eq!(loaded.ofi[0], ofi[0]);
}

#[test]
fn test_fxcache_bf16_roundtrip() {
    let dir = TempDir::new().unwrap();
    let path = dir.path().join("test_bf16.fxcache");
    let key = [0xCDu8; 32];

    let (features, targets, ofi) = make_test_data(50);

    write_fxcache(&path, &features, &targets, &ofi, key, true).unwrap();
    let loaded = load_fxcache(&path).unwrap();

    assert_eq!(loaded.bar_count, 50);
    assert_eq!(loaded.cache_key, key);

    // bf16 has limited precision — check within tolerance
    for j in 0..42 {
        let diff = (loaded.features[0][j] - features[0][j]).abs();
        assert!(diff < 0.02, "feature[0][{j}] diff {diff} too large");
    }
    for j in 0..4 {
        let diff = (loaded.targets[0][j] - targets[0][j]).abs();
        assert!(diff < 1.0, "target[0][{j}] diff {diff} too large (bf16 price precision)");
    }
}

#[test]
fn test_fxcache_find() {
    let dir = TempDir::new().unwrap();
    let key = [0xEFu8; 32];
    let hex_key = hex::encode(key);
    let path = dir.path().join(format!("features_{}.fxcache", hex_key));

    let (features, targets, ofi) = make_test_data(10);
    write_fxcache(&path, &features, &targets, &ofi, key, false).unwrap();

    let found = find_fxcache(dir.path(), &key);
    assert!(found.is_some());
    assert_eq!(found.unwrap(), path);

    // Miss for different key
    let wrong_key = [0x00u8; 32];
    assert!(find_fxcache(dir.path(), &wrong_key).is_none());
}

#[test]
fn test_fxcache_header_validation() {
    let dir = TempDir::new().unwrap();
    let path = dir.path().join("bad.fxcache");

    // Write garbage
    std::fs::write(&path, b"NOT_FXCACHE_DATA_HERE").unwrap();
    let result = load_fxcache(&path);
    assert!(result.is_err());
    assert!(result.unwrap_err().to_string().contains("magic"));
}

#[test]
fn test_fxcache_empty() {
    let dir = TempDir::new().unwrap();
    let path = dir.path().join("empty.fxcache");
    let key = [0x11u8; 32];

    let empty: Vec<[f64; 42]> = vec![];
    let empty_t: Vec<[f64; 4]> = vec![];
    let empty_o: Vec<[f64; 8]> = vec![];

    write_fxcache(&path, &empty, &empty_t, &empty_o, key, false).unwrap();
    let loaded = load_fxcache(&path).unwrap();
    assert_eq!(loaded.bar_count, 0);
}

• Step 2: Run tests

Run: SQLX_OFFLINE=true cargo test -p ml --test fxcache_roundtrip_test -- --nocapture 2>&1 | tail -10 Expected: all 5 tests pass

• Step 3: Commit

git add crates/ml/tests/fxcache_roundtrip_test.rs
git commit -m "test: fxcache roundtrip tests — f64, bf16, find, validation, empty"

---

Task 5: precompute_features Binary

Files:

• Create: crates/ml/examples/precompute_features.rs

• Step 1: Write the precompute binary

This binary reuses the exact same data loading pipeline from DQNTrainer::load_training_data() but writes the output to .fxcache instead of training. The key insight: we instantiate a DQNTrainer with default hyperparams (CPU-only, no CUDA), call its data loading path, then serialize. The constructor is DQNTrainer::new(hyperparams) which takes DQNHyperparameters. The field ofi_features is pub(crate) so the example binary (same crate) can access it.

//! Pre-compute DQN training features to flat binary cache (.fxcache)
//!
//! Reads OHLCV + MBP-10 + trades DBN data, extracts all 54 features per bar
//! (42 market + 4 targets + 8 OFI), and writes a .fxcache file for zero-overhead
//! GPU loading.
//!
//! Usage:
//!     cargo run -p ml --example precompute_features --release -- \
//!         --data-dir test_data/futures-baseline \
//!         --mbp10-data-dir test_data/futures-baseline-mbp10 \
//!         --trades-data-dir test_data/futures-baseline-trades \
//!         --output-dir test_data/feature-cache \
//!         --symbol ES.FUT --yes

use anyhow::{Context, Result};
use clap::Parser;
use sha2::{Digest, Sha256};
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::time::Instant;

use ml::fxcache::write_fxcache;
use ml::trainers::dqn::config::DQNHyperparameters;
use ml::trainers::dqn::trainer::DQNTrainer;

#[derive(Debug, Parser)]
#[command(
    name = "precompute_features",
    about = "Pre-compute DQN features to .fxcache for zero-overhead GPU loading"
)]
struct Opts {
    /// OHLCV data directory (DBN files)
    #[arg(long)]
    data_dir: String,

    /// MBP-10 data directory (DBN files)
    #[arg(long)]
    mbp10_data_dir: String,

    /// Trades data directory (DBN files)
    #[arg(long)]
    trades_data_dir: String,

    /// Output directory for .fxcache files
    #[arg(long, default_value = "test_data/feature-cache")]
    output_dir: String,

    /// Symbol name (for directory structure)
    #[arg(long, default_value = "ES.FUT")]
    symbol: String,

    /// Write bf16 version (version 2) instead of f64 (version 1)
    #[arg(long)]
    bf16: bool,

    /// Skip confirmation prompt
    #[arg(long)]
    yes: bool,
}

/// Compute cache key from all source directories.
fn compute_cache_key(data_dir: &Path, mbp10_dir: &Path, trades_dir: &Path) -> Result<[u8; 32]> {
    let mut hasher = Sha256::new();

    for dir in &[data_dir, mbp10_dir, trades_dir] {
        hasher.update(dir.to_string_lossy().as_bytes());
        if dir.exists() {
            let mut files: Vec<_> = walkdir::WalkDir::new(dir)
                .into_iter()
                .filter_map(|e| e.ok())
                .filter(|e| {
                    let p = e.path().to_string_lossy();
                    p.ends_with(".dbn") || p.ends_with(".dbn.zst")
                })
                .collect();
            files.sort_by_key(|e| e.path().to_path_buf());
            for f in &files {
                hasher.update(f.path().to_string_lossy().as_bytes());
                if let Ok(meta) = f.metadata() {
                    if let Ok(mtime) = meta.modified() {
                        let ts = mtime
                            .duration_since(std::time::SystemTime::UNIX_EPOCH)
                            .unwrap_or_default()
                            .as_secs();
                        hasher.update(&ts.to_le_bytes());
                    }
                }
            }
        }
    }

    let result = hasher.finalize();
    let mut key = [0u8; 32];
    key.copy_from_slice(&result);
    Ok(key)
}

#[tokio::main]
async fn main() -> Result<()> {
    tracing_subscriber::fmt::init();
    let opts = Opts::parse();

    let data_dir = PathBuf::from(&opts.data_dir);
    let mbp10_dir = PathBuf::from(&opts.mbp10_data_dir);
    let trades_dir = PathBuf::from(&opts.trades_data_dir);

    println!("=== Feature Pre-Compute ===");
    println!("OHLCV:   {}", opts.data_dir);
    println!("MBP-10:  {}", opts.mbp10_data_dir);
    println!("Trades:  {}", opts.trades_data_dir);
    println!("Output:  {}", opts.output_dir);
    println!("Symbol:  {}", opts.symbol);
    println!("Format:  {}", if opts.bf16 { "bf16 (v2)" } else { "f64 (v1)" });
    println!();

    if !opts.yes {
        print!("Proceed? (yes/no): ");
        std::io::Write::flush(&mut std::io::stdout())?;
        let mut input = String::new();
        std::io::stdin().read_line(&mut input)?;
        if !input.trim().eq_ignore_ascii_case("yes") && !input.trim().eq_ignore_ascii_case("y") {
            println!("Cancelled.");
            return Ok(());
        }
    }

    // Build a DQNTrainer with minimal config just for data loading
    let mut hyperparams = DQNHyperparameters::default();
    hyperparams.mbp10_data_dir = Some(opts.mbp10_data_dir.clone());
    hyperparams.trades_data_dir = Some(opts.trades_data_dir.clone());

    let mut trainer = DQNTrainer::new(hyperparams)
        .context("Failed to create DQN trainer for feature extraction")?;

    // Load all data (this is the expensive step — DBN streaming + feature extraction + OFI)
    let start = Instant::now();
    println!("Loading and extracting features...");
    let (train_data, val_data) = trainer.load_training_data(&opts.data_dir).await?;

    // Merge train + val back into ordered sequence
    let mut all_data = train_data;
    all_data.extend(val_data);
    let n = all_data.len();

    let load_elapsed = start.elapsed();
    println!("Extracted {} bars in {:.1}s", n, load_elapsed.as_secs_f64());

    // Split into features + targets
    let features: Vec<[f64; 42]> = all_data.iter().map(|(f, _)| *f).collect();
    let targets: Vec<[f64; 4]> = all_data
        .iter()
        .map(|(_, t)| {
            let mut arr = [0.0f64; 4];
            for (i, v) in t.iter().take(4).enumerate() {
                arr[i] = *v;
            }
            arr
        })
        .collect();

    // Get OFI features from the trainer
    let ofi: Vec<[f64; 8]> = if let Some(ref ofi_arc) = trainer.ofi_features {
        // OFI may be shorter or longer than features — pad/truncate to match
        let ofi_slice: &[[f64; 8]] = ofi_arc;
        let mut ofi_vec = Vec::with_capacity(n);
        for i in 0..n {
            ofi_vec.push(if i < ofi_slice.len() { ofi_slice[i] } else { [0.0; 8] });
        }
        ofi_vec
    } else {
        vec![[0.0f64; 8]; n]
    };

    // Compute cache key
    let cache_key = compute_cache_key(&data_dir, &mbp10_dir, &trades_dir)?;

    // Write .fxcache
    let output_path = PathBuf::from(&opts.output_dir)
        .join(&opts.symbol)
        .join(format!("features_{}.fxcache", hex::encode(cache_key)));

    let file_size = write_fxcache(&output_path, &features, &targets, &ofi, cache_key, opts.bf16)?;

    let total_elapsed = start.elapsed();
    println!();
    println!("=== Pre-Compute Complete ===");
    println!("Bars:    {}", n);
    println!("File:    {:?}", output_path);
    println!("Size:    {:.1} MB", file_size as f64 / 1_048_576.0);
    println!("Time:    {:.1}s (extract) + {:.1}s (write) = {:.1}s total",
        load_elapsed.as_secs_f64(),
        total_elapsed.as_secs_f64() - load_elapsed.as_secs_f64(),
        total_elapsed.as_secs_f64(),
    );
    println!("Key:     {}", hex::encode(cache_key));

    Ok(())
}

• Step 2: Add walkdir dependency if not present

Run: grep '^walkdir' crates/ml/Cargo.toml — if not present, add walkdir = "2" to [dependencies].

• Step 3: Verify it compiles

Run: SQLX_OFFLINE=true cargo build -p ml --example precompute_features --release 2>&1 | tail -5 Expected: compiles (may take a while for release build)

• Step 4: Commit

git add crates/ml/examples/precompute_features.rs crates/ml/Cargo.toml
git commit -m "feat: precompute_features binary — DBN to .fxcache pipeline"

---

Task 6: Integrate fxcache Loading into DQN Data Loading

Files:

• Modify: crates/ml/src/trainers/dqn/data_loading.rs

• Step 1: Add fxcache check at the top of load_training_data()

In crates/ml/src/trainers/dqn/data_loading.rs, find the load_training_data method on DQNTrainer (the one that takes dbn_data_dir: &str). At the very beginning of the method body (after the cache check for the old binary format), add:

        // Check for .fxcache (flat binary feature cache) — fastest path
        if let Some(ref cache_dir) = self.feature_cache_dir {
            let data_dir = Path::new(dbn_data_dir);
            let mbp10_dir = self.hyperparams.mbp10_data_dir.as_ref().map(|s| Path::new(s.as_str()));
            let trades_dir = self.hyperparams.trades_data_dir.as_ref().map(|s| Path::new(s.as_str()));

            if let Ok(cache_key) = crate::feature_cache::calculate_dbn_cache_key_full(
                data_dir,
                mbp10_dir,
                trades_dir,
            ) {
                let key_bytes = hex::decode(&cache_key)
                    .ok()
                    .and_then(|b| <[u8; 32]>::try_from(b).ok());

                if let Some(key) = key_bytes {
                    if let Some(fxcache_path) = crate::fxcache::find_fxcache(cache_dir, &key) {
                        match crate::fxcache::load_fxcache(&fxcache_path) {
                            Ok(cached) => {
                                info!(
                                    "fxcache hit: {} bars from {:?}",
                                    cached.bar_count, fxcache_path
                                );

                                // Set OFI features
                                let has_ofi = cached.ofi.iter().any(|o| o.iter().any(|&v| v != 0.0));
                                if has_ofi {
                                    self.ofi_features = Some(Arc::from(cached.ofi));
                                }

                                // Combine features + targets into training pairs
                                let all_data: Vec<([f64; 42], Vec<f64>)> = cached
                                    .features
                                    .into_iter()
                                    .zip(cached.targets.into_iter())
                                    .map(|(f, t)| (f, t.to_vec()))
                                    .collect();

                                // 80/20 split
                                let split = (all_data.len() * 80) / 100;
                                let train = all_data[..split].to_vec();
                                let val = all_data[split..].to_vec();
                                return Ok((train, val));
                            }
                            Err(e) => {
                                debug!("fxcache load failed: {e}, falling through to DBN path");
                            }
                        }
                    }
                }
            }
        }

• Step 2: Verify it compiles

Run: SQLX_OFFLINE=true cargo check -p ml --lib 2>&1 | tail -3 Expected: compiles

• Step 3: Commit

git add crates/ml/src/trainers/dqn/data_loading.rs
git commit -m "feat: DQN data loader checks .fxcache before DBN streaming"

---

Task 7: Argo WorkflowTemplate for Pre-Compute

Files:

• Create: infra/k8s/argo/precompute-features-template.yaml

• Step 1: Write the WorkflowTemplate

# Pre-compute DQN training features to .fxcache format.
#
# Runs after data downloads complete. Reads DBN files from training-data PVC,
# extracts all features (OHLCV 42-dim + targets 4-dim + OFI 8-dim), writes
# flat binary cache for zero-overhead GPU loading.
#
# Usage:
#   argo submit -n foxhunt --from=wftmpl/precompute-features
#   ./scripts/argo-precompute.sh ES.FUT --bf16
---
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: precompute-features
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: precompute-features
    app.kubernetes.io/part-of: foxhunt
spec:
  entrypoint: precompute
  serviceAccountName: argo-workflow
  podMetadata:
    labels:
      app.kubernetes.io/part-of: foxhunt
      app.kubernetes.io/component: precompute-features
  securityContext:
    fsGroup: 1000
  ttlStrategy:
    secondsAfterCompletion: 3600
  activeDeadlineSeconds: 7200  # 2 hours

  arguments:
    parameters:
      - name: symbol
        value: ES.FUT
      - name: data-dir
        value: /data/futures-baseline
      - name: mbp10-dir
        value: /data/futures-baseline-mbp10
      - name: trades-dir
        value: /data/futures-baseline-trades
      - name: output-dir
        value: /data/feature-cache
      - name: node-pool
        value: ci-compile-cpu
      - name: bf16
        value: "false"

  volumes:
    - name: training-data
      persistentVolumeClaim:
        claimName: training-data-pvc

  templates:
    - name: precompute
      inputs:
        parameters:
          - name: symbol
          - name: data-dir
          - name: mbp10-dir
          - name: trades-dir
          - name: output-dir
          - name: node-pool
          - name: bf16
      nodeSelector:
        k8s.scaleway.com/pool-name: "{{inputs.parameters.node-pool}}"
      container:
        image: ubuntu:24.04
        command: ["/bin/bash", "-c"]
        env:
          - name: RUST_LOG
            value: info
        resources:
          requests:
            cpu: "4"
            memory: 8Gi
          limits:
            cpu: "16"
            memory: 32Gi
        volumeMounts:
          - name: training-data
            mountPath: /data
        args:
          - |
            set -e
            BINARY=/data/bin/precompute_features
            chmod +x "$BINARY"

            SYMBOL="{{inputs.parameters.symbol}}"
            DATA_DIR="{{inputs.parameters.data-dir}}"
            MBP10_DIR="{{inputs.parameters.mbp10-dir}}"
            TRADES_DIR="{{inputs.parameters.trades-dir}}"
            OUTPUT_DIR="{{inputs.parameters.output-dir}}"
            BF16="{{inputs.parameters.bf16}}"

            BF16_FLAG=""
            if [ "$BF16" = "true" ]; then
              BF16_FLAG="--bf16"
            fi

            echo "=== Feature Pre-Compute ==="
            echo "Symbol:  $SYMBOL"
            echo "OHLCV:   $DATA_DIR"
            echo "MBP-10:  $MBP10_DIR"
            echo "Trades:  $TRADES_DIR"
            echo "Output:  $OUTPUT_DIR"
            echo "BF16:    $BF16"
            echo "Node:    $(hostname), CPUs: $(nproc), RAM: $(free -g | awk '/Mem:/{print $2}')G"
            echo ""

            $BINARY \
              --data-dir "$DATA_DIR" \
              --mbp10-data-dir "$MBP10_DIR" \
              --trades-data-dir "$TRADES_DIR" \
              --output-dir "$OUTPUT_DIR" \
              --symbol "$SYMBOL" \
              $BF16_FLAG \
              --yes

            echo ""
            echo "=== Output ==="
            find "$OUTPUT_DIR" -name '*.fxcache' -exec ls -lh {} \;
            du -sh "$OUTPUT_DIR"

• Step 2: Validate YAML

Run: python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/precompute-features-template.yaml'))" && echo OK Expected: OK

• Step 3: Commit

git add infra/k8s/argo/precompute-features-template.yaml
git commit -m "feat: Argo WorkflowTemplate for feature pre-compute"

---

Task 8: CLI Wrapper Script

Files:

• Create: scripts/argo-precompute.sh

• Step 1: Write the script

#!/usr/bin/env bash
# Pre-compute DQN features via Argo Workflows.
#
# Usage:
#   ./scripts/argo-precompute.sh ES.FUT
#   ./scripts/argo-precompute.sh ES.FUT --bf16
#   ./scripts/argo-precompute.sh ES.FUT --pool platform --watch
#
# Requires: argo CLI
set -euo pipefail

TEMPLATE="precompute-features"
POOL=""
BF16=""
DATA_DIR=""
MBP10_DIR=""
TRADES_DIR=""
OUTPUT_DIR=""
WATCH=false

usage() {
  cat <<EOF
Usage: $(basename "$0") <symbol> [OPTIONS]

Options:
  --bf16               Write bf16 format (version 2)
  --pool <name>        Node pool (default: ci-compile-cpu)
  --data-dir <dir>     OHLCV data dir (default: /data/futures-baseline)
  --mbp10-dir <dir>    MBP-10 data dir (default: /data/futures-baseline-mbp10)
  --trades-dir <dir>   Trades data dir (default: /data/futures-baseline-trades)
  --output-dir <dir>   Output dir (default: /data/feature-cache)
  --watch              Follow workflow logs after submission
  -h, --help           Show this help
EOF
  exit 0
}

[[ $# -eq 0 ]] && { echo "Error: symbol argument required"; usage; }
[[ "$1" == "-h" || "$1" == "--help" ]] && usage

SYMBOL="$1"; shift

while [[ $# -gt 0 ]]; do
  case $1 in
    --bf16)       BF16="true"; shift ;;
    --pool)       POOL="$2"; shift 2 ;;
    --data-dir)   DATA_DIR="$2"; shift 2 ;;
    --mbp10-dir)  MBP10_DIR="$2"; shift 2 ;;
    --trades-dir) TRADES_DIR="$2"; shift 2 ;;
    --output-dir) OUTPUT_DIR="$2"; shift 2 ;;
    --watch)      WATCH=true; shift ;;
    -h|--help)    usage ;;
    *)            echo "Unknown option: $1"; usage ;;
  esac
done

CMD="argo submit -n foxhunt --from=wftmpl/$TEMPLATE"
CMD="$CMD -p symbol=$SYMBOL"

[[ -n "$BF16" ]]       && CMD="$CMD -p bf16=$BF16"
[[ -n "$POOL" ]]       && CMD="$CMD -p node-pool=$POOL"
[[ -n "$DATA_DIR" ]]   && CMD="$CMD -p data-dir=$DATA_DIR"
[[ -n "$MBP10_DIR" ]]  && CMD="$CMD -p mbp10-dir=$MBP10_DIR"
[[ -n "$TRADES_DIR" ]] && CMD="$CMD -p trades-dir=$TRADES_DIR"
[[ -n "$OUTPUT_DIR" ]] && CMD="$CMD -p output-dir=$OUTPUT_DIR"

if $WATCH; then
  CMD="$CMD --watch"
fi

echo "Submitting feature pre-compute workflow..."
echo "  symbol: $SYMBOL"
[[ -n "$BF16" ]] && echo "  bf16:   $BF16"
[[ -n "$POOL" ]] && echo "  pool:   $POOL"
echo ""
eval "$CMD"

• Step 2: Make executable and test help

Run: chmod +x scripts/argo-precompute.sh && ./scripts/argo-precompute.sh --help Expected: Shows usage

• Step 3: Commit

git add scripts/argo-precompute.sh
git commit -m "feat: argo-precompute.sh CLI wrapper"

---

Task 9: Update train-dqn Template

Files:

• Modify: infra/k8s/argo/train-dqn-template.yaml

• Step 1: Add feature-cache-dir parameter

In infra/k8s/argo/train-dqn-template.yaml, add to the arguments.parameters list:

      - name: feature-cache-dir
        value: /data/feature-cache

And pass it to the training binary invocation wherever --data-dir, --mbp10-data-dir, --trades-data-dir are passed, add:

--feature-cache-dir "{{inputs.parameters.feature-cache-dir}}"

• Step 2: Verify YAML

Run: python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/train-dqn-template.yaml'))" && echo OK Expected: OK

• Step 3: Commit

git add infra/k8s/argo/train-dqn-template.yaml
git commit -m "feat: train-dqn template passes feature-cache-dir to trainer"

---

Task 10: Local Validation — Copy Subset + Test

Files: No code changes — validation steps.

• Step 1: Copy trades Q1 from PVC (MBP-10 Q1 will be available after download completes)

kubectl run pvc-copy --rm -it --restart=Never -n foxhunt \
  --image=ubuntu:24.04 \
  --overrides='{"spec":{"nodeSelector":{"k8s.scaleway.com/pool-name":"platform"},"containers":[{"name":"c","image":"ubuntu:24.04","command":["bash","-c","cat /data/futures-baseline-trades/ES.FUT/ES.FUT_2024-Q1.dbn.zst"],"volumeMounts":[{"name":"d","mountPath":"/data"}]}],"volumes":[{"name":"d","persistentVolumeClaim":{"claimName":"training-data-pvc"}}]}}' \
  > test_data/futures-baseline-trades/ES.FUT/ES.FUT_2024-Q1.dbn.zst

(Repeat for MBP-10 Q1 once download finishes.)

• Step 2: Run precompute locally with Q1 subset

cargo run -p ml --example precompute_features --release -- \
  --data-dir test_data/futures-baseline \
  --mbp10-data-dir test_data/futures-baseline-mbp10 \
  --trades-data-dir test_data/futures-baseline-trades \
  --output-dir test_data/feature-cache \
  --symbol ES.FUT --yes

Expected: completes successfully, prints bar count and file size.

• Step 3: Validate cache file contents

# Check file exists and has reasonable size
ls -lh test_data/feature-cache/ES.FUT/*.fxcache

# Run roundtrip test with real data
SQLX_OFFLINE=true cargo test -p ml --test fxcache_roundtrip_test -- --nocapture

• Step 4: Run bf16 version

cargo run -p ml --example precompute_features --release -- \
  --data-dir test_data/futures-baseline \
  --mbp10-data-dir test_data/futures-baseline-mbp10 \
  --trades-data-dir test_data/futures-baseline-trades \
  --output-dir test_data/feature-cache \
  --symbol ES.FUT --bf16 --yes

Expected: second .fxcache file created, smaller than f64 version.

• Step 5: Smoke test training with fxcache

FOXHUNT_TEST_DATA=test_data/futures-baseline \
  cargo test -p ml --lib -- fxcache_smoke --ignored --nocapture

Expected: DQN training runs 2 epochs using cached features, completes successfully.