580 lines
21 KiB
Markdown
580 lines
21 KiB
Markdown
# Zero-Copy FxCache Training Pipeline — Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** Eliminate all per-fold CPU waste by building the entire training pipeline on FxCacheData (features[42] + targets[4] + OFI[8] + timestamps). Load once, upload to GPU once, slice by index per fold. Walk-forward uses full MBP-10/trades data via precomputed OFI features.
|
||
|
||
**Architecture:** FxCacheData replaces OHLCVBar as the data backbone. Walk-forward generates index ranges from timestamps. Trainer accepts `(&[[f64; 42]], &[[f64; 4]], &[[f64; 8]])` slices instead of `Vec<(FeatureVector, Vec<f64>)>`. Data uploaded to GPU once; per-fold transitions set index bounds only.
|
||
|
||
**Tech Stack:** Rust 1.85, cudarc 0.17.3, CUDA
|
||
|
||
---
|
||
|
||
## Current Data Flow (per fold)
|
||
|
||
```
|
||
fxcache ─→ (features[42], targets[4], OFI[8], timestamps) # 470MB, loaded once
|
||
│
|
||
▼
|
||
OHLCVBars reconstructed from targets (close prices) # WASTE: fxcache has everything
|
||
│
|
||
▼
|
||
generate_walk_forward_windows(bars) # Clones bars into Vec per fold
|
||
│
|
||
▼
|
||
prepare_fold_data(window) # RE-EXTRACTS features from bars (!)
|
||
│
|
||
▼
|
||
features_to_trainer_format(features, bars) # 4M × Vec<f64> heap allocs
|
||
│
|
||
▼
|
||
DQNTrainer::new() per fold # Rebuilds GPU pipeline
|
||
│
|
||
▼
|
||
init_gpu_data → flatten + upload to GPU # Unpacks tuples, flattens, HtoD
|
||
│
|
||
▼
|
||
Training epochs (GPU) # The only useful work
|
||
```
|
||
|
||
## Target Data Flow
|
||
|
||
```
|
||
fxcache ─→ FxCacheData { features[42], targets[4], OFI[8], timestamps }
|
||
│
|
||
▼ (one-time)
|
||
Upload to GPU: features_cuda, targets_cuda, ofi_cuda
|
||
│
|
||
▼ (one-time)
|
||
generate_walk_forward_indices(timestamps) → Vec<FoldRange>
|
||
│
|
||
▼ (one-time)
|
||
DQNTrainer::new()
|
||
│
|
||
▼ (per fold, ~0.1s)
|
||
trainer.set_training_range(range.train_start, range.train_end, ...)
|
||
trainer.reset_for_fold()
|
||
NormStats from &features[train_start..train_end] # Slice, no copy
|
||
│
|
||
▼
|
||
Training epochs (GPU reads from offset in pre-uploaded arrays)
|
||
```
|
||
|
||
## File Structure
|
||
|
||
### Modified Files
|
||
|
||
| File | Changes |
|
||
|------|---------|
|
||
| `crates/ml/src/walk_forward.rs` | `generate_walk_forward_indices_from_timestamps(&[i64], config) → Vec<FoldRange>` — date slicing on nanosecond timestamps from fxcache, no OHLCVBar dependency |
|
||
| `crates/ml/src/trainers/dqn/trainer/mod.rs` | New `train_with_fxcache_data()` API accepting `&FxCacheData`. Remove `Vec<(FeatureVector, Vec<f64>)>` from the hot path. Add `upload_fxcache_to_gpu()` for one-time upload |
|
||
| `crates/ml/src/trainers/dqn/trainer/training_loop.rs` | `init_gpu_raw_buffers_from_slices()` accepting `(&[[f64; 42]], &[[f64; 4]], &[[f64; 8]])`. Remove per-element flattening loops |
|
||
| `crates/ml/src/cuda_pipeline/mod.rs` | `DqnGpuData::upload_slices()` accepting contiguous `&[[f64; 42]]` + `&[[f64; 4]]` — no tuple unpacking |
|
||
| `crates/ml/examples/train_baseline_rl.rs` | Complete rewrite of fold loop: load fxcache → upload GPU → index ranges → per-fold: set range, reset, train. Delete `prepare_fold_data`, `FoldData`, `features_to_trainer_format`, double-buffer, prefetch thread |
|
||
|
||
---
|
||
|
||
### Task 1: Walk-forward from timestamps (already partially done)
|
||
|
||
**Files:**
|
||
- Modify: `crates/ml/src/walk_forward.rs`
|
||
|
||
`generate_walk_forward_indices` (added in previous commit) uses `&[OHLCVBar]` for date slicing. Add a variant that works directly on `&[i64]` timestamps from fxcache — no OHLCVBar needed.
|
||
|
||
- [ ] **Step 1: Add `generate_walk_forward_indices_from_timestamps`**
|
||
|
||
```rust
|
||
/// Generate walk-forward index ranges from nanosecond timestamps.
|
||
/// Works directly on fxcache timestamps — no OHLCVBar conversion needed.
|
||
pub fn generate_walk_forward_indices_from_timestamps(
|
||
timestamps_ns: &[i64],
|
||
config: &WalkForwardConfig,
|
||
) -> Vec<FoldRange> {
|
||
if timestamps_ns.is_empty() {
|
||
return Vec::new();
|
||
}
|
||
let ts_to_date = |ts: i64| -> NaiveDate {
|
||
chrono::DateTime::from_timestamp_nanos(ts).date_naive()
|
||
};
|
||
let data_start = ts_to_date(timestamps_ns[0]);
|
||
let data_end = ts_to_date(*timestamps_ns.last().unwrap());
|
||
// ... same fold logic as generate_walk_forward_indices
|
||
// but uses partition_point on timestamps instead of bars
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 2: Add `compute_difficulty_from_features` that reads ADX from features[40]**
|
||
|
||
The current `compute_difficulty(&[OHLCVBar])` computes ADX from raw bars. With fxcache, ADX is already at feature index 40. Add:
|
||
|
||
```rust
|
||
pub fn compute_difficulty_from_features(features: &[[f64; 42]]) -> f64 {
|
||
if features.is_empty() { return 0.0; }
|
||
let adx_idx = 40;
|
||
let sum: f64 = features.iter().map(|f| f[adx_idx]).sum();
|
||
sum / features.len() as f64
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 3: Compile**
|
||
|
||
```bash
|
||
SQLX_OFFLINE=true cargo check -p ml
|
||
```
|
||
|
||
- [ ] **Step 4: Commit**
|
||
|
||
---
|
||
|
||
### Task 2: DqnGpuData::upload_slices — contiguous array upload
|
||
|
||
**Files:**
|
||
- Modify: `crates/ml/src/cuda_pipeline/mod.rs`
|
||
|
||
The current `upload()` takes `&[([f64; 42], Vec<f64>)]` and iterates element-by-element to flatten. Add `upload_slices()` that accepts pre-contiguous arrays from fxcache.
|
||
|
||
- [ ] **Step 1: Add `upload_slices` method**
|
||
|
||
```rust
|
||
/// Upload contiguous feature/target slices to GPU (zero per-element iteration).
|
||
/// Accepts fxcache arrays directly — no tuple unpacking, no Vec<f64> allocation.
|
||
pub fn upload_slices(
|
||
features: &[[f64; 42]],
|
||
targets: &[[f64; 4]],
|
||
ofi: &[[f64; 8]],
|
||
stream: &Arc<CudaStream>,
|
||
) -> Result<Self, MLError> {
|
||
let num_bars = features.len();
|
||
let feature_dim = 42;
|
||
|
||
// Cast f64 → f32 in one contiguous pass (no per-element Vec push)
|
||
let flat_features: Vec<f32> = features.iter()
|
||
.flat_map(|f| f.iter().map(|&v| v as f32))
|
||
.collect();
|
||
let flat_targets: Vec<f32> = targets.iter()
|
||
.flat_map(|t| t.iter().map(|&v| v as f32))
|
||
.collect();
|
||
|
||
let features_gpu = clone_htod_f32_to_bf16(stream, &flat_features)?;
|
||
let targets_gpu = clone_htod_f32_to_bf16(stream, &flat_targets)?;
|
||
|
||
let mut data = Self {
|
||
features: features_gpu,
|
||
targets: targets_gpu,
|
||
ofi_features: None,
|
||
num_bars,
|
||
feature_dim,
|
||
aligned_state_dim: None,
|
||
};
|
||
|
||
// Upload OFI if present (non-zero)
|
||
if !ofi.is_empty() && ofi.iter().any(|o| o.iter().any(|&v| v != 0.0)) {
|
||
let flat_ofi: Vec<f32> = ofi.iter()
|
||
.flat_map(|o| o.iter().map(|&v| v as f32))
|
||
.collect();
|
||
let ofi_gpu = clone_htod_f32_to_bf16(stream, &flat_ofi)?;
|
||
data.ofi_features = Some(ofi_gpu);
|
||
}
|
||
|
||
Ok(data)
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 2: Compile**
|
||
|
||
```bash
|
||
SQLX_OFFLINE=true cargo check -p ml
|
||
```
|
||
|
||
- [ ] **Step 3: Commit**
|
||
|
||
---
|
||
|
||
### Task 3: Trainer API — `init_from_fxcache`
|
||
|
||
**Files:**
|
||
- Modify: `crates/ml/src/trainers/dqn/trainer/mod.rs`
|
||
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs`
|
||
|
||
Add `init_from_fxcache()` that uploads the full fxcache dataset to GPU ONCE. This replaces the per-fold `init_gpu_data` + `init_gpu_raw_buffers` pair.
|
||
|
||
- [ ] **Step 1: Add `init_from_fxcache` on DQNTrainer**
|
||
|
||
```rust
|
||
/// Upload full fxcache dataset to GPU ONCE. Call before the fold loop.
|
||
/// Replaces per-fold init_gpu_data + init_gpu_raw_buffers.
|
||
pub async fn init_from_fxcache(
|
||
&mut self,
|
||
features: &[[f64; 42]],
|
||
targets: &[[f64; 4]],
|
||
ofi: &[[f64; 8]],
|
||
) -> Result<()> {
|
||
let stream = self.cuda_stream.as_ref()
|
||
.ok_or_else(|| anyhow::anyhow!("CUDA stream required"))?;
|
||
|
||
// Upload via DqnGpuData::upload_slices (contiguous, zero tuple unpacking)
|
||
let gpu_data = DqnGpuData::upload_slices(features, targets, ofi, stream)?;
|
||
let ofi_enabled = gpu_data.ofi_features.is_some();
|
||
let raw_dim = if ofi_enabled { 53 } else { 45 };
|
||
let aligned_dim = (raw_dim + 7) & !7;
|
||
info!("init_from_fxcache: {} bars uploaded to GPU ({:.1} MB)",
|
||
features.len(),
|
||
(features.len() * (42 + 4 + 8) * 4) as f64 / 1_048_576.0);
|
||
self.gpu_data = Some(gpu_data);
|
||
|
||
// Also upload raw buffers for experience collector
|
||
self.init_gpu_raw_buffers_from_slices(features, targets).await?;
|
||
|
||
Ok(())
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 2: Add `init_gpu_raw_buffers_from_slices` in training_loop.rs**
|
||
|
||
```rust
|
||
/// Upload raw features + targets from contiguous slices (no tuple unpacking).
|
||
pub(crate) async fn init_gpu_raw_buffers_from_slices(
|
||
&mut self,
|
||
features: &[[f64; 42]],
|
||
targets: &[[f64; 4]],
|
||
) -> Result<()> {
|
||
if self.targets_raw_cuda.is_some() {
|
||
return Ok(());
|
||
}
|
||
let stream = match self.cuda_stream {
|
||
Some(ref s) => Arc::clone(s),
|
||
None => return Ok(()),
|
||
};
|
||
|
||
let num_bars = features.len();
|
||
let flat_targets: Vec<f32> = targets.iter()
|
||
.flat_map(|t| t.iter().map(|&v| v as f32))
|
||
.collect();
|
||
let flat_features: Vec<f32> = features.iter()
|
||
.flat_map(|f| f.iter().map(|&v| v as f32))
|
||
.collect();
|
||
|
||
self.targets_raw_cuda = Some(
|
||
crate::cuda_pipeline::clone_htod_f32_to_bf16(&stream, &flat_targets)?
|
||
);
|
||
self.features_raw_cuda = Some(
|
||
crate::cuda_pipeline::clone_htod_f32_to_bf16(&stream, &flat_features)?
|
||
);
|
||
|
||
info!("CUDA raw buffers uploaded: {} bars x 42 features + 4 targets", num_bars);
|
||
Ok(())
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 3: Add `set_val_data_from_slices` for validation**
|
||
|
||
The trainer needs val_data for epoch-end Sharpe computation. Instead of `Vec<(FeatureVector, Vec<f64>)>`, accept slices:
|
||
|
||
```rust
|
||
/// Set validation data from contiguous slices (zero Vec<f64> allocation).
|
||
pub fn set_val_data_from_slices(
|
||
&mut self,
|
||
features: &[[f64; 42]],
|
||
targets: &[[f64; 4]],
|
||
ofi_val_offset: usize,
|
||
) {
|
||
// Convert to the legacy format for now — val set is small (~50K bars)
|
||
// Full GPU-native validation is a future task
|
||
self.val_data = features.iter().zip(targets.iter())
|
||
.map(|(f, t)| (*f, t.to_vec()))
|
||
.collect();
|
||
self.ofi_val_offset = ofi_val_offset;
|
||
self.val_features_gpu = None;
|
||
self.val_closes_gpu = None;
|
||
self.val_ofi_gpu = None;
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 4: Compile**
|
||
|
||
```bash
|
||
SQLX_OFFLINE=true cargo check -p ml
|
||
```
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
---
|
||
|
||
### Task 4: Rewrite train_baseline_rl fold loop
|
||
|
||
**Files:**
|
||
- Modify: `crates/ml/examples/train_baseline_rl.rs`
|
||
|
||
This is the integration task. Replace the entire fold loop to:
|
||
1. Load fxcache ONCE
|
||
2. Create DQNTrainer ONCE
|
||
3. Upload to GPU ONCE via `init_from_fxcache`
|
||
4. Generate index ranges from timestamps
|
||
5. Per fold: slice features/targets, normalize, set_training_range, reset_for_fold, train
|
||
|
||
- [ ] **Step 1: Replace data loading section (lines ~798-891)**
|
||
|
||
Replace the fxcache + DBN loading + OHLCVBar reconstruction with:
|
||
|
||
```rust
|
||
// Load fxcache — contains features[42] + targets[4] + OFI[8] + timestamps
|
||
// MBP-10 orderbook and trades data are pre-embedded in OFI features
|
||
let fxcache = load_fxcache_data(&args)?;
|
||
info!("Loaded fxcache: {} bars, features[42] + targets[4] + OFI[8]",
|
||
fxcache.bar_count);
|
||
```
|
||
|
||
Where `load_fxcache_data` is a helper that does the cache lookup (same as current code but returns `FxCacheData` directly, no OHLCVBar conversion).
|
||
|
||
- [ ] **Step 2: Replace walk-forward window generation**
|
||
|
||
```rust
|
||
let fold_ranges = generate_walk_forward_indices_from_timestamps(
|
||
&fxcache.timestamps, &wf_config,
|
||
);
|
||
info!("Generated {} walk-forward folds", fold_ranges.len());
|
||
```
|
||
|
||
- [ ] **Step 3: Create DQN trainer + upload to GPU ONCE**
|
||
|
||
```rust
|
||
let hp = load_hyperopt_params(&args.hyperopt_params, "dqn");
|
||
let hyperparams = build_dqn_hyperparams(args, &hp, &gpu_profile);
|
||
let mut trainer = DQNTrainer::new(hyperparams)?;
|
||
|
||
// One-time GPU upload of full dataset
|
||
let rt = tokio::runtime::Builder::new_current_thread()
|
||
.enable_all().build()?;
|
||
rt.block_on(trainer.init_from_fxcache(
|
||
&fxcache.features, &fxcache.targets, &fxcache.ofi,
|
||
))?;
|
||
```
|
||
|
||
- [ ] **Step 4: Replace fold loop body**
|
||
|
||
```rust
|
||
for range in &fold_ranges {
|
||
info!("--- Fold {} ---", range.fold);
|
||
|
||
// Slice features for normalization (zero copy — just index bounds)
|
||
let train_feat = &fxcache.features[range.train_start..range.train_end];
|
||
let val_feat = &fxcache.features[range.val_start..range.val_end];
|
||
let train_targets = &fxcache.targets[range.train_start..range.train_end];
|
||
let val_targets = &fxcache.targets[range.val_start..range.val_end];
|
||
|
||
// NormStats from training slice (CPU, ~10ms for 500K bars)
|
||
let norm_stats = NormStats::from_features(train_feat);
|
||
let train_norm = norm_stats.normalize_batch(train_feat);
|
||
let val_norm = norm_stats.normalize_batch(val_feat);
|
||
|
||
// Set fold range on trainer (GPU arrays stay put, just index bounds change)
|
||
trainer.set_training_range(
|
||
range.train_start, range.train_end,
|
||
range.val_start, range.val_end,
|
||
);
|
||
|
||
// Set validation data for epoch-end Sharpe
|
||
trainer.set_val_data_from_slices(val_feat, val_targets, range.train_end - range.train_start);
|
||
|
||
// Reset per-fold state (keep GPU infra)
|
||
rt.block_on(trainer.reset_for_fold())?;
|
||
|
||
// Checkpoint callback
|
||
let output_dir = args.output_dir.clone();
|
||
let fold = range.fold;
|
||
let checkpoint_callback = move |epoch: usize, data: Vec<u8>, is_best: bool| -> Result<String> {
|
||
let suffix = if is_best { "best" } else { &format!("epoch{}", epoch) };
|
||
let path = output_dir.join(format!("dqn_fold{}_{}.safetensors", fold, suffix));
|
||
std::fs::write(&path, &data)?;
|
||
Ok(path.to_string_lossy().into_owned())
|
||
};
|
||
|
||
// Train — trainer reads from GPU arrays at fold range offset
|
||
// NOTE: still passes train_norm as Vec<(FeatureVector, Vec<f64>)> for now.
|
||
// Full elimination of this conversion requires changing train_with_data_full_loop
|
||
// to read directly from GPU buffers — tracked as follow-up.
|
||
let training_data = features_to_trainer_format_fast(&train_norm, train_targets);
|
||
let val_data_fmt = features_to_trainer_format_fast(&val_norm, val_targets);
|
||
|
||
match rt.block_on(trainer.train_with_preloaded_data(training_data, val_data_fmt, checkpoint_callback)) {
|
||
Ok(metrics) => {
|
||
info!("Fold {} complete: loss={:.6}, epochs={}", fold, metrics.loss, metrics.epochs_trained);
|
||
dqn_results.push((fold, metrics.loss));
|
||
}
|
||
Err(e) => error!("Fold {} failed: {:#}", fold, e),
|
||
}
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 5: Add `features_to_trainer_format_fast` helper**
|
||
|
||
Faster version that avoids per-bar `vec![open, high, low, close]`:
|
||
|
||
```rust
|
||
fn features_to_trainer_format_fast(
|
||
features: &[[f64; 42]],
|
||
targets: &[[f64; 4]],
|
||
) -> Vec<(FeatureVector, Vec<f64>)> {
|
||
features.iter().zip(targets.iter())
|
||
.map(|(f, t)| (*f, t.to_vec()))
|
||
.collect()
|
||
}
|
||
```
|
||
|
||
This still allocates `Vec<f64>` per bar (4 elements), but eliminates:
|
||
- Bar cloning (320MB)
|
||
- Feature re-extraction (30s CPU)
|
||
- Trainer re-creation (3s GPU init)
|
||
- Data re-upload (1s HtoD)
|
||
|
||
The `Vec<f64>` allocation (~32 bytes per bar × 500K bars = 16MB) is the last remaining waste. Eliminating it requires changing `train_with_data_full_loop` to accept `&[[f64; 4]]` — tracked as Task 6.
|
||
|
||
- [ ] **Step 6: Delete dead code**
|
||
|
||
Remove:
|
||
- `prepare_fold_data` function
|
||
- `FoldData` type alias
|
||
- `features_to_trainer_format` function (replaced by `_fast` variant)
|
||
- `prefetched_data` and fold prefetch thread
|
||
- `dqn_gpu_staged` double-buffer
|
||
- `DqnGpuData::upload` from double-buffer path
|
||
|
||
- [ ] **Step 7: Compile and test locally**
|
||
|
||
```bash
|
||
SQLX_OFFLINE=true FOXHUNT_TEST_DATA=test_data/futures-baseline cargo run --release -p ml --example train_baseline_rl -- --model dqn --epochs 3 --data-dir test_data/futures-baseline 2>&1 | tail -20
|
||
```
|
||
|
||
Expected: 3 folds × 3 epochs in <10s (was 159s).
|
||
|
||
- [ ] **Step 8: Commit**
|
||
|
||
---
|
||
|
||
### Task 5: Eliminate last Vec<f64> allocation (trainer API)
|
||
|
||
**Files:**
|
||
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs`
|
||
- Modify: `crates/ml/src/trainers/dqn/trainer/mod.rs`
|
||
|
||
Change `train_with_data_full_loop` to accept `(&[[f64; 42]], &[[f64; 4]])` instead of `&[(FeatureVector, Vec<f64>)]`. This eliminates the 4M × `Vec<f64>` allocation.
|
||
|
||
- [ ] **Step 1: Add `train_fold_from_slices` on DQNTrainer**
|
||
|
||
```rust
|
||
/// Train one fold from contiguous feature/target slices.
|
||
/// Zero Vec<f64> allocation — reads targets directly as &[[f64; 4]].
|
||
pub async fn train_fold_from_slices<F>(
|
||
&mut self,
|
||
features: &[[f64; 42]],
|
||
targets: &[[f64; 4]],
|
||
checkpoint_callback: F,
|
||
) -> Result<TrainingMetrics>
|
||
where
|
||
F: FnMut(usize, Vec<u8>, bool) -> Result<String> + Send,
|
||
{
|
||
// GPU data already uploaded via init_from_fxcache — skip re-upload
|
||
// Training loop uses self.features_raw_cuda / self.targets_raw_cuda
|
||
// which are the FULL dataset on GPU. Fold range selects the slice.
|
||
|
||
let num_bars = features.len();
|
||
info!("train_fold_from_slices: {} bars (GPU range {}..{})",
|
||
num_bars, self.fold_train_start, self.fold_train_end);
|
||
|
||
// The training loop still needs training_data for:
|
||
// 1. .len() → use fold range
|
||
// 2. Curriculum ADX filter → read features[bar][40]
|
||
// 3. Vol normalizer → read targets for returns
|
||
// Build a lightweight wrapper that avoids Vec<f64>
|
||
let training_data_ref: Vec<([f64; 42], [f64; 4])> = features.iter()
|
||
.zip(targets.iter())
|
||
.map(|(f, t)| (*f, *t))
|
||
.collect();
|
||
|
||
self.train_with_data_full_loop_slices(&training_data_ref, checkpoint_callback).await
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 2: Add `train_with_data_full_loop_slices` in training_loop.rs**
|
||
|
||
Mirror of `train_with_data_full_loop` but accepts `&[([f64; 42], [f64; 4])]`. The internal code only needs `.len()`, `features[bar][adx_idx]`, and `targets[bar]` for vol normalizer — all work with fixed-size arrays.
|
||
|
||
- [ ] **Step 3: Update fold loop in train_baseline_rl to use new API**
|
||
|
||
Replace `features_to_trainer_format_fast` + `train_with_preloaded_data` with:
|
||
|
||
```rust
|
||
match rt.block_on(trainer.train_fold_from_slices(&train_norm, train_targets, checkpoint_callback)) {
|
||
```
|
||
|
||
- [ ] **Step 4: Compile and test**
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
---
|
||
|
||
### Task 6: Validate locally and on H100
|
||
|
||
- [ ] **Step 1: Local validation**
|
||
|
||
```bash
|
||
SQLX_OFFLINE=true FOXHUNT_TEST_DATA=test_data/futures-baseline cargo run --release -p ml --example train_baseline_rl -- --model dqn --epochs 3 --data-dir test_data/futures-baseline 2>&1 | tail -20
|
||
```
|
||
|
||
Expected: <10s for 3 folds × 3 epochs.
|
||
|
||
- [ ] **Step 2: Smoketest**
|
||
|
||
```bash
|
||
SQLX_OFFLINE=true FOXHUNT_TEST_DATA=test_data/futures-baseline cargo test -p ml --lib -- smoke_tests::training_stability::test_gpu_collector_auto_initializes --ignored --nocapture 2>&1 | tail -5
|
||
```
|
||
|
||
Expected: PASS, <60s.
|
||
|
||
- [ ] **Step 3: H100 submission**
|
||
|
||
```bash
|
||
./scripts/argo-train.sh dqn --epochs 50 --trials 0 --gpu-pool ci-training-h100
|
||
```
|
||
|
||
Expected:
|
||
- fxcache hit (precomputed on PVC)
|
||
- One-time GPU upload: <2s
|
||
- Per-fold transition: <0.5s
|
||
- Per-epoch (batch=8192, 488 steps): <5s
|
||
- Total 50 epochs × 3 folds: <15 minutes
|
||
|
||
---
|
||
|
||
## Performance Summary
|
||
|
||
| Operation | Before | After |
|
||
|-----------|--------|-------|
|
||
| Data load (fxcache hit) | ~5s | ~5s (unchanged) |
|
||
| Walk-forward window gen | ~200ms + 320MB cloned bars | ~1ms (index ranges only) |
|
||
| Feature extraction per fold | ~30s CPU | 0s (pre-computed in fxcache) |
|
||
| Trainer creation per fold | ~3s GPU init | 0s (reused) |
|
||
| Data upload per fold | ~1s HtoD | 0s (stays on GPU) |
|
||
| Vec<f64> allocs per fold | ~16MB (4M × 4 elements) | 0 (Task 5: fixed-size arrays) |
|
||
| NormStats per fold | ~10ms (unchanged) | ~10ms (slice, no copy) |
|
||
| **Total fold transition** | **~34s** | **~10ms** |
|
||
| **3-fold overhead** | **~102s** | **~30ms** |
|
||
|
||
## Data Quality Improvement
|
||
|
||
The walk-forward now uses the FULL precomputed feature set from fxcache:
|
||
- **42 market features**: OHLCV derivatives + technical indicators + ADX + CUSUM
|
||
- **8 OFI features**: from MBP-10 order book (OFI L1/L5, depth imbalance, VPIN, Kyle's Lambda, bid/ask slope, trade imbalance)
|
||
- **4 targets**: raw close prices for portfolio simulation
|
||
|
||
This matches production: the model trains on exactly the same feature set it will see in real-time trading, including orderbook microstructure from MBP-10 and trade flow data.
|
||
|
||
## Risks
|
||
|
||
| Risk | Impact | Mitigation |
|
||
|------|--------|-----------|
|
||
| NormStats computed per fold on CPU | ~10ms — negligible | Future: GPU reduction kernel |
|
||
| val_data still uses legacy format | Small alloc (~50K × Vec<f64>) | `set_val_data_from_slices` minimizes, full GPU val in follow-up |
|
||
| OHLCVBar removal breaks PPO path | PPO uses same fold loop | Keep PPO using legacy path for now, separate refactor |
|
||
| train_with_data_full_loop signature change | Touches many callers | Task 5 adds new method, doesn't change existing |
|