Adds `has_ofi: bool` to `FxCacheData` and the fxcache binary header
(stored in `reserved[0]`). Propagates through load_fxcache, write_fxcache,
all callers (precompute_features, train_baseline_rl, hyperopt dqn adapter),
existing roundtrip tests, and adds two new has_ofi-specific roundtrip tests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Different instruments (ES.FUT vs NQ.FUT) and data source modes (ohlcv vs mbp10)
now produce distinct .fxcache keys, preventing silent overwrites and wrong-bar-type
loads at cache lookup time.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- precompute_features early-exits if {cache_key}.fxcache already exists
(cache key = SHA256 of filenames + sizes, only changes when data changes)
- compile-and-train copies binaries to PVC /data/bin/ after compile
- precompute step added to compile-and-train DAG (runs after compile,
before training, parallel with GPU warmup)
- precompute template reads binary from PVC (no GitLab download)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
collect_dbn_files_filtered() checks for symbol subdirectory first,
falls back to filename matching. Precompute now loads only the
target symbol's data instead of all 4 instruments mixed together.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Normalize features once in precompute_features (single source of truth).
NormStats saved alongside .fxcache for inference denormalization.
Removed per-fold NormStats computation from train_baseline_rl, hyperopt
adapter, and smoketests — fxcache is pre-normalized, consumers use as-is.
NOTE: features still show raw price values (max=18000+) because
test_data/futures-baseline contains multiple symbols (ES, NQ, ZN, 6E)
mixed into one dataset. The feature extraction pipeline needs
investigation — log returns between different symbols produce garbage.
This is tracked as a separate task.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause of NaN at step 22: cuBLAS out-of-bounds reads in the GPU
experience collector and training forward pass, caused by buffer
dimension mismatches that produced garbage states → garbage Q-values →
NaN loss.
Bugs fixed:
1. GpuDqnTrainConfig::default() had bottleneck_dim=2, market_dim=42 —
collector computed s1_input_dim=40 instead of state_dim=80, cuBLAS
read 2x past W1 weight buffer (10KB OOB per SGEMM)
2. batch_states allocated with state_dim=80 but cuBLAS ldb=128 (CUTLASS
K-tile alignment) — 48 bf16/f32 values read past end per row
3. Validation state tensor had 59 elements/row (42+3+8+6) instead of
128 (state_dim_padded) — missing portfolio/MTF features + padding
4. CUTLASS K-tile overread on all hidden activation buffers (K=64 but
tile=128) — added 128-element padding to 14 bf16 buffers
5. init_from_fxcache never set self.ofi_features — collector OFI buffer
was 1-element placeholder, state_gather kernel read OOB
6. collect_gpu_experiences_slices missing *2 counterfactual multiplier —
half of collected experiences never inserted into PER
Dead code removed (~1228 lines):
- train_with_data_full_loop (old Vec<(FeatureVector, Vec<f64>)> loop)
- init_gpu_data, init_gpu_raw_buffers, collect_gpu_experiences,
run_training_steps (old helpers only called by above)
- train_with_preloaded_data, train_with_shared_data (old hyperopt APIs)
- train() and train_walk_forward() now convert to fixed-size arrays
and route through init_from_fxcache + train_fold_from_slices
Other fixes:
- precompute_features defaults output-dir to sibling of data-dir
- workspace_root() + feature_cache_dir() helpers for reliable path
resolution (works from any cwd, with/without CARGO_MANIFEST_DIR)
- Production smoketest uses fxcache (no slow DBN parsing)
- compute-sanitizer: 0 errors (was 2539)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each record now starts with an i64 timestamp (nanoseconds since epoch)
before the feature/target/OFI data. v1 records grow from 432 to 440
bytes, v2 from 112 to 120 bytes. The timestamp is always i64 even in
bf16 mode. train_baseline_rl reconstructs bars with real timestamps
instead of placeholders so the walk-forward windower can split by month.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Precompute binary now uses calculate_dbn_cache_key_full from the
library instead of its own compute_cache_key. Canonicalize all
paths in the key function so relative/absolute produce the same
key. Walk up parent dirs for sibling feature-cache discovery.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract standalone functions from DQNTrainer:
- extract_features_from_bars(): 42-dim feature extraction
- extract_ohlcv_bars_from_dbn(): DBN to OHLCVBar conversion
- collect_dbn_files_recursive(): file discovery
DQNTrainer delegates to these, ensuring consistency.
Precompute binary no longer initializes CUDA — runs on any
CPU node (ci-compile-cpu at EUR 0.85/hr vs GPU at EUR 12.60/hr).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DQNTrainer constructor requires CUDA for regime-conditional heads.
Use DQNTrainer::new() (GPU) instead of new_with_device(Cpu).
Set buffer_size=100_000 to satisfy PER minimum threshold.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CLI binary that reads OHLCV + MBP-10 + trades DBN data via DQNTrainer,
runs the full feature extraction pipeline, and writes a .fxcache file
for zero-overhead GPU loading during training. Uses CPU device to avoid
CUDA dependency for data-only workloads.
Also promotes load_training_data() and ofi_features to pub visibility
so the example binary can access them.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
