OFI features at bar t were computed over [close(bar_t), close(bar_{t+1}))
— bar t+1's formation period — so 31 of 32 OFI dims at the policy's
input for bar t carried next-bar microstructure data. Per audit
`docs/lookahead-bias-audit-2026-04-28.md` §3, this is a DEFINITE leak
contaminating both training inputs and validation backtest.
Fix: shift the window backward to (close(bar_{t-1}), close(bar_t)] so
OFI at bar t uses only data accumulated during bar t's own formation.
Mirrors the correct `log_bar_duration` convention at
precompute_features.rs:567-575 (audit's smoking-gun citation).
Per feedback_no_partial_refactor, both write sites migrated in lockstep:
- crates/ml/examples/precompute_features.rs:441-475 (precompute path)
- crates/ml/src/trainers/dqn/data_loading.rs:396-448 (DBN fallback)
FXCACHE_VERSION bumped 6 → 7. PVC auto-regen via schema-hash check on
next deploy. Local regen:
./target/release/examples/precompute_features \
--data-dir test_data/futures-baseline \
--mbp10-data-dir test_data/futures-baseline-mbp10 \
--trades-data-dir test_data/futures-baseline-trades \
--output-dir test_data/feature-cache \
--symbol ES.FUT --data-source mbp10 --yes
Regression test added: tests/ofi_features_test.rs::
test_ofi_window_alignment_uses_formation_interval validates the new
partition_point predicates against a synthetic 5-bar dataset, including
an explicit anti-regression assertion that bar t+1 snapshots are NEVER
picked up for bar t.
dqn-wire-up-audit.md updated to reflect new contract for
`trainers/dqn/data_loading.rs`. Audit report
`docs/lookahead-bias-audit-2026-04-28.md` checked in alongside the fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Smoke and localdev profiles previously used data_source = "ohlcv", divergent
from production (mbp10). Mismatch silently produced cache-key collisions in
the SHA256 hash path: smoke fxcache could not be loaded by production-shape
training without an explicit override. Local-dev fxcache regen also required
remembering to pass --data-source ohlcv to match.
Globalize mbp10 as the default everywhere it isn't deliberately overridden:
- config/training/dqn-smoketest.toml: data_source = "mbp10"
- config/training/dqn-localdev.toml: data_source = "mbp10"
- training_profile.rs: doc Default → "mbp10"
- trainers/dqn/config.rs: Default impl → "mbp10"
- hyperopt/adapters/dqn.rs: default → "mbp10"
- examples/precompute_features.rs: doc updated
- fxcache.rs / feature_cache.rs: discover_and_load + cache-key tests
use "mbp10" arguments
- docs/dqn-wire-up-audit.md: new entry per Invariant 7
Documentation strings retained "ohlcv" only where they document the two
available choices (config.rs:946, training_profile.rs:83).
Local fxcache regenerated to v6 mbp10:
test_data/feature-cache/13c0b086a975cc7e2384377a2cd0e97738c9410292fcfecb5807c29bf885cb48.fxcache
(175874 bars, 55 MB, OFI_DIM=32). Stale v5 ohlcv fxcache untracked
from git index (already gitignored post-79578bbaf).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two-part fix for a class of bugs causing un-normalised features to
silently flow into training.
(1) data_source mismatch between precompute writer and trainer reader.
precompute_features.rs:214,633 hardcoded "ohlcv"
train_baseline_rl.rs:582 hardcoded "ohlcv"
config/training/dqn-production.toml: data_source = "mbp10"
Production runs the trainer with the production profile (data_source
= mbp10), but the actual cache lookup hardcoded "ohlcv". Smoke worked
by accident (smoke profile is also "ohlcv"). Any future profile with
a different data_source silently mismatches → cache MISS → DBN
fallback path.
Both call sites now hardcode "mbp10" (the canonical production data
source per CLAUDE.md). precompute_features adds a `--data-source`
CLI override for the rare case a smoke flow needs to regenerate
the local "ohlcv" fxcache; default is "mbp10".
(2) DBN-fallback path didn't normalise features.
precompute_features.rs:629 applies NormStats::normalize_batch on the
canonical fxcache write path. The fallback in train_baseline_rl.rs
(cache-miss → load DBN files → extract features → upload to GPU) did
NOT normalise. Any cache miss (data_source drift, schema-hash mismatch,
missing file) silently uploaded RAW features. Raw close prices
(~$5180 ES futures) flowed into next_states[:, 0]; the aux next-bar
head's label_scale EMA latched onto raw-price magnitude (~5443 vs
expected ~1.0 z-score); the shared trunk learned to predict next-bar
prices; the policy effectively traded with future-price knowledge →
train-h5gxb epoch-0 Sharpe = 141 with 0.32% max-drawdown over 214k
bars (impossibly good = oracle leak).
DBN fallback now applies the same z-score normalisation unconditionally
as defence-in-depth, so a future cache-miss cannot reintroduce raw
values into training.
Audit entry updated.
fix kernel-read gap
Adds 12 features to the DQN input pipeline:
- 10 MicrostructureState::snapshot()[0..10] slots that were previously computed
every bar and then discarded before reaching fxcache: ofi_trajectory,
realized_variance, hawkes_intensity, book_pressure (weighted 10-level),
spread_dynamics, aggression_ratio, queue_depletion_asymmetry,
order_count_flux, intra_bar_momentum, regime_score.
- 2 TLOB-novel slots derived directly from Mbp10Snapshot:
order_count_imbalance = (Σbid_ct − Σask_ct) / Σ(bid_ct + ask_ct),
microprice_residual = (weighted_mid − mid) / mid.
Also fixes a production gap: ofi_acceleration (slot 18) and
toxicity_gradient (slot 19) were persisted to fxcache via OFI_DIM=20
but the OFI embed kernel (experience_kernels.cu:6146-6173) only read
[0..18), silently discarding them every bar. Kernel extended to
consume full SL_OFI_DIM=32.
Dimension bumps (all 8-aligned):
OFI_DIM 20 → 32
FXCACHE_VERSION 4 → 5 (invalidates existing caches; regen via
precompute_features)
STATE_DIM 96 → 104
PADDING_DIM 4 → 0 (OFI expansion consumed padding, still 8-aligned)
STATE_DIM_PADDED 128 (unchanged)
OFI_EMBED_IN 18 → 32 (MLP input width; W/grad/Adam/m/v buffers
resized in lockstep via named constants)
fxcache regen results (175874 bars ES.FUT 2024-Q1):
deltas_nonzero: 175781 / 175874 (99.9 percent)
book_aggression: 102137 / 175874 (58.1 percent)
microstructure[20-30): 175874 / 175874 (100 percent)
tlob_novel[30-32): 133615 / 175874 (76.0 percent)
Compile status: SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo check
--workspace --tests passes cleanly (0 errors, pre-existing warnings
only).
Test results:
fxcache roundtrip (unit + integration): PASS (4+6 tests)
magnitude_distribution smoke: ran through epoch 1 successfully
(OFI_DIAG fires, state_dim=104 confirmed, feature_dim=74 in
validation kernel); epoch 2 OOM on local RTX 3050 Ti (4 GB) —
expected hardware limit from state_dim growth. Full 20-epoch run
requires L40S/H100 CI verification.
multi_fold_convergence smoke: not verified locally (same VRAM
ceiling applies). L40S/H100 CI verification required.
The new slots follow the existing OFICalculator/MicrostructureState
pattern and consume signals already computed by ml-features — no new
crate, no ONNX, no stubs. All 12 sources were audited against their
implementation before persistence; every slot traces back to real
Mbp10Snapshot or MicrostructureState math.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The v2 fxcache on PVC passed has_ofi=true validation (mbp10_dir was present
when built) but contained all-zero OFI data. The old binary set the flag
based on directory existence, not actual computed values.
Now counts non-zero OFI rows before accepting a cache — forces rebuild
when MBP-10 data is available but OFI content is all zeros.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OFI (Order Flow Imbalance) features are mandatory — the model is worthless
without MBP-10 order book data. Every silent fallback that degraded to zeros
has been converted to a hard error.
Critical fixes:
- build_batch_states used *8 instead of *20 for OFI dimensions — every
walk-forward backtest was reading corrupted OFI features from adjacent memory
- precompute_features early exit skipped cache rebuild when stale v2/v3
cache existed with has_ofi=false — now validates has_ofi before skipping
- 14 silent OFI fallback paths converted to hard errors across data loading,
training loop, experience collector, state construction, metrics, hyperopt
Dead code removed (-751 lines):
- DoubleBufferedLoader (superseded by init_from_fxcache)
- GpuBufferPool (superseded by init_from_fxcache)
- DqnGpuData::upload legacy method (no OFI support)
- CPU training fallback path (CUDA always required)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
FXCACHE_VERSION 3→4. Three precomputed features added to OFI region:
- ofi[8..16): temporal deltas (ofi[bar] - ofi[bar-1]) for 8 features
- ofi[16]: book aggression (MBP-10 10-level center-of-mass asymmetry)
- ofi[17]: log bar duration (imbalance bar formation time, normalized)
Previously 10 of 18 OFI embed MLP inputs were zero. Now all 18 have
real data: raw_ofi(8) + delta_ofi(8) + book_aggression(1) + log_duration(1).
Added read_state_sample() diagnostic for GPU state verification.
Legacy v3 fxcache handled by zeroing new slots (v2→v4 graceful upgrade).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
target_dim expansion: adds raw_open (OHLCV) and mid_price_open
(MBP-10 midpoint at bar formation) to fxcache targets. FXCACHE_VERSION
2→3 for auto-rebuild. Legacy v2 files handled with close-price fallback.
Spec v5 adds 3 pearls:
- Bar duration encoding in Mamba2 (continuous-time SSM awareness)
- Order book center of mass from all 10 MBP-10 levels (aggression signal)
- Retrospective hold quality bonus (teaches exit timing)
Plus: Hold action (4th direction), DSR Sharpe EMA fix, counterfactual
magnitude/order sign fix, MFT mid-price mark-to-market.
OFI embed MLP now 18→10 (was 16→8). Mamba2 width SH2+10 (was SH2+8).
Attention width D+10 (was D+8).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
12 new tick-level features computed per bar from MBP-10 snapshots:
OFI trajectory, realized variance, Hawkes intensity, book pressure,
spread dynamics, aggression, queue depletion, order count flux,
intra-bar momentum, regime score, OFI acceleration, toxicity gradient.
Combined with existing 8 OFI into [f64; 20] per bar in fxcache.
Also fixes pre-existing bug: no-MBP-10 fallback was [0.0; 8] not [0.0; 20].
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
fxcache: OFI_DIM=20 (pub const), RECORD_F64_COUNT=66, 272 bytes/bar.
constructor: state_dim 74→86 (aligned 88) with OFI.
experience collector: ofi_dim detection 8→20.
All [f64; 8] → [f64; 20]. Existing 8 features preserved at 0-7.
New 12 features zero-padded until precompute is extended.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Global filter selected only ONE quarter's instrument_id, producing 3 months
of data instead of 2+ years. Now filters within each file so quarterly
rolls are handled correctly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace extract_ohlcv_bars_from_dbn() with trades-based volume bar pipeline
(load_trades_sync -> filter_front_month -> build_volume_bars) in both the
precompute binary and the runtime fallback path, eliminating fake price jumps
from interleaved contract months in OHLCV DBN files.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds `has_ofi: bool` to `FxCacheData` and the fxcache binary header
(stored in `reserved[0]`). Propagates through load_fxcache, write_fxcache,
all callers (precompute_features, train_baseline_rl, hyperopt dqn adapter),
existing roundtrip tests, and adds two new has_ofi-specific roundtrip tests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Different instruments (ES.FUT vs NQ.FUT) and data source modes (ohlcv vs mbp10)
now produce distinct .fxcache keys, preventing silent overwrites and wrong-bar-type
loads at cache lookup time.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- precompute_features early-exits if {cache_key}.fxcache already exists
(cache key = SHA256 of filenames + sizes, only changes when data changes)
- compile-and-train copies binaries to PVC /data/bin/ after compile
- precompute step added to compile-and-train DAG (runs after compile,
before training, parallel with GPU warmup)
- precompute template reads binary from PVC (no GitLab download)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
collect_dbn_files_filtered() checks for symbol subdirectory first,
falls back to filename matching. Precompute now loads only the
target symbol's data instead of all 4 instruments mixed together.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Normalize features once in precompute_features (single source of truth).
NormStats saved alongside .fxcache for inference denormalization.
Removed per-fold NormStats computation from train_baseline_rl, hyperopt
adapter, and smoketests — fxcache is pre-normalized, consumers use as-is.
NOTE: features still show raw price values (max=18000+) because
test_data/futures-baseline contains multiple symbols (ES, NQ, ZN, 6E)
mixed into one dataset. The feature extraction pipeline needs
investigation — log returns between different symbols produce garbage.
This is tracked as a separate task.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause of NaN at step 22: cuBLAS out-of-bounds reads in the GPU
experience collector and training forward pass, caused by buffer
dimension mismatches that produced garbage states → garbage Q-values →
NaN loss.
Bugs fixed:
1. GpuDqnTrainConfig::default() had bottleneck_dim=2, market_dim=42 —
collector computed s1_input_dim=40 instead of state_dim=80, cuBLAS
read 2x past W1 weight buffer (10KB OOB per SGEMM)
2. batch_states allocated with state_dim=80 but cuBLAS ldb=128 (CUTLASS
K-tile alignment) — 48 bf16/f32 values read past end per row
3. Validation state tensor had 59 elements/row (42+3+8+6) instead of
128 (state_dim_padded) — missing portfolio/MTF features + padding
4. CUTLASS K-tile overread on all hidden activation buffers (K=64 but
tile=128) — added 128-element padding to 14 bf16 buffers
5. init_from_fxcache never set self.ofi_features — collector OFI buffer
was 1-element placeholder, state_gather kernel read OOB
6. collect_gpu_experiences_slices missing *2 counterfactual multiplier —
half of collected experiences never inserted into PER
Dead code removed (~1228 lines):
- train_with_data_full_loop (old Vec<(FeatureVector, Vec<f64>)> loop)
- init_gpu_data, init_gpu_raw_buffers, collect_gpu_experiences,
run_training_steps (old helpers only called by above)
- train_with_preloaded_data, train_with_shared_data (old hyperopt APIs)
- train() and train_walk_forward() now convert to fixed-size arrays
and route through init_from_fxcache + train_fold_from_slices
Other fixes:
- precompute_features defaults output-dir to sibling of data-dir
- workspace_root() + feature_cache_dir() helpers for reliable path
resolution (works from any cwd, with/without CARGO_MANIFEST_DIR)
- Production smoketest uses fxcache (no slow DBN parsing)
- compute-sanitizer: 0 errors (was 2539)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each record now starts with an i64 timestamp (nanoseconds since epoch)
before the feature/target/OFI data. v1 records grow from 432 to 440
bytes, v2 from 112 to 120 bytes. The timestamp is always i64 even in
bf16 mode. train_baseline_rl reconstructs bars with real timestamps
instead of placeholders so the walk-forward windower can split by month.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Precompute binary now uses calculate_dbn_cache_key_full from the
library instead of its own compute_cache_key. Canonicalize all
paths in the key function so relative/absolute produce the same
key. Walk up parent dirs for sibling feature-cache discovery.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract standalone functions from DQNTrainer:
- extract_features_from_bars(): 42-dim feature extraction
- extract_ohlcv_bars_from_dbn(): DBN to OHLCVBar conversion
- collect_dbn_files_recursive(): file discovery
DQNTrainer delegates to these, ensuring consistency.
Precompute binary no longer initializes CUDA — runs on any
CPU node (ci-compile-cpu at EUR 0.85/hr vs GPU at EUR 12.60/hr).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DQNTrainer constructor requires CUDA for regime-conditional heads.
Use DQNTrainer::new() (GPU) instead of new_with_device(Cpu).
Set buffer_size=100_000 to satisfy PER minimum threshold.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CLI binary that reads OHLCV + MBP-10 + trades DBN data via DQNTrainer,
runs the full feature extraction pipeline, and writes a .fxcache file
for zero-overhead GPU loading during training. Uses CPU device to avoid
CUDA dependency for data-only workloads.
Also promotes load_training_data() and ofi_features to pub visibility
so the example binary can access them.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
