fix(ofi): align OFI window to bar t's formation interval — kill 31/32-slot lookahead
OFI features at bar t were computed over [close(bar_t), close(bar_{t+1}))
— bar t+1's formation period — so 31 of 32 OFI dims at the policy's
input for bar t carried next-bar microstructure data. Per audit
`docs/lookahead-bias-audit-2026-04-28.md` §3, this is a DEFINITE leak
contaminating both training inputs and validation backtest.
Fix: shift the window backward to (close(bar_{t-1}), close(bar_t)] so
OFI at bar t uses only data accumulated during bar t's own formation.
Mirrors the correct `log_bar_duration` convention at
precompute_features.rs:567-575 (audit's smoking-gun citation).
Per feedback_no_partial_refactor, both write sites migrated in lockstep:
- crates/ml/examples/precompute_features.rs:441-475 (precompute path)
- crates/ml/src/trainers/dqn/data_loading.rs:396-448 (DBN fallback)
FXCACHE_VERSION bumped 6 → 7. PVC auto-regen via schema-hash check on
next deploy. Local regen:
./target/release/examples/precompute_features \
--data-dir test_data/futures-baseline \
--mbp10-data-dir test_data/futures-baseline-mbp10 \
--trades-data-dir test_data/futures-baseline-trades \
--output-dir test_data/feature-cache \
--symbol ES.FUT --data-source mbp10 --yes
Regression test added: tests/ofi_features_test.rs::
test_ofi_window_alignment_uses_formation_interval validates the new
partition_point predicates against a synthetic 5-bar dataset, including
an explicit anti-regression assertion that bar t+1 snapshots are NEVER
picked up for bar t.
dqn-wire-up-audit.md updated to reflect new contract for
`trainers/dqn/data_loading.rs`. Audit report
`docs/lookahead-bias-audit-2026-04-28.md` checked in alongside the fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -440,26 +440,44 @@ async fn main() -> Result<()> {
|
||||
|
||||
for i in 0..n {
|
||||
let bar = &all_bars[i + WARMUP];
|
||||
// Bar timestamp = close-time of bar (volume bars: last trade in bucket;
|
||||
// see `crates/ml-features/src/trades_loader.rs:124-176`). The OFI window
|
||||
// for bar t is therefore the *formation interval*
|
||||
// `(close(bar_{t-1}), close(bar_t)]` — strictly historical relative to
|
||||
// the policy's decision time at bar t. The previous implementation used
|
||||
// `[close(bar_t), close(bar_{t+1}))` which leaks bar t+1's microstructure
|
||||
// (audit `docs/lookahead-bias-audit-2026-04-28.md` §3, 31/32 OFI dims).
|
||||
// Mirrors the backward-looking convention already used by
|
||||
// `log_bar_duration` below at lines 567-575.
|
||||
let bar_ts = bar.timestamp.timestamp_nanos_opt().unwrap_or(0) as u64;
|
||||
let bar_end_ts = all_bars.get(i + WARMUP + 1)
|
||||
.map(|b| b.timestamp.timestamp_nanos_opt().unwrap_or(0) as u64)
|
||||
.unwrap_or(bar_ts + 60_000_000_000);
|
||||
let bar_duration_ns = bar_end_ts - bar_ts;
|
||||
let bar_start_ts = if i + WARMUP >= 1 {
|
||||
all_bars[i + WARMUP - 1].timestamp.timestamp_nanos_opt().unwrap_or(0) as u64
|
||||
} else {
|
||||
// No prior bar — emit empty window. log_bar_duration uses the same
|
||||
// sentinel pathway (60s default) for bar 0; here we simply skip
|
||||
// population so OFI[0] stays zero (consistent with existing
|
||||
// first-bar delta handling at lines 558-562).
|
||||
bar_ts
|
||||
};
|
||||
let bar_duration_ns = bar_ts.saturating_sub(bar_start_ts);
|
||||
|
||||
let mut micro_state = MicrostructureState::new(bar_ts, bar_duration_ns);
|
||||
let mut micro_state = MicrostructureState::new(bar_start_ts, bar_duration_ns);
|
||||
|
||||
// Feed trades for this bar window (OFICalculator + MicrostructureState)
|
||||
// Feed trades for this bar window (OFICalculator + MicrostructureState).
|
||||
// Window is half-open at the start, closed at the end:
|
||||
// `(close(bar_{t-1}), close(bar_t)]` so trade-at-close (which formed bar t)
|
||||
// is included.
|
||||
if let Some(ref trades) = all_trades {
|
||||
for trade in get_trades_for_bar(trades, bar_ts, bar_end_ts) {
|
||||
for trade in get_trades_for_bar(trades, bar_start_ts, bar_ts.saturating_add(1)) {
|
||||
calculator.feed_trade(trade.price, trade.volume, trade.is_buy);
|
||||
micro_state.update_trade(trade.price, trade.volume, trade.is_buy, trade.timestamp);
|
||||
}
|
||||
}
|
||||
|
||||
// Get ALL MBP-10 snapshots within this bar window for tick-level features.
|
||||
// Binary search for bar range: [bar_ts, bar_end_ts)
|
||||
let snap_start = all_snapshots.partition_point(|s| s.timestamp < bar_ts);
|
||||
let snap_end = all_snapshots.partition_point(|s| s.timestamp < bar_end_ts);
|
||||
// Binary search for bar range: (bar_start_ts, bar_ts] — formation interval.
|
||||
let snap_start = all_snapshots.partition_point(|s| s.timestamp <= bar_start_ts);
|
||||
let snap_end = all_snapshots.partition_point(|s| s.timestamp <= bar_ts);
|
||||
let bar_snapshots = &all_snapshots[snap_start..snap_end];
|
||||
|
||||
// Feed every snapshot to MicrostructureState for tick-level resolution
|
||||
|
||||
@@ -56,7 +56,14 @@ const HEADER_SIZE: usize = 72;
|
||||
/// compile-time `FEATURE_SCHEMA_HASH` const (built from the bytes of
|
||||
/// extraction.rs / fxcache.rs / state_layout.rs by `build.rs`).
|
||||
/// Recover-from-stale path is identical: validate() bails, Argo regens.
|
||||
pub const FXCACHE_VERSION: u16 = 6;
|
||||
/// v7: OFI window alignment fix — OFI features at bar t now use the formation
|
||||
/// interval `(close(bar_{t-1}), close(bar_t)]` instead of the leaked
|
||||
/// `[close(bar_t), close(bar_{t+1}))`. Wire-format unchanged; semantic
|
||||
/// contract changed. Per audit `docs/lookahead-bias-audit-2026-04-28.md`
|
||||
/// §3 the previous layout leaked bar t+1's microstructure into 31/32 OFI
|
||||
/// dims at bar t. Caches written before this fix carry contaminated OFI
|
||||
/// and MUST be regenerated. Strict-checked via `validate()`.
|
||||
pub const FXCACHE_VERSION: u16 = 7;
|
||||
|
||||
/// Compile-time fingerprint over the feature-schema source files. Bumps
|
||||
/// automatically whenever `features/extraction.rs`, `fxcache.rs`, or
|
||||
|
||||
@@ -395,21 +395,40 @@ impl DQNTrainer {
|
||||
|
||||
for i in 0..feature_vectors.len() {
|
||||
let bar = &all_ohlcv_bars[i + WARMUP];
|
||||
// Bar timestamp = close-time of bar (volume bars). The OFI
|
||||
// window for bar t is the formation interval
|
||||
// `(close(bar_{t-1}), close(bar_t)]` — strictly historical at
|
||||
// the policy's decision time. Previous implementation used
|
||||
// `[close(bar_t), close(bar_{t+1}))` which leaks bar t+1's
|
||||
// microstructure (audit `docs/lookahead-bias-audit-2026-04-28.md`
|
||||
// §3). Migrated in lockstep with the canonical precompute
|
||||
// path at `crates/ml/examples/precompute_features.rs:441-475`.
|
||||
let bar_ts = bar.timestamp.timestamp_nanos_opt().unwrap_or(0) as u64;
|
||||
let bar_start_ts = if i + WARMUP >= 1 {
|
||||
all_ohlcv_bars[i + WARMUP - 1].timestamp.timestamp_nanos_opt().unwrap_or(0) as u64
|
||||
} else {
|
||||
bar_ts
|
||||
};
|
||||
|
||||
if let Some(ref all_trades) = trades {
|
||||
let bar_end_ts = all_ohlcv_bars
|
||||
.get(i + WARMUP + 1)
|
||||
.map(|b| b.timestamp.timestamp_nanos_opt().unwrap_or(0) as u64)
|
||||
.unwrap_or(bar_ts + 60_000_000_000);
|
||||
let bar_trades = get_trades_for_bar(all_trades, bar_ts, bar_end_ts);
|
||||
// Half-open at start, closed at end: trade-at-close
|
||||
// included in bar t (which it formed).
|
||||
let bar_trades = get_trades_for_bar(all_trades, bar_start_ts, bar_ts.saturating_add(1));
|
||||
for trade in bar_trades {
|
||||
ofi_calculator.feed_trade(trade.price, trade.volume, trade.is_buy);
|
||||
}
|
||||
}
|
||||
|
||||
let window = get_snapshots_for_timestamp(&all_snapshots, bar_ts, 1);
|
||||
if let Some(snap) = window.first() {
|
||||
// Formation-interval snapshot: latest snapshot at-or-before
|
||||
// close(bar_t) AND strictly after close(bar_{t-1}).
|
||||
let snap_end = all_snapshots.partition_point(|s| s.timestamp <= bar_ts);
|
||||
let snap_start = all_snapshots.partition_point(|s| s.timestamp <= bar_start_ts);
|
||||
let window = if snap_end > snap_start {
|
||||
&all_snapshots[snap_start..snap_end]
|
||||
} else {
|
||||
&all_snapshots[0..0]
|
||||
};
|
||||
if let Some(snap) = window.last() {
|
||||
match ofi_calculator.calculate(snap) {
|
||||
Ok(features) => {
|
||||
if features.is_valid() {
|
||||
|
||||
@@ -658,3 +658,90 @@ fn test_ofi_feature_names_ordering() {
|
||||
assert_eq!(val, (i + 1) as f64, "Feature '{}' at index {} has wrong value", ofi_names[i], i);
|
||||
}
|
||||
}
|
||||
|
||||
/// OFI window alignment regression — `precompute_features.rs:441-475` and
|
||||
/// `data_loading.rs:396-448` build the per-bar OFI snapshot window via
|
||||
/// `partition_point` predicates. The post-fix contract (audit
|
||||
/// `docs/lookahead-bias-audit-2026-04-28.md` §3) is:
|
||||
///
|
||||
/// window(bar_t) = `(close(bar_{t-1}), close(bar_t)]`
|
||||
///
|
||||
/// i.e. the formation interval — strictly historical at the policy's decision
|
||||
/// time. The pre-fix contract was `[close(bar_t), close(bar_{t+1}))`, which
|
||||
/// leaked bar t+1's microstructure into 31/32 OFI dims. This test exercises the
|
||||
/// new partition_point predicates against a synthetic 5-bar dataset with
|
||||
/// strictly monotonic snapshot timestamps and asserts the slice indexes pick up
|
||||
/// only snapshots in the legal interval.
|
||||
#[test]
|
||||
fn test_ofi_window_alignment_uses_formation_interval() {
|
||||
// Bar close-times (= bar.timestamp for volume bars; see trades_loader.rs:124-176).
|
||||
let bar_close_ts: [u64; 5] = [1_000, 2_000, 3_000, 4_000, 5_000];
|
||||
|
||||
// Synthetic snapshots strictly inside each bar's formation interval.
|
||||
// Bar t formation interval = (bar_close_ts[t-1], bar_close_ts[t]].
|
||||
// Bar 0 has no prior bar (interval empty) — no snapshot.
|
||||
let snapshot_ts: Vec<u64> = vec![
|
||||
// Bar 1 formation: (1000, 2000]
|
||||
1_500,
|
||||
// Bar 2 formation: (2000, 3000]
|
||||
2_400, 2_700,
|
||||
// Bar 3 formation: (3000, 4000]
|
||||
3_500,
|
||||
// Bar 4 formation: (4000, 5000]
|
||||
4_100, 4_900, 5_000, // 5_000 is bar-4 close itself — included (closed end)
|
||||
];
|
||||
|
||||
// Predicates mirror the new precompute_features.rs / data_loading.rs code:
|
||||
// snap_start = partition_point(|t| t <= bar_start_ts)
|
||||
// snap_end = partition_point(|t| t <= bar_ts)
|
||||
// window = snapshots[snap_start..snap_end]
|
||||
for t in 1..bar_close_ts.len() {
|
||||
let bar_ts = bar_close_ts[t];
|
||||
let bar_start_ts = bar_close_ts[t - 1];
|
||||
|
||||
let snap_start = snapshot_ts.partition_point(|&ts| ts <= bar_start_ts);
|
||||
let snap_end = snapshot_ts.partition_point(|&ts| ts <= bar_ts);
|
||||
|
||||
// Every snapshot in the window MUST lie in (bar_start_ts, bar_ts].
|
||||
for &ts in &snapshot_ts[snap_start..snap_end] {
|
||||
assert!(
|
||||
ts > bar_start_ts && ts <= bar_ts,
|
||||
"bar t={t}: snapshot ts={ts} outside formation interval ({bar_start_ts}, {bar_ts}]"
|
||||
);
|
||||
}
|
||||
|
||||
// Conversely, every snapshot in the legal interval must be present.
|
||||
for &ts in &snapshot_ts {
|
||||
let in_window = ts > bar_start_ts && ts <= bar_ts;
|
||||
let picked_up = snapshot_ts[snap_start..snap_end].contains(&ts);
|
||||
assert_eq!(
|
||||
in_window, picked_up,
|
||||
"bar t={t}: snapshot ts={ts} in_window={in_window} but picked_up={picked_up}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// Bar 0 (no prior bar) — bar_start_ts == bar_ts means an empty window
|
||||
// (consistent with the first-bar handling in precompute_features.rs).
|
||||
let bar_ts = bar_close_ts[0];
|
||||
let bar_start_ts = bar_ts;
|
||||
let snap_start = snapshot_ts.partition_point(|&ts| ts <= bar_start_ts);
|
||||
let snap_end = snapshot_ts.partition_point(|&ts| ts <= bar_ts);
|
||||
assert_eq!(
|
||||
snap_start, snap_end,
|
||||
"bar 0 must have an empty OFI window (no prior bar to bound the formation interval)"
|
||||
);
|
||||
|
||||
// Anti-regression: confirm a snapshot at bar t+1's formation period is NEVER
|
||||
// picked up for bar t (this was the leak being fixed).
|
||||
let bar_t = 2_usize;
|
||||
let bar_ts = bar_close_ts[bar_t]; // 3000
|
||||
let bar_start_ts = bar_close_ts[bar_t - 1]; // 2000
|
||||
let snap_start = snapshot_ts.partition_point(|&ts| ts <= bar_start_ts);
|
||||
let snap_end = snapshot_ts.partition_point(|&ts| ts <= bar_ts);
|
||||
let leaky_snap_ts = 3_500_u64; // belongs to bar 3's formation interval
|
||||
assert!(
|
||||
!snapshot_ts[snap_start..snap_end].contains(&leaky_snap_ts),
|
||||
"bar t=2 must NOT pick up snapshot ts=3500 (that belongs to bar 3's formation interval)"
|
||||
);
|
||||
}
|
||||
|
||||
@@ -561,7 +561,7 @@ P5T5 Phase E (2026-04-26): per-fold reset for `MetricBandsRegistry` regression-d
|
||||
|---|---|---|---|---|
|
||||
| `trainers/dqn/mod.rs` | `train_baseline_rl.rs`, `ml_training_service`, `hyperopt/adapters/dqn.rs` | Wired | Entry point for DQN trainer | — |
|
||||
| `trainers/dqn/config.rs` (`DQNHyperparameters`, `DQNAgentType`) | `trainer/constructor.rs`, `fused_training.rs`, smoke tests, hyperopt adapter | Wired | Core config struct | — |
|
||||
| `trainers/dqn/data_loading.rs` | re-exported via `dqn::mod`; consumed by `train_baseline_rl.rs` | Wired | DBN + fxcache loading | — |
|
||||
| `trainers/dqn/data_loading.rs` | re-exported via `dqn::mod`; consumed by `train_baseline_rl.rs` | Wired | DBN + fxcache loading. OFI window for bar t uses formation interval `(close(bar_{t-1}), close(bar_t)]` — strictly historical at the policy's decision time. Migrated 2026-04-28 in lockstep with `crates/ml/examples/precompute_features.rs:441-475` to kill the lookahead leak documented in `docs/lookahead-bias-audit-2026-04-28.md` §3 (FXCACHE_VERSION 6→7). | — |
|
||||
| `trainers/dqn/early_stopping.rs` (`EarlyStopping`) | `trainer/mod.rs`, `trainer/constructor.rs` | Wired | Patience-based stopping | — |
|
||||
| `trainers/dqn/features.rs` | `trainer/constructor.rs` via `extract_features_from_bars` | Wired | Feature extraction for DQN | — |
|
||||
| `trainers/dqn/financials.rs` | `trainer/metrics.rs` via `compute_epoch_financials` | Wired | Sharpe / drawdown per epoch | — |
|
||||
|
||||
129
docs/lookahead-bias-audit-2026-04-28.md
Normal file
129
docs/lookahead-bias-audit-2026-04-28.md
Normal file
@@ -0,0 +1,129 @@
|
||||
# Lookahead Bias Audit — train-multi-seed-fgg9x Sharpe=196 Investigation
|
||||
|
||||
## Executive summary
|
||||
|
||||
- Surfaces audited: 7 (feature extraction, target slots, OFI/MBP-10 timing, walk-forward split, validation backtest, reward, CVaR feedback)
|
||||
- Findings: 1 DEFINITE leak, 2 SUSPICIOUS, 4 CLEAN
|
||||
- Recommendation: **further investigate before publishing**. The OFI alignment in `precompute_features.rs:441-475` and the symmetric DBN fallback at `data_loading.rs:396-448` both use `[close(bar_t), close(bar_{t+1}))` as the OFI window, contaminating bar t's OFI features with snapshots/trades from bar t+1's formation period. Combined with a 2x annualization scaling error from val subsampling, this is sufficient to inflate annualised Sharpe by a large factor.
|
||||
|
||||
## 1. Feature extraction (crates/ml/src/features/extraction.rs)
|
||||
|
||||
- **CONFIDENCE**: clean
|
||||
- Function audited: `extract_ml_features` at `extraction.rs:77`; orchestrates `FeatureExtractor::update` (line 163) then `extract_current_features_v2` (line 200) at each bar.
|
||||
- Data flow: `update()` `push_back`s bar t into a `VecDeque` (line 165). All sub-extractors (`extract_ohlcv_features`, `extract_technical_features_v2`, `extract_price_patterns_v2`, `extract_volume_features_v2`, `extract_time_features_v2`, `extract_statistical_features_v2`) read either `self.bars.back()` (current = t) or `self.bars[len-2]` (t-1) or windowed slices ending at `len-1`. No index ever accesses `len()` or beyond.
|
||||
- Spot checks:
|
||||
- `extraction.rs:281` `let close = self.bars.back().map(|b| b.close)` — current bar only
|
||||
- `extraction.rs:307-313` returns: `prev = self.bars[len-2]`, `bar = self.bars.back()` — strictly t and t-1
|
||||
- `extraction.rs:484-491` autocorr lags 1/5/10 — built from `returns[i]` series ending at len-1
|
||||
- `extraction.rs:741-753` `compute_consecutive_highs` iterates `for i in (0..len-1).rev()` comparing `bars[i+1].close > bars[i].close` — `i+1` here only walks within the rolling window, max value is `len-1`. No future leak.
|
||||
- `extraction.rs:797-810` price acceleration uses `bars.back()`, `bars[len-2]`, `bars[len-3]`. Backward-looking only.
|
||||
- ADX (`last_adx`, line 181) and CUSUM (`last_cusum_direction`, line 190) are produced via `regime_adx.update(bar)` and `regime_cusum.update(close_return)` which are stateful and only fed bar t. Confirmed clean.
|
||||
- No "next_*" or `bars[i+k]` calls in the feature path.
|
||||
|
||||
## 2. Target slots (fxcache.rs / precompute_features.rs)
|
||||
|
||||
- **CONFIDENCE**: clean (with caveat that project-memory comment is stale)
|
||||
- Layout: `TARGET_DIM = 6` per `fxcache.rs:85` with documented order `[close, next_close, raw_close, raw_next, raw_open, mid_price_open]`. Project-memory entry "TARGETS=4 elements" is stale — `feedback_trust_code_not_docs.md` rules apply; the code is the truth. Slot count was bumped to 6 with `mid_price_open` (slot 5) added.
|
||||
- Writes: `precompute_features.rs:354-360` writes `[raw_curr, raw_next, raw_curr, raw_next, raw_open, raw_open]` where `raw_curr = all_bars[i+WARMUP].close` and `raw_next = all_bars[i+WARMUP+1].close`. Targets ARE labels, so referencing bar t+1 is legitimate.
|
||||
- The 4-bar gap between feature row index `i` and bar index `i+WARMUP` is consistent across features, targets, OFI and timestamps (precompute_features.rs:351-365).
|
||||
- DBN fallback parity: `train_baseline_rl.rs:642-643` writes `[b.close, b.close, b.close, b.close, b.open, b.open]` — the `next_close` slot incorrectly aliases `b.close` here. This is a fallback-only quality bug (model loses `next_close` signal) but **NOT lookahead** (it under-uses available info; doesn't leak future). The fxcache fast-path (used by `train-multi-seed-fgg9x`) writes correct values.
|
||||
- Trainer reads target slots only for label/reward computation, never as state inputs (verified by absence of `targets[*][N]` references in `experience_state_gather` / `backtest_state_gather` at `experience_kernels.cu:375` and `experience_kernels.cu:6803`).
|
||||
|
||||
## 3. OFI / MBP-10 timing — DEFINITE LEAK
|
||||
|
||||
- **CONFIDENCE**: DEFINITE-LEAK
|
||||
- File: `crates/ml/examples/precompute_features.rs:441-475` (canonical fxcache path) and `crates/ml/src/trainers/dqn/data_loading.rs:396-448` (DBN fallback)
|
||||
|
||||
### Root cause
|
||||
|
||||
`build_volume_bars` (`crates/ml-features/src/trades_loader.rs:124-176`) sets each bar's timestamp to **the last trade in the bucket** (`bar_ts = trade.timestamp` at line 153, written into the bar at line 162-169). For volume bars this means `bar.timestamp == close-time of bar = trigger time`.
|
||||
|
||||
Then `precompute_features.rs:443-446` constructs the OFI/MBP-10 window as:
|
||||
|
||||
```rust
|
||||
let bar_ts = bars[i+WARMUP].timestamp.timestamp_nanos_opt()...; // close(bar_t)
|
||||
let bar_end_ts = bars[i+WARMUP+1].timestamp.timestamp_nanos_opt()...; // close(bar_{t+1})
|
||||
```
|
||||
|
||||
This window `[close(bar_t), close(bar_{t+1}))` covers the **formation period of bar t+1**, NOT the formation period of bar t. The MBP-10 snapshots and trades consumed inside this window are *future* relative to bar t's decision time.
|
||||
|
||||
### Affected feature slots (per OFI row attached to feature row t)
|
||||
|
||||
- `ofi_row[0..8]` — raw OFI from `last_snap = bar_snapshots.last()` (`precompute_features.rs:471`, `:478`) — **leak**
|
||||
- `ofi_row[8..16]` — lag-1 deltas of leaked OFI (`:558-562`) — **leak (derived)**
|
||||
- `ofi_row[16]` — book_aggression from `bar_snapshots.last()` (`:534-552`) — **leak**
|
||||
- `ofi_row[17]` — log_bar_duration uses `bars[i+WARMUP] - bars[i+WARMUP-1]` (`:567-575`) — **CLEAN** (correctly backward-looking)
|
||||
- `ofi_row[18..30]` — MicrostructureState slots fed inside the leaked window (`:466-468`, `:497-501`) — **leak**
|
||||
- `ofi_row[30..32]` — TLOB-novel slots derived from `last_snap` (`:504-523`) — **leak**
|
||||
|
||||
31 of 32 OFI features carry next-bar microstructure signal that the policy receives at bar t.
|
||||
|
||||
### Symmetric leak in DBN fallback
|
||||
|
||||
`crates/ml/src/trainers/dqn/data_loading.rs:401-411` repeats the same pattern: `bar_end_ts = bars[i+WARMUP+1].timestamp` and `get_snapshots_for_timestamp(&all_snapshots, bar_ts, 1)` (defined in `crates/ml-features/src/mbp10_loader.rs:779-796`) returns the first snapshot **at or after** `bar_ts` — i.e. the snapshot from immediately after bar t closed.
|
||||
|
||||
### Why this is high-impact
|
||||
|
||||
The validation backtest and the training experience-replay BOTH gather features from the same `features_buf`/`market_features` arrays via `experience_state_gather` (`experience_kernels.cu:6803`) and `backtest_state_gather` (`experience_kernels.cu:6803`), which read `features[bar_idx * feat_dim + market_dim + k]` for OFI columns (`experience_kernels.cu:703`, `:6826-6830`). State[t] at training time = OHLCV+technical features of bar t (clean, see §1) plus OFI features that already encode bar t+1's microstructure. The aux next-bar head loss target `next_states[:, 0]` (`aux_heads_loss_ema_kernel.cu:56`) is column 0 (log return); the trunk's OFI inputs are essentially predicting their own near-future label.
|
||||
|
||||
### Cross-check: log_bar_duration is backward-looking
|
||||
|
||||
`precompute_features.rs:567-575` uses `bars[i+WARMUP] - bars[i+WARMUP-1]` for duration. Same file, same loop, two different time conventions. This is internal evidence that the OFI window endpoints are off-by-one (the duration would have been written as `bars[i+WARMUP+1] - bars[i+WARMUP]` if "bar t spans `[t, t+1)`" were the intended convention; instead it's "bar t spans `[t-1, t]`" for duration but "bar t spans `[t, t+1)`" for OFI body).
|
||||
|
||||
## 4. Walk-forward fold split (crates/ml/src/walk_forward.rs)
|
||||
|
||||
- **CONFIDENCE**: suspicious
|
||||
- `generate_walk_forward_indices` at `walk_forward.rs:454`: train `[0..train_end_idx)`, val `[val_start..val_end_bound)` with `val_start = train_end_idx` (`walk_forward.rs:495`). Same in `generate_walk_forward_indices_from_timestamps` (`walk_forward.rs:584`).
|
||||
- **No purge/embargo gap** between train and val. For per-bar labels with `target = next_close` this leaks at most one bar (the last training bar's `next_close` target = first val bar's `close`). On 4M-bar datasets that single-bar overlap is negligible for Sharpe inflation, but worth flagging.
|
||||
- More serious: **`NormStats::from_features` is computed on the FULL dataset (train + val combined) at `precompute_features.rs:647`** before the walk-forward split happens. The fold loop then reads pre-normalised features from the fxcache. The mean/std used to z-score training data already incorporate validation-period statistics. Defence-in-depth normalization in the DBN fallback (`train_baseline_rl.rs:631`) repeats the same pattern — `from_features(&all_features_raw)` over the entire span.
|
||||
- This is a textbook normalisation leak. The contract documented at `walk_forward.rs:25` ("Compute stats from training data only") is violated by the actual call site. `train_baseline_supervised.rs:466-468` does it correctly (per-fold `NormStats::from_features(&train_features)`); RL training does not.
|
||||
- Magnitude of impact is bounded (mean/std drift across a 4M-bar futures series is small, sub-percent for stationary features), but for non-stationary slots like price-derived ratios it can shift the input distribution by ~0.1σ. Not the primary explanation for Sharpe=196 by itself, but compounds with §3.
|
||||
|
||||
## 5. Validation backtest (gpu_backtest_evaluator + backtest_env_kernel)
|
||||
|
||||
- **CONFIDENCE**: clean (decision/fill timing) but with one orthogonal concern
|
||||
- File: `crates/ml/src/cuda_pipeline/backtest_env_kernel.cu` and `crates/ml/src/cuda_pipeline/trade_physics.cuh`
|
||||
- Decision/fill alignment: at step t the kernel reads `prices[base..base+4]` = OHLC of bar t (`backtest_env_kernel.cu:158-162`), then `unified_env_step_core` at `trade_physics.cuh:580` decodes the action and executes a trade at `close = prices[base+3]` (line 240, line 583, line 744). Cash update: `*cash -= delta * price` (`trade_physics.cuh:211`).
|
||||
- Mark-to-market: `new_value = cash + position * close` (`trade_physics.cuh:771`); `step_return = (new_value - prev_equity) / prev_equity` (`trade_physics.cuh:775-777`). `prev_equity = portfolio_state[ps+0]` from the previous step's `new_value` write (`backtest_env_kernel.cu:301`).
|
||||
- Algebra: `step_ret_t = (prev_position * close_t + cash_{t-1} - tx_cost - prev_equity) / prev_equity = (prev_position * (close_t - close_{t-1}) - tx_cost) / prev_equity`. **Position from t-1 graded against t's price move. Lookahead-free.**
|
||||
- Price source for backtest: `metrics.rs:521-522` builds `close = target[0]` where `target[0] = raw_curr = bars[i+WARMUP].close` (current bar's close, not next bar's). Confirmed by `precompute_features.rs:355`.
|
||||
- **Orthogonal concern (not lookahead but inflates Sharpe)**: `metrics.rs:518` subsamples val to every 4th bar. The metrics kernel computes `mean/std` over the subsampled series (`backtest_metrics_kernel.cu:208-211`) and multiplies by `annualization_factor = sqrt(bars_per_day * 252)` (`gpu_backtest_evaluator.rs:674`, `backtest_metrics_kernel.cu:277`). After 4× subsampling, each per-step return spans 4 bars, so `effective_bars_per_day = bars_per_day / 4`. The annualization factor should be `sqrt((bars_per_day/4) * 252) = ann_factor / 2`. The code uses `ann_factor` unscaled. **This is a 2x Sharpe inflation from subsampling.** With ann_factor = 1704, the corrected ann_factor would be 852, halving the reported Sharpe. Comment at `metrics.rs:517` claims "<2% Sharpe error (CLT)" — this is wrong; CLT only adjusts variance, not the bars-per-day count used for annualization.
|
||||
- Aux next-bar head label: `aux_heads_loss_ema_kernel.cu:56-80` reads `label = next_states[:, 0]` which is log return (feature column 0 = `safe_log_return(open, prev_close)` per `extraction.rs:528`). The label itself is clean (uses bar t+1 only at the open vs bar t close). The concern is that the trunk *input* at bar t already carries bar t+1's OFI (§3), making the label trivially predictable.
|
||||
|
||||
## 6. Reward / step_return computation
|
||||
|
||||
- **CONFIDENCE**: clean
|
||||
- The unified-env helper at `trade_physics.cuh:580` is the single source of truth for both training (`experience_env_step`) and validation (`backtest_env_step`). step_return formula `(new_value - prev_equity) / prev_equity` (line 775) where `new_value = cash + position * close` (line 771).
|
||||
- `close` here is the current bar's close (not bar t+1's close). The position field reflects the policy's choice executed at this same close. Old position value (carried in `prev_equity`) was MTM'd at the previous step's close. Ratio captures the bar-over-bar P&L correctly.
|
||||
- Capital-floor check at line 267 fires AFTER the trade and computes step_return identically (`backtest_env_kernel.cu:279-281`). No fundamental difference.
|
||||
- No "next bar" appears anywhere in the reward path.
|
||||
|
||||
## 7. CVaR / Q-variance feedback
|
||||
|
||||
- **CONFIDENCE**: clean
|
||||
- `q_variance` is computed from C51 distributional outputs at the current step's forward pass. It feeds `var_scale` for adaptive position sizing in `experience_kernels.cu` (per project memory + grep for `var_scale`). This is a self-referential signal — the network's own uncertainty about state[t] — not a future observation. As long as state[t] itself is built from `bars[..=t]` (clean from §1) and the C51 output is the network's own forward computation on state[t], no future information enters.
|
||||
- The OFI leak from §3 contaminates state[t], so q_variance computed on contaminated state is itself contaminated. But that's a downstream consequence of §3, not a separate leak.
|
||||
|
||||
## Cross-cutting findings
|
||||
|
||||
- The OFI leak in §3 is the primary substantive lookahead concern. It silently affects every consumer that reads from the `features` array: training (`experience_state_gather`), validation (`backtest_state_gather`), aux heads (next-bar label), curiosity (`curiosity_inference_kernel.cu:96`), MTF block, and TLOB pearl input.
|
||||
- The norm-stats global-fit in §4 compounds the leak by leaking val-distribution statistics into training-time scaling.
|
||||
- The val-subsampling annualization scaling bug in §5 is independent of lookahead but contributes a factor-2 multiplicative inflation to reported Sharpe.
|
||||
- Combined effect (lookahead + 2× annualization scaling): a per-bar Sharpe that should annualise to ~50-100 (already strong but plausible for HFT) annualises to ~100-200 in this run. Sharpe=196 is consistent with this combination.
|
||||
- The training and validation paths use IDENTICAL OFI rows (same fxcache, same kernel, same `bar_idx` indexing — `experience_kernels.cu:703` for training vs `:6826-6830` for backtest). So the policy was trained ON the leak and evaluated WITH the leak. The leak is "consistent" in the sense that it doesn't create train/val distribution mismatch, but BOTH are corrupt — the model has learned to exploit a feature that isn't available in live trading.
|
||||
|
||||
## Conclusion
|
||||
|
||||
**Sharpe=196 has a definite lookahead component.** The OFI alignment bug in `precompute_features.rs:441-475` and `data_loading.rs:396-448` makes 31 of 32 OFI feature slots at bar t depend on MBP-10 / trades data from bar t+1's formation period. Combined with a 2× annualization scaling error from val subsampling (`gpu_backtest_evaluator.rs:674` + `metrics.rs:517-518`) and a smaller normalisation-stats leak (`precompute_features.rs:647`), the reported number is highly suspect.
|
||||
|
||||
A live trading deployment of this model is structurally guaranteed to underperform the backtest — the next-bar microstructure is not observable at decision time in production.
|
||||
|
||||
### Top 3 recommended follow-ups
|
||||
|
||||
1. **Fix the OFI window** (highest priority): in `precompute_features.rs:441-446` and `data_loading.rs:401-404`, change the window to `[close(bar_{t-1}), close(bar_t)]` so OFI at bar t uses ONLY the trades and snapshots that occurred while bar t was forming. Concretely: `bar_start_ts = bars[i+WARMUP-1].timestamp` (or `bars[0]` for first bar), `bar_end_ts = bars[i+WARMUP].timestamp`. Then bump `FXCACHE_VERSION` to invalidate all existing caches and regenerate via Argo workflow `precompute-fxcache`. Re-run the multi-seed validation; expect Sharpe to drop substantially (likely to ~30-80 range, depending on OFI's true marginal value).
|
||||
|
||||
2. **Fix val annualization scaling**: at `gpu_backtest_evaluator.rs:674`, when val uses subsampling stride `S` (currently hardcoded `S=4` at `metrics.rs:518`), divide `bars_per_day` by `S` before multiplying by `trading_days_per_year`. Either thread the subsample stride through `GpuBacktestConfig`, or eliminate subsampling in val (it's a 4× perf shortcut with measurable accuracy cost — a full-density val on H100 with the chunked SGEMM should fit in epoch time budget). Independent of (1), this alone halves the reported Sharpe.
|
||||
|
||||
3. **Move NormStats fit to per-fold training data**: in `precompute_features.rs:647-648`, drop the global normalisation. Compute `NormStats::from_features(&train_slice)` per fold inside the training loop (mirroring `train_baseline_supervised.rs:466`), serialise per-fold `norm_stats_fold{N}.json` (the helper at `fxcache.rs:593` already supports this layout). Re-normalise val features at fold-boundary using train-only stats. The fxcache then stores RAW unnormalised features, with normalisation applied after the fold split.
|
||||
|
||||
After all three fixes are deployed, re-run `train-multi-seed-fgg9x` and report annualised Sharpe distribution across folds and seeds. A range of 0.5-2.5 is the realistic expectation for an HFT system on ES futures with non-leaky OFI.
|
||||
Reference in New Issue
Block a user