foxhunt/docs/lookahead-bias-audit-2026-04-28.md

Lookahead Bias Audit — train-multi-seed-fgg9x Sharpe=196 Investigation

Executive summary

• Surfaces audited: 7 (feature extraction, target slots, OFI/MBP-10 timing, walk-forward split, validation backtest, reward, CVaR feedback)
• Findings: 1 DEFINITE leak, 2 SUSPICIOUS, 4 CLEAN
• Recommendation: further investigate before publishing. The OFI alignment in precompute_features.rs:441-475 and the symmetric DBN fallback at data_loading.rs:396-448 both use [close(bar_t), close(bar_{t+1})) as the OFI window, contaminating bar t's OFI features with snapshots/trades from bar t+1's formation period. Combined with a 2x annualization scaling error from val subsampling, this is sufficient to inflate annualised Sharpe by a large factor.

1. Feature extraction (crates/ml/src/features/extraction.rs)

• CONFIDENCE: clean
• Function audited: extract_ml_features at extraction.rs:77; orchestrates FeatureExtractor::update (line 163) then extract_current_features_v2 (line 200) at each bar.
• Data flow: update() push_backs bar t into a VecDeque (line 165). All sub-extractors (extract_ohlcv_features, extract_technical_features_v2, extract_price_patterns_v2, extract_volume_features_v2, extract_time_features_v2, extract_statistical_features_v2) read either self.bars.back() (current = t) or self.bars[len-2] (t-1) or windowed slices ending at len-1. No index ever accesses len() or beyond.
• Spot checks:
  • extraction.rs:281 let close = self.bars.back().map(|b| b.close) — current bar only
  • extraction.rs:307-313 returns: prev = self.bars[len-2], bar = self.bars.back() — strictly t and t-1
  • extraction.rs:484-491 autocorr lags 1/5/10 — built from returns[i] series ending at len-1
  • extraction.rs:741-753 compute_consecutive_highs iterates for i in (0..len-1).rev() comparing bars[i+1].close > bars[i].close — i+1 here only walks within the rolling window, max value is len-1. No future leak.
  • extraction.rs:797-810 price acceleration uses bars.back(), bars[len-2], bars[len-3]. Backward-looking only.
• ADX (last_adx, line 181) and CUSUM (last_cusum_direction, line 190) are produced via regime_adx.update(bar) and regime_cusum.update(close_return) which are stateful and only fed bar t. Confirmed clean.
• No "next_*" or bars[i+k] calls in the feature path.

2. Target slots (fxcache.rs / precompute_features.rs)

• CONFIDENCE: clean (with caveat that project-memory comment is stale)
• Layout: TARGET_DIM = 6 per fxcache.rs:85 with documented order [close, next_close, raw_close, raw_next, raw_open, mid_price_open]. Project-memory entry "TARGETS=4 elements" is stale — feedback_trust_code_not_docs.md rules apply; the code is the truth. Slot count was bumped to 6 with mid_price_open (slot 5) added.
• Writes: precompute_features.rs:354-360 writes [raw_curr, raw_next, raw_curr, raw_next, raw_open, raw_open] where raw_curr = all_bars[i+WARMUP].close and raw_next = all_bars[i+WARMUP+1].close. Targets ARE labels, so referencing bar t+1 is legitimate.
• The 4-bar gap between feature row index i and bar index i+WARMUP is consistent across features, targets, OFI and timestamps (precompute_features.rs:351-365).
• DBN fallback parity: train_baseline_rl.rs:642-643 writes [b.close, b.close, b.close, b.close, b.open, b.open] — the next_close slot incorrectly aliases b.close here. This is a fallback-only quality bug (model loses next_close signal) but NOT lookahead (it under-uses available info; doesn't leak future). The fxcache fast-path (used by train-multi-seed-fgg9x) writes correct values.
• Trainer reads target slots only for label/reward computation, never as state inputs (verified by absence of targets[*][N] references in experience_state_gather / backtest_state_gather at experience_kernels.cu:375 and experience_kernels.cu:6803).

3. OFI / MBP-10 timing — DEFINITE LEAK

• CONFIDENCE: DEFINITE-LEAK
• File: crates/ml/examples/precompute_features.rs:441-475 (canonical fxcache path) and crates/ml/src/trainers/dqn/data_loading.rs:396-448 (DBN fallback)

Root cause

build_volume_bars (crates/ml-features/src/trades_loader.rs:124-176) sets each bar's timestamp to the last trade in the bucket (bar_ts = trade.timestamp at line 153, written into the bar at line 162-169). For volume bars this means bar.timestamp == close-time of bar = trigger time.

Then precompute_features.rs:443-446 constructs the OFI/MBP-10 window as:

let bar_ts     = bars[i+WARMUP].timestamp.timestamp_nanos_opt()...;     // close(bar_t)
let bar_end_ts = bars[i+WARMUP+1].timestamp.timestamp_nanos_opt()...;   // close(bar_{t+1})

This window [close(bar_t), close(bar_{t+1})) covers the formation period of bar t+1, NOT the formation period of bar t. The MBP-10 snapshots and trades consumed inside this window are future relative to bar t's decision time.

Affected feature slots (per OFI row attached to feature row t)

• ofi_row[0..8] — raw OFI from last_snap = bar_snapshots.last() (precompute_features.rs:471, :478) — leak
• ofi_row[8..16] — lag-1 deltas of leaked OFI (:558-562) — leak (derived)
• ofi_row[16] — book_aggression from bar_snapshots.last() (:534-552) — leak
• ofi_row[17] — log_bar_duration uses bars[i+WARMUP] - bars[i+WARMUP-1] (:567-575) — CLEAN (correctly backward-looking)
• ofi_row[18..30] — MicrostructureState slots fed inside the leaked window (:466-468, :497-501) — leak
• ofi_row[30..32] — TLOB-novel slots derived from last_snap (:504-523) — leak

31 of 32 OFI features carry next-bar microstructure signal that the policy receives at bar t.

Symmetric leak in DBN fallback

crates/ml/src/trainers/dqn/data_loading.rs:401-411 repeats the same pattern: bar_end_ts = bars[i+WARMUP+1].timestamp and get_snapshots_for_timestamp(&all_snapshots, bar_ts, 1) (defined in crates/ml-features/src/mbp10_loader.rs:779-796) returns the first snapshot at or after bar_ts — i.e. the snapshot from immediately after bar t closed.

Why this is high-impact

The validation backtest and the training experience-replay BOTH gather features from the same features_buf/market_features arrays via experience_state_gather (experience_kernels.cu:6803) and backtest_state_gather (experience_kernels.cu:6803), which read features[bar_idx * feat_dim + market_dim + k] for OFI columns (experience_kernels.cu:703, :6826-6830). State[t] at training time = OHLCV+technical features of bar t (clean, see §1) plus OFI features that already encode bar t+1's microstructure. The aux next-bar head loss target next_states[:, 0] (aux_heads_loss_ema_kernel.cu:56) is column 0 (log return); the trunk's OFI inputs are essentially predicting their own near-future label.

Cross-check: log_bar_duration is backward-looking

precompute_features.rs:567-575 uses bars[i+WARMUP] - bars[i+WARMUP-1] for duration. Same file, same loop, two different time conventions. This is internal evidence that the OFI window endpoints are off-by-one (the duration would have been written as bars[i+WARMUP+1] - bars[i+WARMUP] if "bar t spans [t, t+1)" were the intended convention; instead it's "bar t spans [t-1, t]" for duration but "bar t spans [t, t+1)" for OFI body).

4. Walk-forward fold split (crates/ml/src/walk_forward.rs)

• CONFIDENCE: suspicious — RESOLVED 2026-05-09
• generate_walk_forward_indices at walk_forward.rs:454: train [0..train_end_idx), val [val_start..val_end_bound) with val_start = train_end_idx (walk_forward.rs:495). Same in generate_walk_forward_indices_from_timestamps (walk_forward.rs:584).
• No purge/embargo gap between train and val. For per-bar labels with target = next_close this leaks at most one bar (the last training bar's next_close target = first val bar's close). On 4M-bar datasets that single-bar overlap is negligible for Sharpe inflation, but worth flagging.
  • RESOLVED 2026-05-09: walk_forward.rs::PURGE_BARS = 5 constant inserted between train_end and val_start in all four fold-construction sites: generate_walk_forward_indices, generate_walk_forward_indices_from_timestamps, generate_walk_forward_windows (date-based, drops first 5 bars of val window), and gpu_walk_forward.rs::GpuWalkForwardConfig::generate_folds (+ both stratified variants). 5 bars chosen as the maximum lookback of any per-bar feature (autocorr lags 1/5/10 at extraction.rs:484-491, price-acceleration window 3 at extraction.rs:797-810). Behavioral test: test_walk_forward_purge_gap_indices_from_timestamps confirms val_start - train_end == PURGE_BARS for all generated folds.
• More serious: NormStats::from_features is computed on the FULL dataset (train + val combined) at precompute_features.rs:647 before the walk-forward split happens. The fold loop then reads pre-normalised features from the fxcache. The mean/std used to z-score training data already incorporate validation-period statistics. Defence-in-depth normalization in the DBN fallback (train_baseline_rl.rs:631) repeats the same pattern — from_features(&all_features_raw) over the entire span.
• This is a textbook normalisation leak. The contract documented at walk_forward.rs:25 ("Compute stats from training data only") is violated by the actual call site. train_baseline_supervised.rs:466-468 does it correctly (per-fold NormStats::from_features(&train_features)); RL training does not.
• Magnitude of impact is bounded (mean/std drift across a 4M-bar futures series is small, sub-percent for stationary features), but for non-stationary slots like price-derived ratios it can shift the input distribution by ~0.1σ. Not the primary explanation for Sharpe=196 by itself, but compounds with §3.
  • RESOLVED 2026-05-09: train_baseline_rl.rs fold loop now refits NormStats from train-only RAW features per fold via NormStats::from_features_slice, renormalises the entire dataset with the fold-specific stats, and re-uploads to GPU via init_from_fxcache at fold boundary. RAW features are recovered by reading the fxcache sidecar norm_stats.json (saved at precompute time) and applying denormalize_batch. Per-fold norm_stats_fold{N}.json is saved to the output directory (replacing the legacy single-stats copy) so evaluate_baseline finds train-only-fit stats per fold. Behavioral test: test_norm_stats_per_fold_fit_train_only constructs a synthetic dataset where the train slice has mean 1.0 and val slice has mean 9.0; from_features_slice returns the train-only mean to ε=1e-5 while from_features returns the global mean of 5.0. Caveats: (1) fxcache features at the clamp boundary |z|=20 (rare, corruption only) round-trip lossy through the denormalize path; (2) the DBN-fallback path and ensemble-trainer constructor block still use global-fit features — flagged as a known follow-up. The canonical primary training path is fixed.

5. Validation backtest (gpu_backtest_evaluator + backtest_env_kernel)

• CONFIDENCE: clean (decision/fill timing) but with one orthogonal concern
• File: crates/ml/src/cuda_pipeline/backtest_env_kernel.cu and crates/ml/src/cuda_pipeline/trade_physics.cuh
• Decision/fill alignment: at step t the kernel reads prices[base..base+4] = OHLC of bar t (backtest_env_kernel.cu:158-162), then unified_env_step_core at trade_physics.cuh:580 decodes the action and executes a trade at close = prices[base+3] (line 240, line 583, line 744). Cash update: *cash -= delta * price (trade_physics.cuh:211).
• Mark-to-market: new_value = cash + position * close (trade_physics.cuh:771); step_return = (new_value - prev_equity) / prev_equity (trade_physics.cuh:775-777). prev_equity = portfolio_state[ps+0] from the previous step's new_value write (backtest_env_kernel.cu:301).
• Algebra: step_ret_t = (prev_position * close_t + cash_{t-1} - tx_cost - prev_equity) / prev_equity = (prev_position * (close_t - close_{t-1}) - tx_cost) / prev_equity. Position from t-1 graded against t's price move. Lookahead-free.
• Price source for backtest: metrics.rs:521-522 builds close = target[0] where target[0] = raw_curr = bars[i+WARMUP].close (current bar's close, not next bar's). Confirmed by precompute_features.rs:355.
• Orthogonal concern (not lookahead but inflates Sharpe): metrics.rs:518 subsamples val to every 4th bar. The metrics kernel computes mean/std over the subsampled series (backtest_metrics_kernel.cu:208-211) and multiplies by annualization_factor = sqrt(bars_per_day * 252) (gpu_backtest_evaluator.rs:674, backtest_metrics_kernel.cu:277). After 4× subsampling, each per-step return spans 4 bars, so effective_bars_per_day = bars_per_day / 4. The annualization factor should be sqrt((bars_per_day/4) * 252) = ann_factor / 2. The code uses ann_factor unscaled. This is a 2x Sharpe inflation from subsampling. With ann_factor = 1704, the corrected ann_factor would be 852, halving the reported Sharpe. Comment at metrics.rs:517 claims "<2% Sharpe error (CLT)" — this is wrong; CLT only adjusts variance, not the bars-per-day count used for annualization.
• Aux next-bar head label: aux_heads_loss_ema_kernel.cu:56-80 reads label = next_states[:, 0] which is log return (feature column 0 = safe_log_return(open, prev_close) per extraction.rs:528). The label itself is clean (uses bar t+1 only at the open vs bar t close). The concern is that the trunk input at bar t already carries bar t+1's OFI (§3), making the label trivially predictable.

6. Reward / step_return computation

• CONFIDENCE: clean
• The unified-env helper at trade_physics.cuh:580 is the single source of truth for both training (experience_env_step) and validation (backtest_env_step). step_return formula (new_value - prev_equity) / prev_equity (line 775) where new_value = cash + position * close (line 771).
• close here is the current bar's close (not bar t+1's close). The position field reflects the policy's choice executed at this same close. Old position value (carried in prev_equity) was MTM'd at the previous step's close. Ratio captures the bar-over-bar P&L correctly.
• Capital-floor check at line 267 fires AFTER the trade and computes step_return identically (backtest_env_kernel.cu:279-281). No fundamental difference.
• No "next bar" appears anywhere in the reward path.

7. CVaR / Q-variance feedback

• CONFIDENCE: clean
• q_variance is computed from C51 distributional outputs at the current step's forward pass. It feeds var_scale for adaptive position sizing in experience_kernels.cu (per project memory + grep for var_scale). This is a self-referential signal — the network's own uncertainty about state[t] — not a future observation. As long as state[t] itself is built from bars[..=t] (clean from §1) and the C51 output is the network's own forward computation on state[t], no future information enters.
• The OFI leak from §3 contaminates state[t], so q_variance computed on contaminated state is itself contaminated. But that's a downstream consequence of §3, not a separate leak.

Cross-cutting findings

• The OFI leak in §3 is the primary substantive lookahead concern. It silently affects every consumer that reads from the features array: training (experience_state_gather), validation (backtest_state_gather), aux heads (next-bar label), curiosity (curiosity_inference_kernel.cu:96), MTF block, and TLOB pearl input.
• The norm-stats global-fit in §4 compounds the leak by leaking val-distribution statistics into training-time scaling.
• The val-subsampling annualization scaling bug in §5 is independent of lookahead but contributes a factor-2 multiplicative inflation to reported Sharpe.
• Combined effect (lookahead + 2× annualization scaling): a per-bar Sharpe that should annualise to ~50-100 (already strong but plausible for HFT) annualises to ~100-200 in this run. Sharpe=196 is consistent with this combination.
• The training and validation paths use IDENTICAL OFI rows (same fxcache, same kernel, same bar_idx indexing — experience_kernels.cu:703 for training vs :6826-6830 for backtest). So the policy was trained ON the leak and evaluated WITH the leak. The leak is "consistent" in the sense that it doesn't create train/val distribution mismatch, but BOTH are corrupt — the model has learned to exploit a feature that isn't available in live trading.

Conclusion

Sharpe=196 has a definite lookahead component. The OFI alignment bug in precompute_features.rs:441-475 and data_loading.rs:396-448 makes 31 of 32 OFI feature slots at bar t depend on MBP-10 / trades data from bar t+1's formation period. Combined with a 2× annualization scaling error from val subsampling (gpu_backtest_evaluator.rs:674 + metrics.rs:517-518) and a smaller normalisation-stats leak (precompute_features.rs:647), the reported number is highly suspect.

A live trading deployment of this model is structurally guaranteed to underperform the backtest — the next-bar microstructure is not observable at decision time in production.

Top 3 recommended follow-ups

1. Fix the OFI window (highest priority): in precompute_features.rs:441-446 and data_loading.rs:401-404, change the window to [close(bar_{t-1}), close(bar_t)] so OFI at bar t uses ONLY the trades and snapshots that occurred while bar t was forming. Concretely: bar_start_ts = bars[i+WARMUP-1].timestamp (or bars[0] for first bar), bar_end_ts = bars[i+WARMUP].timestamp. Then bump FXCACHE_VERSION to invalidate all existing caches and regenerate via Argo workflow precompute-fxcache. Re-run the multi-seed validation; expect Sharpe to drop substantially (likely to ~30-80 range, depending on OFI's true marginal value).

2. Fix val annualization scaling: at gpu_backtest_evaluator.rs:674, when val uses subsampling stride S (currently hardcoded S=4 at metrics.rs:518), divide bars_per_day by S before multiplying by trading_days_per_year. Either thread the subsample stride through GpuBacktestConfig, or eliminate subsampling in val (it's a 4× perf shortcut with measurable accuracy cost — a full-density val on H100 with the chunked SGEMM should fit in epoch time budget). Independent of (1), this alone halves the reported Sharpe.

3. Move NormStats fit to per-fold training data: in precompute_features.rs:647-648, drop the global normalisation. Compute NormStats::from_features(&train_slice) per fold inside the training loop (mirroring train_baseline_supervised.rs:466), serialise per-fold norm_stats_fold{N}.json (the helper at fxcache.rs:593 already supports this layout). Re-normalise val features at fold-boundary using train-only stats. The fxcache then stores RAW unnormalised features, with normalisation applied after the fold split.

After all three fixes are deployed, re-run train-multi-seed-fgg9x and report annualised Sharpe distribution across folds and seeds. A range of 0.5-2.5 is the realistic expectation for an HFT system on ES futures with non-leaky OFI.