# Lookahead Bias Audit — train-multi-seed-fgg9x Sharpe=196 Investigation

## Executive summary

- Surfaces audited: 7 (feature extraction, target slots, OFI/MBP-10 timing, walk-forward split, validation backtest, reward, CVaR feedback)
- Findings: 1 DEFINITE leak, 2 SUSPICIOUS, 4 CLEAN
- Recommendation: **further investigate before publishing**. The OFI alignment in `precompute_features.rs:441-475` and the symmetric DBN fallback at `data_loading.rs:396-448` both use `[close(bar_t), close(bar_{t+1}))` as the OFI window, contaminating bar t's OFI features with snapshots/trades from bar t+1's formation period. Combined with a 2x annualization scaling error from val subsampling, this is sufficient to inflate annualised Sharpe by a large factor.

## 1. Feature extraction (crates/ml/src/features/extraction.rs)

- **CONFIDENCE**: clean
- Function audited: `extract_ml_features` at `extraction.rs:77`; orchestrates `FeatureExtractor::update` (line 163) then `extract_current_features_v2` (line 200) at each bar.
- Data flow: `update()` `push_back`s bar t into a `VecDeque` (line 165). All sub-extractors (`extract_ohlcv_features`, `extract_technical_features_v2`, `extract_price_patterns_v2`, `extract_volume_features_v2`, `extract_time_features_v2`, `extract_statistical_features_v2`) read either `self.bars.back()` (current = t) or `self.bars[len-2]` (t-1) or windowed slices ending at `len-1`. No index ever accesses `len()` or beyond.
- Spot checks:
  - `extraction.rs:281` `let close = self.bars.back().map(|b| b.close)` — current bar only
  - `extraction.rs:307-313` returns: `prev = self.bars[len-2]`, `bar = self.bars.back()` — strictly t and t-1
  - `extraction.rs:484-491` autocorr lags 1/5/10 — built from `returns[i]` series ending at len-1
  - `extraction.rs:741-753` `compute_consecutive_highs` iterates `for i in (0..len-1).rev()` comparing `bars[i+1].close > bars[i].close` — `i+1` here only walks within the rolling window, max value is `len-1`. No future leak.
  - `extraction.rs:797-810` price acceleration uses `bars.back()`, `bars[len-2]`, `bars[len-3]`. Backward-looking only.
- ADX (`last_adx`, line 181) and CUSUM (`last_cusum_direction`, line 190) are produced via `regime_adx.update(bar)` and `regime_cusum.update(close_return)` which are stateful and only fed bar t. Confirmed clean.
- No "next_*" or `bars[i+k]` calls in the feature path.

## 2. Target slots (fxcache.rs / precompute_features.rs)

- **CONFIDENCE**: clean (with caveat that project-memory comment is stale)
- Layout: `TARGET_DIM = 6` per `fxcache.rs:85` with documented order `[close, next_close, raw_close, raw_next, raw_open, mid_price_open]`. Project-memory entry "TARGETS=4 elements" is stale — `feedback_trust_code_not_docs.md` rules apply; the code is the truth. Slot count was bumped to 6 with `mid_price_open` (slot 5) added.
- Writes: `precompute_features.rs:354-360` writes `[raw_curr, raw_next, raw_curr, raw_next, raw_open, raw_open]` where `raw_curr = all_bars[i+WARMUP].close` and `raw_next = all_bars[i+WARMUP+1].close`. Targets ARE labels, so referencing bar t+1 is legitimate.
- The 4-bar gap between feature row index `i` and bar index `i+WARMUP` is consistent across features, targets, OFI and timestamps (precompute_features.rs:351-365).
- DBN fallback parity: `train_baseline_rl.rs:642-643` writes `[b.close, b.close, b.close, b.close, b.open, b.open]` — the `next_close` slot incorrectly aliases `b.close` here. This is a fallback-only quality bug (model loses `next_close` signal) but **NOT lookahead** (it under-uses available info; doesn't leak future). The fxcache fast-path (used by `train-multi-seed-fgg9x`) writes correct values.
- Trainer reads target slots only for label/reward computation, never as state inputs (verified by absence of `targets[*][N]` references in `experience_state_gather` / `backtest_state_gather` at `experience_kernels.cu:375` and `experience_kernels.cu:6803`).

## 3. OFI / MBP-10 timing — DEFINITE LEAK

- **CONFIDENCE**: DEFINITE-LEAK
- File: `crates/ml/examples/precompute_features.rs:441-475` (canonical fxcache path) and `crates/ml/src/trainers/dqn/data_loading.rs:396-448` (DBN fallback)

### Root cause

`build_volume_bars` (`crates/ml-features/src/trades_loader.rs:124-176`) sets each bar's timestamp to **the last trade in the bucket** (`bar_ts = trade.timestamp` at line 153, written into the bar at line 162-169). For volume bars this means `bar.timestamp == close-time of bar = trigger time`.

Then `precompute_features.rs:443-446` constructs the OFI/MBP-10 window as:

```rust
let bar_ts     = bars[i+WARMUP].timestamp.timestamp_nanos_opt()...;     // close(bar_t)
let bar_end_ts = bars[i+WARMUP+1].timestamp.timestamp_nanos_opt()...;   // close(bar_{t+1})
```

This window `[close(bar_t), close(bar_{t+1}))` covers the **formation period of bar t+1**, NOT the formation period of bar t. The MBP-10 snapshots and trades consumed inside this window are *future* relative to bar t's decision time.

### Affected feature slots (per OFI row attached to feature row t)

- `ofi_row[0..8]` — raw OFI from `last_snap = bar_snapshots.last()` (`precompute_features.rs:471`, `:478`) — **leak**
- `ofi_row[8..16]` — lag-1 deltas of leaked OFI (`:558-562`) — **leak (derived)**
- `ofi_row[16]` — book_aggression from `bar_snapshots.last()` (`:534-552`) — **leak**
- `ofi_row[17]` — log_bar_duration uses `bars[i+WARMUP] - bars[i+WARMUP-1]` (`:567-575`) — **CLEAN** (correctly backward-looking)
- `ofi_row[18..30]` — MicrostructureState slots fed inside the leaked window (`:466-468`, `:497-501`) — **leak**
- `ofi_row[30..32]` — TLOB-novel slots derived from `last_snap` (`:504-523`) — **leak**

31 of 32 OFI features carry next-bar microstructure signal that the policy receives at bar t.

### Symmetric leak in DBN fallback

`crates/ml/src/trainers/dqn/data_loading.rs:401-411` repeats the same pattern: `bar_end_ts = bars[i+WARMUP+1].timestamp` and `get_snapshots_for_timestamp(&all_snapshots, bar_ts, 1)` (defined in `crates/ml-features/src/mbp10_loader.rs:779-796`) returns the first snapshot **at or after** `bar_ts` — i.e. the snapshot from immediately after bar t closed.

### Why this is high-impact

The validation backtest and the training experience-replay BOTH gather features from the same `features_buf`/`market_features` arrays via `experience_state_gather` (`experience_kernels.cu:6803`) and `backtest_state_gather` (`experience_kernels.cu:6803`), which read `features[bar_idx * feat_dim + market_dim + k]` for OFI columns (`experience_kernels.cu:703`, `:6826-6830`). State[t] at training time = OHLCV+technical features of bar t (clean, see §1) plus OFI features that already encode bar t+1's microstructure. The aux next-bar head loss target `next_states[:, 0]` (`aux_heads_loss_ema_kernel.cu:56`) is column 0 (log return); the trunk's OFI inputs are essentially predicting their own near-future label.

### Cross-check: log_bar_duration is backward-looking

`precompute_features.rs:567-575` uses `bars[i+WARMUP] - bars[i+WARMUP-1]` for duration. Same file, same loop, two different time conventions. This is internal evidence that the OFI window endpoints are off-by-one (the duration would have been written as `bars[i+WARMUP+1] - bars[i+WARMUP]` if "bar t spans `[t, t+1)`" were the intended convention; instead it's "bar t spans `[t-1, t]`" for duration but "bar t spans `[t, t+1)`" for OFI body).

## 4. Walk-forward fold split (crates/ml/src/walk_forward.rs)

- **CONFIDENCE**: suspicious — **RESOLVED 2026-05-09**
- `generate_walk_forward_indices` at `walk_forward.rs:454`: train `[0..train_end_idx)`, val `[val_start..val_end_bound)` with `val_start = train_end_idx` (`walk_forward.rs:495`). Same in `generate_walk_forward_indices_from_timestamps` (`walk_forward.rs:584`).
- **No purge/embargo gap** between train and val. For per-bar labels with `target = next_close` this leaks at most one bar (the last training bar's `next_close` target = first val bar's `close`). On 4M-bar datasets that single-bar overlap is negligible for Sharpe inflation, but worth flagging.
  - **RESOLVED 2026-05-09**: `walk_forward.rs::PURGE_BARS = 5` constant inserted between `train_end` and `val_start` in all four fold-construction sites:
    `generate_walk_forward_indices`, `generate_walk_forward_indices_from_timestamps`,
    `generate_walk_forward_windows` (date-based, drops first 5 bars of val window),
    and `gpu_walk_forward.rs::GpuWalkForwardConfig::generate_folds` (+ both stratified variants).
    5 bars chosen as the maximum lookback of any per-bar feature (autocorr lags 1/5/10 at
    `extraction.rs:484-491`, price-acceleration window 3 at `extraction.rs:797-810`).
    Behavioral test: `test_walk_forward_purge_gap_indices_from_timestamps` confirms
    `val_start - train_end == PURGE_BARS` for all generated folds.
- More serious: **`NormStats::from_features` is computed on the FULL dataset (train + val combined) at `precompute_features.rs:647`** before the walk-forward split happens. The fold loop then reads pre-normalised features from the fxcache. The mean/std used to z-score training data already incorporate validation-period statistics. Defence-in-depth normalization in the DBN fallback (`train_baseline_rl.rs:631`) repeats the same pattern — `from_features(&all_features_raw)` over the entire span.
- This is a textbook normalisation leak. The contract documented at `walk_forward.rs:25` ("Compute stats from training data only") is violated by the actual call site. `train_baseline_supervised.rs:466-468` does it correctly (per-fold `NormStats::from_features(&train_features)`); RL training does not.
- Magnitude of impact is bounded (mean/std drift across a 4M-bar futures series is small, sub-percent for stationary features), but for non-stationary slots like price-derived ratios it can shift the input distribution by ~0.1σ. Not the primary explanation for Sharpe=196 by itself, but compounds with §3.
  - **RESOLVED 2026-05-09**: `train_baseline_rl.rs` fold loop now refits `NormStats`
    from train-only RAW features per fold via `NormStats::from_features_slice`,
    renormalises the entire dataset with the fold-specific stats, and re-uploads
    to GPU via `init_from_fxcache` at fold boundary. RAW features are recovered
    by reading the fxcache sidecar `norm_stats.json` (saved at precompute time)
    and applying `denormalize_batch`. Per-fold `norm_stats_fold{N}.json` is
    saved to the output directory (replacing the legacy single-stats copy)
    so `evaluate_baseline` finds train-only-fit stats per fold.
    Behavioral test: `test_norm_stats_per_fold_fit_train_only` constructs a
    synthetic dataset where the train slice has mean 1.0 and val slice has
    mean 9.0; `from_features_slice` returns the train-only mean to ε=1e-5
    while `from_features` returns the global mean of 5.0. **Caveats**: (1)
    fxcache features at the clamp boundary `|z|=20` (rare, corruption only)
    round-trip lossy through the `denormalize` path; (2) the DBN-fallback
    path and ensemble-trainer constructor block still use global-fit features —
    flagged as a known follow-up. The canonical primary training path is
    fixed.

## 5. Validation backtest (gpu_backtest_evaluator + backtest_env_kernel)

- **CONFIDENCE**: clean (decision/fill timing) but with one orthogonal concern
- File: `crates/ml/src/cuda_pipeline/backtest_env_kernel.cu` and `crates/ml/src/cuda_pipeline/trade_physics.cuh`
- Decision/fill alignment: at step t the kernel reads `prices[base..base+4]` = OHLC of bar t (`backtest_env_kernel.cu:158-162`), then `unified_env_step_core` at `trade_physics.cuh:580` decodes the action and executes a trade at `close = prices[base+3]` (line 240, line 583, line 744). Cash update: `*cash -= delta * price` (`trade_physics.cuh:211`).
- Mark-to-market: `new_value = cash + position * close` (`trade_physics.cuh:771`); `step_return = (new_value - prev_equity) / prev_equity` (`trade_physics.cuh:775-777`). `prev_equity = portfolio_state[ps+0]` from the previous step's `new_value` write (`backtest_env_kernel.cu:301`).
- Algebra: `step_ret_t = (prev_position * close_t + cash_{t-1} - tx_cost - prev_equity) / prev_equity = (prev_position * (close_t - close_{t-1}) - tx_cost) / prev_equity`. **Position from t-1 graded against t's price move. Lookahead-free.**
- Price source for backtest: `metrics.rs:521-522` builds `close = target[0]` where `target[0] = raw_curr = bars[i+WARMUP].close` (current bar's close, not next bar's). Confirmed by `precompute_features.rs:355`.
- **Orthogonal concern (not lookahead but inflates Sharpe)**: `metrics.rs:518` subsamples val to every 4th bar. The metrics kernel computes `mean/std` over the subsampled series (`backtest_metrics_kernel.cu:208-211`) and multiplies by `annualization_factor = sqrt(bars_per_day * 252)` (`gpu_backtest_evaluator.rs:674`, `backtest_metrics_kernel.cu:277`). After 4× subsampling, each per-step return spans 4 bars, so `effective_bars_per_day = bars_per_day / 4`. The annualization factor should be `sqrt((bars_per_day/4) * 252) = ann_factor / 2`. The code uses `ann_factor` unscaled. **This is a 2x Sharpe inflation from subsampling.** With ann_factor = 1704, the corrected ann_factor would be 852, halving the reported Sharpe. Comment at `metrics.rs:517` claims "<2% Sharpe error (CLT)" — this is wrong; CLT only adjusts variance, not the bars-per-day count used for annualization.
- Aux next-bar head label: `aux_heads_loss_ema_kernel.cu:56-80` reads `label = next_states[:, 0]` which is log return (feature column 0 = `safe_log_return(open, prev_close)` per `extraction.rs:528`). The label itself is clean (uses bar t+1 only at the open vs bar t close). The concern is that the trunk *input* at bar t already carries bar t+1's OFI (§3), making the label trivially predictable.

## 6. Reward / step_return computation

- **CONFIDENCE**: clean
- The unified-env helper at `trade_physics.cuh:580` is the single source of truth for both training (`experience_env_step`) and validation (`backtest_env_step`). step_return formula `(new_value - prev_equity) / prev_equity` (line 775) where `new_value = cash + position * close` (line 771).
- `close` here is the current bar's close (not bar t+1's close). The position field reflects the policy's choice executed at this same close. Old position value (carried in `prev_equity`) was MTM'd at the previous step's close. Ratio captures the bar-over-bar P&L correctly.
- Capital-floor check at line 267 fires AFTER the trade and computes step_return identically (`backtest_env_kernel.cu:279-281`). No fundamental difference.
- No "next bar" appears anywhere in the reward path.

## 7. CVaR / Q-variance feedback

- **CONFIDENCE**: clean
- `q_variance` is computed from C51 distributional outputs at the current step's forward pass. It feeds `var_scale` for adaptive position sizing in `experience_kernels.cu` (per project memory + grep for `var_scale`). This is a self-referential signal — the network's own uncertainty about state[t] — not a future observation. As long as state[t] itself is built from `bars[..=t]` (clean from §1) and the C51 output is the network's own forward computation on state[t], no future information enters.
- The OFI leak from §3 contaminates state[t], so q_variance computed on contaminated state is itself contaminated. But that's a downstream consequence of §3, not a separate leak.

## Cross-cutting findings

- The OFI leak in §3 is the primary substantive lookahead concern. It silently affects every consumer that reads from the `features` array: training (`experience_state_gather`), validation (`backtest_state_gather`), aux heads (next-bar label), curiosity (`curiosity_inference_kernel.cu:96`), MTF block, and TLOB pearl input.
- The norm-stats global-fit in §4 compounds the leak by leaking val-distribution statistics into training-time scaling.
- The val-subsampling annualization scaling bug in §5 is independent of lookahead but contributes a factor-2 multiplicative inflation to reported Sharpe.
- Combined effect (lookahead + 2× annualization scaling): a per-bar Sharpe that should annualise to ~50-100 (already strong but plausible for HFT) annualises to ~100-200 in this run. Sharpe=196 is consistent with this combination.
- The training and validation paths use IDENTICAL OFI rows (same fxcache, same kernel, same `bar_idx` indexing — `experience_kernels.cu:703` for training vs `:6826-6830` for backtest). So the policy was trained ON the leak and evaluated WITH the leak. The leak is "consistent" in the sense that it doesn't create train/val distribution mismatch, but BOTH are corrupt — the model has learned to exploit a feature that isn't available in live trading.

## Conclusion

**Sharpe=196 has a definite lookahead component.** The OFI alignment bug in `precompute_features.rs:441-475` and `data_loading.rs:396-448` makes 31 of 32 OFI feature slots at bar t depend on MBP-10 / trades data from bar t+1's formation period. Combined with a 2× annualization scaling error from val subsampling (`gpu_backtest_evaluator.rs:674` + `metrics.rs:517-518`) and a smaller normalisation-stats leak (`precompute_features.rs:647`), the reported number is highly suspect.

A live trading deployment of this model is structurally guaranteed to underperform the backtest — the next-bar microstructure is not observable at decision time in production.

### Top 3 recommended follow-ups

1. **Fix the OFI window** (highest priority): in `precompute_features.rs:441-446` and `data_loading.rs:401-404`, change the window to `[close(bar_{t-1}), close(bar_t)]` so OFI at bar t uses ONLY the trades and snapshots that occurred while bar t was forming. Concretely: `bar_start_ts = bars[i+WARMUP-1].timestamp` (or `bars[0]` for first bar), `bar_end_ts = bars[i+WARMUP].timestamp`. Then bump `FXCACHE_VERSION` to invalidate all existing caches and regenerate via Argo workflow `precompute-fxcache`. Re-run the multi-seed validation; expect Sharpe to drop substantially (likely to ~30-80 range, depending on OFI's true marginal value).

2. **Fix val annualization scaling**: at `gpu_backtest_evaluator.rs:674`, when val uses subsampling stride `S` (currently hardcoded `S=4` at `metrics.rs:518`), divide `bars_per_day` by `S` before multiplying by `trading_days_per_year`. Either thread the subsample stride through `GpuBacktestConfig`, or eliminate subsampling in val (it's a 4× perf shortcut with measurable accuracy cost — a full-density val on H100 with the chunked SGEMM should fit in epoch time budget). Independent of (1), this alone halves the reported Sharpe.

3. **Move NormStats fit to per-fold training data**: in `precompute_features.rs:647-648`, drop the global normalisation. Compute `NormStats::from_features(&train_slice)` per fold inside the training loop (mirroring `train_baseline_supervised.rs:466`), serialise per-fold `norm_stats_fold{N}.json` (the helper at `fxcache.rs:593` already supports this layout). Re-normalise val features at fold-boundary using train-only stats. The fxcache then stores RAW unnormalised features, with normalisation applied after the fold split.

After all three fixes are deployed, re-run `train-multi-seed-fgg9x` and report annualised Sharpe distribution across folds and seeds. A range of 0.5-2.5 is the realistic expectation for an HFT system on ES futures with non-leaky OFI.
