fix(fxcache): target column [0:1] log-return-normalized, not raw price

Bug 1 of the eval-Hold-collapse diagnosis. The fxcache target schema
documented in experience_kernels.cu:1556 + cuda_pipeline/mod.rs:508 specifies:

  target[0] = preproc_close  — log-return-normalized close (network input)
  target[1] = preproc_next   — log-return-normalized next close
  target[2] = raw_close      — raw price for portfolio simulation
  target[3] = raw_next       — raw price
  target[4] = raw_open       — raw price
  target[5] = mid_price_open — MBP-10 midpoint (fallback raw_open)

Both writers — `precompute_features.rs:360` and `data_loading.rs:510` —
violated the contract by storing raw OHLCV close prices in slots [0:1].
Empirical fxcache inspection: target[0..4] mean=$5967, stddev=$582 (raw
prices throughout). The raw-price values at target[0:1] were never directly
consumed by training (production aux head reads next_states[i][0] = MARKET
feat[0] = log_return), but they corrupted any code reading targets per the
documented contract.

Both writers now compute (raw_curr / prev_close).ln() and (raw_next /
raw_curr).ln() for the preproc columns. FXCACHE_VERSION bumped 7→8 to
invalidate existing caches and trigger ensure-fxcache regen.

A second bug — eval label_scale=5300 (raw price magnitude) at production
binary despite source state[0] tracing back to z-normalized log_return —
remains unresolved. Bug 2 instrumentation lands in the next commit; that
runtime trace will pin which production-binary code path injects raw_close
into state[0] post-gather.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

crates/ml/examples/precompute_features.rs

@@ -348,6 +348,16 @@ async fn main() -> Result<()> {
     info!("Extracted {} feature vectors in {:.1}s", feature_vectors.len(), t1.elapsed().as_secs_f64());
 
     // ── Step 3: Build targets [preproc_close, preproc_next, raw_close, raw_next, raw_open, mid_price_open]
+    //   [0:1] = log-return-normalized close/next (network input; matches contract
+    //           in experience_kernels.cu:1556 + cuda_pipeline/mod.rs:508)
+    //   [2:3] = raw dollar prices for portfolio simulation P&L + tx costs
+    //   [4]   = raw open price
+    //   [5]   = mid-price at bar open (MBP-10 midpoint; fallback to raw_open below)
+    //
+    // Prior implementation wrote raw_curr/raw_next to slots [0:1] in violation of
+    // the documented contract — the network-input columns ended up containing raw
+    // prices (~$5000+ for ES futures). Fixed here so the fxcache matches its own
+    // documented schema.
     let n = feature_vectors.len().saturating_sub(1); // need next bar for target
     let features: Vec<[f64; 42]> = feature_vectors[..n].to_vec();
     let mut targets: Vec<[f64; 6]> = Vec::with_capacity(n);
@@ -355,9 +365,18 @@ async fn main() -> Result<()> {
         let raw_curr = all_bars[i + WARMUP].close;
         let raw_next = all_bars[i + WARMUP + 1].close;
         let raw_open = all_bars[i + WARMUP].open;
+        // prev_close: 1-bar lag for log-return computation. WARMUP ≥ 50 so
+        // all_bars[i + WARMUP - 1] is always valid.
+        let prev_close = all_bars[i + WARMUP - 1].close;
+        let preproc_close = if prev_close > 0.0 {
+            (raw_curr / prev_close).ln()
+        } else { 0.0 };
+        let preproc_next = if raw_curr > 0.0 {
+            (raw_next / raw_curr).ln()
+        } else { 0.0 };
         // mid_price_open will be filled from MBP-10 data below if available;
         // default to raw_open (OHLCV-only fallback).
-        targets.push([raw_curr, raw_next, raw_curr, raw_next, raw_open, raw_open]);
+        targets.push([preproc_close, preproc_next, raw_curr, raw_next, raw_open, raw_open]);
     }
 
     // ── Step 3b: Build per-bar timestamps ─────────────────────────────────

crates/ml/src/fxcache.rs

@@ -63,7 +63,15 @@ const HEADER_SIZE: usize = 72;
 ///     §3 the previous layout leaked bar t+1's microstructure into 31/32 OFI
 ///     dims at bar t. Caches written before this fix carry contaminated OFI
 ///     and MUST be regenerated. Strict-checked via `validate()`.
-pub const FXCACHE_VERSION: u16 = 7;
+/// v8: Target column semantic fix — slots [0:1] (`preproc_close`/`preproc_next`)
+///     now hold log-return-normalized values per the documented contract in
+///     `experience_kernels.cu:1556` and `cuda_pipeline/mod.rs:508`. Prior writers
+///     (`precompute_features.rs:360`, `data_loading.rs:510`) stored raw prices
+///     in all 6 target slots, leaving the network-input columns at ~$5000+ ES
+///     futures price level instead of unit-scale log returns. Wire-format
+///     unchanged; semantic contract enforced. Caches written before this fix
+///     carry raw prices in `preproc_*` columns and MUST be regenerated.
+pub const FXCACHE_VERSION: u16 = 8;
 
 /// Compile-time fingerprint over the feature-schema source files. Bumps
 /// automatically whenever `features/extraction.rs`, `fxcache.rs`, or

crates/ml/src/trainers/dqn/data_loading.rs

@@ -496,28 +496,45 @@ impl DQNTrainer {
 
         // Create training data pairs (features, target)
         // Target layout: [preproc_close, preproc_next, raw_close, raw_next, raw_open, mid_price_open]
-        //   [0:1] = network input (raw prices here; preprocessing applied in feature vector)
+        //   [0:1] = log-return-normalized close/next (network input; matches contract
+        //           in experience_kernels.cu:1556 + cuda_pipeline/mod.rs:508)
         //   [2:3] = raw dollar prices for portfolio simulation P&L + tx costs
         //   [4]   = raw open price
         //   [5]   = mid-price at bar open (MBP-10 midpoint; fallback to raw_open)
+        //
+        // Prior implementation wrote raw prices to all 6 columns (target[0:2] should
+        // have been preproc_close / preproc_next per documented contract). The
+        // raw-price values at target[0:1] were never consumed — production aux head
+        // reads next_states[i][0] (= MARKET feat[0] = log_return), not targets.
+        // Fixed here so the on-disk fxcache matches its own documented schema.
         let mut training_data = Vec::new();
         for i in 0..feature_vectors.len().saturating_sub(1) {
             let raw_current = all_ohlcv_bars[i + 50].close;
             let raw_next = all_ohlcv_bars[i + 1 + 50].close;
             let raw_open = all_ohlcv_bars[i + 50].open;
+            // prev_close: 1-bar lag for log-return computation. Within the 50-bar
+            // warmup window we are guaranteed to have all_ohlcv_bars[i + 49] valid
+            // (extraction needs ≥50 bars of history before emitting feat 0).
+            let prev_close = all_ohlcv_bars[i + 49].close;
+            let preproc_close = if prev_close > 0.0 {
+                (raw_current / prev_close).ln()
+            } else { 0.0 };
+            let preproc_next = if raw_current > 0.0 {
+                (raw_next / raw_current).ln()
+            } else { 0.0 };
             training_data.push((
                 feature_vectors[i],
-                vec![raw_current, raw_next, raw_current, raw_next, raw_open, raw_open],
+                vec![preproc_close, preproc_next, raw_current, raw_next, raw_open, raw_open],
             ));
         }
-        // Last sample targets itself
+        // Last sample targets itself — preproc returns are 0.0 (no future bar).
         if !feature_vectors.is_empty() {
             let idx = all_ohlcv_bars.len() - 1;
             let raw_close = all_ohlcv_bars[idx].close;
             let raw_open = all_ohlcv_bars[idx].open;
             training_data.push((
                 feature_vectors[feature_vectors.len() - 1],
-                vec![raw_close, raw_close, raw_close, raw_close, raw_open, raw_open],
+                vec![0.0, 0.0, raw_close, raw_close, raw_open, raw_open],
             ));
         }
 

docs/dqn-gpu-hot-path-audit.md

@@ -268,3 +268,12 @@ Also removed the now-unused `DevicePtr` import and updated the module docstring
 **Scope.** This commit eliminates 2 of the 47 production `*_via_pinned` call sites (the two inside the deleted orphan). The remaining 45 sites across 14 files are cold-path init (constructors, per-fold setup, hyperopt adapters, walk-forward backtest setup) — they don't break operationally because they're outside any captured graph, but they're still graph-capture-fragile and host-stalling. The guard now enforces no NEW calls; the existing 45 will be migrated in subsequent atomic per-buffer commits. After all 45 are converted, the four helpers themselves get deleted from `mapped_pinned.rs`.
 
 **Validation.** Smoke `smoke-test-82fjk` at the τ-buffer-fix HEAD `facbf76eb` succeeded (`phase=Succeeded progress=1/1`, 5-epoch L40S `multi_fold_convergence`). Magnitude differentiation restored (`q_full=0.462 > q_half=0.409 > q_quarter=0.350` vs baseline frozen Pascal-triangle 0.225/0.280/0.495). Eval distribution unfrozen (`eq=0.596, eh=0.404, ef=0.000` vs baseline single-action collapse). Validates that the SP5+SP6 architecture works once the τ buffer DtoD violation is gone.
+
+### Fix 21 — fxcache target schema fix + Bug 2 state[0] instrumentation (2026-05-02)
+Two related changes addressing the EVAL_DIST=Hold collapse observed in `train-multi-seed-hc6kt` 50-epoch validation (terminated epoch 5).
+
+**Bug 1 — fxcache target column semantics violation (FIXED).** `precompute_features.rs:360` and `data_loading.rs:510` both stored raw OHLCV close prices in target slots [0:1] (`preproc_close` / `preproc_next`), violating the documented schema in `experience_kernels.cu:1556` and `cuda_pipeline/mod.rs:508`. Empirically confirmed via direct fxcache inspection: `target[0..4]` had mean=$5967, stddev=$582 — raw prices throughout. The contract specifies log-return-normalized values for the network-input columns. Both writers now compute `(raw_curr / prev_close).ln()` and `(raw_next / raw_curr).ln()` for the preproc columns. `FXCACHE_VERSION` bumped 7 → 8 to invalidate existing caches and force regen via the ensure-fxcache Argo step. The raw-price values at target[0:1] were never directly consumed by training (aux head reads `next_states[i][0]` = MARKET feat[0] = log_return), but they corrupted any downstream code that read targets per the documented contract.
+
+**Bug 2 — state[0]=raw_close suspected; instrumented for runtime confirmation.** Production runs show `aux_label_scale=5300` (raw price magnitude) but the source path traces back through `next_states_buf[i][0]` ← `experience_state_gather(market_features[bar_idx][0])` ← fxcache feat[0] (z-normalized, stddev=1.0). The discrepancy can't be resolved by static code reading alone — there must be a runtime path that injects raw_close into state[0] between the gather and the aux read. Added one-shot debug print in `training_loop.rs` after first `collect_experiences_gpu` return: downloads first 5 samples × 5 columns of both `gpu_batch.states` and `gpu_batch.next_states`, computes mean/std/mean_abs over col 0, prints under `target: "DIAG_BUG2"`. Triggers once via static `AtomicBool`. f32 dtoh download is slow but pre-graph-capture so no hot-path impact. Interpretation key embedded in the log line: `mean_abs ~0.001 = log_return ✓ | mean_abs ~5000 = raw price ✗`.
+
+**Why the smoke test passes but production fails.** The smoke harness `multi_fold_convergence::test_multi_fold_convergence` runs the same DQN trainer but bypasses the production `train_baseline_rl` rollout machinery. The smoke at facbf76eb showed label_scale~0.1 (consistent with normalized log_return). The production binary at the same commit shows label_scale~5300 (raw price). One of the production-only code paths is mutating state[0] post-gather. Bug 2 instrumentation will pin the exact mutation site at next L40S run.