# Data Pipeline Gaps + Diagnostic Logging Fix Spec

## Problems Found

### P1: Book aggression (state[82]) is zero — never precomputed
The micro-reward reads `state[82]` for book aggression (order book center-of-mass from 10-level MBP-10 depth). But `precompute_features` never computes this. The feature slot is always 0.0.

**Fix:** In `precompute_features.rs`, after OFI feature computation, compute book aggression from MBP-10 snapshots:
```
buy_com = sum(level_i × bid_depth_i for i in 1..10) / sum(bid_depth_i)
sell_com = sum(level_i × ask_depth_i for i in 1..10) / sum(ask_depth_i)
book_aggression = sell_com - buy_com  // positive = aggressive buying
```
Store at feature index 82 in the state vector. Bump FXCACHE_VERSION 3→4.

**Data source:** MBP-10 snapshots are already loaded per-bar in `precompute_features.rs` (used for OFI computation). The 10-level depth is available.

### P2: Log bar duration (state[83]) is zero — never precomputed
The Mamba2 enrichment should see bar duration for continuous-time awareness. But `precompute_features` never computes this.

**Fix:** In `precompute_features.rs`, compute `log_duration = ln(max(bar_duration_secs, 0.1)) / 10.0` from consecutive bar timestamps. Store at feature index 83. Same FXCACHE_VERSION bump.

**Data source:** Bar timestamps are available from the OHLCV data. Duration = `timestamp[bar] - timestamp[bar-1]`.

### P3: OFI deltas (state[74..82)) are zero in replay buffer
The env_step kernel computes deltas from `ps[30..37]` for the micro-reward, but CANNOT write them to the output state buffer because `batch_states` is read-only in env_step.

**Fix options:**

**Option A (recommended): Precompute deltas in fxcache.**
Compute `delta_ofi[k] = ofi[bar, k] - ofi[bar-1, k]` for all 8 OFI features during `precompute_features`. Store at feature indices 74-81. First bar gets delta=0.

Advantages: deltas are always available in the state vector, no runtime computation needed, works correctly in replay buffer (each sample has its own pre-computed delta).

**Option B: Scatter kernel after env_step.**
Add a kernel that writes deltas from `ps[30..37]` to the output state. Requires a new kernel and wiring.

Disadvantages: deltas in the replay buffer would be from experience collection time, not the actual temporal context during training (samples are shuffled by PER).

**Option A is correct** because replay buffer samples are drawn from different episodes/times — the delta relative to the PREVIOUS BAR in the original sequence is the right value, not a runtime computation on shuffled samples.

### P4: Hold action evaluates 3 magnitude options — should collapse to 1
Hold(dir=1) keeps current position unchanged regardless of magnitude. But the model evaluates Hold×Small, Hold×Half, Hold×Full — 3 redundant Q-values that dilute the Hold signal.

**Fix:** In the branching action selector kernel and Q-value computation:
- When `dir_idx == 1` (Hold), force `mag_idx = 0` regardless of the magnitude branch output
- The Q-value for Hold should be a SINGLE value, not 3
- In the action encoding: Hold always encodes as `1 * 27 + 0 * 9 + ord * 3 + urg = 27 + ord*3 + urg`

**Alternative (simpler):** Keep the 3 Hold×magnitude combos but make them produce IDENTICAL Q-values by sharing the bias term. This way the model can't over-fit to Hold×Full vs Hold×Small. This is simpler than restructuring the action space.

**Simplest fix:** In `compute_target_position_4branch`, Hold already returns `current_position` regardless of magnitude. So the 3 combos already produce identical outcomes. The only waste is Q-evaluation capacity. This is cosmetic — not blocking convergence. **Skip for now.**

### P5: No diagnostic logging for OFI features or embed gradient
We can't verify OFI is non-zero, embed gradient is flowing, or micro-reward is producing non-zero values.

**Fix:** Add logging at epoch end in `training_loop.rs`:
```rust
// Log OFI feature diagnostics every 10 epochs
if epoch % 10 == 0 {
    if let Some(ref fused) = self.fused_ctx {
        let ofi_stats = fused.ofi_embed_diagnostic();
        info!("OFI embed: input_mean={:.4}, input_std={:.4}, output_mean={:.4}, grad_norm={:.6}",
              ofi_stats.input_mean, ofi_stats.input_std, ofi_stats.output_mean, ofi_stats.grad_norm);
    }
}
```

The diagnostic reads a small sample from the OFI input/output buffers and the gradient norm. This requires a new method `ofi_embed_diagnostic()` that does a small DtoH readback (non-hot-path, every 10 epochs only).

### P6: Micro-reward non-zero verification
We need to verify that the micro-reward is actually producing non-zero values and entering the replay buffer.

**Fix:** In `gpu_experience_collector.rs`, after experience collection, compute a simple diagnostic:
```rust
// Count non-zero rewards (should be >> 0 with micro-reward active)
let reward_stats = self.compute_reward_stats()?;
info!("Rewards: non_zero={}/{} ({:.1}%), mean={:.6}, std={:.6}",
      reward_stats.non_zero, reward_stats.total,
      reward_stats.non_zero as f64 / reward_stats.total as f64 * 100.0,
      reward_stats.mean, reward_stats.std);
```

This already exists partially in `gpu_monitoring.rs`. Wire it to log at epoch boundary.

### P7: OFI embed MLP input buffer should use precomputed deltas
Currently `ofi_embed_build_input` reads raw OFI from state[66..74) and deltas from state[74..82). After fixing P3, the deltas will be precomputed in fxcache. The kernel doesn't need changes — it just reads from the correct indices.

## Implementation Order

1. **P1 + P2 + P3 together** (fxcache precompute): Compute book_aggression, log_duration, and OFI deltas during fxcache creation. FXCACHE_VERSION 3→4. One rebuild.
2. **P5 + P6 together** (logging): Add OFI embed diagnostic + reward stats logging.
3. **P4** (Hold magnitude): Skip for now — cosmetic, not blocking.

## Files Modified

| File | Change |
|------|--------|
| `precompute_features.rs` | Compute book_aggression, log_duration, OFI deltas |
| `fxcache.rs` | FXCACHE_VERSION 3→4, state layout docs |
| `data_loading.rs` | Populate state[74..83] from precomputed features |
| `training_loop.rs` | OFI embed diagnostic logging every 10 epochs |
| `gpu_dqn_trainer.rs` | `ofi_embed_diagnostic()` method |
| `gpu_experience_collector.rs` | Reward stats logging |

## Success Criteria

1. State[74..82) (OFI deltas) non-zero in training — verified by diagnostic log
2. State[82] (book_aggression) non-zero — verified by diagnostic log
3. State[83] (log_duration) non-zero — verified by diagnostic log
4. OFI embed grad_norm > 0 — gradient flowing
5. Micro-reward non-zero count > 50% of positioned bars
6. Trade count continues to drop below 1M by epoch 50
