# Unified State Layout + Single State Assembly Kernel

## Problem

Training and validation use different code paths to assemble state vectors, making generalization impossible:

- **Training** (`experience_state_gather`): 96-dim states with 14 portfolio features (8 base + 6 plan/ISV), MTF at [56..72), OFI hardcoded at index 66 (collides with MTF)
- **Validation** (`gather_states`): 88-dim states with 8 portfolio features (missing plan/ISV), MTF at [50..66), OFI at [66..86)

The model learns Q-values for one state distribution and is evaluated on a completely different one. MTF momentum at position 60 in training becomes OFI data at position 60 in validation. No amount of training can overcome this.

Additionally, `state_dim` is independently configurable in 4 systems (DQN config, PER config, experience collector, backtest evaluator), enabling silent mismatches.

## Solution

### 1. Compile-Time State Layout (`state_layout.cuh`)

A single CUDA header defines the canonical state layout as named constants:

```c
// state_layout.cuh — single source of truth for all state assembly
#pragma once

// ── Dimensions ──
#define STATE_DIM          96
#define MARKET_DIM         42
#define OFI_DIM            20
#define MTF_DIM            16    // 4 lookbacks x 4 features
#define PORTFOLIO_BASE_DIM  8
#define PORTFOLIO_PLAN_DIM  6    // plan_progress, pnl_vs_target, pnl_vs_stop, conviction, conviction_drift, regime_shift
#define PADDING_DIM         4

// ── Offsets (group by update frequency: fast-changing first) ──
#define MARKET_START        0                                    // [0..42)
#define OFI_START           (MARKET_START + MARKET_DIM)          // [42..62)
#define MTF_START           (OFI_START + OFI_DIM)                // [62..78)
#define PORTFOLIO_START     (MTF_START + MTF_DIM)                // [78..86)
#define PLAN_ISV_START      (PORTFOLIO_START + PORTFOLIO_BASE_DIM) // [86..92)
#define PADDING_START       (PLAN_ISV_START + PORTFOLIO_PLAN_DIM)  // [92..96)

// Compile-time validation
_Static_assert(PADDING_START + PADDING_DIM == STATE_DIM,
    "State layout dimensions must sum to STATE_DIM");
_Static_assert(STATE_DIM % 8 == 0,
    "STATE_DIM must be 8-aligned for tensor core cuBLAS");
```

### 2. Shared `__device__` State Assembly Function

The inner state assembly logic lives in `state_layout.cuh` as an inline device function. Both the experience collector and backtest evaluator call it:

```c
__device__ __forceinline__ void assemble_state(
    float* __restrict__ out,           // [STATE_DIM] output row
    const float* market_features,      // [MARKET_DIM] pre-loaded market data
    const float* ofi_row,              // [OFI_DIM] — unconditional, never NULL
    const float* mtf_features,         // [MTF_DIM] pre-computed MTF
    const float* portfolio_base,       // [PORTFOLIO_BASE_DIM]
    const float* plan_isv              // [PORTFOLIO_PLAN_DIM] — unconditional, never NULL
);

// Derived constant for cuBLAS K-tile alignment
#define STATE_DIM_PADDED   128   // pad128(STATE_DIM) — cuBLAS ldb stride
```

Both kernels:
1. Prepare their data (market features from their respective buffers, portfolio from their respective simulation)
2. Call `assemble_state()` with the same function, producing identical layouts
3. Handle their own outer loop (episodes x timesteps vs windows x bars)

### 3. Rust Compile-Time Constant

```rust
// ml-core/src/state.rs (or similar central location)
pub const STATE_DIM: usize = 96;
pub const MARKET_DIM: usize = 42;
pub const OFI_DIM: usize = 20;
pub const MTF_DIM: usize = 16;
pub const PORTFOLIO_BASE_DIM: usize = 8;
pub const PORTFOLIO_PLAN_DIM: usize = 6;

// Feature offsets (must match state_layout.cuh)
pub const MARKET_START: usize = 0;
pub const OFI_START: usize = 42;
pub const MTF_START: usize = 62;
pub const PORTFOLIO_START: usize = 78;
pub const PLAN_ISV_START: usize = 86;
```

### 4. Remove All Configurable `state_dim`

Every system that currently has `state_dim` as a config field changes to use the constant:

| System | Current | After |
|--------|---------|-------|
| `DQNConfig.state_dim` | Configurable (default 48) | `STATE_DIM` constant |
| `GpuReplayBufferConfig.state_dim` | Passed at construction | `STATE_DIM` constant |
| `GpuExperienceCollector.state_dim` | From `kernel_dims` tuple | `STATE_DIM` constant |
| `GpuBacktestConfig.ofi_dim` | Configurable (default 0) | `OFI_DIM` constant |
| `GpuDqnTrainer.config.state_dim` | From DQNConfig | `STATE_DIM` constant |

### 5. Delete Backtest Gather Kernel

`backtest_gather_kernel.cu` is deleted entirely. The backtest evaluator:
1. Loads the experience kernel module (already compiled as cubin)
2. Calls the same `experience_state_gather` entry point (or a new `assemble_states` entry point that wraps the shared device function with a simpler outer loop for window-based iteration)
3. Passes the 6 plan/ISV features (computed from its own portfolio simulation)

If the outer loop structure genuinely differs (episodes x timesteps vs flat windows), a thin wrapper kernel in `experience_kernels.cu` provides the backtest entry point while calling the same `assemble_state()` device function.

### 6. Wire Plan/ISV Features in Backtest

The backtest evaluator currently has 8 portfolio features. It gains 6 more:

- `plan_progress`: track progress through the simulated trade (0 at entry, 1 at target)
- `pnl_vs_target`: current PnL / target PnL (how close to take-profit)
- `pnl_vs_stop`: current PnL / stop distance (how close to stop-loss)
- `conviction`: from ISV signals (backtest has access to ISV via pinned memory)
- `conviction_drift`: delta of conviction between bars
- `regime_shift`: ISV regime stability signal

All 6 are computable from the backtest's portfolio state + ISV signals, which are already available during validation.

## Canonical State Layout

```
Index    Feature                          Source
─────────────────────────────────────────────────────
[0..42)  Market features (OHLCV+tech)     fxcache features
[42..62) OFI (20-dim order flow)          fxcache OFI / GPU ofi_gpu buffer
[62..78) MTF (4 lookbacks x 4 features)   computed from price history
[78..86) Portfolio base (8 features)       portfolio simulation state
[86..92) Plan/ISV (6 features)             trade plan + ISV signals
[92..96) Zero padding                      tensor core alignment
─────────────────────────────────────────────────────
Total: 96 (STATE_DIM)
```

## Files Changed

| File | Action |
|------|--------|
| `state_layout.cuh` (new) | Constants + `assemble_state()` device function |
| `experience_kernels.cu` | Include `state_layout.cuh`, refactor inner assembly to call shared function |
| `backtest_gather_kernel.cu` | **Delete** |
| `gpu_backtest_evaluator.rs` | Use experience kernel module, add plan/ISV features |
| `ml-core/src/state.rs` or `ml-core/src/lib.rs` | `pub const STATE_DIM: usize = 96` + all offset constants |
| `ml-dqn/src/dqn.rs` | Remove `state_dim` from `DQNConfig`, use constant |
| `ml-dqn/src/gpu_replay_buffer.rs` | Remove `state_dim` from `GpuReplayBufferConfig`, use constant |
| `gpu_experience_collector.rs` | Remove `state_dim` field, use constant |
| `gpu_dqn_trainer.rs` | Remove `config.state_dim`, use constant |
| `constructor.rs` | Remove `state_dim` computation, use constant |
| `training_loop.rs` | Remove `state_dim` from kernel_dims tuple |
| `fused_training.rs` | Use constant for `read_state_sample` stride |
| `OFI_DIAG` in `training_loop.rs` | Update indices: `[42..50)` raw OFI, `[50..58)` delta, `[58]` book_agg, `[59]` log_dur |
| `build.rs` (ml-dqn, ml) | Include `state_layout.cuh` in cubin compilation |
| Checkpoint loading (`dqn.rs`) | Accept `state_dim=96` only — old checkpoints are invalid (trained on garbage PER data anyway) |

## Testing

1. **Layout match test**: After state_gather (training) and after backtest gather (validation), readback both states for the same bar index, assert they are identical within floating-point tolerance.
2. **OFI_DIAG update**: Indices change from `[66..84)` to `[42..62)` — must show non-zero on H100.
3. **Smoke test**: `test_fxcache_zero_copy_training` — passes with new layout, OFI at correct positions.
4. **Generalization test**: val_Sharpe should no longer be catastrophically worse than train Sharpe (distribution mismatch eliminated).
5. **Regression**: Existing walk-forward and training stability smoke tests must pass.

## Success Criteria

- Single `STATE_DIM = 96` constant used by all 5 systems
- Zero configurable `state_dim` fields remaining
- One state assembly function called by both training and validation
- `backtest_gather_kernel.cu` deleted
- Training and validation produce identical state vectors for the same bar
- OFI_DIAG non-zero on H100
- val_Sharpe trajectory no longer diverges from train Sharpe
