# Reward v8: Comprehensive Training Overhaul — Master Plan

**Goal:** Break through the 30% win rate ceiling to 50%+ on ES futures by fixing the bootstrapping trap, adding per-branch gradient attribution, and implementing novel training techniques. Full CUDA, no CPU path, no memory copies, zero regressions.

**Architecture:** 18 items across 5 categories: bug fixes (A), bootstrap trap (B), initialization (C), advanced techniques (D), dead code cleanup (E). All GPU-side. 4 new CUDA kernels, 3 modified kernels, ~12 new config fields, 4 removed dead fields.

**Tech Stack:** CUDA (nvcc → cubin via build.rs), Rust (cudarc), cuBLAS SGEMM, TOML config

---

## Category A: v7.1 Bug Fixes & Wiring (9 items)

### A1-A5: Already Committed (Tasks 1-3)

| Item | Description | Status |
|------|-------------|--------|
| A1 | Exit timing `closing_sign` | DONE (commit 28f8fc74b) |
| A2 | Kelly prior seeding (Bayesian) | DONE (commit 28f8fc74b) |
| A3 | Reward noise magnitude scaling | DONE (commit 28f8fc74b) |
| A4 | Best exposure output buffer (`out_best_exposure`) | DONE (commit 44ca6147a) |
| A5 | Exposure aux grad kernel (cubin) | DONE (commit 3ac6daf3d) |

### A6: Exposure Auxiliary GEMM Wiring

Wire the `exposure_aux_grad_kernel` (A5) into the training pipeline with a full backward GEMM for weight gradients.

**Components:**

**a) GpuDqnTrainer** (`gpu_dqn_trainer.rs`):
- Load `exposure_aux_grad_kernel` from DQN_UTILITY_CUBIN
- Allocate scratch buffer `exposure_aux_scratch: CudaSlice<f32>` [batch × b0 × num_atoms]
- Store `exposure_aux_grad_kernel: CudaFunction` field
- `launch_exposure_aux_grad(&mut self, targets, aux_weight)` method:
  1. Zero scratch
  2. Launch aux kernel → f32 logit gradients in scratch
  3. f32→bf16 cast (reuse `f32_to_bf16_kernel`)
  4. `backward_fc_layer(staging_bf16, save_h_b0, _, grad_w_b0out, grad_b_b0out, 0, b0*na, adv_h, batch)` — cuBLAS SGEMM for weight gradients + bias sum
- `exposure_bias_offset()` helper: `sum(compute_param_sizes()[..11])` (byte offset for b_b0out)

**b) FusedTrainingCtx** (`fused_training.rs`):
- `exposure_aux_weight: f64` field
- `pending_exposure_targets: Option<CudaSlice<i32>>` field
- Call `self.trainer.launch_exposure_aux_grad(targets, weight)` in `run_full_step()` between backward and Adam — same injection point as IQN/CQL

**c) Training loop** (`training_loop.rs`):
- Compute decayed weight per epoch:
  ```rust
  let warmup = self.hyperparams.exposure_aux_warmup_epochs.max(1) as f64;
  let progress = (self.current_epoch as f64 / warmup).min(1.0);
  let aux_w = self.hyperparams.exposure_aux_weight * (1.0 - 0.9 * progress);
  fused.exposure_aux_weight = aux_w;
  ```

### A7: CEA 25% Linear Warmup

In `training_loop.rs`, replace the current CEA warmup (hard cutoff at 3 epochs) with linear decay over 25% of training:

```rust
cea_weight: {
    let base = self.hyperparams.cea_weight as f32;
    let warmup_end = (self.hyperparams.epochs as f32 * 0.25).max(3.0);
    if (self.current_epoch as f32) < warmup_end {
        let progress = self.current_epoch as f32 / warmup_end;
        1.0_f32 * (1.0 - progress) + base * progress
    } else { base }
},
```

### A8: OFI Epoch Gate

Disable OFI reward multiplier for the first 5 epochs. In `training_loop.rs`:

```rust
ofi_reward_weight: {
    let base = self.hyperparams.ofi_reward_weight as f32;
    if self.current_epoch < 5 { 0.0 } else { base }
},
```

### A9: Dead v6 Code Removal

Remove from `ExperienceCollectorConfig` (struct + Default + training_loop constructor):
- `loss_aversion: f32`
- `beta_penalty: f32`
- `regret_blend: f32`
- `trade_clustering_penalty: f32`

Remove stale references in `training_profile.rs`. Keep the fields in `DQNHyperparameters` for TOML backward compat (zeroed defaults).

---

## Category B: Bootstrap Trap Fixes (3 items)

### B1: GPU-Side Cosine Epsilon Schedule

Replace host-provided epsilon with GPU-computed cosine decay.

**Kernel change** in `experience_action_select` (`experience_kernels.cu`):

Add 3 new parameters: `float eps_start`, `float eps_end`, `int total_epochs`, `int current_epoch`.

Inside the kernel, replace `float epsilon` read from param with:

```cuda
float cosine_progress = (float)current_epoch / fmaxf((float)total_epochs, 1.0f);
float epsilon = eps_end + 0.5f * (eps_start - eps_end)
              * (1.0f + cosf(3.14159265f * cosine_progress));
```

Remove the old `float epsilon` parameter from `experience_env_step` — the action selection kernel now owns epsilon.

**Rust wiring**: Pass `eps_start`, `eps_end`, `total_epochs`, `current_epoch` to the action select kernel launch. Update `ExperienceCollectorConfig` to hold these instead of a single `epsilon`.

**Config**: `epsilon_start` (default 0.3, hyperopt [0.1, 0.5]), `epsilon_end` (default 0.02, hyperopt [0.01, 0.05]).

### B2: n-step Default Increase

Config-only change:
- Default `n_steps`: 3 → 5
- Hyperopt range: [3, 5] → [3, 7]
- `nstep_kernel.cu` already supports arbitrary n. Zero kernel changes.

### B3: Dense Directional Micro-Reward (Adaptive Vol)

In `experience_env_step`, OUTSIDE the `segment_complete` block, BEFORE the drawdown penalty:

```cuda
/* ── Dense micro-reward: directional bootstrap signal ── */
if (fabsf(position) > 0.001f && micro_reward_scale > 0.0f) {
    float price_change = (raw_next - raw_close) / fmaxf(raw_close, 1.0f);
    float direction_signal = (position > 0.0f) ? price_change : -price_change;
    /* Adaptive: smaller in volatile markets, larger in calm */
    float vol_ratio = vol_proxy / 0.005f;
    float adaptive_scale = micro_reward_scale / fmaxf(sqrtf(vol_ratio), 0.5f);
    reward += adaptive_scale * direction_signal / vol_proxy;
}
```

New kernel param: `float micro_reward_scale` (default 0.001, hyperopt [0.0, 0.005]).

Note: `vol_proxy` is only computed inside `segment_complete`. For the micro-reward (outside that block), compute a local vol_proxy from ATR:

```cuda
float micro_vol_proxy = 0.005f;  /* default 0.5% */
if (features != NULL && bar_idx < total_bars && market_dim > 9) {
    float atr_n = __bfloat162float(features[(long long)bar_idx * market_dim + 9]);
    float log_a = atr_n * 16.0f - 7.0f;
    micro_vol_proxy = fmaxf(expf(log_a) / fmaxf(raw_close, 1.0f), 0.0001f);
}
```

---

## Category C: Initialization & Pre-Training (3 items)

### C1: Supervised Pre-Training Kernel (Epoch 0)

New CUDA kernel `exposure_pretrain_step` in `dqn_utility_kernels.cu`. Functionally identical to `exposure_aux_grad_kernel` but operates on raw bars instead of replay buffer samples:

```cuda
extern "C" __global__ void exposure_pretrain_step(
    const __nv_bfloat16* __restrict__ targets,  /* [total_bars, 4] raw prices */
    const float* __restrict__ adv_logits,       /* [batch, b0_size * num_atoms] from forward */
    float*       __restrict__ grad_scratch,     /* [batch, b0_size * num_atoms] output */
    int b0_size, int num_atoms, int batch_size,
    float max_position
) {
    int sample = blockIdx.x * blockDim.x + threadIdx.x;
    if (sample >= batch_size) return;

    /* Compute target: which exposure maximizes single-bar PnL? */
    /* targets[bar, 2] = raw_close, targets[bar, 3] = raw_next */
    /* (bar index is implicit — caller maps batch samples to bar indices) */

    /* Same cross-entropy gradient as exposure_aux_grad_kernel */
    /* ... (identical math, different data source) */
}
```

**Training flow** (in `training_loop.rs`, before epoch 0):
1. Sample random batches from the GPU-resident feature/target data
2. cuBLAS forward → exposure branch logits
3. Launch `exposure_pretrain_step` → logit gradients in scratch
4. f32→bf16 cast + `backward_fc_layer` GEMM → weight gradients in grad_buf
5. `replay_adam()` → Adam update
6. Repeat for 100 batches (covers ~800K bars at batch=8192)
7. Normal RL training begins at epoch 1

This uses the SAME backward GEMM infrastructure as A6 — identical code path.

### C2: Pessimistic Q-Value Initialization

One-time operation in `GpuDqnTrainer::new()`, after weight initialization:

```rust
// Shift value head output bias (b_v2) to slightly negative
// Makes V(s) ≈ -0.1 for all states → forces exploration
let sizes = compute_param_sizes(&self.config);
let b_v2_offset = sizes[..7].iter().sum::<usize>();  // offset to b_v2
let pessimistic_bias = vec![-0.1_f32; self.config.num_atoms];
unsafe {
    cudarc::driver::sys::cuMemcpyHtoDAsync_v2(
        self.grad_buf.raw_ptr() + (b_v2_offset * 4) as u64,  // NO — write to params_f32, not grad_buf
        pessimistic_bias.as_ptr().cast(),
        pessimistic_bias.len() * 4,
        self.stream.cu_stream(),
    );
}
// Then f32→bf16 cast to sync params_bf16
```

Actually, write directly to the f32 master params buffer at the b_v2 bias offset, then cast f32→bf16. One HtoD copy (num_atoms × 4 bytes = ~400 bytes) + one cast kernel launch. Happens once at construction.

### C3: Combined Schedule (Pre-train → Bootstrap → RL)

All schedules computed GPU-side from `epoch` and `total_epochs`:

| Phase | Epochs | aux_weight | CEA weight | Epsilon | OFI | Micro-reward |
|-------|--------|-----------|------------|---------|-----|--------------|
| Pre-train | 0 | 1.0 | 0.0 | 1.0 | off | off |
| Bootstrap | 1-25% | 0.8→0.1 | 1.0→base | cosine 0.3→0.05 | off (epoch<5) | on |
| RL-only | 25%+ | 0.05 floor | base | cosine →0.02 | on | on |

The training loop passes `epoch` and `total_epochs` to both the experience kernel and the action select kernel. The kernels compute their own schedules internally — no CPU scheduling needed.

---

## Category D: Advanced Techniques (4 items)

### D1: TD(λ) Truncated Returns (Replaces n-step kernel)

Replace `nstep_kernel.cu` with a TD(λ) kernel that computes geometrically-weighted multi-horizon returns:

```cuda
extern "C" __global__ void td_lambda_kernel(
    const float* __restrict__ rewards,    /* [N * L] */
    const float* __restrict__ dones,      /* [N * L] */
    const float* __restrict__ q_next,     /* [N * L] bootstrapped Q(s') */
    float*       __restrict__ out_returns, /* [N * L] overwritten with G^λ */
    float gamma, float lambda,
    int max_trace, int L, int N
) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    if (idx >= N * L) return;

    int ep = idx / L;
    int t  = idx % L;
    int base = ep * L;

    /* Truncated TD(λ): blend n-step returns for n=1..max_trace */
    float G_lambda = 0.0f;
    float weight_sum = 0.0f;
    float lambda_pow = 1.0f;

    for (int n = 1; n <= max_trace; n++) {
        /* Compute n-step return G^(n) */
        float G_n = 0.0f;
        float gamma_pow = 1.0f;
        int valid = 1;
        for (int k = 0; k < n && (t + k) < L; k++) {
            if (dones[base + t + k] > 0.5f && k > 0) { valid = 0; break; }
            G_n += gamma_pow * rewards[base + t + k];
            gamma_pow *= gamma;
        }
        if (!valid || (t + n) >= L) {
            /* Terminal or out of bounds — use last valid step */
            G_lambda += lambda_pow * G_n;
            weight_sum += lambda_pow;
            break;
        }
        /* Add bootstrap */
        G_n += gamma_pow * q_next[base + t + n - 1];

        /* Accumulate into λ-weighted sum */
        float w = (n < max_trace) ? (1.0f - lambda) * lambda_pow : lambda_pow;
        G_lambda += w * G_n;
        weight_sum += w;
        lambda_pow *= lambda;
    }

    out_returns[idx] = (weight_sum > 0.0f) ? G_lambda / weight_sum : rewards[idx];
}
```

**Config**: `td_lambda` (default 0.9, hyperopt [0.5, 0.99]), `max_trace_length` (default 7, hyperopt [5, 10]).

**Replaces**: the current `nstep_kernel.cu`. The old kernel is kept but unused (TD(λ) is a strict generalization). The Rust n-step launch code switches to the new kernel.

### D2: PopArt Reward Normalization

Adaptive reward normalization with weight rescaling for C51 stability.

**GPU-resident running statistics** (2 f32 scalars + 1 counter):
```rust
popart_mean: CudaSlice<f32>,      // [1] running mean of rewards
popart_var: CudaSlice<f32>,       // [1] running variance
popart_count: CudaSlice<f32>,     // [1] sample count
```

**New kernel** `popart_normalize_kernel`:
```cuda
extern "C" __global__ void popart_normalize_kernel(
    float* __restrict__ rewards,       /* [N] in-place normalize */
    float* __restrict__ mean_buf,      /* [1] running mean */
    float* __restrict__ var_buf,       /* [1] running variance */
    float* __restrict__ count_buf,     /* [1] sample count */
    int N
) {
    /* Thread 0: update running stats with Welford's algorithm */
    /* All threads: normalize rewards in-place */
    float mu = *mean_buf;
    float sigma = sqrtf(*var_buf / fmaxf(*count_buf, 1.0f));
    sigma = fmaxf(sigma, 0.01f);

    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < N) {
        rewards[i] = (rewards[i] - mu) / sigma;
    }
}
```

**Weight rescaling** (once per epoch, after stats update): adjust value head last layer:
```
W_v2_new = W_v2_old * (sigma_old / sigma_new)
b_v2_new = (b_v2_old * sigma_old + mu_old - mu_new) / sigma_new
```

This is a SAXPY-like kernel on the value head weights. Launched once per epoch. Eliminates need for manual v_min/v_max tuning.

**Config**: `popart_enabled` (default true), `popart_warmup` (default 100 — minimum samples before normalizing).

### D3: Curriculum Learning (Regime-Ordered Episodes)

**Pre-computation** (once at init):

New kernel `compute_difficulty_scores`:
```cuda
extern "C" __global__ void compute_difficulty_scores(
    const __nv_bfloat16* __restrict__ targets,  /* [total_bars, 4] */
    float* __restrict__ scores,                  /* [total_bars] output */
    int total_bars
) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i >= total_bars - 5) return;

    /* Difficulty = inverse of directional clarity over 5 bars */
    float close_0 = __bfloat162float(targets[i * 4 + 2]);
    float close_5 = __bfloat162float(targets[(i + 5) * 4 + 2]);
    float ret_5 = fabsf(close_5 - close_0) / fmaxf(close_0, 1.0f);
    /* Low difficulty = strong trend, high difficulty = choppy */
    scores[i] = 1.0f / (ret_5 + 0.001f);
}
```

**Episode ordering** (in Rust, once at init):
1. Download difficulty scores from GPU (one-time DtoH)
2. Sort bar indices by difficulty (ascending = easy first)
3. Upload sorted indices as `curriculum_starts: CudaSlice<i32>`

**Per-epoch curriculum** (in training_loop.rs):
```rust
let curriculum_fraction = (self.current_epoch as f32 / self.hyperparams.epochs as f32)
    .min(1.0).max(0.0);
// Phase 1 (0-30%): use easiest 30% of bars
// Phase 2 (30-60%): use easiest 60%
// Phase 3 (60%+): use all bars
let usable_bars = ((0.3 + 0.7 * curriculum_fraction) * total_bars as f32) as usize;
```

Modify `episode_starts` buffer to only sample from the first `usable_bars` of sorted indices.

**Config**: `curriculum_enabled` (default true), `curriculum_warmup_fraction` (default 0.6 — fraction of training before full dataset).

### D4: Hindsight Optimal Exit Relabeling

New kernel `hindsight_relabel_kernel`:
```cuda
extern "C" __global__ void hindsight_relabel_kernel(
    const __nv_bfloat16* __restrict__ targets,  /* [total_bars, 4] raw prices */
    const int*   __restrict__ actions,           /* [batch] taken actions */
    const int*   __restrict__ episode_starts,    /* [batch] bar index of entry */
    float*       __restrict__ rewards,           /* [batch] in-place relabel */
    const float* __restrict__ relabel_mask,      /* [batch] 1.0 = relabel, 0.0 = keep */
    int b0_size, int b1_size, int b2_size,
    float max_position, int lookahead,
    int total_bars, float vol_proxy_default,
    int batch_size
) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i >= batch_size) return;
    if (relabel_mask[i] < 0.5f) return;  /* keep original reward */

    int bar = episode_starts[i];
    int exposure_idx = actions[i] / (b1_size * b2_size);
    float pos = compute_target_position(exposure_idx, b0_size, max_position);
    if (fabsf(pos) < 0.001f) return;  /* flat — nothing to relabel */

    float entry_price = __bfloat162float(targets[bar * 4 + 2]);

    /* Find optimal exit within lookahead */
    float best_pnl = -1e30f;
    for (int k = 1; k <= lookahead && (bar + k) < total_bars; k++) {
        float future_price = __bfloat162float(targets[(bar + k) * 4 + 2]);
        float pnl = pos * (future_price - entry_price);
        if (pnl > best_pnl) best_pnl = pnl;
    }

    /* Relabel reward with optimal exit PnL (vol-normalized, soft-clamped) */
    float opt_return = best_pnl / fmaxf(entry_price, 1.0f);
    float opt_reward = asymmetric_soft_clamp(10.0f * opt_return / vol_proxy_default);
    rewards[i] = opt_reward;
}
```

**Relabel mask**: Generated per batch using deterministic hash (same as reward noise):
```cuda
float mask = ((hash(i) & 0xFF) < (int)(hindsight_fraction * 255.0f)) ? 1.0f : 0.0f;
```

Launched AFTER batch sampling from PER, BEFORE C51 loss computation. Modifies `rewards_buf` in-place for the selected fraction.

**Config**: `hindsight_fraction` (default 0.1, hyperopt [0.0, 0.3]), `hindsight_lookahead` (default 10, hyperopt [5, 20]).

---

## Category E: Dead Code & Refactoring

### E1: Remove Dead v6 Fields

From `ExperienceCollectorConfig`:
- `loss_aversion: f32` → REMOVE (soft-clamp replaces)
- `beta_penalty: f32` → REMOVE (disabled, dead)
- `regret_blend: f32` → REMOVE (CEA replaces)
- `trade_clustering_penalty: f32` → REMOVE (removed in v7)

From the kernel launch arg chain: remove corresponding `.arg()` lines.

Keep in `DQNHyperparameters` (for TOML backward compat, zeroed defaults).

### E2: Stale Code Comments

Update all references to "reward v6" or "reward v7" to "reward v8" in kernel comments.

---

## New Config Fields Summary

| Field | Type | Default | Hyperopt Range | Category |
|-------|------|---------|---------------|----------|
| `exposure_aux_weight` | f64 | 0.5 | [0.1, 1.0] | A6 |
| `exposure_aux_warmup_epochs` | usize | 5 | [3, 10] | A6 |
| `epsilon_start` | f64 | 0.3 | [0.1, 0.5] | B1 |
| `epsilon_end` | f64 | 0.02 | [0.01, 0.05] | B1 |
| `micro_reward_scale` | f64 | 0.001 | [0.0, 0.005] | B3 |
| `td_lambda` | f64 | 0.9 | [0.5, 0.99] | D1 |
| `max_trace_length` | usize | 7 | [5, 10] | D1 |
| `popart_enabled` | bool | true | — | D2 |
| `curriculum_enabled` | bool | true | — | D3 |
| `curriculum_warmup_fraction` | f64 | 0.6 | [0.3, 0.8] | D3 |
| `hindsight_fraction` | f64 | 0.1 | [0.0, 0.3] | D4 |
| `hindsight_lookahead` | usize | 10 | [5, 20] | D4 |

## Removed Fields

| Field | Reason |
|-------|--------|
| `loss_aversion` | Replaced by asymmetric_soft_clamp |
| `beta_penalty` | Was disabled (0.0) |
| `regret_blend` | Replaced by CEA |
| `trade_clustering_penalty` | Removed in v7 |

---

## Files Modified

| File | Changes |
|------|---------|
| `experience_kernels.cu` | B1 (epsilon params), B3 (micro-reward), A4 updates |
| `dqn_utility_kernels.cu` | C1 (pretrain kernel), A5 (already done) |
| `nstep_kernel.cu` | D1 (TD(λ) replacement) |
| `gpu_dqn_trainer.rs` | A6 (aux GEMM), C2 (pessimistic init), D2 (PopArt buffers) |
| `gpu_experience_collector.rs` | B1 (epsilon wiring), B3 (micro-reward), A9 (dead fields), D3 (curriculum) |
| `fused_training.rs` | A6 (aux wiring), D4 (hindsight relabel) |
| `training_loop.rs` | A7 (CEA warmup), A8 (OFI gate), C1 (pretrain phase), C3 (schedule), D3 (curriculum) |
| `config.rs` | All new fields, dead field cleanup |
| `training_profile.rs` | Dead field refs cleanup |
| `config/training/*.toml` (×4) | All new params |

## New CUDA Kernels (4)

| Kernel | File | Purpose |
|--------|------|---------|
| `exposure_pretrain_step` | dqn_utility_kernels.cu | Supervised direction pre-training |
| `td_lambda_kernel` | nstep_kernel.cu (or new file) | Multi-horizon λ-returns |
| `popart_normalize_kernel` | dqn_utility_kernels.cu | Adaptive reward normalization |
| `compute_difficulty_scores` | experience_kernels.cu | Curriculum learning scores |
| `hindsight_relabel_kernel` | experience_kernels.cu | Optimal exit relabeling |

## Testing

- `cargo check --workspace` — zero compilation errors
- `cargo test -p ml --lib` — 891+ pass, 0 fail, 0 regressions
- Explicit v8 unit tests:
  - `test_asymmetric_soft_clamp_properties` — f(0)=0, f(5)=5, f(-5)≈-3.93, f(15)=10
  - `test_kelly_prior_neutral` — with no trades, Kelly recommends 0.0 (no bet)
  - `test_td_lambda_degenerates_to_nstep` — λ=0 gives 1-step, λ=1 gives MC
  - `test_popart_normalization` — mean→0, std→1 after normalization
  - `test_cosine_epsilon_schedule` — epoch 0 = eps_start, epoch N = eps_end
  - `test_curriculum_ordering` — easy bars (trending) sorted before hard bars (choppy)
  - `test_hindsight_relabel_improves_reward` — relabeled reward ≥ original for winners
  - `test_micro_reward_adaptive_scale` — smaller in volatile, larger in calm
  - `test_exposure_aux_gradient_unique` — each of 9 outputs gets different gradient

## Performance Budget

| Item | Per-step cost | Notes |
|------|--------------|-------|
| Aux GEMM (A6) | ~0.1ms | 1 SGEMM + 1 cast + 1 kernel |
| Micro-reward (B3) | ~0.01ms | 1 conditional per thread |
| Cosine epsilon (B1) | ~0ms | 2 float ops per thread |
| TD(λ) (D1) | ~0.5ms | Replaces n-step (~0.1ms), 5× more computation |
| PopArt (D2) | ~0.05ms | Per-batch normalization |
| Hindsight (D4) | ~0.1ms | 10% of batch × 10 bar lookahead |
| Pre-train (C1) | ~2s total | 100 batches × 20ms, epoch 0 only |
| Curriculum (D3) | ~0ms runtime | Sort at init only |
| **Total overhead** | **~0.8ms/step** | From 35s → ~36s per epoch (+3%) |
