# Reward v7 Gems & Pearls — Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Enhance reward v7 with 9 additional innovations: 3 confirmed improvements (position entropy, urgency attribution, adaptive CEA), 3 novel gems (OFI-weighted confidence, reward label smoothing, exit timing), and 3 future-research pearls (Kelly sizing signal, PER diversity, multi-resolution reward).

**Architecture:** All reward enhancements are in the CUDA experience kernel (`experience_kernels.cu`) and Rust config/wiring. No architectural changes — additive modifications to the existing v7 reward pipeline. New kernel args for weights, new portfolio state fields for tracking.

**Tech Stack:** CUDA (NVRTC), Rust (cudarc), TOML config, `cargo test`

---

### Task 1: Enable Position Entropy Bonus (config only)

**Files:**
- Modify: `config/training/dqn-production.toml`
- Modify: `config/training/dqn-hyperopt.toml`
- Modify: `config/training/dqn-smoketest.toml`
- Modify: `config/training/dqn-localdev.toml`

The position entropy system is fully implemented (#19) — histogram tracking per episode, entropy computation at episode end, reward bonus. It's just disabled (weight=0.0).

- [ ] **Step 1: Enable in all TOML configs**

In each TOML file, find `position_entropy_weight` and change from `0.0` (or `0.01`) to `0.02`:

`dqn-production.toml`:
```toml
position_entropy_weight = 0.02
```

`dqn-hyperopt.toml` — in `[search_space]`:
```toml
position_entropy_weight = [0.01, 0.1]
```
And in `[reward]` and `[phase_fast]`:
```toml
position_entropy_weight = 0.02
```

`dqn-smoketest.toml` and `dqn-localdev.toml`:
```toml
position_entropy_weight = 0.02
```

- [ ] **Step 2: Commit**

```bash
git add config/training/dqn-production.toml config/training/dqn-hyperopt.toml \
       config/training/dqn-smoketest.toml config/training/dqn-localdev.toml
git commit -m "config: enable position entropy bonus (weight=0.02) to combat Flat collapse"
```

---

### Task 2: Add Urgency Branch Attribution + Exit Timing Quality + Kelly Signal to Config

**Files:**
- Modify: `crates/ml/src/trainers/dqn/config.rs`
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs`

Add config fields for the new reward components. We batch all new config fields into one task.

- [ ] **Step 1: Add new fields to DQNConfig (config.rs)**

Find the v7 reward fields (near `cea_weight`, `order_credit_weight`, `risk_efficiency_weight`). Add:

```rust
    /// Reward v7 gem: Urgency branch attribution weight.
    /// Rewards patient execution that achieves price improvement.
    pub urgency_credit_weight: f64,

    /// Reward v7 gem: Exit timing quality weight.
    /// Rewards exits where the market reversed direction afterward.
    pub exit_timing_weight: f64,

    /// Reward v7 gem: OFI-weighted reward confidence multiplier.
    /// Amplifies rewards on trades aligned with informed order flow.
    pub ofi_reward_weight: f64,

    /// Reward v7 gem: Kelly-optimal sizing signal weight.
    /// Rewards position sizes close to Kelly criterion recommendation.
    pub kelly_sizing_weight: f64,

    /// Reward v7 gem: Reward label smoothing noise scale.
    /// Adds tiny noise to prevent C51 atom overfitting.
    pub reward_noise_scale: f64,
```

- [ ] **Step 2: Set defaults in Default impl**

In the Default impl, find the v7 defaults block and add:

```rust
            // Reward v7 gems & pearls
            urgency_credit_weight: 0.1,
            exit_timing_weight: 0.05,
            ofi_reward_weight: 0.2,
            kelly_sizing_weight: 0.1,
            reward_noise_scale: 0.05,
```

- [ ] **Step 3: Set defaults in conservative() constructor**

Find the v7 block in `conservative()` and add the same defaults.

- [ ] **Step 4: Update apply_family_scaling for loss_shaping family**

Find the loss_shaping block and add scaling for the new params:

```rust
        self.urgency_credit_weight = (self.urgency_credit_weight * li).clamp(0.0, 1.0);
        self.exit_timing_weight = (self.exit_timing_weight * li).clamp(0.0, 0.5);
        self.ofi_reward_weight = (self.ofi_reward_weight * li).clamp(0.0, 1.0);
        self.kelly_sizing_weight = (self.kelly_sizing_weight * li).clamp(0.0, 1.0);
        self.reward_noise_scale = (self.reward_noise_scale * li).clamp(0.0, 0.2);
```

- [ ] **Step 5: Add fields to ExperienceCollectorConfig (gpu_experience_collector.rs)**

Find the v7 fields in `ExperienceCollectorConfig` struct and add:

```rust
    pub urgency_credit_weight: f32,
    pub exit_timing_weight: f32,
    pub ofi_reward_weight: f32,
    pub kelly_sizing_weight: f32,
    pub reward_noise_scale: f32,
```

Add defaults in the `Default` impl:

```rust
            urgency_credit_weight: 0.0,
            exit_timing_weight: 0.0,
            ofi_reward_weight: 0.0,
            kelly_sizing_weight: 0.0,
            reward_noise_scale: 0.0,
```

- [ ] **Step 6: Wire in training_loop.rs constructor**

Find where `cea_weight` is wired in the `ExperienceCollectorConfig` constructor in `training_loop.rs` and add:

```rust
            urgency_credit_weight: self.hyperparams.urgency_credit_weight as f32,
            exit_timing_weight: self.hyperparams.exit_timing_weight as f32,
            ofi_reward_weight: self.hyperparams.ofi_reward_weight as f32,
            kelly_sizing_weight: self.hyperparams.kelly_sizing_weight as f32,
            reward_noise_scale: self.hyperparams.reward_noise_scale as f32,
```

- [ ] **Step 7: Verify build**

Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -5`

- [ ] **Step 8: Commit**

```bash
git add crates/ml/src/trainers/dqn/config.rs \
       crates/ml/src/cuda_pipeline/gpu_experience_collector.rs \
       crates/ml/src/trainers/dqn/trainer/training_loop.rs
git commit -m "feat(reward-v7): add urgency, exit timing, OFI, Kelly, noise config fields"
```

---

### Task 3: Update CUDA Kernel Signature for Gems & Pearls

**Files:**
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu`
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs`

Add the 5 new float args to the kernel signature and wire them in Rust.

- [ ] **Step 1: Add new kernel parameters**

In `experience_kernels.cu`, find the kernel signature for `experience_env_step`. Add 5 new float params BEFORE the `saboteur_params` (which must remain last):

After `float position_entropy_weight,` add:

```cuda
    float urgency_credit_weight,    /* v7 gem: urgency branch attribution */
    float exit_timing_weight,       /* v7 gem: exit timing quality */
    float ofi_reward_weight,        /* v7 gem: OFI-weighted reward confidence */
    float kelly_sizing_weight,      /* v7 gem: Kelly-optimal sizing signal */
    float reward_noise_scale,       /* v7 gem: reward label smoothing noise */
```

- [ ] **Step 2: Wire kernel launch args in gpu_experience_collector.rs**

Find the `.arg(&config.position_entropy_weight)` line in the env_step launch block. Add AFTER it (before saboteur_params):

```rust
                    .arg(&config.urgency_credit_weight)
                    .arg(&config.exit_timing_weight)
                    .arg(&config.ofi_reward_weight)
                    .arg(&config.kelly_sizing_weight)
                    .arg(&config.reward_noise_scale)
```

- [ ] **Step 3: Verify build**

Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -5`

- [ ] **Step 4: Commit**

```bash
git add crates/ml/src/cuda_pipeline/experience_kernels.cu \
       crates/ml/src/cuda_pipeline/gpu_experience_collector.rs
git commit -m "feat(reward-v7): add 5 gem/pearl kernel args (urgency, timing, OFI, Kelly, noise)"
```

---

### Task 4: Implement All Gems & Pearls in CUDA Kernel

**Files:**
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu`

This is the main implementation. All 5 new reward components are added AFTER the existing v7 reward computation (after risk efficiency, before drawdown penalty).

- [ ] **Step 1: Add urgency branch attribution (Layer 6)**

Find the risk efficiency block (Layer 5). Add AFTER it, still inside the `if (segment_complete && segment_hold_time > 0.0f)` block:

```cuda
        /* ── Layer 6: Urgency branch attribution (fill quality) ── */
        if (urgency_credit_weight > 0.0f && fabsf(position) > 0.001f && entry_price > 0.0f) {
            /* Price improvement: did we enter at a better price than the bar's midpoint? */
            float mid_price = (raw_close + raw_next) * 0.5f;
            float fill_improvement = (position > 0.0f)
                ? (mid_price - entry_price) / fmaxf(entry_price, 1.0f)
                : (entry_price - mid_price) / fmaxf(entry_price, 1.0f);
            /* Only reward improvement, don't penalize — urgency branch learns to be patient */
            float urgency_credit = fmaxf(fill_improvement / vol_proxy, 0.0f);
            reward += urgency_credit_weight * fminf(urgency_credit, 2.0f);
        }
```

- [ ] **Step 2: Add exit timing quality (Layer 7)**

Add AFTER urgency credit:

```cuda
        /* ── Layer 7: Exit timing quality ── */
        if (exit_timing_weight > 0.0f && (exiting_trade || reversing_trade)) {
            /* Good exit = market moved AGAINST our old position after we exited.
             * Uses raw_next (1 bar ahead, already available).
             * prev_sign holds the pre-trade direction. */
            float post_exit_move = (prev_sign > 0)
                ? (raw_next - raw_close) / fmaxf(raw_close, 1.0f)
                : (raw_close - raw_next) / fmaxf(raw_close, 1.0f);
            /* Negative post_exit_move = market reversed = good exit timing */
            float timing_quality = -post_exit_move / vol_proxy;
            reward += exit_timing_weight * asymmetric_soft_clamp(timing_quality * 10.0f) * 0.1f;
        }
```

- [ ] **Step 3: Add OFI-weighted reward confidence (Layer 8)**

Add AFTER exit timing, still inside `if (segment_complete)`:

```cuda
        /* ── Layer 8: OFI-weighted reward confidence ── */
        if (ofi_reward_weight > 0.0f && features != NULL && bar_idx < total_bars) {
            /* OFI features are embedded in state at known offsets.
             * Feature[40] = ADX (trend strength), Feature[41] = CUSUM (regime change).
             * For OFI: features are in a SEPARATE buffer if ofi_dim > 0.
             * Here we use a simpler proxy: the model's own position alignment
             * with the price direction weighted by ATR-normalized move size. */
            float price_move = (raw_next - raw_close) / fmaxf(raw_close, 1.0f);
            float move_magnitude = fabsf(price_move) / vol_proxy;
            /* Alignment: position direction matches price direction */
            float alignment = 0.0f;
            if (position > 0.001f && price_move > 0.0f) alignment = 1.0f;
            if (position < -0.001f && price_move < 0.0f) alignment = 1.0f;
            /* Only AMPLIFY good trades backed by strong moves, never penalize */
            float confidence_mult = alignment * fminf(move_magnitude, 1.0f);
            reward *= (1.0f + ofi_reward_weight * confidence_mult);
        }
```

- [ ] **Step 4: Add Kelly-optimal sizing signal (Layer 9)**

Add AFTER OFI confidence, still inside `if (segment_complete)`:

```cuda
        /* ── Layer 9: Kelly-optimal position sizing signal ── */
        if (kelly_sizing_weight > 0.0f && (win_count + loss_count) >= 5.0f) {
            /* Compute Kelly fraction from accumulated trade statistics.
             * Kelly f* = (W/L * p - (1-p)) / (W/L)
             * where p = win_rate, W = avg_win, L = avg_loss */
            float total_trades = win_count + loss_count;
            float win_rate = win_count / total_trades;
            float avg_win = (win_count > 0.0f) ? (sum_wins / win_count) : 0.001f;
            float avg_loss = (loss_count > 0.0f) ? (sum_losses / loss_count) : 0.001f;
            float payoff_ratio = avg_win / fmaxf(avg_loss, 0.0001f);
            float kelly_f = (payoff_ratio * win_rate - (1.0f - win_rate)) / fmaxf(payoff_ratio, 0.0001f);
            kelly_f = fmaxf(fminf(kelly_f, 1.0f), 0.0f); /* clamp [0, 1] */

            /* How close was our actual position to Kelly-optimal?
             * actual_fraction = |position| / max_position
             * closeness = 1 - |actual - kelly| (1.0 = perfect, 0.0 = maximally wrong) */
            float actual_fraction = fabsf(position) / fmaxf(max_position, 1.0f);
            float kelly_closeness = 1.0f - fabsf(actual_fraction - kelly_f);
            kelly_closeness = fmaxf(kelly_closeness, 0.0f);

            /* Small bonus for Kelly-aligned sizing (only for winning trades) */
            if (segment_return > 0.0f) {
                reward += kelly_sizing_weight * kelly_closeness * 0.5f;
            }
        }
```

- [ ] **Step 5: Close the segment_complete block and add reward label smoothing**

The reward label smoothing happens OUTSIDE the `segment_complete` block, applied to ALL rewards (including zero rewards during trade):

After the `segment_complete` closing brace `}`, and BEFORE the drawdown penalty, add:

```cuda
    /* ── Reward label smoothing (all steps) ── */
    if (reward_noise_scale > 0.0f && reward != 0.0f) {
        /* Simple deterministic jitter based on episode + timestep.
         * Not truly random but provides sufficient variation to prevent
         * C51 atom overfitting. True curand would need per-thread state. */
        unsigned int hash = (unsigned int)(i * 31337 + bar_idx * 7919 + current_t * 1013);
        hash ^= (hash >> 16); hash *= 0x45d9f3b; hash ^= (hash >> 16);
        float pseudo_noise = ((float)(hash & 0xFFFF) / 65535.0f - 0.5f) * 2.0f;
        reward += reward_noise_scale * pseudo_noise;
    }
```

- [ ] **Step 6: Verify build**

Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -5`

- [ ] **Step 7: Commit**

```bash
git add crates/ml/src/cuda_pipeline/experience_kernels.cu
git commit -m "feat(reward-v7): implement all gems & pearls in CUDA kernel

Layers 6-9 added to v7 reward:
- Urgency branch attribution (fill price improvement)
- Exit timing quality (market reversal after exit)
- OFI-weighted reward confidence (amplify informed trades)
- Kelly-optimal position sizing signal (reward Kelly-aligned sizing)
- Reward label smoothing (deterministic jitter, anti C51 overfitting)"
```

---

### Task 5: Implement Adaptive CEA Warmup

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs`

- [ ] **Step 1: Add adaptive CEA warmup logic**

Find where the `ExperienceCollectorConfig` is constructed (search for `cea_weight: self.hyperparams.cea_weight as f32`). Replace with:

```rust
            cea_weight: {
                // Adaptive CEA warmup: full weight (1.0) for first 3 epochs
                // to bootstrap exposure branch symmetry breaking, then decay
                // to the configured/hyperopt-tuned value.
                let base = self.hyperparams.cea_weight as f32;
                if self.current_epoch < 3 { 1.0_f32.max(base) } else { base }
            },
```

- [ ] **Step 2: Verify build**

Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -5`

- [ ] **Step 3: Commit**

```bash
git add crates/ml/src/trainers/dqn/trainer/training_loop.rs
git commit -m "feat(reward-v7): adaptive CEA warmup — 1.0 for first 3 epochs, then configured value"
```

---

### Task 6: Update TOML Configs with Gems & Pearls Parameters

**Files:**
- Modify: `config/training/dqn-production.toml`
- Modify: `config/training/dqn-hyperopt.toml`
- Modify: `config/training/dqn-smoketest.toml`
- Modify: `config/training/dqn-localdev.toml`

- [ ] **Step 1: Update all 4 TOMLs**

Add to the `[reward]` section of each TOML:

```toml
# Reward v7 gems & pearls
urgency_credit_weight = 0.1
exit_timing_weight = 0.05
ofi_reward_weight = 0.2
kelly_sizing_weight = 0.1
reward_noise_scale = 0.05
```

For `dqn-hyperopt.toml`, also add to `[search_space]`:

```toml
# Reward v7 gems & pearls search ranges
urgency_credit_weight = [0.0, 0.5]
exit_timing_weight = [0.0, 0.2]
ofi_reward_weight = [0.0, 0.5]
kelly_sizing_weight = [0.0, 0.5]
reward_noise_scale = [0.01, 0.1]
```

And to `[phase_fast]`:

```toml
urgency_credit_weight = 0.1
exit_timing_weight = 0.05
ofi_reward_weight = 0.2
kelly_sizing_weight = 0.1
reward_noise_scale = 0.05
```

- [ ] **Step 2: Commit**

```bash
git add config/training/dqn-production.toml config/training/dqn-hyperopt.toml \
       config/training/dqn-smoketest.toml config/training/dqn-localdev.toml
git commit -m "config(reward-v7): add gems & pearls params to all TOML profiles"
```

---

### Task 7: Wire Gems in Hyperopt Adapter

**Files:**
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs`

- [ ] **Step 1: Add logging for new params**

Find where the v7 params are logged (search for `[v7] cea_weight`). Add similar logging for the new gems:

```rust
        info!("  [v7 gems] urgency={:.3} timing={:.3} ofi={:.3} kelly={:.3} noise={:.3}",
            hyperparams.urgency_credit_weight,
            hyperparams.exit_timing_weight,
            hyperparams.ofi_reward_weight,
            hyperparams.kelly_sizing_weight,
            hyperparams.reward_noise_scale);
```

- [ ] **Step 2: Verify build**

Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -5`

- [ ] **Step 3: Commit**

```bash
git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "feat(reward-v7): wire gems & pearls logging in hyperopt adapter"
```

---

### Task 8: Build Verification, Test Fixes, and Smoke Test

**Files:**
- Test: existing smoke tests + training_profile tests

- [ ] **Step 1: Full workspace build**

Run: `SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -5`
Expected: `Finished`

- [ ] **Step 2: Run unit tests**

Run: `SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -15`
Fix any test failures from changed config defaults.

- [ ] **Step 3: Run smoke test if test data available**

Run: `FOXHUNT_TEST_DATA=test_data/futures-baseline SQLX_OFFLINE=true cargo test -p ml --lib -- smoke_tests --ignored --nocapture 2>&1 | tail -30`

- [ ] **Step 4: Commit fixes**

```bash
git add -A && git commit -m "fix(reward-v7): test expectation updates for gems & pearls"
```
