# Reward v7: Counterfactual Branch Attribution — Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Replace the v6 reward function (95% breakeven win rate) with v7 featuring asymmetric soft-clamp, counterfactual exposure advantage, order microstructure credit, and intra-trade risk efficiency — dropping breakeven to ~52%.

**Architecture:** Modify the CUDA experience kernel's reward section, update Rust config/wiring, update all TOML configs. Portfolio state expands from 20→23 fields (add `intra_trade_max_dd`, repurpose 2 reserved slots for new params). Kernel signature changes to replace 4 removed args with 3 new ones.

**Tech Stack:** CUDA (NVRTC), Rust (cudarc), TOML config, `cargo test`

---

### Task 1: Expand Portfolio State and Add `asymmetric_soft_clamp` Device Function

**Files:**
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu:23-52` (header + PORTFOLIO_STRIDE)
- Modify: `crates/ml/src/cuda_pipeline/trade_stats_kernel.cu:12` (PORTFOLIO_STRIDE constant)

- [ ] **Step 1: Update PORTFOLIO_STRIDE from 20 → 23 and document new fields**

In `experience_kernels.cu`, update the header comment and `#define`:

```cuda
/* Portfolio state layout used by experience kernels ([N, PORTFOLIO_STRIDE=23]):
 *   [0]  position        — current contract position (signed)
 *   [1]  cash            — cash balance
 *   [2]  portfolio_value — mark-to-market total value (cash + position * price)
 *   [3]  (reserved)
 *   [4]  (reserved)
 *   [5]  (reserved)
 *   [6]  (reserved)
 *   [7]  peak_equity     — high-water mark (init to initial_capital)
 *   [8]  flat_counter    — consecutive flat steps
 *   [9]  prev_equity     — equity at previous step (init to initial_capital)
 *   [10] hold_time       — consecutive steps with position
 *   [11] realized_pnl    — cumulative realized PnL
 *   [12] entry_price     — trade entry price (raw)
 *   [13] trade_start_pnl — realized_pnl at trade entry
 *   [14] win_count       — Kelly: number of profitable trade exits
 *   [15] loss_count      — Kelly: number of losing trade exits
 *   [16] sum_wins        — Kelly: cumulative profit from winners
 *   [17] sum_losses      — Kelly: cumulative loss (absolute) from losers
 *   [18] sum_returns     — running sum of segment returns
 *   [19] sum_sq_returns  — running sum of squared segment returns
 *   [20] intra_trade_max_dd — worst unrealized drawdown during current trade (v7)
 *   [21] last_trade_t    — timestep of last trade exit (for clustering, retained)
 *   [22] interval_sum    — (retained from v6, used internally)
 */
#define PORTFOLIO_STRIDE 23
```

Also update `trade_stats_kernel.cu` line 12:
```cuda
#define PORTFOLIO_STRIDE 23
```

- [ ] **Step 2: Add `asymmetric_soft_clamp` device function**

Add this device function BEFORE the `experience_env_step` kernel (after the `#include` / device function section, near line 55):

```cuda
/* ── Reward v7: Asymmetric soft-clamp ──────────────────────────────────
 * Gains:  linear, capped at +10 (full gradient for positive rewards)
 * Losses: exponential compression (smooth natural risk aversion)
 *
 * f(+5)=+5.0, f(-5)=-3.93, f(-15)=-7.77, f(+15)=+10.0
 * Replaces loss_aversion + tanh from v6. No separate parameter needed. */
__device__ __forceinline__ float asymmetric_soft_clamp(float x) {
    if (x >= 0.0f) return fminf(x, 10.0f);
    return -10.0f * (1.0f - expf(x / 10.0f));
}
```

- [ ] **Step 3: Update PORTFOLIO_STRIDE in gpu_experience_collector.rs**

In `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs`, update the constant at line 54:

```rust
const PORTFOLIO_STRIDE: usize = 23;
```

- [ ] **Step 4: Verify build compiles**

Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -3`
Expected: `Finished` (no errors — PORTFOLIO_STRIDE is a constant used in allocation, no signature changes yet)

- [ ] **Step 5: Commit**

```bash
git add crates/ml/src/cuda_pipeline/experience_kernels.cu \
       crates/ml/src/cuda_pipeline/trade_stats_kernel.cu \
       crates/ml/src/cuda_pipeline/gpu_experience_collector.rs
git commit -m "feat(reward-v7): expand PORTFOLIO_STRIDE 20→23, add asymmetric_soft_clamp device function"
```

---

### Task 2: Add v7 Config Fields and Remove v6 Penalty Fields

**Files:**
- Modify: `crates/ml/src/trainers/dqn/config.rs`

- [ ] **Step 1: Add new v7 reward fields to DQNConfig**

Find the existing `loss_aversion` field (line ~774) and the nearby reward fields. Add the 3 new fields in the same region. Search for `pub loss_aversion` and add after the nearby reward fields:

```rust
    /// Reward v7: Counterfactual Exposure Advantage weight.
    /// Blends per-branch advantage signal into the scalar reward.
    /// 0.0 = disabled, 1.0 = full advantage signal.
    pub cea_weight: f64,

    /// Reward v7: Order type microstructure credit weight.
    /// Rewards cost-efficient execution (LimitMaker > Market).
    pub order_credit_weight: f64,

    /// Reward v7: Intra-trade risk efficiency weight.
    /// Rewards clean winning trades with minimal intra-trade drawdown.
    pub risk_efficiency_weight: f64,
```

- [ ] **Step 2: Set defaults for new fields, zero out removed fields**

Find the `Default` implementation for DQNConfig (search for the block containing `loss_aversion: 1.5`). Update:

```rust
            // Reward v6 penalties — REMOVED in v7 (zeroed for backward compat)
            loss_aversion: 1.0,  // 1.0 = no asymmetry (soft-clamp handles it)
            regret_blend: 0.0,   // replaced by CEA
            trade_clustering_penalty: 0.0,  // removed
            beta_penalty_strength: 0.0,     // removed (was already 0)

            // Reward v7 — Counterfactual Branch Attribution
            cea_weight: 0.3,
            order_credit_weight: 0.1,
            risk_efficiency_weight: 0.1,
```

- [ ] **Step 3: Update `apply_family_scaling` for loss_shaping family**

Find the `loss_shaping_intensity` block (line ~1489). Replace:

```rust
        // Loss shaping family — v7: scale CEA + order credit + risk efficiency
        let li = self.loss_shaping_intensity;
        self.cea_weight = (self.cea_weight * li).clamp(0.0, 2.0);
        self.order_credit_weight = (self.order_credit_weight * li).clamp(0.0, 1.0);
        self.risk_efficiency_weight = (self.risk_efficiency_weight * li).clamp(0.0, 1.0);
        self.position_entropy_weight *= li;
        // v6 legacy (zeroed, no-op but kept for TOML backward compat)
        self.asymmetric_dd_weight *= li;
```

- [ ] **Step 4: Verify build**

Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -3`
Expected: `Finished`

- [ ] **Step 5: Commit**

```bash
git add crates/ml/src/trainers/dqn/config.rs
git commit -m "feat(reward-v7): add cea_weight, order_credit_weight, risk_efficiency_weight config fields"
```

---

### Task 3: Update CUDA Kernel Signature and Wire New Args

**Files:**
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu:812-858` (kernel signature)
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs:1794-1843` (kernel launch)

- [ ] **Step 1: Update kernel signature — replace removed args with new ones**

In `experience_kernels.cu`, find the `experience_env_step` kernel signature (line ~812). Replace the `loss_aversion` parameter and the 4 removed penalty parameters with the 3 new v7 parameters:

Replace `float loss_aversion,` (line 825) with:
```cuda
    float cea_weight,               /* v7: counterfactual exposure advantage weight */
```

Replace `float beta_penalty,` (line 845) with:
```cuda
    float order_credit_weight,      /* v7: order type microstructure credit weight */
```

Replace `float trade_clustering_penalty,` (line 846) with:
```cuda
    float risk_efficiency_weight,   /* v7: intra-trade risk efficiency weight */
```

Remove `float regret_blend,` (line 853) entirely — this was the last penalty arg before saboteur_params.

The final kernel args in order (after `position_entropy_weight`):
```cuda
    float cea_weight,               /* v7: counterfactual exposure advantage */
    float order_credit_weight,      /* v7: order microstructure credit */
    float risk_efficiency_weight,   /* v7: risk efficiency bonus */
    const float* __restrict__ saboteur_params
```

- [ ] **Step 2: Update kernel launch in gpu_experience_collector.rs**

In `gpu_experience_collector.rs`, find the env_step launch block (line ~1789). Replace the arg wiring:

Replace `let rw_loss_av = config.loss_aversion;` and its `.arg(&rw_loss_av)` with:
```rust
            let cea_w = config.cea_weight as f32;
```

Then in the `.arg()` chain, replace `.arg(&rw_loss_av)` (line ~1809) with:
```rust
                    .arg(&cea_w)
```

Replace `.arg(&config.beta_penalty)` (line ~1829) with:
```rust
                    .arg(&(config.order_credit_weight as f32))
```

Replace `.arg(&config.trade_clustering_penalty)` (line ~1830) with:
```rust
                    .arg(&(config.risk_efficiency_weight as f32))
```

Remove the `.arg(&config.regret_blend)` line (line ~1834) entirely.

- [ ] **Step 3: Verify build**

Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -3`
Expected: `Finished`

- [ ] **Step 4: Commit**

```bash
git add crates/ml/src/cuda_pipeline/experience_kernels.cu \
       crates/ml/src/cuda_pipeline/gpu_experience_collector.rs
git commit -m "feat(reward-v7): update kernel signature — replace v6 penalty args with v7 CEA/order/risk args"
```

---

### Task 4: Implement Reward v7 Core Logic in CUDA Kernel

**Files:**
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu:1198-1410` (reward computation section)

This is the main implementation task. Replace the v6 reward section with v7 logic.

- [ ] **Step 1: Add intra-trade max DD tracking (per-bar update)**

Find the line where `hold_time` is updated (line ~1189: `hold_time = update_hold_time(...)`). Add immediately AFTER it:

```cuda
    /* ── v7: Track intra-trade max drawdown for risk efficiency ── */
    float intra_trade_max_dd = __bfloat162float(ps[20]);
    if (entering_trade) {
        intra_trade_max_dd = 0.0f;  /* reset at trade entry */
    }
    if (reversing_trade) {
        intra_trade_max_dd = 0.0f;  /* reset for new segment */
    }
    if (fabsf(position) > 0.001f && entry_price > 0.0f) {
        float unrealized = (position > 0.0f)
            ? (raw_close - entry_price) / entry_price
            : (entry_price - raw_close) / entry_price;
        float current_dd = fminf(unrealized, 0.0f);  /* negative when underwater */
        intra_trade_max_dd = fminf(intra_trade_max_dd, current_dd);
    }
```

- [ ] **Step 2: Replace v6 reward section with v7**

Find the entire reward v6 block starting at line 1198 (`/* ================================================================`) through the end of the trade_clustering_penalty block (line ~1374). Replace the ENTIRE block (from `float reward = 0.0f;` through the end of the clustering penalty) with:

```cuda
    /* ================================================================
     * REWARD v7: Counterfactual Branch Attribution
     *
     * Fixes: v6 penalty stacking required 95% win rate breakeven.
     * v7 drops breakeven to ~52% via:
     *   1. Asymmetric soft-clamp (natural risk aversion, no loss_aversion param)
     *   2. Counterfactual Exposure Advantage (per-branch gradient signal)
     *   3. Order type microstructure credit (execution quality signal)
     *   4. Intra-trade risk efficiency (path quality, not just endpoint)
     *
     * Kept from v6: sparse trade-completion, vol-normalization, drawdown
     * penalty, capital floor.
     *
     * Removed from v6: loss_aversion, regret_blend, hold_scale,
     * trade_clustering_penalty, beta_penalty.
     * ================================================================ */

    float reward = 0.0f;

    if (segment_complete && segment_hold_time > 0.0f) {
        float segment_pnl = __bfloat162float(ps[11]) + raw_pnl - trade_start_pnl;
        float segment_return = segment_pnl / (prev_equity > 1.0f ? prev_equity : 1.0f);

        /* Kelly statistics (unchanged from v6) */
        if (exiting_trade) {
            if (segment_return > 0.0f) {
                win_count += 1.0f;
                sum_wins += segment_return;
            } else {
                loss_count += 1.0f;
                sum_losses += fabsf(segment_return);
            }
            sum_returns += segment_return;
            sum_sq_returns += segment_return * segment_return;
        }
        if (reversing_trade) {
            segment_return = reversal_return;
        }

        /* Vol normalization (unchanged from v6) */
        float atr_norm = 0.0f;
        if (features != NULL && bar_idx < total_bars && market_dim > 9) {
            atr_norm = __bfloat162float(features[(long long)bar_idx * market_dim + 9]);
        }
        float log_atr = atr_norm * 16.0f - 7.0f;
        float atr_pct = expf(log_atr) / fmaxf(raw_close, 1.0f);
        float vol_proxy = fmaxf(atr_pct, 0.0001f);
        float vol_norm = vol_proxy * sqrtf(fmaxf(segment_hold_time, 1.0f));
        float vol_normalized_return = segment_return / vol_norm;

        /* ── Layer 2: Asymmetric soft-clamp (replaces loss_aversion + tanh) ── */
        float base_reward = 10.0f * vol_normalized_return;
        reward = asymmetric_soft_clamp(base_reward);

        /* ── Layer 3: Counterfactual Exposure Advantage (CEA) ── */
        if (cea_weight > 0.0f) {
            float sum_alt_rewards = 0.0f;
            float price_delta = raw_next - raw_close;
            for (int k = 0; k < b0_size; k++) {
                float alt_pos = compute_target_position(k, b0_size, max_position);
                float alt_pnl = alt_pos * price_delta;
                float alt_return = alt_pnl / (prev_equity > 1.0f ? prev_equity : 1.0f);
                float alt_vol_norm = alt_return / vol_norm;
                float alt_reward = asymmetric_soft_clamp(10.0f * alt_vol_norm);
                sum_alt_rewards += alt_reward;
            }
            float mean_alt_reward = sum_alt_rewards / (float)b0_size;
            float exposure_advantage = reward - mean_alt_reward;
            reward += cea_weight * exposure_advantage;
        }

        /* ── Layer 4: Order type microstructure credit ── */
        if (order_credit_weight > 0.0f && fabsf(position) > 0.001f) {
            float market_cost = compute_tx_cost(position, raw_close, tx_cost_multiplier,
                                                spread_cost, max_position, 0/*Market*/, -1.0f);
            float taken_cost  = compute_tx_cost(position, raw_close, tx_cost_multiplier,
                                                spread_cost, max_position, order_type_idx, spread_scale);
            float order_credit = (market_cost - taken_cost) / (prev_equity > 1.0f ? prev_equity : 1.0f);
            reward += order_credit_weight * fminf(order_credit * 100.0f, 2.0f);
        }

        /* ── Layer 5: Intra-trade risk efficiency (path quality) ── */
        if (risk_efficiency_weight > 0.0f && segment_return > 0.0f && intra_trade_max_dd < -0.001f) {
            float risk_eff = segment_return / fabsf(intra_trade_max_dd);
            reward += risk_efficiency_weight * fminf(risk_eff, 5.0f);
        }
    }

    /* ── Layer 6: Drawdown penalty (dense, every bar — unchanged from v6) ── */
```

Note: Everything AFTER the drawdown penalty (capital floor, position entropy, episode done, portfolio state writeback) stays unchanged from v6. Only the reward computation section (layers 1-5) is replaced.

- [ ] **Step 3: Save `intra_trade_max_dd` back to portfolio state**

Find where portfolio state fields are written back (search for `ps_base` or the block that writes `ps[10]`, `ps[11]`, etc.). In the portfolio state writeback section, add:

```cuda
    portfolio_states[ps_base + 20] = __float2bfloat16(intra_trade_max_dd);
```

- [ ] **Step 4: Update the kernel docstring**

Replace the kernel's doc comment (lines ~784-811) header to say v7:

```cuda
/**
 * Portfolio simulation, reward v7 (Counterfactual Branch Attribution), and done detection.
```

- [ ] **Step 5: Verify build**

Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -3`
Expected: `Finished`

- [ ] **Step 6: Commit**

```bash
git add crates/ml/src/cuda_pipeline/experience_kernels.cu
git commit -m "feat(reward-v7): implement CEA, asymmetric soft-clamp, order credit, risk efficiency in CUDA kernel"
```

---

### Task 5: Update Config TOML Files

**Files:**
- Modify: `config/training/dqn-production.toml`
- Modify: `config/training/dqn-hyperopt.toml`
- Modify: `config/training/dqn-smoketest.toml`
- Modify: `config/training/dqn-localdev.toml`

- [ ] **Step 1: Update dqn-production.toml**

Find the `[reward]` section and update:

```toml
[reward]
w_dsr = 1.0
w_pnl = 0.3
w_dd = 1.0
w_idle = 0.0
dd_threshold = 0.01
loss_aversion = 1.0          # v7: no-op (soft-clamp handles asymmetry)
time_decay_rate = 0.0005
q_gap_threshold = 0.1
# Reward v7 — Counterfactual Branch Attribution
cea_weight = 0.3
order_credit_weight = 0.1
risk_efficiency_weight = 0.1
```

- [ ] **Step 2: Update dqn-hyperopt.toml**

In the `[search_space]` section, add v7 ranges and update removed params:

```toml
# Reward v7 — Counterfactual Branch Attribution
cea_weight = [0.1, 1.0]
order_credit_weight = [0.0, 0.5]
risk_efficiency_weight = [0.0, 0.5]

# v6 legacy (fixed at no-op values, not searched)
loss_aversion = [1.0, 1.0]
```

In the `[reward]` section:

```toml
[reward]
w_dsr = 1.0
w_pnl = 0.3
w_dd = 1.0
w_idle = 0.0
dd_threshold = 0.01
loss_aversion = 1.0
time_decay_rate = 0.0005
q_gap_threshold = 0.1
cea_weight = 0.3
order_credit_weight = 0.1
risk_efficiency_weight = 0.1
```

Also update `[phase_fast]`:

```toml
loss_aversion = 1.0
cea_weight = 0.3
order_credit_weight = 0.1
risk_efficiency_weight = 0.1
```

- [ ] **Step 3: Update dqn-smoketest.toml**

Find or add reward section, add v7 params:

```toml
cea_weight = 0.3
order_credit_weight = 0.1
risk_efficiency_weight = 0.1
loss_aversion = 1.0
```

- [ ] **Step 4: Update dqn-localdev.toml**

Same changes:

```toml
cea_weight = 0.3
order_credit_weight = 0.1
risk_efficiency_weight = 0.1
loss_aversion = 1.0
```

- [ ] **Step 5: Commit**

```bash
git add config/training/dqn-production.toml \
       config/training/dqn-hyperopt.toml \
       config/training/dqn-smoketest.toml \
       config/training/dqn-localdev.toml
git commit -m "config(reward-v7): add cea_weight, order_credit_weight, risk_efficiency_weight to all TOML profiles"
```

---

### Task 6: Wire Hyperopt Adapter for New Parameters

**Files:**
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs`

- [ ] **Step 1: Find where hyperopt params are read from TOML and applied to config**

Search for `loss_aversion` in the hyperopt adapter to find where params are extracted. The adapter uses `apply_family_scaling` which we already updated in Task 2. But the hyperopt search space extraction may need updating.

Search for where `search_space` values are read and applied. The adapter reads ranges from `[search_space]` in the TOML and generates trial params. Find where `loss_aversion` is read as a search param and add the new v7 params in the same pattern:

```rust
            // Reward v7 params
            if let Some(v) = trial_params.get("cea_weight") {
                config.cea_weight = *v;
            }
            if let Some(v) = trial_params.get("order_credit_weight") {
                config.order_credit_weight = *v;
            }
            if let Some(v) = trial_params.get("risk_efficiency_weight") {
                config.risk_efficiency_weight = *v;
            }
```

- [ ] **Step 2: Verify build**

Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -3`
Expected: `Finished`

- [ ] **Step 3: Commit**

```bash
git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "feat(reward-v7): wire cea_weight, order_credit_weight, risk_efficiency_weight in hyperopt adapter"
```

---

### Task 7: Run Local Smoke Test and Validate

**Files:**
- Test: existing smoke tests in `crates/ml/src/trainers/dqn/smoke_tests/`

- [ ] **Step 1: Run cargo check for full workspace**

Run: `SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -5`
Expected: `Finished` with no errors

- [ ] **Step 2: Run ML crate unit tests**

Run: `SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -10`
Expected: All tests pass. Any compilation errors from the CUDA kernel changes will surface here.

- [ ] **Step 3: Run smoke test (if test data available)**

Run: `FOXHUNT_TEST_DATA=test_data/futures-baseline SQLX_OFFLINE=true cargo test -p ml --lib -- smoke_tests --ignored --nocapture 2>&1 | tail -30`
Expected: Training runs, rewards are non-zero, no panics.

- [ ] **Step 4: Verify reward v7 diagnostics in smoke test output**

Check the output for:
- `mean_reward` should NOT be stuck at 0.0 (model is generating trades)
- Action diversity should show >1 exposure level by epoch 2-3
- No NaN/Inf in rewards

- [ ] **Step 5: Commit any test fixes if needed**

```bash
git add -A
git commit -m "fix(reward-v7): address smoke test issues"
```

---

### Task 8: Deploy and Run Hyperopt

- [ ] **Step 1: Deploy to Argo**

```bash
./scripts/argo-compile-deploy.sh --branch main
```

- [ ] **Step 2: Launch hyperopt with v7 reward**

```bash
./scripts/argo-train.sh --model dqn --config dqn-hyperopt
```

- [ ] **Step 3: Monitor early metrics**

Check first epoch of first trial:
- Win rate should be >30% (breaking the random baseline)
- Action diversity should be >1/9 exposure levels
- Q-values should NOT show i%3 pattern
- Rewards should have both positive and negative values

```bash
kubectl logs -l workflow=train-dqn --tail=50 | grep -E "Epoch.*Sharpe|Action diversity|Per-action Q"
```
